Bone-based registration method and device, surgical robot, electronic device

Abstract

This application discloses a skeleton-based registration method and apparatus, a surgical robot, and an electronic device. The method includes: acquiring a first three-dimensional point cloud and a second three-dimensional point cloud, wherein both the first and second three-dimensional point clouds include information corresponding to the skeleton of a first object. The first three-dimensional point cloud is obtained based on first information, and the second three-dimensional point cloud is obtained based on second information. The first information includes target information acquired by a first acquisition device at a first moment, and the second information includes target information acquired by a second acquisition device at a second moment. The target information includes information related to the first object, and the first moment and the second moment are different. The first and second three-dimensional point clouds are then registered to obtain a first registration parameter. The first registration parameter is used to register the target information acquired by the first acquisition device and the target information acquired by the second acquisition device. The method provided in this application can improve the accuracy of the first registration parameter.

Description

Technical Field

[0001] This application relates to the field of medical image processing, and in particular to a bone-based registration method and apparatus, a surgical robot, and electronic equipment. Background Technology

[0002] In the medical field, to obtain more information about a subject, different acquisition devices are typically used to collect that information. However, because the location of the subject-related information differs within the data collected by different acquisition devices, it is necessary to register the subject-related information collected by different devices.

[0003] Current techniques typically involve registering object-related information acquired from different acquisition devices to obtain registration parameters. These parameters are used to register the object-related information from different devices. However, the accuracy of registration parameters obtained using current techniques is low. Summary of the Invention

[0004] This application provides a bone-based registration method and apparatus, a surgical robot, and an electronic device to improve the accuracy of registration parameters.

[0005] Firstly, a skeleton-based registration method is provided, the method comprising:

[0006] Acquire a first 3D point cloud and a second 3D point cloud. Both the first 3D point cloud and the second 3D point cloud include information corresponding to the skeleton of the first object. The first 3D point cloud is obtained based on the first information, and the second 3D point cloud is obtained based on the second information. The first information includes target information acquired by the first acquisition device at a first moment, and the second information includes the target information acquired by the second acquisition device at a second moment. The target information includes information related to the first object. The first moment and the second moment are different.

[0007] The first 3D point cloud and the second 3D point cloud are registered to obtain a first registration parameter. The first registration parameter is used to register the target information acquired by the first acquisition device and the target information acquired by the second acquisition device.

[0008] In conjunction with any embodiment of this application, the registration of the first 3D point cloud and the second 3D point cloud to obtain the first registration parameters includes:

[0009] The first covariance matrix is ​​obtained based on the first three-dimensional point cloud;

[0010] The second covariance matrix is ​​obtained based on the second three-dimensional point cloud;

[0011] Based on the first covariance matrix, a first eigenvector is obtained. The eigenvalue corresponding to the first eigenvector is the nth largest eigenvalue among the eigenvalues ​​corresponding to the eigenvectors of the first covariance matrix, where n is a positive integer.

[0012] Based on the second covariance matrix, a second eigenvector is obtained, and the eigenvalue corresponding to the second eigenvector is the nth largest eigenvalue among the eigenvalues ​​corresponding to the eigenvectors of the second covariance matrix.

[0013] Based on the first feature vector and the second feature vector, the first 3D point cloud and the second 3D point cloud are registered so that the first feature vector and the second feature vector satisfy a first condition, and the first registration parameter is obtained. The first condition includes that the angle between the first feature vector and the second feature vector is less than a first angle threshold or greater than a second angle threshold.

[0014] In conjunction with any embodiment of this application, before registering the first 3D point cloud and the second 3D point cloud based on the first feature vector and the second feature vector, so that the first feature vector and the second feature vector satisfy a first condition, and obtaining the first registration parameter, the method further includes:

[0015] Based on the first three-dimensional point cloud, first bending information is obtained, which includes information related to the bending direction of the skeleton of the first object;

[0016] Based on the second three-dimensional point cloud, second bending information is obtained, which includes information related to the bending direction of the skeleton of the first object;

[0017] Based on the first curvature information and the second curvature information, a second registration parameter is obtained; wherein, after translating and / or rotating the first three-dimensional point cloud based on the second registration parameter, the curvature direction of the skeleton of the first object indicated by the first three-dimensional point cloud matches the curvature direction of the skeleton of the first object indicated by the second three-dimensional point cloud.

[0018] The registration of the first 3D point cloud and the second 3D point cloud based on the first feature vector and the second feature vector, so that the first feature vector and the second feature vector satisfy a first condition, and obtaining the first registration parameters, includes:

[0019] Based on the first feature vector and the second feature vector, the first 3D point cloud and the second 3D point cloud are registered so that the first feature vector and the second feature vector satisfy the first condition, and the third registration parameter is obtained.

[0020] The first registration parameter is obtained based on the second registration parameter and the third registration parameter.

[0021] In any embodiment of this application, the step of registering the first 3D point cloud and the second 3D point cloud based on the first feature vector and the second feature vector, so that the first feature vector and the second feature vector satisfy a first condition, to obtain the first registration parameters, includes:

[0022] Based on the first feature vector and the second feature vector, the first 3D point cloud and the second 3D point cloud are registered so that the first feature vector and the second feature vector satisfy the first condition, and the third registration parameter is obtained.

[0023] Based on the third registration parameters, the pose of the first three-dimensional point cloud is adjusted to obtain the third three-dimensional point cloud;

[0024] The third 3D point cloud and the second 3D point cloud are registered to align the points in the third 3D point cloud with the points in the second 3D point cloud, thus obtaining the fourth registration parameter;

[0025] If the registration error corresponding to the fourth registration parameter is greater than the error threshold, the first three-dimensional point cloud is rotated around the first feature vector by a preset angle to obtain the fourth three-dimensional point cloud. The difference between 180 degrees and the preset angle is less than or equal to the third angle threshold.

[0026] The fourth 3D point cloud and the second 3D point cloud are registered to align the points in the fourth 3D point cloud with the points in the second 3D point cloud, thus obtaining the fifth registration parameter;

[0027] The first registration parameter is obtained based on the fourth registration parameter and the fifth registration parameter.

[0028] In conjunction with any embodiment of this application, obtaining the first registration parameter based on the fourth registration parameter and the fifth registration parameter includes:

[0029] If the registration error corresponding to the fifth registration parameter is less than or equal to the error threshold, the first registration parameter is obtained based on the fifth registration parameter.

[0030] If the registration error corresponding to the fifth registration parameter is greater than the error threshold, the first registration parameter is obtained based on the registration parameter with the smaller registration error among the fourth and fifth registration parameters.

[0031] In any embodiment of this application, obtaining the first covariance matrix based on the first three-dimensional point cloud includes:

[0032] Determine the first coordinates of the centroid of the first 3D point cloud;

[0033] Determine the second coordinates of the centroid of the second three-dimensional point cloud;

[0034] Determine the absolute value of the difference between the first coordinate and the second coordinate;

[0035] Based on the absolute value, the coordinates of the points in the first three-dimensional point cloud are corrected to obtain the corrected first three-dimensional point cloud.

[0036] Based on the corrected first three-dimensional point cloud, the first covariance matrix is ​​obtained.

[0037] In any embodiment of this application, the first acquisition device includes a CT scanning device, the second acquisition device includes an ultrasound scanning device, and the method further includes:

[0038] Acquire three-dimensional CT images and three-dimensional ultrasound images. The three-dimensional CT images are acquired by the CT scanning device at a third time moment, and the three-dimensional ultrasound images are acquired by the ultrasound scanning device at a fourth time moment. Both the three-dimensional CT images and the three-dimensional ultrasound images include the soft tissue of the first object. The third time moment and the fourth time moment are different.

[0039] Based on the first registration parameter, the three-dimensional CT image and the three-dimensional ultrasound image are registered so that the soft tissue in the three-dimensional CT image is aligned with the soft tissue in the three-dimensional ultrasound image, thereby obtaining a registered three-dimensional CT image.

[0040] Secondly, a skeleton-based registration device is provided, the skeleton-based registration device comprising:

[0041] An acquisition unit is used to acquire a first three-dimensional point cloud and a second three-dimensional point cloud. Both the first three-dimensional point cloud and the second three-dimensional point cloud include information corresponding to the skeleton of the first object. The first three-dimensional point cloud is obtained based on the target information acquired by the first acquisition device, and the second three-dimensional point cloud is obtained based on the target information acquired by the second acquisition device. The target information includes information related to the first object.

[0042] The processing unit is used to register the first three-dimensional point cloud and the second three-dimensional point cloud to obtain a first registration parameter. The first registration parameter is used to register the target information acquired by the first acquisition device and the target information acquired by the second acquisition device.

[0043] In conjunction with any embodiment of this application, the processing unit is further configured to:

[0044] The first covariance matrix is ​​obtained based on the first three-dimensional point cloud;

[0045] The second covariance matrix is ​​obtained based on the second three-dimensional point cloud;

[0046] Based on the first covariance matrix, a first eigenvector is obtained. The eigenvalue corresponding to the first eigenvector is the nth largest eigenvalue among the eigenvalues ​​corresponding to the eigenvectors of the first covariance matrix, where n is a positive integer.

[0047] Based on the second covariance matrix, a second eigenvector is obtained, and the eigenvalue corresponding to the second eigenvector is the nth largest eigenvalue among the eigenvalues ​​corresponding to the eigenvectors of the second covariance matrix.

[0048] Based on the first feature vector and the second feature vector, the first 3D point cloud and the second 3D point cloud are registered so that the first feature vector and the second feature vector satisfy a first condition, and the first registration parameter is obtained. The first condition includes that the angle between the first feature vector and the second feature vector is less than a first angle threshold or greater than a second angle threshold.

[0049] In conjunction with any embodiment of this application, the processing unit is further configured to:

[0050] Based on the first three-dimensional point cloud, first bending information is obtained, which includes information related to the bending direction of the skeleton of the first object;

[0051] Based on the second three-dimensional point cloud, second bending information is obtained, which includes information related to the bending direction of the skeleton of the first object;

[0052] Based on the first curvature information and the second curvature information, a second registration parameter is obtained; wherein, after translating and / or rotating the first three-dimensional point cloud based on the second registration parameter, the curvature direction of the skeleton of the first object indicated by the first three-dimensional point cloud matches the curvature direction of the skeleton of the first object indicated by the second three-dimensional point cloud.

[0053] Based on the first feature vector and the second feature vector, the first 3D point cloud and the second 3D point cloud are registered so that the first feature vector and the second feature vector satisfy the first condition, and the third registration parameter is obtained.

[0054] The first registration parameter is obtained based on the second registration parameter and the third registration parameter.

[0055] In conjunction with any embodiment of this application, the processing unit is further configured to:

[0056] Based on the first feature vector and the second feature vector, the first 3D point cloud and the second 3D point cloud are registered so that the first feature vector and the second feature vector satisfy the first condition, and the third registration parameter is obtained.

[0057] Based on the third registration parameters, the pose of the first three-dimensional point cloud is adjusted to obtain the third three-dimensional point cloud;

[0058] The third 3D point cloud and the second 3D point cloud are registered to align the points in the third 3D point cloud with the points in the second 3D point cloud, thus obtaining the fourth registration parameter;

[0059] If the registration error corresponding to the fourth registration parameter is greater than the error threshold, the first three-dimensional point cloud is rotated around the first feature vector by a preset angle to obtain the fourth three-dimensional point cloud. The difference between 180 degrees and the preset angle is less than or equal to the third angle threshold.

[0060] The fourth 3D point cloud and the second 3D point cloud are registered to align the points in the fourth 3D point cloud with the points in the second 3D point cloud, thus obtaining the fifth registration parameter;

[0061] The first registration parameter is obtained based on the fourth registration parameter and the fifth registration parameter.

[0062] In conjunction with any embodiment of this application, the processing unit is further configured to:

[0063] If the registration error corresponding to the fifth registration parameter is less than or equal to the error threshold, the first registration parameter is obtained based on the fifth registration parameter.

[0064] If the registration error corresponding to the fifth registration parameter is greater than the error threshold, the first registration parameter is obtained based on the registration parameter with the smaller registration error among the fourth and fifth registration parameters.

[0065] In conjunction with any embodiment of this application, the processing unit is further configured to:

[0066] Determine the first coordinates of the centroid of the first 3D point cloud;

[0067] Determine the second coordinates of the centroid of the second three-dimensional point cloud;

[0068] Determine the absolute value of the difference between the first coordinate and the second coordinate;

[0069] Based on the absolute value, the coordinates of the points in the first three-dimensional point cloud are corrected to obtain the corrected first three-dimensional point cloud.

[0070] Based on the corrected first three-dimensional point cloud, the first covariance matrix is ​​obtained.

[0071] In any embodiment of this application, the first acquisition device includes a CT scanning device, the second acquisition device includes an ultrasound scanning device, and the acquisition unit is further configured to: acquire a three-dimensional CT image and a three-dimensional ultrasound image, wherein the three-dimensional CT image is acquired by the CT scanning device at a third time, the three-dimensional ultrasound image is acquired by the ultrasound scanning device at a fourth time, and both the three-dimensional CT image and the three-dimensional ultrasound image include the soft tissue of the first object, and the third time is different from the fourth time;

[0072] The processing unit is further configured to register the three-dimensional CT image and the three-dimensional ultrasound image based on the first registration parameters, so that the soft tissue in the three-dimensional CT image is aligned with the soft tissue in the three-dimensional ultrasound image, thereby obtaining a registered three-dimensional CT image.

[0073] Thirdly, a surgical robot is provided, including a bone-based registration device as described in the second aspect. In this third aspect, the surgical robot can perform a bone-based registration method using the bone-based registration device, thereby improving the accuracy of the registration parameters.

[0074] Fourthly, an electronic device is provided, comprising: a processor and a memory, the memory for storing computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs a method as described in the first aspect above and any possible implementation thereof.

[0075] Fifthly, another electronic device is provided, comprising: a processor, a transmitting device, an input device, an output device, and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs a method as described in the first aspect above and any possible implementation thereof.

[0076] In a sixth aspect, a computer-readable storage medium is provided, wherein a computer program is stored therein, the computer program including program instructions that, when executed by a processor, cause the processor to perform a method as described in the first aspect above and any possible implementation thereof.

[0077] In a seventh aspect, a computer program product is provided, the computer program product comprising a computer program or instructions, wherein, when the computer program or instructions are executed on a computer, the computer performs the method described in the first aspect and any possible implementation thereof.

[0078] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application.

[0079] In this embodiment, the first information and the second information are information collected by the first acquisition device and the second acquisition device at different times, and both the first information and the second information include information related to the first object. Because the information related to the first object includes information corresponding to the skeleton of the first object, both the first 3D point cloud obtained based on the first information and the second 3D point cloud obtained based on the second information include information corresponding to the skeleton of the first object.

[0080] After acquiring the first and second 3D point clouds, the registration device registers them to obtain the first registration parameters. These parameters can be derived based on information corresponding to the skeleton of the first object. Considering that the first object's respiratory state and / or position vary at different times, both of which can cause deformation and / or displacement of the soft tissue, registration based on the soft tissue information at different times is prone to large errors. However, the shape and / or position of the first object's skeleton are less affected by its respiratory state and / or position. Therefore, by obtaining the first registration parameters based on information corresponding to the skeleton, the accuracy of the first registration parameters can be improved. Attached Figure Description

[0081] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments of this application will be described below.

[0082] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0083] Figure 1 A schematic flowchart of a skeleton-based registration method provided in an embodiment of this application;

[0084] Figure 2 A schematic diagram of a first three-dimensional point cloud provided for an embodiment of this application;

[0085] Figure 3 A schematic diagram of a second three-dimensional point cloud provided for an embodiment of this application;

[0086] Figure 4 A schematic diagram of three dimensions of a first three-dimensional point cloud provided for an embodiment of this application;

[0087] Figure 5 A schematic diagram illustrating the relationship between a first three-dimensional point cloud and a first sphere provided in an embodiment of this application;

[0088] Figure 6 This is another schematic diagram illustrating the relationship between a first three-dimensional point cloud and a first sphere, provided in an embodiment of this application.

[0089] Figure 7 A schematic diagram of a first three-dimensional point cloud and a first bounding box provided for an embodiment of this application;

[0090] Figure 8 A schematic diagram of a skeleton-based registration device provided in an embodiment of this application;

[0091] Figure 9 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0092] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0093] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0094] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. It should be understood that in this application, "at least one" means one or more, "more" means two or more, and "at least two" means two or three or more.

[0095] The execution subject of this application embodiment is a skeleton-based registration device (hereinafter referred to as the registration device), wherein the registration device can be any electronic device capable of executing the technical solutions disclosed in the method embodiments of this application. Optionally, the registration device can be one of the following: a computer, a server.

[0096] It should be understood that the method embodiments of this application can also be implemented by a processor executing computer program code. The embodiments of this application are described below with reference to the accompanying drawings. Please refer to... Figure 1 , Figure 1 This is a schematic flowchart of a skeleton-based registration method provided in an embodiment of this application.

[0097] 101. Acquire a first three-dimensional point cloud and a second three-dimensional point cloud, wherein both the first three-dimensional point cloud and the second three-dimensional point cloud include information corresponding to the skeleton of the first object. The first three-dimensional point cloud is obtained based on the first information, and the second three-dimensional point cloud is obtained based on the second information. The first information includes target information acquired by the first acquisition device at a first moment, and the second information includes target information acquired by the second acquisition device at a second moment. The target information includes information related to the first object. The first moment and the second moment are different.

[0098] In this embodiment, the three-dimensional point cloud (including the aforementioned first and second three-dimensional point clouds, and the third, fourth, fifth, and sixth three-dimensional point clouds mentioned below) is a collection of discrete sampling points in three-dimensional space, each sampling point containing three-dimensional spatial coordinate information. Both the first and second three-dimensional point clouds include information corresponding to the skeleton of the first object; that is, at least a portion of the point clouds in both the first and second three-dimensional point clouds correspond to the skeleton of the first object. Optionally, the first object is a person.

[0099] The first acquisition device mentioned above includes a computed tomography (CT) scanning device, wherein the CT scanning device is used to perform CT scans, and optionally, the CT scanning device is a CT scanner. The second acquisition device mentioned above includes an ultrasound scanning device, and optionally, the second acquisition device is an ultrasound probe.

[0100] Both the first information and the second information include information related to the first object. The first information is information collected by the first acquisition device at a first moment, and the second information is information collected by the second acquisition device at a second moment. The first moment and the second moment are different. In other words, the first information and the second information are information collected by the first acquisition device and the second acquisition device at different moments.

[0101] In one possible scenario, a CT scan is performed on a first object using a CT scanner at a first moment, obtaining CT scan data. A first 3D point cloud is obtained based on the CT scan data. At a second moment, an ultrasound scan is performed on the first object using an ultrasound scanner, obtaining ultrasound scan data. A second 3D point cloud is obtained based on the ultrasound scan data. Optionally, the first moment is earlier than the second moment. For example... Figure 2 This is a schematic diagram of a first three-dimensional point cloud provided in an embodiment of this application. Figure 3 This is a schematic diagram of a second three-dimensional point cloud provided in an embodiment of this application. Wherein, Figure 2 The first 3D point cloud shown is obtained based on CT scan data. Figure 3 The second three-dimensional point cloud shown is obtained based on ultrasonic scanning data.

[0102] Optionally, the skeleton of the first object includes the ribs of the first object.

[0103] 102. Register the first 3D point cloud and the second 3D point cloud to obtain the first registration parameter, wherein the first registration parameter is used to register the target information acquired by the first acquisition device and the target information acquired by the second acquisition device.

[0104] In this embodiment of the application, the registration parameters (including the first registration parameter mentioned above, and the second, third, fourth and fifth registration parameters to be mentioned below) are used to register the target information collected by the first acquisition device and the target information collected by the second acquisition device.

[0105] In one optional implementation, the first acquisition device includes a CT scanning device, and the three-dimensional image obtained based on the target information acquired by the first acquisition device is a three-dimensional CT image. The second acquisition device includes an ultrasound scanning device, and the three-dimensional image obtained based on the target information acquired by the second acquisition device is a three-dimensional ultrasound image. Both the three-dimensional CT image and the three-dimensional ultrasound image include a target region, wherein the target region is the image region corresponding to the first object. Registration parameters are used to register the target region in the three-dimensional CT image and the target region in the three-dimensional ultrasound image; that is, the registration parameters are used to align the target region in the three-dimensional CT image and the target region in the three-dimensional ultrasound image. Specifically, the position of the target region in the three-dimensional CT image differs from its position in the three-dimensional ultrasound image. By registering the target region in the three-dimensional CT image and the target region in the three-dimensional ultrasound image based on the registration parameters, the difference in position between the target region in the three-dimensional CT image and the target region in the three-dimensional ultrasound image can be reduced, thereby aligning the target region in the three-dimensional CT image and the target region in the three-dimensional ultrasound image.

[0106] Optionally, the registration parameters can be one of the following: rigid transformation, similarity transformation, affine transformation, projection transformation, or nonlinear transformation. Optionally, the registration parameters can include one or more of the following: translation parameters and rotation parameters, wherein the translation parameters are used to translate the target region in the image, and the rotation parameters are used to rotate the target region in the image. Optionally, the translation parameters are used to translate the target region in the image within the image's pixel coordinate system, and the rotation parameters are used to rotate the target region in the image within the image's pixel coordinate system.

[0107] In one optional implementation, registration parameters are used to register soft tissue information in target information acquired by a first acquisition device and soft tissue information in target information acquired by a second acquisition device, wherein the soft tissue information corresponds to the soft tissue of the first object. Optionally, the target information includes soft tissue information and hard tissue information, wherein the hard tissue information corresponds to the hard tissue of the first object. A first three-dimensional point cloud can be obtained based on the target information acquired by the first acquisition device, and a second three-dimensional point cloud can be obtained based on the target information acquired by the second acquisition device. When the first registration parameters are obtained based on the first three-dimensional point cloud and the second three-dimensional point cloud, the first registration parameters can be used to register the soft tissue information of the first object acquired by the first acquisition device and the soft tissue information acquired by the second acquisition device. For example, the soft tissue of the first object includes at least one of the following: blood vessels, organs, muscles, tendons, ligaments, fascia, and skin, such as the soft tissue of the first object including the kidney. The hard tissue of the first object includes bone.

[0108] exist Figure 1 In the skeleton-based registration method, the first information and the second information are information acquired by the first acquisition device and the second acquisition device at different times, and both the first information and the second information include information related to the first object. Because the information related to the first object includes information corresponding to the skeleton of the first object, both the first 3D point cloud obtained based on the first information and the second 3D point cloud obtained based on the second information include information corresponding to the skeleton of the first object.

[0109] After acquiring the first and second 3D point clouds, the registration device registers them to obtain the first registration parameters. These parameters can be derived based on information corresponding to the skeleton of the first object. Considering that the first object's respiratory state and / or position vary at different times, both of which can cause deformation and / or displacement of the soft tissue, registration based on the soft tissue information at different times is prone to large errors. However, the shape and / or position of the first object's skeleton are less affected by its respiratory state and / or position. Therefore, by obtaining the first registration parameters based on information corresponding to the skeleton, the accuracy of the first registration parameters can be improved.

[0110] As an optional implementation, registering a first 3D point cloud and a second 3D point cloud to obtain a first registration parameter includes the following steps: obtaining a first covariance matrix based on the first 3D point cloud; obtaining a second covariance matrix based on the second 3D point cloud; obtaining a first eigenvector based on the first covariance matrix, wherein the eigenvalue corresponding to the first eigenvector is the nth largest eigenvalue among the eigenvalues ​​corresponding to the eigenvectors of the first covariance matrix, where n is a positive integer; obtaining a second eigenvector based on the second covariance matrix, wherein the eigenvalue corresponding to the second eigenvector is the nth largest eigenvalue among the eigenvalues ​​corresponding to the eigenvectors of the second covariance matrix; and registering the first 3D point cloud and the second 3D point cloud based on the first and second eigenvectors to ensure that the first eigenvector and the second eigenvector satisfy a first condition, thereby obtaining the first registration parameter, wherein the first condition includes that the angle between the first eigenvector and the second eigenvector is less than a first angle threshold or greater than a second angle threshold.

[0111] In this embodiment, the registration device first obtains a first covariance matrix and a second covariance matrix based on the first 3D point cloud and the second 3D point cloud, respectively. Then, it can obtain a first eigenvector based on the first covariance matrix and a second eigenvector based on the second covariance matrix. For example, the eigenvectors of the first covariance matrix include vectors v1, v2, and v3, where the eigenvalue corresponding to vector v1 is t1, the eigenvalue corresponding to vector v2 is t2, and the eigenvalue corresponding to vector v3 is t3, wherein t1 is greater than t2, and t2 is greater than t3. The eigenvectors of the second covariance matrix include vectors v4, v5, and v6, where the eigenvalue corresponding to vector v4 is t4, the eigenvalue corresponding to vector v5 is t5, and the eigenvalue corresponding to vector v6 is t6, wherein t4 is greater than t5, and t5 is greater than t6. If n=1, then the eigenvalue corresponding to the first eigenvector is the maximum value among the eigenvalues ​​of the first covariance matrix, and the eigenvalue corresponding to the second eigenvector is the maximum value among the eigenvalues ​​of the second covariance matrix. In this case, the first eigenvector is v1, and the second eigenvector is v4. If n=2, then the eigenvalue corresponding to the first eigenvector is the second largest eigenvalue of the first covariance matrix, and the eigenvalue corresponding to the second eigenvector is the second largest eigenvalue of the second covariance matrix. In this case, the first eigenvector is v2, and the second eigenvector is v5. If n=3, then the eigenvalue corresponding to the first eigenvector is the third largest eigenvalue of the first covariance matrix, and the eigenvalue corresponding to the second eigenvector is the third largest eigenvalue of the second covariance matrix. In this case, the first eigenvector is v3, and the second eigenvector is v6.

[0112] Optionally, the skeleton of the first object is the ribs of the first object, n=1. Because the ribs of the first object are curved and elongated, the ribs of the first object include three dimensions: length, thickness, and width. The length dimension of the skeleton of the first object is the longest. The direction corresponding to the length dimension of the ribs of the first object matches the horizontal direction of the first object, which is the direction from the left side of the first object to the right side of the first object.

[0113] In one optional implementation, the registration device calculates the eigenvectors of a first covariance matrix based on principal component analysis (PCA), and then determines a first eigenvector from the eigenvectors of the first covariance matrix. The registration device also calculates the eigenvectors of a second covariance matrix based on PCA, and then determines a second eigenvector from the eigenvectors of the second covariance matrix. Optionally, the eigenvectors of the first covariance matrix can be obtained based on PCA, and then the three dimensional directions corresponding to the three dimensions of the first object in the first 3D point cloud can be determined based on the eigenvectors of the first covariance matrix. For example, Figure 4This application provides a schematic diagram of three dimensions of a first three-dimensional point cloud, as shown in the embodiments of the present application. Figure 4 In the diagram, the horizontal, vertical, and angular axes each correspond to an eigenvector of the first covariance matrix, and the direction of the horizontal axis (i.e., ...) Figure 4 (horizontal axis direction) and vertical axis direction (i.e.) Figure 4 The direction of the vertical axis (i.e., the direction of the vertical axis). Figure 4 The vertical axis direction corresponds to the direction of an eigenvector of the first covariance matrix, and the horizontal, vertical, and axial axes all pass through the centroid of the first three-dimensional point cloud.

[0114] Because a larger eigenvalue of the eigenvector of the covariance matrix indicates a greater amount of information carried by the eigenvector, and the amount of information in the skeleton varies across different dimensions, the first and second eigenvectors determined by the registration device in this embodiment correspond to the same dimension of the skeleton of the first object. For example, the skeleton of the first object is its ribs, and n is 1. In this case, both the first and second eigenvectors correspond to the length dimension of the ribs. Therefore, after determining the first and second eigenvectors, the registration device registers the first and second 3D point clouds by reducing the angle between the first and second eigenvectors, thereby obtaining the first registration parameter.

[0115] In this implementation, the first feature vector and the second feature vector satisfy a first condition, indicating that the angle between the first feature vector and the second feature vector is small. Specifically, the angle between the first feature vector and the second feature vector is less than a first angle threshold, or the angle between the first feature vector and the second feature vector is greater than a second angle threshold. Optionally, the first angle threshold is 1 degree, and the second angle threshold is 179 degrees. For example, when the angle between the first feature vector and the second feature vector is 0 degrees, the angle between the first feature vector and the second feature vector is less than the first threshold. When the angle between the first feature vector and the second feature vector is 180 degrees, the angle between the first feature vector and the second feature vector is greater than the second threshold.

[0116] As an optional implementation, obtaining the first covariance matrix based on the first 3D point cloud includes the following steps: determining the first coordinates of the centroid of the first 3D point cloud; determining the second coordinates of the centroid of the second 3D point cloud; determining the absolute value of the difference between the first and second coordinates; correcting the coordinates of points in the first 3D point cloud based on the absolute value to obtain a corrected first 3D point cloud, thereby reducing the difference between the coordinates of points in the first and second 3D point clouds; optionally, subtracting the absolute value from the coordinates of points in the first 3D point cloud to obtain the corrected first 3D point cloud; and obtaining the first covariance matrix based on the corrected first 3D point cloud.

[0117] As an optional implementation, before performing the step "registering the first 3D point cloud and the second 3D point cloud based on the first feature vector and the second feature vector, so that the first feature vector and the second feature vector satisfy a first condition to obtain the first registration parameter", the registration device further performs the following steps: obtaining first curvature information based on the first 3D point cloud, wherein the first curvature information includes information related to the curvature direction of the skeleton of the first object. Obtaining second curvature information based on the second 3D point cloud, wherein the second curvature information includes information related to the curvature direction of the skeleton of the first object. Obtaining second registration parameter based on the first curvature information and the second curvature information, wherein after translating and / or rotating the first 3D point cloud based on the second registration parameter, the curvature direction of the skeleton of the first object indicated by the first 3D point cloud matches the curvature direction of the skeleton of the first object indicated by the second 3D point cloud.

[0118] After obtaining the second registration parameters, the following steps are performed during the execution of the step "Registering the first 3D point cloud and the second 3D point cloud based on the first feature vector and the second feature vector so that the first feature vector and the second feature vector satisfy the first condition, thereby obtaining the first registration parameters": Registering the first 3D point cloud and the second 3D point cloud based on the first feature vector and the second feature vector so that the first feature vector and the second feature vector satisfy the first condition, thereby obtaining the third registration parameters. The first registration parameters are then obtained based on the second registration parameters and the third registration parameters.

[0119] Because the skeleton of the first object is curved, the registration device can register the first 3D point cloud and the second 3D point cloud based on the curvature direction of the skeleton, thereby obtaining the second registration parameter. The second registration parameter is used to match the curvature direction of the point cloud in the first 3D point cloud corresponding to the skeleton of the first object with the curvature direction of the point cloud in the second 3D point cloud corresponding to the skeleton of the first object. Specifically, after translating and / or rotating the first 3D point cloud based on the second registration parameter, the curvature direction of the skeleton of the first object indicated by the first 3D point cloud matches the curvature direction of the skeleton of the first object indicated by the second 3D point cloud.

[0120] After obtaining the second registration parameters, the first and second 3D point clouds are first registered based on the first and second eigenvectors to ensure that the first and second eigenvectors satisfy a first condition, thus obtaining the third registration parameters. Then, based on the second and third registration parameters, the first registration parameters are obtained. This allows for the initial registration of the first and second 3D point clouds based on the first and second eigenvectors to obtain the third registration parameters. Then, the third registration parameters are corrected based on the second registration parameters to obtain the first registration parameters, thereby improving the accuracy of the first registration parameters. Optionally, both the second and third registration parameters are matrices including translation and / or rotation information. The registration device determines the product of the second and third registration parameters to obtain the first registration parameters.

[0121] Optionally, the registration device obtains a first sphere by performing spherical fitting on points in a first 3D point cloud, wherein the surface of the first sphere includes the point cloud in the first 3D point cloud corresponding to the skeleton of the first object. A second sphere is obtained by performing spherical fitting on points in a second 3D point cloud, wherein the surface of the second sphere includes the point cloud in the second 3D point cloud corresponding to the skeleton of the first object. A first direction vector is determined, pointing from the centroid of the first 3D point cloud to the center of the first sphere, and a second direction vector is determined, pointing from the centroid of the second 3D point cloud to the center of the second sphere. First curvature information is obtained based on the first direction vector, and second curvature information is obtained based on the second direction vector, wherein the first direction vector is the normal vector of the point cloud in the first 3D point cloud corresponding to the skeleton of the first object, and the second direction vector is the normal vector of the point cloud in the second 3D point cloud corresponding to the skeleton of the first object.

[0122] For example, Figure 5 This is a schematic diagram illustrating the relationship between a first three-dimensional point cloud and a first sphere, provided as an embodiment of this application. Figure 5 In the diagram, the horizontal axis is the direction of the horizontal axis, the vertical axis is the direction of the vertical axis, and the center of gravity is the direction of the vertical axis. The horizontal, vertical, and center axes correspond to the three dimensions of the first object, and all three axes pass through the centroid of the first 3D point cloud. For example... Figure 5 As shown, the vertical axis of the first three-dimensional point cloud deviates significantly from the center of the first sphere. Figure 6 This is another schematic diagram illustrating the relationship between a first three-dimensional point cloud and a first sphere, provided as an embodiment of this application. Figure 6 The first 3D point cloud in the image is obtained by translating and / or rotating the first 3D point cloud based on the second registration parameters. Through comparison... Figure 5 and Figure 6It can be seen that by translating and / or rotating the first three-dimensional point cloud based on the second registration parameters, the deviation between the longitudinal axis of the first three-dimensional point cloud and the center of the first sphere can be reduced, thereby improving the matching degree between the bending direction of the skeleton of the first object indicated by the first three-dimensional point cloud and the bending direction of the skeleton of the first object indicated by the second three-dimensional point cloud.

[0123] Optionally, the registration device performs spherical fitting using the following formula: ,in,( ) represents the coordinates of a point in a 3D point cloud. Let be the coordinates of the center of the sphere to be fitted. Let be the radius of the sphere to be fitted.

[0124] As an optional implementation, registration is performed on a first 3D point cloud and a second 3D point cloud based on a first feature vector and a second feature vector, so that the first feature vector and the second feature vector satisfy a first condition to obtain a first registration parameter. This includes the following steps: Registering the first 3D point cloud and the second 3D point cloud based on the first feature vector and the second feature vector, so that the first feature vector and the second feature vector satisfy the first condition to obtain a third registration parameter. Adjusting the pose of the first 3D point cloud based on the third registration parameter to obtain a third 3D point cloud. Registering the third 3D point cloud and the second 3D point cloud to align the points in the third 3D point cloud with the points in the second 3D point cloud to obtain a fourth registration parameter. If the registration error corresponding to the fourth registration parameter is greater than an error threshold, rotating the first 3D point cloud around the first feature vector by a preset angle to obtain a fourth 3D point cloud, wherein the difference between 180 degrees and the preset angle is less than or equal to a third angle threshold. Registering the fourth 3D point cloud and the second 3D point cloud to align the points in the fourth 3D point cloud with the points in the second 3D point cloud to obtain a fifth registration parameter. Based on the fourth and fifth registration parameters, the first registration parameter is obtained.

[0125] In this embodiment, the registration error corresponding to the registration parameter refers to the root mean square error between the coordinates of points in the first 3D point cloud after adjusting the pose of the first 3D point cloud based on the registration parameter and the coordinates of points in the second 3D point cloud. For example, the pose of points in the first 3D point cloud is adjusted based on the fourth registration parameter to obtain the first 3D point cloud after the pose adjustment. In this case, the registration error corresponding to the fourth registration parameter is the root mean square error between the coordinates of points in the first 3D point cloud and the coordinates of points in the second 3D point cloud after the pose adjustment.

[0126] Optionally, the coarse registration mentioned above refers to aligning the global poses of two 3D point clouds, such as aligning the contours of two 3D point clouds. The fine registration mentioned above refers to aligning the points within two 3D point clouds.

[0127] For ease of explanation, the dimensions corresponding to the first and second feature vectors in the skeleton of the first object will be referred to as the first dimension, and the dimensions other than the first dimension in the skeleton of the first object will be referred to as the second and third dimensions. For example, the skeleton of the first object is its ribs, and n is 1. In this case, the first dimension is the length dimension of the ribs, the second dimension is the thickness dimension of the ribs, and the third dimension is the width dimension of the ribs. Since the probability of a large first deviation after coarse registration based on the first and second feature vectors is low, where the first deviation is the deviation between the information corresponding to the first dimension in the first 3D point cloud and the information corresponding to the first dimension in the second 3D point cloud, the registration error corresponding to the fourth registration parameter is more likely to originate from the second deviation and / or the third deviation. The second deviation is the deviation between the information corresponding to the second dimension in the first 3D point cloud and the information corresponding to the second dimension in the second 3D point cloud, and the third deviation is the deviation between the information corresponding to the third dimension in the first 3D point cloud and the information corresponding to the third dimension in the second 3D point cloud.

[0128] Based on this, when the registration error corresponding to the fourth registration parameter is large, the registration device adjusts the pose of the first 3D point cloud by rotating it around the first feature vector by a preset angle to obtain the fourth 3D point cloud. Then, it performs fine registration on the fourth and second 3D point clouds to obtain the fifth registration parameter, thereby improving the efficiency of fine registration. Finally, based on the fourth and fifth registration parameters, the first registration parameter is obtained, which can improve the accuracy of the first registration parameter.

[0129] In this embodiment, the registration device first performs coarse registration on the first and second 3D point clouds based on the first and second feature vectors to obtain the third registration parameter. Then, it adjusts the pose of the first 3D point cloud based on the third registration parameter to obtain the third 3D point cloud. Next, it performs fine registration on the third and second 3D point clouds to obtain the fourth registration parameter. If the registration error of the fourth registration parameter is greater than an error threshold, it indicates a large registration error. Therefore, the pose of the first 3D point cloud is adjusted to obtain the fifth registration parameter. Finally, the first registration parameter is obtained based on the fourth and fifth registration parameters. This improves both the efficiency and accuracy of obtaining the first registration parameter.

[0130] Optionally, the third threshold is 0, in which case the preset angle is 180 degrees. Rotating the first 3D point cloud around the first feature vector by the preset angle in this case achieves the flipping of the first 3D point cloud around the first feature vector.

[0131] Optionally, registering the first 3D point cloud and the second 3D point cloud using this implementation method can be understood as aligning the first bounding box corresponding to the first 3D point cloud and the second bounding box corresponding to the second 3D point cloud. For example, Figure 7This diagram illustrates a first 3D point cloud and a first bounding box, provided as an embodiment of this application. The pose of the first 3D point cloud is obtained by adjusting a first registration parameter, which is obtained based on a fourth registration parameter and a fifth registration parameter. The fourth registration parameter can be obtained by aligning the first bounding box and the second bounding box.

[0132] As an optional implementation, obtaining the first registration parameter based on the fourth and fifth registration parameters includes the following steps: If the registration error corresponding to the fifth registration parameter is less than or equal to an error threshold, the first registration parameter is obtained based on the fifth registration parameter. If the registration error corresponding to the fifth registration parameter is greater than the error threshold, the first registration parameter is obtained based on the registration parameter with the smaller registration error among the fourth and fifth registration parameters.

[0133] In this implementation, if the registration error corresponding to the fifth registration parameter is less than or equal to the error threshold, it indicates that the registration error corresponding to the fifth registration parameter is small, and also that the registration error corresponding to the fifth registration parameter is smaller than the registration error corresponding to the fourth registration parameter. In this case, obtaining the first registration parameter based on the fifth registration parameter can improve the accuracy of the first registration parameter. If the registration error corresponding to the fifth registration parameter is greater than the error threshold, it indicates that the registration error corresponding to the fifth registration parameter is large. In this case, it is necessary to further compare the magnitudes of the registration errors corresponding to the fourth and fifth registration parameters, and obtain the first registration parameter based on the registration parameter with the smaller registration error, which can improve the accuracy of the first registration parameter.

[0134] As an optional implementation, the registration device can perform coarse registration of the first and second 3D point clouds based on the first and second feature vectors to obtain a third registration parameter. Based on the first and second curvature information, it can then perform coarse registration of the first and second 3D point clouds to obtain a second registration parameter. Finally, based on the second and third registration parameters, it can perform fine registration of the first and second 3D point clouds to obtain a first registration parameter, thereby improving the accuracy of the first registration parameter.

[0135] Optionally, the step "obtaining the first registration parameter based on the second and third registration parameters" specifically includes the following steps: obtaining a sixth registration parameter based on the second and third registration parameters; adjusting the pose of the first 3D point cloud based on the sixth registration parameter to obtain a fifth 3D point cloud; registering the fifth and second 3D point clouds to align the points in the fifth and second 3D point clouds to obtain a seventh registration parameter; if the registration error corresponding to the seventh registration parameter is greater than the error threshold, rotating the first 3D point cloud around the first feature vector by a preset angle to obtain a sixth 3D point cloud; registering the sixth and second 3D point clouds to align the points in the sixth and second 3D point clouds to obtain an eighth registration parameter; and obtaining the first registration parameter based on the eighth and sixth registration parameters.

[0136] Optionally, the first registration parameter is obtained based on the eighth registration parameter and the sixth registration parameter, including the following steps: if the registration error corresponding to the eighth registration parameter is less than or equal to an error threshold, the first registration parameter is obtained based on the eighth registration parameter. If the registration error corresponding to the eighth registration parameter is greater than the error threshold, the first registration parameter is obtained based on the sixth registration parameter and the registration parameter with the smaller registration error among the eighth registration parameters.

[0137] As an optional implementation, the first acquisition device includes a CT scanning device, and the second acquisition device includes an ultrasound scanning device. After obtaining the first registration parameters, the registration device further performs the following steps: acquiring a three-dimensional CT image and a three-dimensional ultrasound image, wherein the three-dimensional CT image is acquired by the CT scanning device at a third time, and the three-dimensional ultrasound image is acquired by the ultrasound scanning device at a fourth time. Both the three-dimensional CT image and the three-dimensional ultrasound image include the soft tissue of the first object, and the third time and the fourth time are different. Based on the first registration parameters, the three-dimensional CT image and the three-dimensional ultrasound image are registered to align the soft tissue in the three-dimensional CT image with the soft tissue in the three-dimensional ultrasound image, thereby obtaining a registered three-dimensional CT image.

[0138] As mentioned earlier, considering that the breathing state and / or position of the first object vary at different times, both of which can lead to deformation and / or displacement of the soft tissue, registration based on the soft tissue information at different times is prone to large errors. However, the shape and / or position of the first object's bones are less affected by its breathing state and / or position. Therefore, after obtaining the first registration parameters, the registration device can align the soft tissue in the 3D CT image and the 3D ultrasound image based on these parameters, thus improving registration accuracy.

[0139] In one possible scenario, a 3D CT image is used to determine a target point in the soft tissue of a first object, and a 3D ultrasound image is used to determine a starting point in the skin region of the first object. Based on this starting point and the target point, a target path can be determined, wherein the target path is a path from the skin region of the first object to the target point within the body of the first object. For example, the soft tissue of the first object includes the lungs of the first object, and the target path is a path from the skin region of the first object to the target point in the lungs of the first object.

[0140] After obtaining the registered 3D CT image based on this implementation method, the target path can be determined based on the position of the target point in the registered 3D CT image and the position of the starting point in the 3D ultrasound image, thereby improving the accuracy of the target path.

[0141] In another possible scenario, after obtaining the registered 3D CT image based on this implementation method, the registered 3D CT image and the 3D ultrasound image can be weighted and fused to obtain a fused image. Thus, the information in the 3D CT image and the information in the 3D ultrasound image can be superimposed and displayed through the fused image.

[0142] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0143] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, while using clear signs / information to inform users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, personal information processing may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0144] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0145] Please see Figure 8 , Figure 8This is a schematic diagram of a skeleton-based registration device provided in an embodiment of this application. The skeleton-based registration device 1 includes: an acquisition unit 11 and a processing unit 12, wherein:

[0146] The acquisition unit 11 is used to acquire a first three-dimensional point cloud and a second three-dimensional point cloud. Both the first three-dimensional point cloud and the second three-dimensional point cloud include information corresponding to the skeleton of the first object. The first three-dimensional point cloud is obtained based on the target information acquired by the first acquisition device, and the second three-dimensional point cloud is obtained based on the target information acquired by the second acquisition device. The target information includes information related to the first object.

[0147] The processing unit 12 is used to register the first three-dimensional point cloud and the second three-dimensional point cloud to obtain a first registration parameter. The first registration parameter is used to register the target information acquired by the first acquisition device and the target information acquired by the second acquisition device.

[0148] In conjunction with any embodiment of this application, the processing unit 12 is further configured to:

[0149] The first covariance matrix is ​​obtained based on the first three-dimensional point cloud;

[0150] The second covariance matrix is ​​obtained based on the second three-dimensional point cloud;

[0151] Based on the first covariance matrix, a first eigenvector is obtained. The eigenvalue corresponding to the first eigenvector is the nth largest eigenvalue among the eigenvalues ​​corresponding to the eigenvectors of the first covariance matrix, where n is a positive integer.

[0152] Based on the second covariance matrix, a second eigenvector is obtained, and the eigenvalue corresponding to the second eigenvector is the nth largest eigenvalue among the eigenvalues ​​corresponding to the eigenvectors of the second covariance matrix.

[0153] Based on the first feature vector and the second feature vector, the first 3D point cloud and the second 3D point cloud are registered so that the first feature vector and the second feature vector satisfy a first condition, and the first registration parameter is obtained. The first condition includes that the angle between the first feature vector and the second feature vector is less than a first angle threshold or greater than a second angle threshold.

[0154] In conjunction with any embodiment of this application, the processing unit 12 is further configured to:

[0155] Based on the first three-dimensional point cloud, first bending information is obtained, which includes information related to the bending direction of the skeleton of the first object;

[0156] Based on the second three-dimensional point cloud, second bending information is obtained, which includes information related to the bending direction of the skeleton of the first object;

[0157] Based on the first curvature information and the second curvature information, a second registration parameter is obtained; wherein, after translating and / or rotating the first three-dimensional point cloud based on the second registration parameter, the curvature direction of the skeleton of the first object indicated by the first three-dimensional point cloud matches the curvature direction of the skeleton of the first object indicated by the second three-dimensional point cloud.

[0158] Based on the first feature vector and the second feature vector, the first 3D point cloud and the second 3D point cloud are registered so that the first feature vector and the second feature vector satisfy the first condition, and the third registration parameter is obtained.

[0159] The first registration parameter is obtained based on the second registration parameter and the third registration parameter.

[0160] In conjunction with any embodiment of this application, the processing unit 12 is further configured to:

[0161] Based on the first feature vector and the second feature vector, the first 3D point cloud and the second 3D point cloud are registered so that the first feature vector and the second feature vector satisfy the first condition, and the third registration parameter is obtained.

[0162] Based on the third registration parameters, the pose of the first three-dimensional point cloud is adjusted to obtain the third three-dimensional point cloud;

[0163] The third 3D point cloud and the second 3D point cloud are registered to align the points in the third 3D point cloud with the points in the second 3D point cloud, thus obtaining the fourth registration parameter;

[0164] If the registration error corresponding to the fourth registration parameter is greater than the error threshold, the first three-dimensional point cloud is rotated around the first feature vector by a preset angle to obtain the fourth three-dimensional point cloud. The difference between 180 degrees and the preset angle is less than or equal to the third angle threshold.

[0165] The fourth 3D point cloud and the second 3D point cloud are registered to align the points in the fourth 3D point cloud with the points in the second 3D point cloud, thus obtaining the fifth registration parameter;

[0166] The first registration parameter is obtained based on the fourth registration parameter and the fifth registration parameter.

[0167] In conjunction with any embodiment of this application, the processing unit 12 is further configured to:

[0168] If the registration error corresponding to the fifth registration parameter is less than or equal to the error threshold, the first registration parameter is obtained based on the fifth registration parameter.

[0169] If the registration error corresponding to the fifth registration parameter is greater than the error threshold, the first registration parameter is obtained based on the registration parameter with the smaller registration error among the fourth and fifth registration parameters.

[0170] In conjunction with any embodiment of this application, the processing unit 12 is further configured to:

[0171] Determine the first coordinates of the centroid of the first 3D point cloud;

[0172] Determine the second coordinates of the centroid of the second three-dimensional point cloud;

[0173] Determine the absolute value of the difference between the first coordinate and the second coordinate;

[0174] Based on the absolute value, the coordinates of the points in the first three-dimensional point cloud are corrected to obtain the corrected first three-dimensional point cloud.

[0175] Based on the corrected first three-dimensional point cloud, the first covariance matrix is ​​obtained.

[0176] In any embodiment of this application, the first acquisition device includes a CT scanning device, the second acquisition device includes an ultrasound scanning device, and the acquisition unit 11 is further configured to: acquire a three-dimensional CT image and a three-dimensional ultrasound image, wherein the three-dimensional CT image is acquired by the CT scanning device at a third time, the three-dimensional ultrasound image is acquired by the ultrasound scanning device at a fourth time, and both the three-dimensional CT image and the three-dimensional ultrasound image include the soft tissue of the first object, and the third time is different from the fourth time;

[0177] The processing unit 12 is further configured to register the three-dimensional CT image and the three-dimensional ultrasound image based on the first registration parameters, so that the soft tissue in the three-dimensional CT image is aligned with the soft tissue in the three-dimensional ultrasound image, thereby obtaining a registered three-dimensional CT image.

[0178] In this embodiment, the first information and the second information are information collected by the first acquisition device and the second acquisition device at different times, and both the first information and the second information include information related to the first object. Because the information related to the first object includes information corresponding to the skeleton of the first object, both the first 3D point cloud obtained based on the first information and the second 3D point cloud obtained based on the second information include information corresponding to the skeleton of the first object.

[0179] After acquiring the first and second 3D point clouds, the registration device registers them to obtain the first registration parameters. These parameters can be derived based on information corresponding to the skeleton of the first object. Considering that the first object's respiratory state and / or position vary at different times, both of which can cause deformation and / or displacement of the soft tissue, registration based on the soft tissue information at different times is prone to large errors. However, the shape and / or position of the first object's skeleton are less affected by its respiratory state and / or position. Therefore, by obtaining the first registration parameters based on information corresponding to the skeleton, the accuracy of the first registration parameters can be improved.

[0180] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0181] Figure 9 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device 2 includes a processor 21 and a memory 22. Optionally, the electronic device 2 also includes an input device 23 and an output device 24. The processor 21, memory 22, input device 23, and output device 24 are coupled together via connectors, which include various interfaces, transmission lines, or buses, etc., and are not limited in this embodiment. It should be understood that in the various embodiments of this application, coupling refers to mutual connection in a specific way, including direct connection or indirect connection through other devices, such as through various interfaces, transmission lines, buses, etc.

[0182] Processor 21 can be one or more graphics processing units (GPUs). If processor 21 is a GPU, the GPU can be a single-core GPU or a multi-core GPU. Optionally, processor 21 can be a processor group composed of multiple GPUs, with the multiple processors coupled to each other via one or more buses. Optionally, the processor can also be other types of processors, etc., which are not limited in this embodiment.

[0183] The memory 22 can be used to store computer program instructions, as well as various types of computer program code, including program code for executing the scheme of this application. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), which is used for related instructions and data.

[0184] Input device 23 is used to input data and / or signals, and output device 24 is used to output data and / or signals. Input device 23 and output device 24 can be independent devices or an integrated device.

[0185] It is understood that in this embodiment of the application, the memory 22 can be used not only to store related instructions, but also to store related data. For example, the memory 22 can be used to store the first three-dimensional point cloud and the second three-dimensional point cloud obtained through the input device 23, or the memory 22 can also be used to store the first registration parameters obtained through the processor 21, etc. This embodiment of the application does not limit the specific data stored in the memory.

[0186] Understandable Figure 9 This is merely a simplified design of an electronic device. In practical applications, the electronic device may also include other necessary components, including, but not limited to, any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of this application are within the protection scope of this application.

[0187] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0188] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will also readily understand that the various embodiments of this application have different focuses, and for the sake of convenience and brevity, the same or similar parts may not be repeated in different embodiments. Therefore, parts not described or not described in detail in one embodiment can be referred to the descriptions in other embodiments.

[0189] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0190] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0191] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0192] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0193] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A bone-based registration method, characterized in that, The method includes: Acquire a first 3D point cloud and a second 3D point cloud. Both the first 3D point cloud and the second 3D point cloud include information corresponding to the skeleton of the first object. The first 3D point cloud is obtained based on the first information, and the second 3D point cloud is obtained based on the second information. The first information includes target information acquired by the first acquisition device at a first moment, and the second information includes the target information acquired by the second acquisition device at a second moment. The target information includes information related to the first object. The first moment and the second moment are different. The first 3D point cloud and the second 3D point cloud are registered to obtain a first registration parameter. The first registration parameter is used to register the target information acquired by the first acquisition device and the target information acquired by the second acquisition device. The registration of the first 3D point cloud and the second 3D point cloud to obtain the first registration parameters includes: obtaining a first covariance matrix based on the first 3D point cloud; obtaining a second covariance matrix based on the second 3D point cloud; obtaining a first eigenvector based on the first covariance matrix, wherein the eigenvalue corresponding to the first eigenvector is the nth largest eigenvalue among the eigenvalues ​​corresponding to the eigenvectors of the first covariance matrix, where n is a positive integer; obtaining a second eigenvector based on the second covariance matrix, wherein the eigenvalue corresponding to the second eigenvector is the nth largest eigenvalue among the eigenvalues ​​corresponding to the eigenvectors of the second covariance matrix; obtaining first bending information based on the first 3D point cloud, wherein the first bending information includes information related to the bending direction of the skeleton of the first object; and obtaining the first eigenvector based on the second 3D point cloud. The first object has two bending information, including information related to the bending direction of its skeleton. Based on the first and second bending information, a second registration parameter is obtained. After translating and / or rotating the first 3D point cloud based on the second registration parameter, the bending direction of the skeleton of the first object indicated by the first 3D point cloud matches the bending direction of the skeleton of the first object indicated by the second 3D point cloud. Based on the first and second feature vectors, the first and second 3D point clouds are registered so that the first and second feature vectors satisfy a first condition, thus obtaining the first registration parameter. The first condition includes the angle between the first and second feature vectors being less than a first angle threshold or greater than a second angle threshold. The step of registering the first 3D point cloud and the second 3D point cloud based on the first feature vector and the second feature vector, so that the first feature vector and the second feature vector satisfy a first condition, and obtaining the first registration parameter, includes: registering the first 3D point cloud and the second 3D point cloud based on the first feature vector and the second feature vector, so that the first feature vector and the second feature vector satisfy the first condition, and obtaining a third registration parameter; and obtaining the first registration parameter based on the second registration parameter and the third registration parameter.

2. The method of claim 1, wherein, The registration of the first 3D point cloud and the second 3D point cloud based on the first feature vector and the second feature vector, so that the first feature vector and the second feature vector satisfy a first condition, and obtaining the first registration parameters, includes: Based on the first feature vector and the second feature vector, the first 3D point cloud and the second 3D point cloud are registered so that the first feature vector and the second feature vector satisfy the first condition, and the third registration parameter is obtained. Based on the third registration parameters, the pose of the first three-dimensional point cloud is adjusted to obtain the third three-dimensional point cloud; The third 3D point cloud and the second 3D point cloud are registered to align the points in the third 3D point cloud with the points in the second 3D point cloud, thus obtaining the fourth registration parameter; If the registration error corresponding to the fourth registration parameter is greater than the error threshold, the first three-dimensional point cloud is rotated around the first feature vector by a preset angle to obtain the fourth three-dimensional point cloud. The difference between 180 degrees and the preset angle is less than or equal to the third angle threshold. The fourth 3D point cloud and the second 3D point cloud are registered to align the points in the fourth 3D point cloud with the points in the second 3D point cloud, thus obtaining the fifth registration parameter; The first registration parameter is obtained based on the fourth registration parameter and the fifth registration parameter.

3. The method of claim 2, wherein, The process of obtaining the first registration parameter based on the fourth registration parameter and the fifth registration parameter includes: If the registration error corresponding to the fifth registration parameter is less than or equal to the error threshold, the first registration parameter is obtained based on the fifth registration parameter. If the registration error corresponding to the fifth registration parameter is greater than the error threshold, the first registration parameter is obtained based on the registration parameter with the smaller registration error among the fourth and fifth registration parameters.

4. The method according to any one of claims 1 to 3, characterized in that, The process of obtaining the first covariance matrix based on the first 3D point cloud includes: Determine the first coordinates of the centroid of the first 3D point cloud; Determine the second coordinates of the centroid of the second three-dimensional point cloud; Determine the absolute value of the difference between the first coordinate and the second coordinate; Based on the absolute value, the coordinates of the points in the first three-dimensional point cloud are corrected to obtain the corrected first three-dimensional point cloud. Based on the corrected first three-dimensional point cloud, the first covariance matrix is ​​obtained.

5. The method according to any one of claims 1 to 3, characterized in that, The first acquisition device includes a CT scanning device, the second acquisition device includes an ultrasound scanning device, and the method further includes: Acquire three-dimensional CT images and three-dimensional ultrasound images. The three-dimensional CT images are acquired by the CT scanning device at a third time moment, and the three-dimensional ultrasound images are acquired by the ultrasound scanning device at a fourth time moment. Both the three-dimensional CT images and the three-dimensional ultrasound images include the soft tissue of the first object. The third time moment and the fourth time moment are different. Based on the first registration parameter, the three-dimensional CT image and the three-dimensional ultrasound image are registered so that the soft tissue in the three-dimensional CT image is aligned with the soft tissue in the three-dimensional ultrasound image, thereby obtaining a registered three-dimensional CT image.

6. A skeleton-based registration apparatus, characterized by, The skeleton-based registration device includes: An acquisition unit is used to acquire a first three-dimensional point cloud and a second three-dimensional point cloud. Both the first three-dimensional point cloud and the second three-dimensional point cloud include information corresponding to the skeleton of the first object. The first three-dimensional point cloud is obtained based on the target information acquired by the first acquisition device, and the second three-dimensional point cloud is obtained based on the target information acquired by the second acquisition device. The target information includes information related to the first object. The processing unit is used to register the first three-dimensional point cloud and the second three-dimensional point cloud to obtain a first registration parameter. The first registration parameter is used to register the target information acquired by the first acquisition device and the target information acquired by the second acquisition device. The registration of the first 3D point cloud and the second 3D point cloud to obtain the first registration parameters includes: obtaining a first covariance matrix based on the first 3D point cloud; obtaining a second covariance matrix based on the second 3D point cloud; obtaining a first eigenvector based on the first covariance matrix, wherein the eigenvalue corresponding to the first eigenvector is the nth largest eigenvalue among the eigenvalues ​​corresponding to the eigenvectors of the first covariance matrix, where n is a positive integer; obtaining a second eigenvector based on the second covariance matrix, wherein the eigenvalue corresponding to the second eigenvector is the nth largest eigenvalue among the eigenvalues ​​corresponding to the eigenvectors of the second covariance matrix; obtaining first bending information based on the first 3D point cloud, wherein the first bending information includes information related to the bending direction of the skeleton of the first object; and obtaining the first eigenvector based on the second 3D point cloud. The first object has two bending information, including information related to the bending direction of its skeleton. Based on the first and second bending information, a second registration parameter is obtained. After translating and / or rotating the first 3D point cloud based on the second registration parameter, the bending direction of the skeleton of the first object indicated by the first 3D point cloud matches the bending direction of the skeleton of the first object indicated by the second 3D point cloud. Based on the first and second feature vectors, the first and second 3D point clouds are registered so that the first and second feature vectors satisfy a first condition, thus obtaining the first registration parameter. The first condition includes the angle between the first and second feature vectors being less than a first angle threshold or greater than a second angle threshold. The step of registering the first 3D point cloud and the second 3D point cloud based on the first feature vector and the second feature vector, so that the first feature vector and the second feature vector satisfy a first condition, and obtaining the first registration parameter, includes: registering the first 3D point cloud and the second 3D point cloud based on the first feature vector and the second feature vector, so that the first feature vector and the second feature vector satisfy the first condition, and obtaining a third registration parameter; and obtaining the first registration parameter based on the second registration parameter and the third registration parameter.

7. A surgical robot, characterised in that, Includes the skeleton-based registration device as described in claim 6.

8. An electronic device, comprising: include: A processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1 to 5.

Images