Multi-camera point cloud non-rigid registration method and system
Through the multi-camera point cloud non-rigid registration method, the problem of inability to accurately monitor subtle position changes in traditional radiation therapy is solved, and radiotherapy with higher accuracy and safety is achieved.
Patent Information
- Application Number
- CN202510250274.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-04
- Publication Date
- 2025-06-13
AI Technical Summary
In traditional radiation therapy, doctors rely on visual evaluation of the patient's real-time position and cannot accurately reflect the subtle displacement caused by physiological movements such as breathing, resulting in insufficient treatment accuracy.
The non-rigid registration method of multi-camera point clouds is adopted, and multiple depth images are obtained through synchronous processing of multiple depth cameras, and the point cloud data is mapped to the three-dimensional space, and rigid rotation, denoising and interpolation deformation are performed to achieve accurate position registration.
This method can more accurately detect and represent subtle changes in position during treatment, improve the accuracy and safety of radiation therapy, and reduce the impact on surrounding healthy tissues.
Smart Images
Figure CN120147381A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a multi-camera point cloud non-rigid registration method and system. Background Art
[0002] Radiation therapy is one of the key methods for tumor treatment today. It aims to precisely focus high-dose radiation on the tumor area to maximize the destruction of cancer cells while minimizing the impact on surrounding healthy tissues. However, during the treatment process, the patient's body position may change due to physiological activities such as breathing, which may lead to a deviation in the radiation irradiation area.
[0003] To overcome this challenge and ensure that each treatment can accurately target the tumor, it is crucial to monitor the patient's position in real time. Traditional methods often rely on doctors to evaluate the patient's real-time position based on personal experience and observation, which obviously lacks objectivity and consistency.
[0004] Therefore, it is very necessary to introduce auxiliary tools that can quantify the patient's body position changes for improving the accuracy and effectiveness of radiation therapy. These tools can help doctors more accurately judge the patient's pose, thereby improving the safety and effectiveness of treatment and ensuring that the radiation can accurately act on the target area. In this way, the radiation therapy process can be further optimized to provide higher-quality medical services for patients.
[0005] During the traditional radiation therapy monitoring process, doctors mainly rely on visual assessment of the patient's real-time body position to judge the accuracy of treatment. In recent years, with the progress of technology, a more advanced method is to use a depth camera to capture information on the patient's optical surface and automatically monitor the patient's body position through rigid registration technology. This method improves the automation level and efficiency of body position monitoring, enabling doctors to more conveniently obtain the patient's body surface information.
[0006] However, the above rigid registration technology is not sensitive enough to the subtle displacements caused by physiological movements such as breathing and cannot accurately reflect these minute changes. To more precisely handle the body position changes during the treatment process, especially those non-rigid displacements caused by breathing, a more suitable registration method or system is urgently needed. Summary of the Invention
[0007] In view of this, it is necessary to provide a multi-camera point cloud non-rigid registration method and system, which can ensure the accuracy and time efficiency of patient treatment while minimizing the workload of doctors and the adverse experience of patients.
[0008] The present invention provides a multi-camera point cloud non-rigid registration method, which includes the following steps: S1, synchronize multiple depth cameras to obtain multiple depth images, map the three-dimensional space point cloud data according to the multiple depth images, and perform rigid rotation on the multiple point cloud data to obtain coarsely fused point cloud data; S2, perform point cloud denoising on each vertex in the coarsely fused point cloud data to obtain finely fused point cloud data; S3, perform uniform sampling on the finely fused point cloud data, search for the K-nearest neighbor relationship, filter and interpolate the K-nearest neighbors, calculate the correspondence and displacement vector, construct an equation set based on the thin plate spline function, solve the parameter matrix, and perform interpolation deformation on the finely fused point cloud data to achieve accurate registration.
[0009] Preferably, in step S4, according to the finely fused point cloud data obtained in step S2 and the point cloud data after interpolation deformation in step S3, calculate the deviation within the ROI region.
[0010] Preferably, step S1 includes:
[0011] Construct a master-slave relationship by connecting multiple cameras, strictly lock the acquisition time of each camera at the same moment to ensure that each camera captures images simultaneously; the depth image obtained by each depth camera is two-dimensional data in pixels, where each pixel records the depth value from the camera; use the internal parameters of the camera to convert the two-dimensional pixel coordinates and depth information into three-dimensional coordinates:
[0012]
[0013] Apply the above formula to each pixel, thereby converting the entire depth image into a point cloud;
[0014] Obtain the rotation matrix R and displacement vector t of each camera relative to the global coordinate system through a calibration method; for the point P obtained by each camera in its camera coordinate system local , use the external parameters to convert it to the global coordinate system:
[0015] P global =R·P local +t
[0016] Apply the above rigid transformation to the point cloud data obtained by all depth cameras respectively to fuse them into the same coordinate system, thereby obtaining coarsely fused point cloud data.
[0017] Preferably, step S2 includes:
[0018] Step S21, perform neighborhood sampling and normal vector update on each vertex in the coarsely fused point cloud data;
[0019] Step S22: For each vertex, construct the comprehensive quadratic error matrix of this vertex;
[0020] Step S23: According to the QEM of each vertex and the current normal vector, perform constrained optimization to solve for the new vertex and update iteratively.
[0021] Preferably, the step S22 includes:
[0022] For the local fitting plane ax + by + cz + d = 0, the extended coordinate is v = [x, y, z, 1] T , and the error is expressed as:
[0023] E(v) = v T Qv
[0024] where:
[0025]
[0026] For each vertex, by accumulating the Q f generated by multiple local planes within its neighborhood, the comprehensive quadratic error matrix Q of this vertex is obtained.
[0027] Preferably, the step S23 includes:
[0028] According to the QEM of each vertex and the current normal vector, solve for the optimal new vertex position to minimize the local quadratic error while satisfying the geometric constraint in the direction of the normal vector; the optimization problem to be solved is expressed as:
[0029]
[0030] Obtain the solution of λ under the optimal conditions through calculation; if the error E(v) of the new vertex is less than the initial error E(v 0 ), then update the vertex coordinates, otherwise maintain the initial vertex coordinates;
[0031] Each time, take the updated vertex position as the new input and repeat local optimization; after several iterations, the noise introduced by depth deviation and stitching error in the original coarsely fused point cloud data is gradually suppressed, and finally a smoothly and finely fused point cloud data is obtained.
[0032] Preferably, the step S3 includes:
[0033] Step S31: Construct an OBB based on the finely fused point cloud data and perform uniform sampling;
[0034] Step S32: Construct KD Trees for the sampled point cloud and the target point cloud respectively, and use GPU acceleration for K-nearest neighbor search to obtain the K-nearest neighbor relationship;
[0035] Step S33: Calculate the corresponding relationship, displacement vector, and target point set based on the above K-nearest neighbor search.
[0036] Step S34: Solve the TPS parameters based on the obtained target point set and perform interpolation on the finely fused point cloud data.
[0037] Preferably, the K-nearest neighbor relationship includes: the K-nearest neighbor relationship of each point in the sampled point cloud, the K-nearest neighbor relationship from the sampled point cloud to the target point cloud, and the K-nearest neighbor relationship from the target point cloud to the sampled point cloud.
[0038] The present invention also provides a multi-camera point cloud non-rigid registration system, which includes a processing module, a denoising module, and an accurate registration module, where:
[0039] The processing module is used to synchronize multiple depth cameras to obtain multiple depth images, map the multiple depth images to obtain point cloud data in three-dimensional space, and perform rigid rotation on the multiple point cloud data to obtain coarsely fused point cloud data.
[0040] The denoising module is used to perform point cloud denoising on each vertex in the coarsely fused point cloud data to obtain finely fused point cloud data.
[0041] The accurate registration module is used to uniformly sample the finely fused point cloud data, search for the K-nearest neighbor relationship, filter and interpolate the K-nearest neighbors, calculate the corresponding relationship and displacement vector, construct an equation system based on the thin plate spline function, solve the parameter matrix, and perform interpolation deformation on the finely fused point cloud data to achieve accurate registration.
[0042] Preferably, the system further includes a deviation calculation module, which is used to calculate the deviation within the ROI area based on the finely fused point cloud data obtained by the denoising module and the point cloud data after interpolation deformation by the accurate registration module.
[0043] This application generates, denoises, performs thin plate spline registration accelerated by cuda, and calculates the deviation within the ROI area on the depth images collected by multiple cameras, and finally obtains the deformation result of the local area of the patient. This application can more sensitively and accurately detect and represent the subtle body position changes that occur during treatment, thereby providing more reliable and refined guidance for doctors, ensuring the safety and effectiveness of radiotherapy, and further improving the treatment accuracy. Description of the Drawings
[0044] Figure 1 It is a flowchart of the multi-camera point cloud non-rigid registration method of the present invention;
[0045] Figure 2This is the hardware architecture diagram of the multi-camera point cloud non-rigid registration system of the present invention. Specific embodiments
[0046] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0047] Refer to Figure 1 As shown, it is the operation flowchart of the preferred embodiment of the multi-camera point cloud non-rigid registration method of the present invention.
[0048] Step S1: Synchronize multiple depth cameras to obtain multiple depth images. According to the multiple depth images, map to obtain point cloud data in three-dimensional space, and perform rigid rotation on the multiple point cloud data to obtain coarsely fused point cloud data. That is:
[0049] Synchronize multiple depth cameras to ensure that the time error between the obtained depth images does not exceed 1 frame. For multiple depth images, use the camera internal parameters to map the two-dimensional depth information to point cloud data in three-dimensional space. Use the camera external parameters to perform rigid rotation transformation on the multiple point cloud data to the same coordinate system to obtain coarsely fused point cloud data. Specifically:
[0050] To ensure that the data collected by each depth camera can correspond to the scene information at the same time point, it is necessary to perform hardware synchronization on multiple cameras. That is, by connecting multiple cameras to establish a master-slave relationship, strictly lock the acquisition time of each camera at the same moment to ensure that each camera captures images simultaneously. The depth image obtained by each depth camera is two-dimensional data in pixels, where each pixel records the depth value from the camera. Use the internal parameters of the camera to convert the two-dimensional pixel coordinates and depth information into three-dimensional coordinates:
[0051]
[0052] Apply the above formula to each pixel, thereby converting the entire depth image into a point cloud.
[0053] Obtain the rotation matrix R and displacement vector t of each camera relative to the global coordinate system through the calibration method. For the point P local obtained in the camera coordinate system of each camera, use the external parameters to convert it to the global coordinate system:
[0054] P global = R·P local + t
[0055] Apply the above rigid transformation to the point cloud data obtained by all depth cameras respectively to fuse them into the same coordinate system, thereby obtaining coarsely fused point cloud data.
[0056] Step S2: Denoise each vertex in the coarsely fused point cloud data to obtain finely fused point cloud data. That is:
[0057] Due to the deviation of depth data and the error in stitching after rigid transformation of multiple point clouds, there is certain noise in the coarsely fused point cloud data. A QEM (Quadic Error Metrics, mesh simplification) method accelerated by cuda is proposed for smoothing to reduce point cloud noise. This method first selects the neighboring points of a vertex as a point set and constructs the fitting plane of this point set, and then obtains the depth value of the optimized vertex by optimizing the quadratic error matrix. This process is performed for each vertex to finally obtain finely fused point cloud data. Specifically:
[0058] Step S21: Perform neighborhood sampling and normal vector update for each vertex in the coarsely fused point cloud data.
[0059] For each vertex v in the coarsely fused point cloud data, first construct a KD Tree and perform radius search to obtain its local neighborhood point set S. Then, accumulate and average the normal vectors of all points in its neighborhood S and normalize them to obtain the updated normal vector.
[0060] Step S22: Construct the comprehensive quadratic error matrix for each vertex.
[0061] For the local fitting plane ax + by + cz + d = 0, the extended coordinate is v = [x, y, z, 1] T , and the error is expressed as:
[0062] E(v) = v T Qv
[0063] Where:
[0064]
[0065] For each vertex, by accumulating the Q f generated by multiple local planes in its neighborhood, obtain the comprehensive quadratic error matrix Q of this vertex.
[0066] Step S23: According to the QEM and the current normal vector of each vertex, perform constrained optimization to solve for the new vertex and update it iteratively.
[0067] According to the QEM and the current normal vector of each vertex, solve for the optimal new vertex position to minimize the local quadratic error while satisfying the geometric constraint of the normal vector direction. The optimization problem to be solved is expressed as:
[0068]
[0069] The solution of λ under the optimal conditions is obtained through calculation. If the error E(v) of the new vertex is less than the initial error E(v 0 ), the vertex coordinates are updated; otherwise, the initial vertex coordinates are maintained.
[0070] Each time, the updated vertex position is used as the new input, and local optimization is repeated. After several iterations, the noise introduced by depth deviation and stitching error in the original coarsely fused point cloud data is gradually suppressed, and finally a smoothly and finely fused point cloud data is obtained.
[0071] Step S3: Uniformly sample the finely fused point cloud data, search for the K-nearest neighbor relationship, filter and interpolate the K-nearest neighbors, calculate the correspondence and displacement vectors, construct an equation system based on the thin plate spline function, solve the parameter matrix, and perform interpolation deformation on the finely fused point cloud data to achieve precise registration. That is:
[0072] First, calculate the OBB (Oriented Bounding Box) of the finely fused point cloud data and perform voxelization uniform sampling on it; then construct the KDTree of the sampled point cloud and the target point cloud, and use the GPU to perform KNN search to obtain the nearest neighbor relationship; next, filter and interpolate the K-nearest neighbors, calculate the correspondence and displacement vectors, and correct them to the normal direction; finally, construct an equation system based on the thin plate spline function, solve the parameter matrix, and perform interpolation deformation on the finely fused point cloud data to achieve precise registration. Specifically:
[0073] Step S31: Construct the OBB according to the finely fused point cloud data and perform uniform sampling:
[0074] Calculate the OBB according to the finely fused point cloud data to obtain the representation of the point cloud in the new xyz coordinate system. Perform voxelization on the OBB, and randomly select a point in each non-empty voxel as the sampling point to achieve uniform sampling.
[0075] Step S32: Construct the KDTree for the sampled point cloud and the target point cloud respectively, and use the GPU to accelerate the K-nearest neighbor search (KNN search) to obtain the following three types of K-nearest neighbor relationships:
[0076] · The K-nearest neighbor relationship of each point in the sampled point cloud;
[0077] · The K-nearest neighbor relationship from the sampled point cloud to the target point cloud;
[0078] · The K-nearest neighbor relationship from the target point cloud to the sampled point cloud.
[0079] Step S33: Calculate the correspondence, displacement vector, and target point set based on the above K-nearest neighbor search.
[0080] Perform three-step filtering on the KNN results:
[0081] Bidirectional consistency: The candidate corresponding points a and b correspond to each other, that is: a is the corresponding point of b, and b is the corresponding point of a;
[0082] Distance threshold: Satisfy ∥a - b∥ < d thresh ;
[0083] Normal direction threshold: It is required that the normal vectors of a and b satisfy na·nb > t thresh .
[0084] For valid corresponding points, directly calculate the displacement vector:
[0085] d = b - a
[0086] For invalid corresponding points, interpolate through the weighted average displacement of their neighborhood points, and the weights are set as:
[0087]
[0088] Thus, the interpolated displacement is obtained:
[0089]
[0090] Finally, the target point set is composed of the sampling points plus their respective displacements, and is corrected in the normal direction.
[0091] Step S34, according to the obtained target point set, solve the TPS parameters and interpolate the finely fused point cloud data.
[0092] Construct the following linear equations based on the thin plate spline function:
[0093] Among them:
[0094] K ij = U(∥x i - x j ∥),
[0095] P is the affine term matrix, and each row is in the form of [1, x i , y i , z i ;
[0096] Y is the target point set.
[0097] After solving the TPS parameters, perform a calculation transformation on any point X = (x, y, z) in the finely fused point cloud data:
[0098]
[0099] Obtain the deformed point cloud to achieve precise registration. The last step of interpolation calculation is accelerated using CUDA (Compute Unified Device Architecture).
[0100] Step S4: Calculate the deviation within the ROI (region of interest) based on the finely fused point cloud data obtained in step S2 and the point cloud data after interpolation and deformation in step S3.
[0101] Using the finely fused point cloud data obtained in step S2 and the point cloud data after interpolation and deformation in step S3 that already exist, calculate the offset value for each corresponding group. Average all the offset values to obtain the deviation within the ROI region.
[0102] Refer to Figure 2 As shown, it is the hardware architecture diagram of the multi-camera point cloud non-rigid registration system 10 of the present invention. The system includes: a processing module 101, a denoising module 102, a precise registration module 103, and a deviation calculation module 104. Among them:
[0103] The processing module 101 is used to synchronize multiple depth cameras to obtain multiple depth images, map the multiple depth images to obtain point cloud data in three-dimensional space, and perform rigid rotation on the multiple point cloud data to obtain coarsely fused point cloud data. That is:
[0104] The processing module 101 synchronizes multiple depth cameras to ensure that the time error between the acquired depth images does not exceed 1 frame. For multiple depth images, use the camera internal parameters to map the two-dimensional depth information to point cloud data in three-dimensional space. Use the camera external parameters to perform rigid rotation transformation on the multiple point cloud data to the same coordinate system to obtain coarsely fused point cloud data. Specifically:
[0105] To ensure that the data collected by each depth camera can correspond to the scene information at the same time point, it is necessary to perform hardware synchronization on multiple cameras. That is, by connecting multiple cameras to establish a master-slave relationship, strictly lock the acquisition time of each camera at the same moment to ensure that each camera captures images simultaneously. The depth image obtained by each depth camera is two-dimensional data in pixels, where each pixel records the depth value from the camera. Use the internal parameters of the camera to convert the two-dimensional pixel coordinates and depth information into three-dimensional coordinates:
[0106]
[0107] Apply the above formula to each pixel, thereby converting the entire depth image into a point cloud.
[0108] The rotation matrix R and displacement vector t of each camera relative to the global coordinate system are obtained through a calibration method. For the point P in the camera coordinate system obtained by each camera local , it is transformed to the global coordinate system using the external parameters:
[0109] P global = R·P local + t
[0110] The point cloud data obtained by all depth cameras are respectively applied with the above rigid transformation to fuse them into the same coordinate system, thereby obtaining the coarsely fused point cloud data.
[0111] The denoising module 102 is used to perform point cloud denoising on each vertex in the coarsely fused point cloud data to obtain the finely fused point cloud data. That is:
[0112] Due to the deviation of depth data and the error in stitching after the rigid transformation of multiple point clouds, there is certain noise in the coarsely fused point cloud data. The denoising module 102 uses the QEM (Quadic Error Metrics, mesh simplification) method accelerated by cuda for smoothing processing to reduce the point cloud noise. The denoising module 102 first selects the neighboring points of a vertex as a point set and constructs the fitting plane of this point set, and then obtains the depth value of the optimized vertex by optimizing the quadratic error matrix. This process is performed for each vertex to finally obtain the finely fused point cloud data. Specifically:
[0113] The denoising module 102 performs neighborhood sampling and normal vector update for each vertex in the coarsely fused point cloud data.
[0114] For each vertex v in the coarsely fused point cloud data, first construct a KD Tree and perform radius search to obtain its local neighborhood point set S. Then, the normal vectors of all points in its neighborhood S are accumulated and averaged, and normalized to obtain the updated normal vector.
[0115] The denoising module 102 constructs the comprehensive quadratic error matrix of each vertex.
[0116] For the local fitting plane ax + by + cz + d = 0, the extended coordinate is v = [x, y, z, 1] T , and the error is expressed as:
[0117] E(v) = v T Qv
[0118] Where:
[0119]
[0120] For each vertex, the Q generated by accumulating multiple local planes within its neighborhood f is used to obtain the comprehensive quadratic error matrix Q for that vertex.
[0121] The denoising module 102 constrains and optimizes to solve for new vertices and iteratively updates them based on the QEM and the current normal vector of each vertex.
[0122] Based on the QEM and the current normal vector of each vertex, the optimal new vertex position is solved to minimize the local quadratic error while satisfying the geometric constraints of the normal vector direction. The optimization problem to be solved is expressed as:
[0123]
[0124] The solution of λ under optimal conditions is obtained through calculation. If the error E(v) of the new vertex is less than the initial error E(v 0 ), then the vertex coordinates are updated; otherwise, the initial vertex coordinates are maintained.
[0125] Each time, the updated vertex position is used as the new input, and local optimization is repeated. After several iterations, the noise introduced by depth deviation and stitching error in the original coarsely fused point cloud data is gradually suppressed, and finally a smoothly and finely fused point cloud data is obtained.
[0126] The precise registration module 103 is used to perform uniform sampling on the finely fused point cloud data, search for and obtain the K-nearest neighbor relationship, filter and interpolate the K-nearest neighbors, calculate the correspondence and displacement vectors, construct an equation system based on the thin plate spline function, solve the parameter matrix, and perform interpolation deformation on the finely fused point cloud data to achieve precise registration. That is:
[0127] The precise registration module 103 first calculates the OBB (Oriented Bounding Box) of the finely fused point cloud data and performs voxelized uniform sampling on it; then constructs the KD Tree of the sampled point cloud and the target point cloud, and uses the GPU to perform KNN search to obtain the nearest neighbor relationship; then filters and interpolates the K-nearest neighbors, calculates the correspondence and displacement vectors, and corrects them to the normal direction; finally, constructs an equation system based on the thin plate spline function, solves the parameter matrix, and performs interpolation deformation on the finely fused point cloud data to achieve precise registration. Specifically:
[0128] The precise registration module 103 constructs the OBB based on the finely fused point cloud data and performs uniform sampling:
[0129] The precise registration module 103 calculates the OBB based on the finely fused point cloud data, and obtains the representation of the point cloud in the new xyz coordinate system. The OBB is voxelized, and a point is randomly selected as a sampling point in each non-empty voxel to achieve uniform sampling.
[0130] The precise registration module 103 constructs KD Trees for the sampled point cloud and the target point cloud respectively, and uses GPU acceleration for K-nearest neighbor search (KNN search) to obtain the following three types of K-nearest neighbor relationships:
[0131] · The K-nearest neighbor relationship of each point in the sampled point cloud;
[0132] · The K-nearest neighbor relationship from the sampled point cloud to the target point cloud;
[0133] · The K-nearest neighbor relationship from the target point cloud to the sampled point cloud.
[0134] The precise registration module 103 calculates the corresponding relationship, displacement vector, and target point set based on the above K-nearest neighbor search.
[0135] Perform three-step filtering on the KNN results:
[0136] Bidirectional consistency: The candidate corresponding points a and b are corresponding to each other, that is: a is the corresponding point of b, and b is the corresponding point of a;
[0137] Distance threshold: Satisfy ∥a - b∥ < d thresh ;
[0138] Normal direction threshold: It is required that the normal vectors of a and b satisfy na · nb > t thresh .
[0139] For valid corresponding points, directly calculate the displacement vector:
[0140] d = b - a
[0141] For invalid corresponding points, interpolate through the weighted average displacement of their neighborhood points, and the weight is set as:
[0142]
[0143] Thus, the interpolated displacement is obtained:
[0144]
[0145] Finally, the target point set is composed of the sampled points plus their respective displacements, and is corrected in the normal direction.
[0146] The precise registration module 103 solves the TPS parameters based on the obtained target point set, and interpolates the finely fused point cloud data.
[0147] Construct the following system of linear equations based on thin plate spline functions:
[0148] Where:
[0149] K ij = U(∥x i - x j ∥),
[0150] P is the affine term matrix, and each row is in the form of [1, x i , y i , z i ;
[0151] Y is the set of target points.
[0152] After obtaining the TPS parameters, perform a calculation transformation on any point X = (x, y, z) in the finely fused point cloud data:
[0153]
[0154] Obtain the deformed point cloud to achieve accurate registration. The last interpolation calculation is accelerated using cuda (Compute Unified Device Architecture).
[0155] The deviation calculation module 104 is used to calculate the deviation within the ROI (region of interest) according to the finely fused point cloud data obtained by the denoising module 102 and the point cloud data after interpolation deformation by the accurate registration module 103. Specifically:
[0156] Use the finely fused point cloud data obtained by the existing denoising module 102 and the point cloud data after interpolation deformation by the accurate registration module 103 to calculate each set of corresponding offset values. Average all the offset values to obtain the deviation within the ROI region.
[0157] The present invention denoises the point cloud fused by multiple cameras to obtain a fused point cloud with higher quality. At the same time, a more suitable registration method is adopted for non-rigid deformation and GPU acceleration is used, thus achieving higher-precision and more efficient real-time patient pose monitoring. At the same time, the present invention realizes the efficient calculation of non-rigid deformation errors, which is beneficial to the radiotherapy experience of patients and the treatment operations of doctors.
[0158] Although the present invention is described with reference to the current preferred embodiments, those skilled in the art should understand that the above preferred embodiments are only used to illustrate the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle scope of the present invention shall be included within the scope of the present invention's rights protection.
Claims
1. A multi-camera point cloud non-rigid registration method, characterized in that: The method comprises the following steps: S1, synchronously processing multiple depth cameras to obtain multiple depth images, mapping point cloud data in three-dimensional space according to the multiple depth images, and rigidly rotating the multiple point cloud data to obtain roughly fused point cloud data; S2, performing point cloud denoising on each vertex in the coarsely fused point cloud data to obtain finely fused point cloud data; S3, uniformly sampling the finely fused point cloud data, searching for K nearest neighbor relationships, filtering and interpolating the K nearest neighbors, calculating corresponding relationships and displacement vectors, constructing a set of equations based on thin plate spline functions, solving parameter matrices, and interpolating and deforming the finely fused point cloud data to achieve precise alignment.
2. The multi-camera point cloud non-rigid registration method according to claim 1, characterized in that: The method further comprises: Step S4, calculating the deviation within the ROI area according to the finely fused point cloud data obtained in step S2 and the point cloud data after interpolation and deformation in step S3.
3. The multi-camera point cloud non-rigid registration method according to claim 2, characterized in that: The step S1 comprises: By connecting multiple cameras to build a master-slave relationship, the acquisition time of each camera is strictly locked at the same moment to ensure that each camera captures the image at the same time; the depth image obtained by each depth camera is two-dimensional data in pixels, where each pixel records the depth value from the camera; the two-dimensional pixel coordinates and depth information are converted into three-dimensional coordinates using the camera's internal parameters: Apply the above formula to each pixel to convert the entire depth image into a point cloud; The rotation matrix R and displacement vector t of each camera relative to the global coordinate system are obtained by calibration method; for each camera, the point P in its camera coordinate system is obtained local , and use external parameters to transform it to the global coordinate system: P global =R·P local +t The above rigid transformation is applied to the point cloud data acquired by all depth cameras respectively to fuse them into the same coordinate system, thereby obtaining roughly fused point cloud data.
4. The multi-camera point cloud non-rigid registration method according to claim 3, characterized in that: The step S2 comprises: Step S21, performing neighborhood sampling and normal vector update for each vertex in the roughly fused point cloud data; Step S22: for each vertex, construct a comprehensive quadratic error matrix of the vertex; Step S23: According to the QEM and current normal vector of each vertex, constrain optimization is used to solve the new vertex and iteratively update it.
5. The multi-camera point cloud non-rigid registration method according to claim 4, characterized in that: The step S22 comprises: For the local fitting plane ax+by+cz+d=0, the extended coordinates are v=[x,y,z,1] T , the error is expressed as: E(v)=v T Qv in: For each vertex, Q is generated by accumulating multiple local planes in its neighborhood f , and obtain the comprehensive quadratic error matrix Q of the vertex.
6. The multi-camera point cloud non-rigid registration method according to claim 5, characterized in that: The step S23 comprises: According to the QEM and current normal vector of each vertex, the optimal new vertex position is solved to minimize the local quadratic error and satisfy the geometric constraints of the normal vector direction; the optimization problem to be solved is expressed as: The solution of λ under the optimal conditions is obtained by calculation; if the error E(v) of the new vertex is less than the initial error E(v0), the vertex coordinates are updated, otherwise the initial vertex coordinates are maintained; Each time, the updated vertex position is used as a new input and local optimization is repeated; after several iterations, the noise introduced by the depth deviation and stitching error in the original roughly fused point cloud data is gradually suppressed, and finally a smooth and finely fused point cloud data is obtained.
7. The multi-camera point cloud non-rigid registration method according to claim 6, characterized in that: The step S3 comprises: Step S31, constructing an OBB according to the finely fused point cloud data, and performing uniform sampling; Step S32, constructing KD Tree for the sampling point cloud and the target point cloud respectively, and performing K nearest neighbor search using GPU acceleration to obtain K nearest neighbor relationship; Step S33, according to the above K nearest neighbor search, calculate the corresponding relationship, displacement vector and target point set; Step S34: solving TPS parameters according to the obtained target point set, and interpolating the finely fused point cloud data.
8. The multi-camera point cloud non-rigid registration method according to claim 7, characterized in that: The K nearest neighbor relationship includes: the K nearest neighbor relationship of each point in the sampling point cloud, the K nearest neighbor relationship from the sampling point cloud to the target point cloud, and the K nearest neighbor relationship from the target point cloud to the sampling point cloud.
9. A multi-camera point cloud non-rigid registration system, characterized in that: The system includes a processing module, a denoising module, and a precise registration module, wherein: The processing module is used to synchronously process multiple depth cameras to obtain multiple depth images, map point cloud data in three-dimensional space according to the multiple depth images, and rigidly rotate the multiple point cloud data to obtain roughly fused point cloud data; The denoising module is used to perform point cloud denoising on each vertex in the roughly fused point cloud data to obtain finely fused point cloud data; The precise registration module is used to uniformly sample the finely fused point cloud data, search for K nearest neighbor relationships, filter and interpolate the K nearest neighbors, calculate corresponding relationships and displacement vectors, construct a set of equations based on thin plate spline functions, solve the parameter matrix, and interpolate and deform the finely fused point cloud data to achieve precise registration.
10. The multi-camera point cloud non-rigid registration system according to claim 9, characterized in that: The system also has a deviation calculation module, which is used to calculate the deviation in the ROI area based on the finely fused point cloud data obtained by the denoising module and the point cloud data after interpolation and deformation by the precise registration module.