A transcranial magnetic navigation face registration device

Abstract

The application discloses a transcranial magnetic navigation face registration device, and relates to the technical field of computer vision, which collects a depth map and a color map of a target region and records camera intrinsic parameters. Face detection and face segmentation processing are performed on the depth map and the color map to generate a trusted area mask. Based on the camera intrinsic parameters, the depth map and the trusted area mask, a face trusted point cloud is generated; the face trusted point cloud is registered with a head surface model based on an image space to obtain a spatial transformation matrix. When the spatial transformation matrix meets registration requirements, target point navigation of a transcranial magnetic navigation system is performed according to the spatial transformation matrix. Under the condition of not contacting the face of a user, the invasiveness is reduced by analyzing the depth map and the color map collected by the camera. The spatial transformation matrix can be obtained through automatic registration, and registration requirements are set, so that a non-contact, automatic and quantifiable evaluation spatial registration scheme of a TMS navigation system is realized.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a transcranial magnetic navigation facial registration device. Background Technology

[0002] Transcranial magnetic stimulation (TMS) navigation systems in clinical and research applications typically require spatial registration between the patient's magnetic resonance imaging (MRI) space and physical space to achieve precise correspondence between stimulation target points and coil positions. Currently, a common approach involves manually selecting several feature points on the patient's face using an optically labeled probe tool for coarse registration, followed by fine registration through sliding sampling of the probe across the face.

[0003] While this approach can achieve registration, it has significant shortcomings in engineering applications. Probe contact with the face can easily cause discomfort and resistance, especially in anxious individuals, children, or psychiatric patients, resulting in an unpleasant patient experience. Operation relies on operator experience; variations in point selection, contact pressure, and sliding coverage lead to error fluctuations and unstable repeatability. Multiple point selections and sliding sampling increase preparation time, impacting clinical throughput. Registration quality often lacks a clear indicator system and automated judgment mechanism, posing a risk of continuing the procedure despite poor registration. Furthermore, TMS often involves multiple treatment sessions and procedures, and repeated probe registration increases treatment costs.

[0004] It is evident that how to achieve contactless, automated, and quantifiable spatial registration for TMS navigation systems is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a transcranial magnetic navigation facial registration device that can achieve contactless, automated, and quantifiable spatial registration of the TMS navigation system.

[0006] This application provides a transcranial magnetic navigation facial registration device, including an acquisition unit, a mask generation unit, a point cloud generation unit, a registration unit, and a navigation unit; The acquisition unit is used to acquire depth and color maps of the target area and record camera intrinsic parameters; The mask generation unit is used to perform face detection and facial segmentation on the depth map and color map to generate a trusted region mask. The point cloud generation unit is used to generate a believable facial point cloud based on camera intrinsic parameters, depth map, and believable region mask. The registration unit is used to register the reliable facial point cloud with the image space-based head surface model to obtain the spatial transformation matrix; The navigation unit is used to perform target navigation of the transcranial magnetic navigation system based on the spatial transformation matrix, provided that the spatial transformation matrix meets the registration requirements.

[0007] On the one hand, the mask generation unit includes a detection subunit, a segmentation subunit, and a removal subunit; The detection subunit is used to perform face detection on the color image and determine the face region bounding box. The segmentation subunit is used to segment the face region bounding box using the depth map and generate a face mask. The culling sub-unit is used to remove anomalous regions from the face mask to obtain a reliable region mask; wherein, anomalous regions include deep hole regions, noise regions, and deformed regions.

[0008] On the one hand, it also includes a first judgment unit, a first determination unit, a second judgment unit, and a second determination unit; The first judgment unit is used to determine whether the effective depth value of the depth map is greater than or equal to the preset depth value, and whether the depth distance of the face region in the depth map is within the preset range. The first determination unit is used to determine that the depth map is a valid depth map when the effective depth value of the depth map is greater than or equal to the preset depth value and the depth distance of the face region in the depth map is within the preset range. The second judgment unit is used to determine whether the center drift of the face region box in multiple consecutive color images is less than a preset threshold. The second determination unit is used to determine that a color image is a valid color image if the center drift of the face region bounding box in multiple consecutive color images is less than a preset threshold.

[0009] On the one hand, the point cloud generation unit includes a back projection subunit, a fusion subunit, and a filtering subunit; The back-projection subunit is used to back-project multi-frame depth maps based on camera intrinsic parameters to generate multi-frame point clouds. The fusion subunit is used to fuse the preprocessed point clouds from multiple frames to obtain a fused point cloud; The filtering subunit is used to filter the fused point cloud based on the trusted region mask to obtain a trusted facial point cloud.

[0010] On the one hand, the fusion subunit is used to denoise the multi-frame point cloud and perform voxel downsampling on the denoised multi-frame point cloud to obtain the pre-processed multi-frame point cloud. Align the preprocessed multi-frame point clouds based on the changes in face pose between adjacent frames; The aligned point clouds from multiple frames are superimposed onto the same coordinate system, and the overlapping areas are fused to obtain a fused point cloud.

[0011] On one hand, a filtering sub-unit is used to determine whether the pixel position of the target point in the fused point cloud is within the trusted region mask; where the target point is any pixel in the fused point cloud; If the pixel position of the target point is within the trusted region mask, set the value of the target point in the trusted region mask to 1; If the pixel position of the target point is not within the trusted region mask, set the value of the target point in the trusted region mask to 0; The set of all target points with a value of 1 in the trusted region mask is taken as the facial trusted point cloud.

[0012] On the one hand, the registration unit includes an initial registration subunit and an iterative optimization subunit; The initial registration subunit is used to perform initial registration between the reliable facial point cloud and the head surface model based on image space to obtain the initial spatial transformation matrix; The iterative optimization subunit is used to iteratively optimize the facial point cloud and the head surface model based on image space, using the initial spatial transformation matrix as the initial value, to obtain the optimized spatial transformation matrix.

[0013] On the one hand, the initial registration subunit is used to align the facial point cloud and the head surface model based on the image space according to the set initial registration rules to obtain the initial spatial transformation matrix; wherein, the initial registration rules include registration based on 3D key points, registration based on head pose prior, or registration based on local geometric feature matching.

[0014] On the one hand, the iterative optimization subunit is used to take the initial spatial transformation matrix as the initial value. Each time the iterative optimization is performed, the matching error between each point in the optimized facial confidence point cloud and the head surface model based on image space is recalculated. Based on the matching error corresponding to each point in the facial reliable point cloud, the spatial transformation matrix of the current iteration is adjusted after optimizing the facial reliable point cloud, and the next iteration optimization is performed until the iteration termination condition is met, and then the final spatial transformation matrix is ​​output.

[0015] On the one hand, the iterative optimization sub-unit is used to remove the credible target point from the credible point cloud when the matching error corresponding to the credible target point in the credible point cloud is greater than or equal to a set first error threshold; wherein, the credible target point is any point in the credible point cloud of the face; If the matching error corresponding to a credible target point in a facial credible point cloud is less than a first error threshold but greater than a set second error threshold, the weight of the credible target point is reduced; wherein the first error threshold is greater than the second error threshold. If the matching error corresponding to a credible target point in a credible point cloud is less than the second error threshold, the credible target point is retained.

[0016] As can be seen from the above technical solution, depth and color images of the target area are acquired, and camera intrinsic parameters are recorded. To remove interference from the background and non-facial areas, face detection and facial segmentation processing can be performed on the depth and color images to generate a reliable region mask. The reliable region mask is an image representation of a reliable region, which is a more stable and suitable area for registration within the facial region. Based on the camera intrinsic parameters, depth image, and reliable region mask, a reliable facial point cloud is generated; the reliable facial point cloud is then registered with a head surface model based on image space to obtain a spatial transformation matrix. To evaluate the registration quality, registration requirements can be set. If the spatial transformation matrix meets the registration requirements, it indicates that accurate registration from physical space to image space can be achieved based on the spatial transformation matrix. Therefore, target navigation of the transcranial magnetic navigation system can be performed based on the spatial transformation matrix. In this technical solution, reliable acquisition of the user's three-dimensional facial information is achieved by analyzing the depth and color images acquired by the camera without contacting the user's face, reducing invasiveness. The spatial transformation matrix can be obtained through automatic registration between the reliable facial point cloud and the head surface model, reducing manual operation and reliance on experience. Furthermore, registration requirements were set, enabling automatic determination of registration results. Target navigation was only performed based on the spatial transformation matrix when the spatial transformation matrix met the registration requirements, thus improving registration stability and engineering usability. This achieved a contactless, automated, and quantifiable spatial registration scheme for the TMS navigation system. Attached Figure Description

[0017] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic diagram of a transcranial magnetic navigation facial registration device provided in this application embodiment; Figure 2 A flowchart illustrating a method for generating a trusted region mask provided in this application embodiment; Figure 3 A flowchart illustrating a method for generating a believable facial point cloud, provided in an embodiment of this application; Figure 4 A flowchart illustrating a method for determining a spatial transformation matrix provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0020] The terms "comprising" and "having," and any variations thereof, in the specification and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may include steps or units not listed.

[0021] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] In the embodiments of this application, the transformation of different coordinate systems and quality assessment are involved. For ease of explanation, different symbols can be used for description. The symbols involved in this application and their meanings will be explained below.

[0023] Σ_img (MRI image coordinate system): The image space coordinate system defined by the MRI image (DICOM) or the coordinate system defined by the MRI reconstruction model. Σ_pat (Patient physical space coordinate system): The patient's reference coordinate system in the real environment, which can be defined by the patient tracking tool, treatment chair coordinate system, or navigation system reference system. Σ_cam (Depth camera coordinate system): The depth camera's own coordinate system. Point cloud P(x, y, z): A set of three-dimensional points obtained by backprojection from the depth map. MRI head surface model M: A three-dimensional mesh or point cloud model obtained by extracting the skin surface / outer surface of the head from MRI (T1 sequence). T_a←b: Rigid body transformation matrix (including rotation R and translation t) from coordinate system Σ_b to coordinate system Σ_a. T_img←pat: Registration transformation matrix from patient physical space to MRI image space, i.e., the spatial transformation matrix finally output by this application. e_rms: Root mean square distance error (mm) from the registered point cloud to the model surface. e_max: Maximum distance error (mm) from the registered point cloud to the model surface. r_inlier: Inlier percentage, i.e., the percentage (%) of points whose distance after registration is less than the threshold δ. c_cov: Confidential region coverage, used to measure whether the point cloud covers stable regions such as the forehead, bridge of the nose, and cheekbone (%). δ (Inlier threshold): Inlier determination distance threshold, with an example range of 1–3 mm. θ1, θ2, θ3, θ4 (Quality thresholds): Threshold parameters used to determine whether registration is qualified; for example, θ1 corresponds to the upper limit of e_rms, θ3 corresponds to the lower limit of r_inlier, etc. α (Effectiveness ratio threshold): Threshold for the percentage of effective pixels within the ROI, with an example range of 50%–80%. P_hat: The fused point cloud obtained by fusing multiple frame point clouds. The example ranges listed above represent the selectable value ranges for these parameters.

[0024] Next, we will describe in detail a transcranial magnetic navigation facial registration device provided in the embodiments of this application. Figure 1 This is a schematic diagram of a transcranial magnetic navigation facial registration device provided in an embodiment of this application. The device includes an acquisition unit 11, a mask generation unit 12, a point cloud generation unit 13, a registration unit 14, and a navigation unit 15.

[0025] Acquisition unit 11 is used to acquire depth and color maps of the target area and record camera intrinsic parameters.

[0026] The target area can be the user's head region. In practical applications, a depth camera can be fixed on the treatment chair / frame, positioned directly in front of the patient, approximately 0.5–0.8 m away, so that its field of view covers the patient's forehead to jaw area. N consecutive frames of depth images (D) and color images (I) are acquired, and the camera's intrinsic parameters (K) are recorded. The value of N can be flexibly set; for example, N=20.

[0027] In addition to depth cameras acquiring depth and color maps, structured light, TOF, and binocular depth cameras can all achieve 3D acquisition; multi-camera fusion can also be used to improve coverage.

[0028] The mask generation unit 12 is used to perform face detection and facial segmentation processing on the depth map and color map to generate a trusted region mask.

[0029] In addition to generating trusted region masks, the mask generation unit 12 can also generate trusted region filtering rules. By outputting these rules, trusted points in the fused point cloud can be filtered. For ease of explanation, this application will use trusted region masks as an example.

[0030] The point cloud generation unit 13 is used to generate a believable facial point cloud based on camera intrinsic parameters, depth map and believable region mask.

[0031] In this application, point clouds can be generated based on depth maps and camera intrinsic parameters through backprojection. The point clouds are then denoised, voxel downsampled, and multi-frame fused to generate a reliable facial point cloud. For ease of explanation, P can be used to represent the point cloud, and P_trust can be used to represent the reliable facial point cloud.

[0032] The registration unit 14 is used to register the reliable facial point cloud with the head surface model based on the image space to obtain the spatial transformation matrix.

[0033] The process of constructing a head surface model based on image space can include inputting an MRI (T1 sequence, slice thickness example 1mm), performing head outer surface extraction, generating a 3D mesh / point cloud model M_mesh or sampling point cloud Q, and if point-to-surface error is used for fine registration, computing normal vectors for the mesh or point cloud. For ease of explanation, the following content will use the head surface model as a point cloud model M as an example for illustration.

[0034] The registration process can be divided into coarse registration and fine registration.

[0035] For coarse registration methods, T0 can be obtained based on keypoint matching, pose prior, feature descriptor matching, template matching, and other methods.

[0036] For fine registration methods, point-to-point, point-to-surface, probabilistic model, grid registration, or deep learning point cloud matching methods can be used.

[0037] Coarse registration is used to generate an initial spatial transformation matrix, which serves as the initial value for fine registration. Coarse registration helps prevent fine registration from getting trapped in local optima and provides a feasible initial alignment.

[0038] Fine registration is used for iterative optimization to obtain the final spatial transformation matrix.

[0039] For ease of description, we can use T0 to represent the initial spatial transformation matrix, and T... This represents the spatial transformation matrix obtained through precise registration.

[0040] Based on prior estimation of key facial structures or pose, the initial rigid body transformation, i.e., the initial spatial transformation matrix T0, is used to initially align P_trust with the point cloud model M. Using T0 as the initial value, iterative optimization is performed within the trusted region to obtain the final transformation T. .

[0041] In this application, a depth camera is used to non-contactly acquire a 3D point cloud of the user's face, and automatic spatial registration is performed with a head surface model constructed from MRI images to generate a spatial transformation matrix for target navigation or robotic arm control. The entire process does not require probe contact with the face, reducing invasiveness and psychological stress, achieving non-contact automatic registration, improving patient comfort and acceptability, and making it suitable for multiple treatment scenarios.

[0042] The navigation unit 15 is used to perform target navigation of the transcranial magnetic navigation system based on the spatial transformation matrix, provided that the spatial transformation matrix meets the registration requirements.

[0043] To achieve automatic determination of registration results, registration requirements can be set, the calculated error index can be evaluated, and the spatial transformation matrix can be determined as qualified or unqualified.

[0044] Registration requirements may include indicators such as root mean square error (e_rms), maximum error (e_max), proportion of inliers with a distance < δ (r_inlier), and reliable region coverage (c_cov). If the threshold conditions are met, the registration is deemed qualified; otherwise, reacquisition / reregistration is triggered.

[0045] The above metrics refer to the matching quality between the registered point cloud and the MRI model. e_rms represents the distance error from all trusted points participating in the evaluation to the surface of the MRI model; e_max represents the maximum value among these distances; r_inlier represents the number of points whose matching distance is less than the threshold δ; c_cov represents the proportion of trusted regions that are effectively participated in and successfully matched.

[0046] In the specific implementation, registration requirements can be set as follows: e_rms≤θ1 (example: 1.0–2.0mm); e_max≤θ2 (example: 3.0–6.0mm); r_inlier≥θ3 (example: 60%–85%); c_cov≥θ4 (example: 50%–80%). If the registration requirements are met, it is considered qualified, and T_img←pat is output and written to the navigation system registration module. If the registration requirements are not met, it is considered unqualified, and the reason is displayed and re-acquisition / re-registration is triggered.

[0047] For registration requirements, indicators such as error quantiles, local area errors, model coverage, and stability time statistics can be used as substitutes or combinations.

[0048] In this application, by establishing a registration quality evaluation index system, the registration results are automatically determined, providing a closed-loop mechanism for automatic reacquisition / reregistration, thereby improving registration stability and engineering usability. Registration quality can be quantified and form a closed-loop control, avoiding continued treatment due to erroneous registration.

[0049] The output spatial transformation matrix can be either T_img←cam or T_img←pat, for use by the navigation system. When the system only outputs T_img←cam, T_img←pat can be calculated from the calibrated T_cam←pat, i.e., T_img←pat = T_img←cam · T_cam←pat, for use in navigation registration.

[0050] The facial registration device provided in this application can work in conjunction with optical positioning. The optical positioning system tracks the device worn by the patient in real time, defined as Σ_pat. The depth camera obtains T_img←cam, and this device obtains T_cam←pat through calibration; it then calculates T_img←pat. The navigation system uses T_img←pat to complete the mapping of the target point from the image to physical space and calculates the deviation between the coil and the target point in real time.

[0051] The spatial transformation matrix can be directly integrated with navigation software for target navigation and coil deviation calculation, improving the overall positioning accuracy and process efficiency of the system. In addition, it can also be used to integrate with robotic arm control.

[0052] The device automatically completes ROI extraction, point cloud construction, and registration optimization, reducing error fluctuations caused by operator differences, reducing reliance on manual labor, and improving registration consistency and repeatability.

[0053] This application presents a transcranial magnetic navigation-based automatic facial registration method based on depth cameras, replacing the traditional probe-based point selection. A closed-loop process of "coarse registration + fine registration + quality assessment + automatic retry" is employed to ensure engineering usability and stability. A reliable region point cloud strategy is proposed to suppress the influence of outliers caused by glasses, hair, occlusion, etc. Standard transformation matrices such as T_img←pat are output, which can be directly interfaced with TMS navigation and positioning or robotic arm control, suitable for repetitive clinical treatment procedures, and can significantly reduce the invasiveness and operational costs of pre-treatment preparation.

[0054] As can be seen from the above technical solution, depth and color images of the target area are acquired, and camera intrinsic parameters are recorded. To remove interference from the background and non-facial areas, face detection and facial segmentation processing can be performed on the depth and color images to generate a reliable region mask. The reliable region mask is an image representation of a reliable region, which is a more stable and suitable area for registration within the facial region. Based on the camera intrinsic parameters, depth image, and reliable region mask, a reliable facial point cloud is generated; the reliable facial point cloud is then registered with a head surface model based on image space to obtain a spatial transformation matrix. To evaluate the registration quality, registration requirements can be set. If the spatial transformation matrix meets the registration requirements, it indicates that accurate registration from physical space to image space can be achieved based on the spatial transformation matrix. Therefore, target navigation of the transcranial magnetic navigation system can be performed based on the spatial transformation matrix. In this technical solution, reliable acquisition of the user's three-dimensional facial information is achieved by analyzing the depth and color images acquired by the camera without contacting the user's face, reducing invasiveness. The spatial transformation matrix can be obtained through automatic registration between the reliable facial point cloud and the head surface model, reducing manual operation and reliance on experience. Furthermore, registration requirements were set, enabling automatic determination of registration results. Target navigation was only performed based on the spatial transformation matrix when the spatial transformation matrix met the registration requirements, thus improving registration stability and engineering usability. This achieved a contactless, automated, and quantifiable spatial registration scheme for the TMS navigation system.

[0055] Based on the functions required to be implemented by the mask generation unit 12, the mask generation unit 12 may include a detection subunit, a segmentation subunit, and a removal subunit. The detection subunit is used to perform face detection on the color image and determine the face region bounding box. The segmentation subunit is used to segment the face region bounding box using a depth map to generate a face mask. The removal subunit is used to remove abnormal regions from the face mask to obtain a reliable region mask.

[0056] Figure 2 A flowchart of a method for generating a trusted region mask provided in this application embodiment, the method including: S201: Perform face detection on the color image and determine the face region bounding box.

[0057] In a practical implementation, face detection can be performed on the color image I to obtain the face region bounding box, i.e., the ROI box.

[0058] S202: Use the depth map to segment the face region and generate a face mask.

[0059] After determining the Region of Interest (ROI), the ROI can be further segmented using the depth map D to generate a facial mask. For facial segmentation, methods such as keypoint-based face region generation, instance segmentation models, and depth threshold-based foreground extraction can be employed.

[0060] In the specific implementation, the face region within the ROI can be used as the initial foreground; background points that are significantly deviated from the face depth are removed by using depth value range constraints; the foreground boundary is refined by using edge detection, region growing or morphological operations; opening, closing and hole filling operations are performed on the segmentation results to obtain a connected and relatively complete face region; finally, a binary mask of the same size as the ROI is generated, with the face region pixels recorded as 1 and the background and non-face regions recorded as 0.

[0061] S203: Remove abnormal regions from the face mask to obtain a reliable region mask.

[0062] The abnormal regions include deep cavity regions, noise regions, and deformation regions.

[0063] In practical applications, areas with deep voids and strong noise, such as those corresponding to eyeglass reflections or nasal cavity voids, can be removed. Deformable areas such as hair and ears can also be removed; these areas can be eliminated using depth gradients and morphological culling.

[0064] After removing abnormal areas, the remaining areas can include stable areas such as the forehead, bridge of the nose, and cheekbones, which can be considered as reliable areas.

[0065] After removing abnormal regions from the facial mask, a reliable region mask can be output, which can be used to filter reliable points in the fused point cloud.

[0066] Before using depth maps to segment the face region and generate a facial mask, the validity of the acquired images can be determined. Validity determination can include preliminary checks on the acquisition distance, effective pixel ratio, and motion amplitude.

[0067] Based on the functionality required for validity determination, the transcranial magnetic navigation face registration device further includes a first determination unit, a second determination unit, and a third determination unit. The first determination unit is used to determine whether the valid depth value of the depth map is greater than or equal to a preset depth value, and whether the depth distance of the face region in the depth map is within a preset range.

[0068] The first determination unit is used to determine that the depth map is a valid depth map when the effective depth value of the depth map is greater than or equal to the preset depth value and the depth distance of the face region in the depth map is within the preset range.

[0069] The first judgment unit essentially determines the validity of the depth map data quality, including determining how many pixels in the area are valid depth values ​​and whether the depth distance of the face region is within the range that can be normally collected.

[0070] The preset depth value can be represented by α, which can be a value between 50% and 80%.

[0071] The depth distance of the face region can be preset within a range of 0.35–0.90m.

[0072] For example, if the effective depth value of the depth map is ≥ α, and the depth distance of the face region in the depth map is within 0.35–0.90m, then the depth map is considered to be a valid depth map.

[0073] The second judgment unit is used to determine whether the center drift of the face region box in multiple consecutive color images is less than a preset threshold.

[0074] The second determination unit is used to determine that a color image is a valid color image if the center drift of the face region bounding box in multiple consecutive color images is less than a preset threshold.

[0075] The preset threshold value can be a number between 2 and 10 pixels.

[0076] If the above conditions are not met, the patient should be prompted to adjust the distance / posture or remove the obstruction and then collect the data again.

[0077] By determining whether the center drift of the face region bounding box in consecutive frames is less than a preset threshold, it is possible to assess whether the user has significant movement, which can effectively prevent large movements.

[0078] Based on the functions required to be implemented by the point cloud generation unit 13, the point cloud generation unit 13 may include a back-projection subunit, a fusion subunit, and a filtering subunit. The back-projection subunit is used to back-project multiple frames of depth maps based on camera intrinsic parameters to generate multiple frames of point clouds. The fusion subunit is used to fuse the preprocessed multiple frames of point clouds to obtain a fused point cloud. The filtering subunit is used to filter the fused point cloud based on a reliable region mask to obtain a reliable facial point cloud.

[0079] Preprocessing may include denoising and voxel downsampling.

[0080] Figure 3 A flowchart of a method for generating a believable facial point cloud provided in this application embodiment, the method comprising: S301: Back-projecting multi-frame depth maps based on camera intrinsic parameters to generate multi-frame point clouds.

[0081] In the specific implementation, for each frame of depth map, the pixels (u, v) in the ROI can be back-projected into three-dimensional points (x, y, z) based on the camera intrinsic parameter K to obtain the point cloud P.

[0082] S302: Fuse the preprocessed multi-frame point clouds to obtain a fused point cloud.

[0083] Preprocessing may include denoising the point cloud and voxel downsampling.

[0084] Point cloud denoising methods can include identifying and removing outliers, or using radius filtering.

[0085] The voxel side length for voxel downsampling can be 1–3 mm to reduce computational load. For example, in practical applications, the voxel side length can be set to 2 mm.

[0086] In practical applications, denoising can be performed on multi-frame point clouds, and voxel downsampling can be performed on the denoised multi-frame point clouds to obtain pre-processed multi-frame point clouds. Based on the changes in face pose between adjacent frames, the pre-processed multi-frame point clouds are aligned; the aligned multi-frame point clouds are superimposed on the same coordinate system, and the overlapping areas are fused to obtain fused point clouds.

[0087] Taking N-frame point clouds as an example, the fusion of N-frame point clouds can include using the previous frame or the first frame as the reference frame; performing rigid body alignment on each frame's point cloud based on the changes in face pose between adjacent frames; superimposing the aligned multi-frame point clouds onto the same coordinate system; performing average or median fusion on the overlapping areas to reduce random noise; and finally performing voxel downsampling to obtain a smoother and denser fused point cloud P_hat.

[0088] For example, taking the average fusion of overlapping areas as an example, the same bridge of the nose will have 20 similar 3D points in 20 frames. When fusing, the average position of these points can be taken to obtain a more stable result.

[0089] Fusing N frames of point clouds yields a fused point cloud P_hat, which improves the point cloud density and stability.

[0090] S303: Based on the trusted region mask, the fused point cloud is filtered to obtain a trusted facial point cloud.

[0091] In practical applications, it can be used to determine whether the pixel position of a target point in the fused point cloud is within a trusted region mask. Here, the target point is any pixel in the fused point cloud.

[0092] If the pixel position of the target point is within the trusted region mask, the value of the target point in the trusted region mask is set to 1; if the pixel position of the target point is not within the trusted region mask, the value of the target point in the trusted region mask is set to 0; the set of all target points with a value of 1 in the trusted region mask is taken as the facial trusted point cloud.

[0093] In this application, the quality of the point cloud is improved by denoising the multi-frame point cloud. Furthermore, voxel downsampling is performed on the denoised point cloud to reduce computational load. Fusing the preprocessed multi-frame point clouds improves point cloud density and stability. By filtering the fused point cloud using a reliable region mask, a reliable facial point cloud can be obtained, providing high-quality data support for the subsequent registration process.

[0094] Based on the functions required to be implemented by the registration unit 14, the registration unit 14 may include an initial registration subunit and an iterative optimization subunit. The initial registration subunit is used to perform initial registration of the reliable facial point cloud with the image space-based head surface model to obtain an initial spatial transformation matrix. The iterative optimization subunit is used to use the initial spatial transformation matrix as an initial value to iteratively optimize the reliable facial point cloud and the image space-based head surface model to obtain an optimized spatial transformation matrix.

[0095] Figure 4 A flowchart of a method for determining a spatial transformation matrix provided in this application embodiment, the method comprising: S401: Perform initial registration between the reliable facial point cloud and the head surface model based on image space to obtain the initial spatial transformation matrix.

[0096] In this embodiment of the application, the facial point cloud and the head surface model based on the image space can be aligned according to the set initial registration rules to obtain the initial spatial transformation matrix.

[0097] The initial registration rules may include registration based on 3D key points, registration based on head pose priors, or registration based on local geometric feature matching.

[0098] Coarse registration based on 3D keypoints: Extract the set of 3D keypoints {p_i} from P_trust, including the tip of the nose, the center of the eyebrows, and the corners of the eyes, and obtain the corresponding set {q_i} in the MRI model. Solve for T0. {p_i} represents the set of 3D keypoints in the reliable facial point cloud, and {q_i} represents the corresponding set of 3D keypoints in the MRI model.

[0099] In practical applications, we can first establish the correspondence between point pairs, and then solve for a rigid body transformation T0 that minimizes the overall error between the two sets of points. The specific process may include: calculating the centroids of the two sets of points; subtracting the centroid of each point from its own centroid to obtain a decentralized point set; constructing a covariance matrix based on the corresponding points; performing singular value decomposition (SVD) on the covariance matrix; obtaining the optimal rotation matrix R from the decomposition results; and then obtaining the translation vector t by combining the centroids of the two sets of points; finally, we obtain the rigid body transformation, i.e., the initial spatial transformation matrix T0 = [R, t].

[0100] The purpose of coarse registration based on 3D key points is to find an optimal rigid body transformation so that {p_i} is aligned to {q_i} as much as possible.

[0101] Coarse registration based on head pose prior: The head orientation is obtained through face pose estimation. The MRI model is first rotated to approximate the pose, and then translated to align with the center of the face to obtain T0.

[0102] Face pose estimation can be performed based on facial landmarks in color images or on depth point clouds / reliable point clouds; no specific limitation is made here.

[0103] Coarse registration based on local geometric feature matching: matching pairs are formed using local geometric descriptors of point clouds to estimate T0.

[0104] In practical applications, the coarse registration process based on local geometric feature matching can include: selecting feature points in the P_trust and MRI models respectively; calculating a local geometric descriptor for each feature point to characterize the shape features of its neighborhood; establishing initial matching point pairs between the P_trust and MRI models based on descriptor similarity; using robust methods such as RANSAC to eliminate erroneous matches; and solving the rigid body transformation based on the retained matching point pairs to obtain the initial spatial transformation matrix T0. In summary, coarse registration based on local geometric feature matching involves first "finding similar local shapes," then "establishing point pairs," and finally "deriving the rigid body transformation from the point pairs."

[0105] Any one of the three methods described above can be used as a coarse registration method to solve for the initial transformation T0. Furthermore, any one, two, or three methods can be used in combination to improve the stability of the initial alignment.

[0106] S402: Using the initial spatial transformation matrix as the initial value, iteratively optimize the facial point cloud and the head surface model based on the image space to obtain the optimized spatial transformation matrix.

[0107] In practical applications, the initial spatial transformation matrix can be used as the initial value. After each iteration of optimization, the matching error between each point in the optimized facial point cloud and the head surface model based on image space is recalculated.

[0108] Optimization of facial point clouds can include removing or reducing the weight of points whose distance is greater than a threshold, and prioritizing stable regions such as the forehead, bridge of the nose, and cheekbones. For example, points with particularly large errors can be removed directly; points with relatively large errors can have their weight reduced; and points with smaller errors and stable positions can retain higher weights.

[0109] In the specific implementation, if the matching error corresponding to the credible target point in the credible facial point cloud is greater than or equal to the set first error threshold, the credible target point is removed from the credible facial point cloud; where the credible target point is any point in the credible facial point cloud.

[0110] If the matching error corresponding to a credible target point in a facial credible point cloud is less than a first error threshold but greater than a set second error threshold, the weight of the credible target point is reduced; wherein the first error threshold is greater than the second error threshold.

[0111] If the matching error corresponding to a credible target point in a credible point cloud is less than the second error threshold, the credible target point is retained.

[0112] The fine registration process uses T0 as the initial value and iteratively optimizes the P_trust and MRI model. The MRI model is usually kept stationary as a reference; in each iteration, the P_trust continuously updates its position and pose based on the currently obtained transformation; after each update, the nearest neighbor correspondence or point-to-plane correspondence between the P_trust and the MRI model is recalculated; and the transformation parameters are then optimized again based on the new correspondence. Therefore, the core changes are: the transformed position of the P_trust changes, the correspondence changes, and the error value gradually decreases; the MRI model generally remains unchanged.

[0113] By optimizing each point in the reliable facial point cloud according to the magnitude of their matching error with the MRI model, the impact of outliers, residual occlusion points, and local noise points on the fine registration results can be reduced.

[0114] After optimizing the facial point cloud, adjust the spatial transformation matrix for this iteration and perform the next iteration optimization until the iteration termination condition is met, then output the final spatial transformation matrix.

[0115] Iteration termination conditions may include reaching the maximum number of iterations (30–80 times); the pose increment being less than a threshold (e.g., rotation <0.05°, translation <0.1mm); the error decrease being less than a threshold, etc.

[0116] Robustness is enhanced through multi-frame fusion, a trusted region strategy, and a retry mechanism, reducing preparation time and improving efficiency. It outputs a standard spatial transformation matrix, which can be directly integrated into the target display, distance-angle deviation calculation, and robotic arm target pose solving links, facilitating integration with existing TMS navigation / optical positioning / robotic arm systems.

[0117] In this application, occlusion and abnormal scenes can be handled automatically. If glasses are detected to increase the proportion of deep holes within the ROI, the weight of the periorbital area can be automatically reduced, retaining only the forehead and bridge of the nose area; if a mask covers the lower half of the face, the mouth and nose area is disabled, and the forehead + brow bone + cheekbone area is used; if e_rms fails to meet the standard twice consecutively, the patient is prompted to adjust their posture or the camera is repositioned.

[0118] This application focuses on the "spatial registration" stage of TMS navigation, outputting a rigid body transformation matrix that can be used for target navigation, without being limited to a specific hardware manufacturer. This application can work in conjunction with an optical positioning system: optical positioning handles tool tracking, while the device in this application handles contactless registration; the two can be combined to improve the overall system experience. The device in this application is suitable for repetitive clinical treatment procedures, significantly reducing the invasiveness and operational costs of pre-treatment preparation.

[0119] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0120] The foregoing has provided a detailed description of a transcranial magnetic navigation facial registration device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A transcranial magnetic navigation facial registration device, characterized in that, It includes an acquisition unit, a mask generation unit, a point cloud generation unit, a registration unit, and a navigation unit; The acquisition unit is used to acquire depth and color maps of the target area and record camera intrinsic parameters; The mask generation unit is used to perform face detection and facial segmentation processing on the depth map and the color map to generate a trusted region mask. The point cloud generation unit is used to generate a facial believable point cloud based on the camera intrinsic parameters, the depth map, and the believable region mask. The registration unit is used to register the facial reliable point cloud with the head surface model based on image space to obtain a spatial transformation matrix; The navigation unit is used to perform target navigation of the transcranial magnetic navigation system based on the spatial transformation matrix, provided that the spatial transformation matrix meets the registration requirements.

2. The transcranial magnetic navigation facial registration device according to claim 1, characterized in that, The mask generation unit includes a detection subunit, a segmentation subunit, and a rejection subunit; The detection subunit is used to perform face detection on the color image and determine the face region bounding box; The segmentation subunit is used to segment the face region bounding box using the depth map and generate a face mask. The removal subunit is used to remove abnormal regions from the face mask to obtain a reliable region mask; wherein, the abnormal regions include deep hole regions, noise regions, and deformed regions.

3. The transcranial magnetic navigation facial registration device according to claim 2, characterized in that, It also includes a first judgment unit, a first determination unit, a second judgment unit, and a second determination unit; The first judgment unit is used to determine whether the effective depth value of the depth map is greater than or equal to a preset depth value, and whether the depth distance of the face region in the depth map is within a preset range; The first determination unit is used to determine that the depth map is a valid depth map when the effective depth value of the depth map is greater than or equal to a preset depth value and the depth distance of the face region in the depth map is within a preset range. The second judgment unit is used to determine whether the center drift of the face region box in multiple consecutive color images is less than a preset threshold. The second determination unit is used to determine that the color image is a valid color image if the center drift of the face region box in multiple consecutive color images is less than a preset threshold.

4. The transcranial magnetic navigation facial registration device according to claim 1, characterized in that, The point cloud generation unit includes a back projection subunit, a fusion subunit, and a filtering subunit; The back-projection subunit is used to back-project multiple frames of depth maps based on the camera intrinsic parameters to generate multiple frames of point clouds. The fusion subunit is used to fuse the preprocessed multi-frame point clouds to obtain a fused point cloud; The filtering subunit is used to filter the fused point cloud based on the trusted region mask to obtain a trusted facial point cloud.

5. The transcranial magnetic navigation facial registration device according to claim 4, characterized in that, The fusion subunit is used to denoise the multi-frame point cloud and perform voxel downsampling on the denoised multi-frame point cloud to obtain the pre-processed multi-frame point cloud. Align the preprocessed multi-frame point clouds based on the changes in face pose between adjacent frames; The aligned point clouds from multiple frames are superimposed onto the same coordinate system, and the overlapping areas are fused to obtain a fused point cloud.

6. The transcranial magnetic navigation facial registration device according to claim 4, characterized in that, The filtering subunit is used to determine whether the pixel position of the target point in the fused point cloud is within the trusted region mask; wherein, the target point is any pixel in the fused point cloud; If the pixel position of the target point is within the trusted region mask, the value of the target point in the trusted region mask is set to 1; If the pixel position of the target point is not within the trusted region mask, the value of the target point in the trusted region mask is set to 0; The set of all target points in the trusted region mask with a value of 1 is taken as the facial trusted point cloud.

7. The transcranial magnetic navigation facial registration device according to claim 1, characterized in that, The registration unit includes an initial registration subunit and an iterative optimization subunit; The initial registration subunit is used to perform initial registration between the facial reliable point cloud and the head surface model based on image space to obtain an initial spatial transformation matrix; The iterative optimization subunit is used to iteratively optimize the facial credibility point cloud and the head surface model based on image space, using the initial spatial transformation matrix as the initial value, to obtain the optimized spatial transformation matrix.

8. The transcranial magnetic navigation facial registration device according to claim 7, characterized in that, The initial registration subunit is used to align the facial reliable point cloud and the head surface model based on image space according to the set initial registration rules to obtain an initial spatial transformation matrix; wherein, the initial registration rules include registration based on 3D key points, registration based on head pose prior, or registration based on local geometric feature matching.

9. The transcranial magnetic navigation facial registration device according to claim 7, characterized in that, The iterative optimization subunit is used to take the initial spatial transformation matrix as the initial value, and recalculate the matching error between each point in the optimized facial point cloud and the head surface model based on image space for each iteration optimization. Based on the matching error corresponding to each point in the facial reliable point cloud, the spatial transformation matrix of the current iteration is adjusted after optimizing the facial reliable point cloud, and the next iteration optimization is performed until the iteration termination condition is met, and then the final spatial transformation matrix is ​​output.

10. The transcranial magnetic navigation facial registration device according to claim 9, characterized in that, The iterative optimization subunit is used to remove the credible target point from the credible point cloud if the matching error corresponding to the credible target point in the credible point cloud is greater than or equal to a set first error threshold; wherein, the credible target point is any point in the credible point cloud. If the matching error corresponding to a credible target point in the facial credible point cloud is less than the first error threshold and greater than the set second error threshold, the weight of the credible target point is reduced; wherein the first error threshold is greater than the second error threshold. If the matching error corresponding to a credible target point in the facial credible point cloud is less than the second error threshold, the credible target point is retained.

Images