Transcranial magnetic stimulation closed-loop navigation device based on visual positioning

Abstract

The invention discloses a transcranial magnetic stimulation closed-loop navigation device based on visual positioning. The device comprises a binocular camera, a three-axis slide rail system and an upper computer, a face feature point recognition module, a registration conversion module, a transcranial magnetic coil attitude positioning module and an interaction device are arranged in the upper computer; the binocular camera continuously collects images and inputs the images into the upper computer; the face feature point recognition module constructs a three-dimensional face model and recognizes face feature points according to the image, and generates a three-dimensional head model at the same time; the registration conversion module performs rigid registration on the three-dimensional face model and the three-dimensional head model by using the face feature points; a user selects a needed intracranial treatment target through the interaction device and inputs the target into the transcranial magnetic coil attitude positioning module, the transcranial magnetic coil attitude positioning module tracks a treatment point corresponding to the intracranial treatment target in real time and maps the treatment point to the three-dimensional head model, and the attitude of the transcranial magnetic coil is given. According to the invention, the tracking precision of the transcranial magnetic navigation system is improved.

Description

A visually localized transcranial magnetic stimulation closed-loop navigation device Technical Field

[0001] This invention relates to the field of medical robot automation, and more specifically to a visual positioning-based transcranial magnetic stimulation closed-loop navigation device. Background Technology

[0002] Magnetic resonance imaging, or nuclear magnetic resonance imaging (NMRI), utilizes the principle of nuclear magnetic resonance (NMR). Based on the varying attenuation of emitted energy in different structural environments within matter, and by detecting the emitted electromagnetic waves through an applied gradient magnetic field, the location and type of atomic nuclei that make up the object can be determined, allowing for the creation of an image of the object's internal structure. Applying this technology to imaging the internal structures of the human body has resulted in a revolutionary medical diagnostic tool. Magnetic resonance imaging data is stored in the DICOM (Digital Imaging and Communications in Medicine) format.

[0003] DICOM, or Digital Imaging and Communications in Medicine, is an international standard (ISO 12052) for medical images and related information. It defines a medical image format that meets clinical requirements for data exchange. DICOM is widely used in radiology, cardiovascular imaging, and diagnostic radiology equipment (X-ray, CT, MRI, ultrasound, etc.), and is increasingly being applied in other medical fields such as ophthalmology and dentistry. Among tens of thousands of medical imaging devices in use, DICOM is one of the most widely deployed medical information standards. Currently, there are approximately tens of billions of DICOM-compliant medical images used clinically.

[0004] Transcranial magnetic stimulation (TMS) is a painless, non-invasive, and environmentally friendly treatment method. Magnetic signals can penetrate the skull without attenuation to stimulate brain nerves. Through magnetic pulses acting on the central nervous system, it stimulates nerve cells, compensates for necrotic nerve cells, and balances the excitability of the affected and healthy sides, thereby restoring certain functions such as cognitive impairment and speech disorders. The key to this technique lies in the precise localization of the brain region to be stimulated. In most cases, the method used is to have the patient wear a head-positioning cap. However, due to significant individual differences and the inability of patients to remain stationary, the stimulation position must be adjusted in real time according to the patient's condition.

[0005] In existing technologies, such as Chinese patent document CN 118892617 A, a positioning system for assisting transcranial magnetic stimulation (TMS) therapy is disclosed. The advantage of this patent is that it requires local MRI data and relies on facial point cloud recognition, using a robotic arm to complete TMS target point navigation. While simplifying the data acquisition process, the lack of local images leads to insufficient completeness and accuracy of the 3D head model. Its facial point cloud technology does not solve the problem of facial occlusion, resulting in insufficient robustness of feature point recognition. The robotic arm system is costly, restricts patient movement, and lacks a dynamic error compensation mechanism, leading to insufficient positioning stability. Another example is Chinese patent document CN114288559 A, which discloses a TMS navigation method, system, and computer equipment. This patent achieves navigation based on a "standard head model," selecting multiple virtual feature points on the standard head model and registering them with corresponding feature points on the patient to drive the navigation actuator. This patent does not consider individual differences in human head anatomy; the standard head model cannot match individual differences, resulting in significant target point mapping errors. Furthermore, the number of feature points is small, registration stability is poor, and there is a lack of 3D spatial constraints, making it susceptible to acquisition errors. Without a real-time tracking mechanism, it cannot handle dynamic head movements, nor can it determine the precise location of scalp treatment points using a standard model. It can only rely on experience mapping, which results in significant errors. Summary of the Invention

[0006] This invention addresses the problem of transcranial magnetic stimulation (TMS) navigation based on a cranial mapping model. It proposes a coordinate system registration method based on improved BlazeFace neural network facial feature recognition and MRI data fusion to locate the head using facial feature recognition and 3D reconstructed MRI data. To address the issue of facial occlusion, a prediction branch is added to the original BlazeFace network structure, enhancing its ability to predict facial occlusion. A 3D head model of the patient is reconstructed from MRI data, and the patient's MRI data is matched with the patient's actual head. A coordinate system registration algorithm based on facial feature points is employed to transform the camera coordinate system in real space to the model space coordinate system, thereby converting the target point coordinates in 3D space to the target point coordinates of the real human head in real space. This enables real-time, accurate positioning and tracking, solving problems such as low autonomous navigation performance, high cost, cumbersome coil calibration procedures, and restricted patient movement in current TMS navigation systems, thus resolving the issues mentioned in the background.

[0007] The objective of this invention is achieved by at least one of the following technical solutions.

[0008] A visual-based transcranial magnetic stimulation (TMS) closed-loop navigation device includes a binocular camera, a three-axis sliding rail system, and a host computer. The host computer includes a facial feature point recognition module, a registration and conversion module, a TMS coil posture positioning module, and an interactive device. The binocular camera and three-axis sliding rail system continuously acquire two two-dimensional facial images of the patient at the same time and input them into the host computer. The facial feature point recognition module constructs a three-dimensional facial model and identifies facial feature points based on the input two-dimensional facial images. Simultaneously, it reconstructs the patient's resting-state functional magnetic resonance imaging (fMRI) data using three-dimensional modeling based on the patient's MRI images, generating a three-dimensional head model. The registration and conversion module uses facial feature points to rigidly register the three-dimensional facial model with the three-dimensional head model, completing the coordinate system registration and conversion. Through the interactive device on the host computer, the user (medical personnel) selects the desired intracranial treatment target point according to the treatment purpose and patient condition and inputs it into the TMS coil posture positioning module. The TMS coil posture positioning module tracks the treatment point corresponding to the intracranial treatment target point in real time and maps it to the three-dimensional head model, providing the posture of the TMS coil.

[0009] Furthermore, in the facial feature point recognition module, the patient's resting-state functional magnetic resonance imaging (fMRI) data and T1-weighted structural imaging are obtained based on the patient's MRI images; the resting-state fMRI data are then reconstructed in three dimensions using a ray projection algorithm in volume rendering to obtain a three-dimensional head model.

[0010] Furthermore, in the facial feature point recognition module, a stereo matching algorithm is used to extract disparity information from two 2D facial images of the patient at the same time, identify facial feature points in the 2D facial images, calculate a depth map, and generate a 3D facial model. The improved BlazeFace network is used to identify facial feature points in the 2D facial images in real time and achieve head localization. An occlusion detection branch network is added to the backbone network of the improved BlazeFace network structure. Based on the features of the feature map, a binary classification judgment of occlusion is made, and the probability values ​​of the two categories are output to determine whether the current face is occluded.

[0011] Furthermore, the structure of the improved BlazeFace network is as follows: The existing BlazeFace network is directly used as the backbone network, with two 2D face images captured by a binocular camera as input. The two 2D face images are input into the backbone network for feature extraction. By progressively extracting low-level and high-level face features, multi-scale feature maps are generated. An occlusion detection branch network is embedded after the fourth convolutional layer of the backbone network, using the fourth layer of the multi-scale feature map as input, and outputting the occlusion prediction result to subsequent layers of the backbone network as auxiliary information for computation. The occlusion detection branch network includes sequentially connected convolutional layers, 2×2 max-pooling layers, and fully connected layers. The convolutional layers use 3×3 convolutional kernels for feature extraction, generating feature maps. The 2×2 max-pooling layer is used to reduce the spatial size of the feature maps. The pooled feature maps are flattened and output through a fully connected layer, with the occlusion prediction result categorized as 'No Occlusion'. Occlusion; the output of the fully connected layer needs to be converted into a probability distribution through a Softmax layer. The two are usually used together, and the fully connected layer includes a Softmax layer.

[0012] Furthermore, in the registration and conversion module, a mapping relationship between the coordinate systems of the head in real space and the camera is established, specifically as follows: the coordinates of the facial feature points identified by the facial feature point recognition module are received and passed to the calibrated camera API to calculate the depth information of the face; then, the 2D coordinates of the left camera in the binocular camera and the depth information are used together to form the 3D coordinates used by the navigation system. And it is defined as the camera coordinate system in real space. The coordinate information below; the navigation system uses the camera coordinate system. The coordinate information below was used to calculate three points: the center of the left eye, the center of the right eye, and the tip of the nose, using multiple facial feature points. These points were then analyzed using... (Rcamera) P leye , (Rcamera) P reye and (Rcamera) P nose This indicates that a head coordinate system in real space can be established. ; Cranial coordinate system With the coordinates of the tip of the nose as the origin, The axis direction is from the center of the right eye to the center of the left eye. The axis points from bottom to top; then, according to the right-hand rule for establishing a coordinate system, we obtain the cephalic coordinate system. of Axis in camera coordinate system The downward direction is shown in the following formula: ;in, Represents the head coordinate system in real space of Axial direction; Represents the head coordinate system in real space of In the axial direction, in order to and The direction is vertical, and the direction is from the head upwards, through... The normal vectors of the plane formed by the tip of the nose, the center of the left eye, and the center of the right eye are calculated. Represents the head coordinate system in real space of The axis direction is approximately from back to front of the head, using the right-hand rule for establishing the phylogenetic relationship. and The cross product yields a vector; finally, the three resulting vectors are normalized to obtain the camera coordinates in real space. The head coordinate system below of , , The expressions for the three coordinate axes are then used to derive the head coordinate system in real space. To the camera coordinate system in real space rotation matrix As shown in the following formula: The head coordinate system in real space Get a camera in real space Transformation matrix for: During transcranial magnetic stimulation (TMS) therapy, the patient's brain will move involuntarily, requiring real-time calculation of the head coordinate system in real space based on the detected facial feature points. To the camera coordinate system The transformation matrix.

[0013] Furthermore, the model space coordinate system The coordinate system of the 3D head model reconstructed in model space The model space is used for creation, specifically as follows: The 3D head model obtained from the 3D reconstruction needs to be placed in a model space for display, and the 3D coordinates of this model space are determined. Defined in the model space coordinate system The coordinate information below; using the left eye center, right eye center, and nose tip of the 3D head model in model space, respectively, using... (VModel) P leye , (VModel) P reye and (VModel) P noseThis indicates that a 3D head model coordinate system for model space reconstruction is then established. ; Coordinate system of the reconstructed 3D head model in model space With the coordinates of the tip of the nose as the origin, The axis direction is from the center of the right eye to the center of the left eye. The axis points from bottom to top; then, according to the right-hand rule, the coordinate system of the reconstructed 3D head model in model space is obtained. of Axis in model space coordinate system The downward direction is shown in the following formula: ;in, The coordinate system representing the 3D head model reconstructed in model space. of Axial direction; The coordinate system representing the 3D head model reconstructed in model space. of In the axial direction, in order to and Vertical direction at the same time The direction is from the head upwards, through The normal vectors of the plane formed by the tip of the nose, the center of the left eye, and the center of the right eye are calculated. The coordinate system representing the 3D head model reconstructed in model space. of The axis direction is from back to front of the head, using the right-hand rule to establish the system. and The cross product yields a vector; finally, the three resulting vectors are normalized to obtain the vectors in the model space coordinate system. The coordinate system of the reconstructed 3D head model of , The expressions for the three coordinate axes are then used to obtain the coordinate system of the reconstructed 3D head model in model space. To the model space coordinate system Rotation matrix between the two and As shown in the following formula: ; ; Coordinate system of the reconstructed 3D head model in model space Obtain the model space coordinate system Transformation matrix between the two and for: ; .

[0014] Furthermore, with (Rcamera) P leye , (Rcamera) Preye and (Rcamera) P nose Using three facial feature points as a reference, the 3D head model and the 3D face model are superimposed through rigid transformation; because the head generated from MRI data has the same size as the real human head, the head coordinate system in real space is assumed to be... The coordinate system of the 3D head model reconstructed in model space The coordinates of the three reference points—the center of the left eye, the center of the right eye, and the tip of the nose—are consistent in both coordinate systems, mapping the human head in model space to the human head in real space; therefore, the coordinate system of the 3D head model is consistent. Transform to camera coordinate system Transformation matrix for: ; In the defined model space, the coordinate system of the three-dimensional head model Treatment point location Afterwards, Convert the treatment point to the camera coordinate system in real space Treatment point location The details are as follows: .

[0015] Furthermore, in the registration and conversion system, when the subject's position changes, the camera's real-time positioning will produce errors, so it is necessary to compensate for the position coordinates of the treatment point; the Kalman filter algorithm is used to compensate for the error in the position coordinates of the treatment point, thereby improving the positioning accuracy of the treatment point.

[0016] Furthermore, the interactive device includes two interactive spaces: a model space and a DICOM image space. The operator selects the location of the desired intracranial therapeutic target in either the model space or the DICOM image space. This selected target location is then input into the transcranial magnetic coil posture positioning module. The model space displays a 3D head model reconstructed from MRI data, while the DICOM image space displays cross-sectional images of the skull. The DICOM image space is set... shaft and The axis direction is consistent with the direction of resting-state functional magnetic resonance imaging (fMRI) data. Since the unit is pixels, the coordinate values ​​in the DICOM image space range from [value missing]. ;and DICOM image space The axial direction is consistent with the superposition direction of resting-state functional magnetic resonance imaging (fMRI) data, and the value range is related to the number of DICOM image sequences. It is assumed that... The range of values ​​for the image is: ; respectively , and Indicates the pixels in a DICOM image shaft and The pixel spacing on the axis and the spacing between DICOM image layers are determined by... , and Obtain the DICOM image space coordinate system The transformation relationship to the model space coordinate system is shown in the figure: ;in, Represents coordinates in model space. Represents the coordinates in the DICOM image space.

[0017] Furthermore, in the transcranial magnetic coil (TMC) attitude localization module, the tracked treatment point location needs to be on the scalp. When using the interaction between model space and DICOM image space to obtain the coordinates of the intracranial treatment target, the optimal coordinates of the treatment point on the scalp need to be calculated and used as the tracked treatment point. During treatment, the TMC needs to be aligned with the treatment point on the patient's scalp in a specific orientation, with the orientation of the coil plane used as the normal vector of the treatment point. When generating a three-dimensional head model using the patient's MRI data, the vertices of all triangular facets in the model space are calculated, denoted as S. Treatment points and intracranial treatment targets are different concepts. Intracranial treatment targets refer to deep targets selected through DICOM image space or model space, while treatment points are points actually located on the scalp surface. The position of the treatment point is obtained by calculating the coordinates projected from the intracranial treatment target onto the three-dimensional head model. The formulas for calculating the position and normal vector of the treatment point are as follows:

[0018] Among them, P target N represents the coordinates of the treatment point. target Let P be the normal vector of the treatment point. heal This indicates the coordinates of the brain treatment target point selected during interaction. Let be a vertex of one of the triangular faces in the model space; the 3D model is then superimposed on the actual human head, with the superposition references being the left eye, right eye, and nose tip feature points. (Rcamera) P leye , (Rcamera) P reye and (Rcamera) P nose At this point, the model space coordinate system is considered to be... and head coordinate system It is consistent, that is, through the model space coordinate system. and head coordinate system The matching process applies the model space to real human heads. Switch to The down-rotation and translation matrix is:

[0019] The position of the treatment point in model space (Model) P target and normal vector (Model) N target Once determined, calculate the coordinates and normal vector of the treatment point in camera space, as follows:

[0020] After calculating the camera coordinates and normal vector of the treatment point, the interactive device feeds back the information to the user to confirm the correctness of the treatment point and whether the attitude adjustment of the TMS coil is in place.

[0021] Compared to existing technologies, the advantages of this invention are as follows: The system architecture of this invention is mainly divided into two parts: hardware and software. 1. In the hardware part, this invention uses a binocular camera to complete facial feature point recognition and matching, and a three-axis sliding rail system to control the transcranial magnetic stimulation coil for navigation and tracking. 2. In the software part, this invention uses the MVC software structure to build the software framework, and uses Qt and Python to write the system's visual interface and the specific operating logic of each module, meeting the system's usage requirements.

[0022] 2. To reduce the difficulty for doctors to use the system and increase patient comfort without restricting head movement, this invention generates a three-dimensional head model in model space using a light projection algorithm, which facilitates the operator in selecting target points in the brain region. An improved BlazeFace network is used to identify facial feature points in real time to locate the head position.

[0023] 3. This invention achieves the localization and real-time tracking of target points selected in model space by combining MRI data and a real head. By utilizing the principle that the 3D head model generated from the real head and MRI data are identical in size, a transformation relationship is established from the model space coordinate system to the camera coordinate system, thus enabling the selection of target points in model space to be converted to the real space.

[0024] 4. This invention improves the tracking accuracy of the system by filtering the target points after they have been transformed into the real space using the KF algorithm. Attached Figure Description

[0025] Figure 1 is a flowchart of a visual positioning-based transcranial magnetic stimulation closed-loop navigation device according to an embodiment of the present invention.

[0026] Figure 2 is a schematic diagram of the structure of the improved BlazeFace network in an embodiment of the present invention.

[0027] Figure 3 is a schematic diagram of a visual positioning-based transcranial magnetic stimulation closed-loop navigation device according to an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the present invention clearer, the specific implementation of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0029] This embodiment presents a visual positioning-based transcranial magnetic stimulation closed-loop navigation device, as shown in Figure 3, comprising a binocular camera, a three-axis sliding rail system, and a host computer. The host computer includes a facial feature point recognition module, a registration and conversion module, a transcranial magnetic coil posture positioning module, and an interactive device. As shown in Figure 1, the binocular camera and the three-axis sliding rail system continuously acquire two two-dimensional facial images of the patient at the same time and input them into the host computer. The facial feature point recognition module constructs a three-dimensional facial model and identifies facial feature points based on the input two-dimensional facial images. Simultaneously, based on the patient's MRI images, it uses three-dimensional modeling to reconstruct the patient's resting-state functional magnetic resonance imaging (fMRI) data, generating a three-dimensional head model. The registration and conversion module uses facial feature points to rigidly register the three-dimensional facial model with the three-dimensional head model, completing the coordinate system registration and conversion. Through the interactive device of the host computer, the user selects the desired intracranial treatment target point according to the treatment purpose and the patient's condition and inputs it into the transcranial magnetic coil posture positioning module. The transcranial magnetic coil posture positioning module tracks the treatment point corresponding to the intracranial treatment target point in real time and maps it to the three-dimensional head model, and provides the posture of the transcranial magnetic coil.

[0030] In the facial feature point recognition module, resting-state functional magnetic resonance imaging (fMRI) data and high-resolution T1-weighted structural imaging are acquired based on the patient's MRI images. In one embodiment, the resting-state fMRI data acquisition time is at least 5-10 minutes, with a spatial resolution typically around 3-4 mm and a temporal resolution within 2 seconds. The high-resolution T1-weighted structural imaging data should have a spatial resolution of 1 mm or higher to ensure sufficient detail for accurate 3D reconstruction. The resting-state fMRI data is then reconstructed in 3D using a ray projection algorithm in volume rendering to obtain a 3D head model.

[0031] Furthermore, in the facial feature point recognition module, a stereo matching algorithm is used to extract disparity information from two 2D facial images of the patient taken at the same time, identify facial feature points in the 2D facial images, calculate a depth map, and generate a 3D facial model. An improved BlazeFace network is used to identify facial feature points in the 2D facial images in real time and achieve head localization. To locate the head, a transformation relationship between the 3D reconstructed model space and the real space needs to be established, mainly through calibration and matching using facial feature points. While the BlazeFace network offers high real-time performance and accuracy, in practice, it may encounter errors in facial feature recognition due to patients involuntarily scratching their face, rubbing their eyes, or touching their hair during TMS stimulation, or due to discomfort during TMS stimulation leading to actions such as picking their hair or touching their nose. Further improvements to the BlazeFace neural network algorithm are needed to address the occlusion problems that may occur in real-world scenarios. Even with occlusion, the BlazeFace neural network algorithm can still detect feature points. However, the BlazeFace model treats the feature points identified in the occluded areas as original facial feature points, thus processing the face according to normal 3D feature point coordinates. This leads to significant errors in real-time tracking, potentially even causing harm to the patient. Furthermore, because it uses binocular cameras, when the face is significantly averted, one camera might capture a frontal image while the other captures an occluded profile image, resulting in inconsistent facial feature point recognition between the two images. Based on these two issues, further modifications to the BlazeFace algorithm's network structure are needed. A classification and prediction branch network should be added to the feature extraction network of the BlazeFace neural network to detect whether the current face is occluded.

[0032] An occlusion detection branch network is added to the backbone network of the improved BlazeFace network structure. Based on the features of the feature map, a binary classification judgment of occlusion is made, and the probability values ​​of the two categories are output to determine whether the current face is occluded.

[0033] In one embodiment, as shown in Figure 2, the structure of the improved BlazeFace network is as follows: The existing BlazeFace network is directly used as the backbone network. The input data consists of two two-dimensional face images captured by a binocular camera, typically 128×128 pixels in size. The two two-dimensional face images are input into the backbone network for feature extraction. By progressively extracting low-level and high-level features of the face, multi-scale feature maps are generated. In the backbone network, feature maps are progressively extracted through convolutional and pooling layers. The high-level features of the fourth layer are extracted; the feature maps of this layer reflect the basic structure of the face while still maintaining sufficient spatial information. The occlusion detection branch network is embedded after the fourth convolutional layer of the backbone network. It uses the fourth layer of the multi-scale feature map as input and outputs the occlusion prediction result to the subsequent network layers of the backbone network as auxiliary information for the backbone network to participate in the calculation. The occlusion detection branch network includes sequentially connected convolutional layers, 2×2 max pooling layers, and fully connected layers. The convolutional layers use 3×3 convolutional kernels to extract features and generate feature maps. The 2×2 max pooling layer is used to reduce the spatial size of the feature map. The pooled feature map is flattened and the occlusion prediction result is output through the fully connected layer. The categories include 'No Occlusion' and 'Occlusion'. The output of the fully connected layer needs to be converted into a probability distribution through a softmax layer. The two are usually used in combination. The fully connected layer includes a softmax layer.

[0034] The input consists of multi-scale feature maps from the backbone network, with occlusion detection information added. Keypoint coordinate predictions are generated through convolutional operations. The coordinates of facial feature points (e.g., eyes, nose, mouth) are calculated and adjusted appropriately based on occlusion detection results. The final output keypoint coordinates are corrected to reduce errors caused by occlusion and image inconsistencies. The output includes accurate (corrected) coordinates of facial feature points and an occlusion status determination (whether occlusion occurs).

[0035] Furthermore, in the registration and conversion module, a mapping relationship between the coordinate systems of the head in real space and the camera is established, specifically as follows: the coordinates of the facial feature points identified by the facial feature point recognition module are received and passed to the calibrated camera API to calculate the depth information of the face; then, the 2D coordinates of the left camera in the binocular camera and the depth information are used together to form the 3D coordinates used by the navigation system. And it is defined as the camera coordinate system in real space. The coordinate information below; the navigation system uses the camera coordinate system. The coordinate information below was used to calculate three points: the center of the left eye, the center of the right eye, and the tip of the nose, using multiple facial feature points. These points were then analyzed using... (Rcamera) P leye ,(Rcamera) P reye and (Rcamera) P nose This indicates that a head coordinate system in real space can be established. ; Cranial coordinate system With the coordinates of the tip of the nose as the origin, The axis direction is from the center of the right eye to the center of the left eye. The axis points from bottom to top; then, according to the right-hand rule for establishing a coordinate system, we obtain the cephalic coordinate system. of Axis in camera coordinate system The downward direction is shown in the following formula: ;in, Represents the head coordinate system in real space of Axial direction; Represents the head coordinate system in real space of In the axial direction, in order to and The direction is vertical, and the direction is from the head upwards, through... The normal vectors of the plane formed by the tip of the nose, the center of the left eye, and the center of the right eye are calculated. Represents the head coordinate system in real space of The axis direction is approximately from back to front of the head, using the right-hand rule for establishing the phylogenetic relationship. and The cross product yields a vector; finally, the three resulting vectors are normalized to obtain the camera coordinates in real space. The head coordinate system below of , , The expressions for the three coordinate axes are then used to derive the head coordinate system in real space. To the camera coordinate system in real space rotation matrix As shown in the following formula: The head coordinate system in real space Get a camera in real space Transformation matrix for: During transcranial magnetic stimulation (TMS) therapy, the patient's brain will move involuntarily, requiring real-time calculation of the head coordinate system in real space based on the detected facial feature points. To the camera coordinate system The transformation matrix.

[0036] Furthermore, the model space coordinate system The coordinate system of the 3D head model reconstructed in model space The model space is used for creation, specifically as follows: The 3D head model obtained from the 3D reconstruction needs to be placed in a model space for display, and the 3D coordinates of this model space are determined. Defined in the model space coordinate system The coordinate information below; using the left eye center, right eye center, and nose tip of the 3D head model in model space, respectively, using... (VModel) P leye , (VModel) P reye and (VModel) P nose This indicates that a 3D head model coordinate system for model space reconstruction is then established. ; Coordinate system of the reconstructed 3D head model in model space With the coordinates of the tip of the nose as the origin, The axis direction is from the center of the right eye to the center of the left eye. The axis points from bottom to top; then, according to the right-hand rule, the coordinate system of the reconstructed 3D head model in model space is obtained. of Axis in model space coordinate system The downward direction is shown in the following formula: ;in, The coordinate system representing the 3D head model reconstructed in model space. of Axial direction; The coordinate system representing the 3D head model reconstructed in model space. of In the axial direction, in order to and Vertical direction at the same time The direction is from the head upwards, through The normal vectors of the plane formed by the tip of the nose, the center of the left eye, and the center of the right eye are calculated. The coordinate system representing the 3D head model reconstructed in model space. of The axis direction is from back to front of the head, using the right-hand rule to establish the system. and The cross product yields a vector; finally, the three resulting vectors are normalized to obtain the vectors in the model space coordinate system. The coordinate system of the reconstructed 3D head model of , The expressions for the three coordinate axes are then used to obtain the coordinate system of the reconstructed 3D head model in model space. To the model space coordinate system Rotation matrix between the two and As shown in the following formula: ; ; Coordinate system of the reconstructed 3D head model in model space Obtain the model space coordinate system Transformation matrix between the two and for: ; .

[0037] Furthermore, assuming that patients undergo MRI scans before transcranial magnetic stimulation (TMS) navigation, the 3D head model reconstructed from the DICOM images is approximately one-to-one identical in size to the patient's actual head. This is a core principle of this algorithm. Typically, it's assumed that the MRI and TMS treatment occur within the same timeframe, so the patient's head and facial shape are consistent with the MRI data, thus the reconstructed 3D head model matches the actual head size. (Rcamera) P leye , (Rcamera) P reye and (Rcamera) P nose Using three facial feature points as a reference, the 3D head model and the 3D face model are superimposed through rigid transformations such as rotation and translation; because the head generated from MRI data has the same size as the real human head, the head coordinate system in real space can be considered as such. The coordinate system of the 3D head model reconstructed in model space The coordinates of the three reference points—the center of the left eye, the center of the right eye, and the tip of the nose—are consistent in both coordinate systems, thus mapping the human head from model space to the human head in real space. Therefore, the coordinate system of the 3D head model is consistent. Transform to camera coordinate system Transformation matrix for: ; In the defined model space, the coordinate system of the three-dimensional head model Treatment point location Afterwards, Convert the treatment point to the camera coordinate system in real space Treatment point location The details are as follows: .

[0038] Furthermore, in the registration and conversion system, when the subject's position changes, the camera's real-time positioning will produce errors, so it is necessary to compensate for the position coordinates of the treatment point; the Kalman filter algorithm is used to compensate for the error in the position coordinates of the treatment point, thereby improving the positioning accuracy of the treatment point.

[0039] In one embodiment, the interactive device is a tool for users to directly interact with 3D models and medical images, and consists of several modules: a DICOM image browser, a 3D model viewer, a module for confirming intracranial treatment targets and treatment points, and a coil posture adjustment tool. The interactive device allows users to precisely locate the intracranial treatment targets requiring stimulation by clicking on the DICOM image or 3D model. After the interaction is complete, the system automatically calculates the optimal treatment point on the scalp and the corresponding normal vector. The brain treatment target is obtained through an interactive 3D model. Clicking on the 3D model on the interactive device allows the operator to directly obtain the 3D coordinates and normal vector of the treatment point, since the triangular facets and normal vectors of each vertex were calculated during the initial model space generation. The normal vector of the treatment point is then used as the normal vector of the brain treatment target. The interactive device includes two interactive spaces: a 3D model space and a DICOM image space. The operator selects the desired brain treatment target location in either the model space or the DICOM image space. The selected target location is then input into the transcranial magnetic coil (TMC) posture positioning module. DICOM and NIFTI are the two most commonly used formats for storing patient MRI data. In NIFTI, the raw image data is stored as a 3D image, while DICOM stores 2D layers. Both spaces are used to display brain and skull information. The main difference is that the model space uses millimeters (mm) while the DICOM image space uses pixels. Furthermore, the coordinates in the model space are continuous, while those in the DICOM image space are discrete. The model space displays a 3D head model reconstructed from MRI data, while the DICOM image space displays cross-sectional images of the skull. The DICOM image space is set... shaft and The axis direction is consistent with the direction of resting-state functional magnetic resonance imaging (fMRI) data. Since the unit is pixels, the coordinate values ​​in the DICOM image space range from [value missing]. ;and DICOM image space The axial direction is consistent with the superposition direction of resting-state functional magnetic resonance imaging (fMRI) data, and the value range is related to the number of DICOM image sequences. It is assumed that... The range of values ​​for the image is: In DICOM, "Pixel Spacing" and "Spacing between slices" respectively indicate the spacing between pixels. shaft and The pixel spacing on the axis and the spacing between image layers are respectively represented by... , and Indicates the pixels in a DICOM image shaft and The pixel spacing on the axis and the spacing between DICOM image layers are determined by... , and Obtain the DICOM image space coordinate system The transformation relationship to the model space coordinate system is shown in the figure: ;in, Represents coordinates in model space. Represents the coordinates in the DICOM image space.

[0040] Furthermore, in the transcranial magnetic coil (TMC) attitude localization module, the tracked treatment point location needs to be on the scalp. When using the interaction between model space and DICOM image space to obtain the coordinates of the intracranial treatment target, the optimal coordinates of the treatment point on the scalp need to be calculated and used as the tracked treatment point. During treatment, the TMC needs to be aligned with the treatment point on the patient's scalp in a specific orientation, with the orientation of the coil plane used as the normal vector of the treatment point. When generating a three-dimensional head model using the patient's MRI data, the vertices of all triangular facets in the model space are calculated, denoted as S. Treatment points and intracranial treatment targets are different concepts. Intracranial treatment targets refer to deep targets selected through DICOM image space or model space, while treatment points are points actually located on the scalp surface. The position of the treatment point is obtained by calculating the coordinates projected from the intracranial treatment target onto the three-dimensional head model. The formulas for calculating the position and normal vector of the treatment point are as follows:

[0041] Among them, P target N represents the coordinates of the treatment point. target Let P be the normal vector of the treatment point. heal This indicates the coordinates of the brain treatment target point selected during interaction. Let be a vertex of one of the triangular faces in the model space; the 3D model is then superimposed on the actual human head, with the superposition references being the left eye, right eye, and nose tip feature points. (Rcamera) P leye , (Rcamera) P reye and (Rcamera) P nose At this point, the model space coordinate system can be considered as... and head coordinate system It is consistent, that is, through the model space coordinate system. and head coordinate system The matching process applies the model space to real human heads. Switch to The down-rotation and translation matrix is:

[0042] The position of the treatment point in model space (Model) P targetand normal vector (Model) N target Once determined, calculate the coordinates and normal vector of the treatment point in camera space, as follows:

[0043] After calculating the camera coordinates and normal vector of the treatment point, the interactive device feeds back the information to the user, especially the medical operator, to confirm the correctness of the treatment point and whether the attitude adjustment of the TMS coil is in place.

Claims

1. A visually localized transcranial magnetic stimulation closed-loop navigation device, characterized in that, It includes a binocular camera, a three-axis sliding rail system, and a host computer; the host computer is equipped with a facial feature point recognition module, a registration and conversion module, a transcranial magnetic coil posture positioning module, and an interactive device; through the binocular camera and the three-axis sliding rail system, two two-dimensional facial images of the patient at the same time are continuously acquired and input into the host computer; The facial feature point recognition module constructs a 3D facial model and identifies facial feature points based on the input 2D facial image. Simultaneously, it reconstructs the patient's resting-state functional magnetic resonance imaging (fMRI) data using 3D modeling based on the patient's MRI images, generating a 3D head model. The registration and conversion module uses facial feature points to rigidly register the 3D facial model with the 3D head model, completing the coordinate system registration and conversion. Through the interactive device on the host computer, the user selects the desired intracranial treatment target point according to the treatment purpose and patient condition and inputs it into the transcranial magnetic coil (TMC) attitude positioning module. The TMC attitude positioning module tracks the treatment point corresponding to the intracranial treatment target point in real time and maps it to the 3D head model, and provides the attitude of the TMC.

2. The visual-based transcranial magnetic stimulation closed-loop navigation device according to claim 1, characterized in that, In the facial feature point recognition module, the patient's resting-state functional magnetic resonance imaging (fMRI) data and T1-weighted structural imaging are obtained based on the patient's MRI images. The resting-state fMRI data are then reconstructed in three dimensions using a ray projection algorithm in volume rendering to obtain a three-dimensional head model.

3. The visual-based transcranial magnetic stimulation closed-loop navigation system according to claim 2, characterized in that, In the facial feature point recognition module, a stereo matching algorithm is used to extract disparity information from two 2D facial images of the patient at the same time, identify facial feature points in the 2D facial images, calculate a depth map, and generate a 3D facial model. An improved BlazeFace network is used to identify facial feature points in 2D facial images in real time and achieve head localization. An occlusion detection branch network is added to the backbone network of the improved BlazeFace network structure. Based on the features of the feature map, a binary classification judgment of occlusion is made, and the probability values ​​of the two categories are output to determine whether the current face is occluded.

4. A visual-based transcranial magnetic stimulation closed-loop navigation device according to claim 3, characterized in that, The improved BlazeFace network structure is as follows: It directly uses the existing BlazeFace network as the backbone, with input data consisting of two 2D face images captured by a binocular camera. The two 2D face images are input into the backbone for feature extraction. By progressively extracting low-level and high-level face features, a multi-scale feature map is generated. An occlusion detection branch is embedded after the fourth convolutional layer of the backbone, using the fourth layer of the multi-scale feature map as input, and outputting the occlusion prediction result to subsequent layers of the backbone as auxiliary information for computation. The occlusion detection branch includes sequentially connected convolutional layers, max-pooling layers, and fully connected layers. The convolutional layers extract features to generate feature maps. The max-pooling layer reduces the spatial size of the feature maps. The pooled feature maps are flattened and output through the fully connected layer, with categories including 'no occlusion' and 'occlusion'. The fully connected layer includes a Softmax layer, and the output of the fully connected layer is converted into a probability distribution by the Softmax layer.

5. A visual-based transcranial magnetic stimulation closed-loop navigation device according to claim 1, characterized in that, In the registration and conversion module, a mapping relationship between the coordinate systems of the head in real space and the camera is established, specifically as follows: The coordinates of facial feature points identified by the facial feature point recognition module are received and input into the calibrated camera API to calculate the depth information of the face; then, the 2D coordinates of the left camera in the binocular camera and the depth information are used together to form the 3D coordinates used by the navigation system. And it is defined as the camera coordinate system in real space. The coordinate information below; the navigation system uses the camera coordinate system. The coordinate information below was used to calculate three points: the center of the left eye, the center of the right eye, and the tip of the nose, using multiple facial feature points. These points were then analyzed using... (Rcamera) P leye , (Rcamera) P reye and (Rcamera) P nose This indicates that a head coordinate system in real space can be established. ; Cranial coordinate system With the coordinates of the tip of the nose as the origin, The axis direction is from the center of the right eye to the center of the left eye. The axis points from bottom to top; then, according to the right-hand rule for establishing a coordinate system, we obtain the cephalic coordinate system. of Axis in camera coordinate system Downward direction.

6. A visual-based transcranial magnetic stimulation closed-loop navigation device according to claim 5, characterized in that, Model space coordinate system The coordinate system of the 3D head model reconstructed in model space The model space is used for creation, specifically as follows: The 3D head model obtained from the 3D reconstruction needs to be placed in a model space for display, and the 3D coordinates of this model space are determined. Defined in the model space coordinate system The coordinate information below; using the left eye center, right eye center, and nose tip of the 3D head model in model space, respectively, using... (VModel) P leye , (VModel) P reye and (VModel) P nose This indicates that a 3D head model coordinate system for model space reconstruction is then established. ； 3D head model coordinate system reconstructed in model space With the coordinates of the tip of the nose as the origin, The axis direction is from the center of the right eye to the center of the left eye. The axis points from bottom to top; then, according to the right-hand rule, the coordinate system of the reconstructed 3D head model in model space is obtained. of Axis in model space coordinate system Downward direction.

7. A visual-based transcranial magnetic stimulation closed-loop navigation device according to claim 6, characterized in that, by (Rcamera) P leye , (Rcamera) P reye and (Rcamera) P nose Using three facial feature points as a reference, the 3D head model and the 3D face model are superimposed through rigid transformation; because the head generated from MRI data has the same size as the real human head, the head coordinate system in real space is assumed to be... The coordinate system of the 3D head model reconstructed in model space The coordinates of the three reference points—the center of the left eye, the center of the right eye, and the tip of the nose—are consistent in both coordinate systems, mapping the human head in the model space to the human head in the real space.

8. A visual-based transcranial magnetic stimulation closed-loop navigation device according to claim 1, characterized in that, In the registration and conversion system, the real-time positioning of the camera will produce errors when the subject's position changes, so it is necessary to compensate for the position coordinates of the treatment point; the Kalman filter algorithm is used to compensate for the error of the position coordinates of the treatment point and improve the positioning accuracy of the treatment point.

9. A visual-based transcranial magnetic stimulation closed-loop navigation system according to claim 1, characterized in that, The interactive device includes two interactive spaces: a model space and a DICOM image space. The operator selects the location of the desired intracranial therapeutic target in either space, and this selected target location is input into the transcranial magnetic coil posture positioning module. The model space displays a 3D head model reconstructed from MRI data, while the DICOM image space displays cross-sectional images of the skull. The DICOM image space is configured... shaft and The axis direction is consistent with the direction of resting-state functional magnetic resonance imaging (fMRI) data. Since the unit is pixels, the coordinate values ​​in the DICOM image space range from [value missing]. ;and DICOM image space The axial direction is consistent with the superposition direction of resting-state functional magnetic resonance imaging (fMRI) data, and the value range is related to the number of DICOM image sequences. It is assumed that... The range of values ​​for the image is: ; respectively 、 and Indicates the pixels in a DICOM image shaft and The pixel spacing on the axis and the spacing between DICOM image layers are determined by... 、 and Obtain the DICOM image space coordinate system The transformation relationship to the model space coordinate system is shown in the figure: ;in, Represents coordinates in model space. Represents the coordinates in the DICOM image space.

10. A visual-based transcranial magnetic stimulation closed-loop navigation device according to any one of claims 1 to 9, characterized in that, In the transcranial magnetic coil attitude localization module, the location of the tracked treatment point needs to be on the scalp. When using the interaction between model space and DICOM image space to obtain the coordinates of the treatment target point in the brain, it is necessary to calculate the optimal coordinates of the treatment point on the scalp and use them as the tracked treatment point. During the treatment process, the transcranial magnetic coil needs to be aligned with the treatment point on the patient's scalp in a certain posture, with the orientation of the coil plane as the normal vector of the treatment point. When generating a 3D head model using patient MRI data, the vertices of all triangular faces in the model space are calculated, denoted as S. Treatment points and intracranial treatment targets are different concepts. Intracranial treatment targets refer to deep targets selected through DICOM image space or model space, while treatment points are points actually located on the scalp surface. The position of the treatment point is obtained by calculating the coordinates projected from the intracranial treatment target onto the 3D head model. The formulas for calculating the position and normal vector of the treatment point are as follows: Among them, P target N represents the coordinates of the treatment point. target Let P be the normal vector of the treatment point. heal This indicates the coordinates of the brain treatment target point selected during interaction. Let be a vertex of one of the triangular faces in the model space; the 3D model is then superimposed on the actual human head, with the superposition references being the left eye, right eye, and nose tip feature points. (Rcamera) P leye , (Rcamera) P reye and (Rcamera) P nose At this point, the model space coordinate system is considered to be... and head coordinate system It is consistent, that is, through the model space coordinate system. and head coordinate system The matching process applies the model space to real human heads. Switch to The down-rotation and translation matrix is: The position of the treatment point in model space (Model) P target and normal vector (Model) N target Once determined, calculate the coordinates and normal vector of the treatment point in camera space, as follows: After calculating the camera coordinates and normal vector of the treatment point, the interactive device feeds back the information to the user to confirm the correctness of the treatment point and whether the attitude adjustment of the TMS coil is in place.

Images