Transcranial magnetic navigation body surface visualization and autonomous adjustment system

Abstract

The invention discloses a transcranial magnetic navigation body surface visualization and autonomous adjustment system. According to the system, a patient head model is reconstructed through the sMRI technology to obtain a reconstructed head model coordinate system, a real head coordinate system under the view angle of a camera is established through the face recognition technology, and two visual visualization schemes are provided: virtual-real mapping of a target spot is realized through a laser holder; a physician is guided to accurately position the transcranial magnetic coil by using an augmented reality technology, and meanwhile, a visual interface is supported to guide a patient to autonomously adjust the position so as to ensure target attachment. The effectiveness of technologies such as transcranial magnetism can be remarkably improved through accurate target fitting, so that deterioration of nerve diseases is delayed, and the health level and life quality of patients are maintained.

Description

Technical Field

[0001] The present invention relates to the field of medical rehabilitation therapy, and in particular to a transcranial magnetic navigation body surface visualization and autonomous adjustment system. Background Art

[0002] Brain dysfunction refers to cognitive or motor dysfunction caused by neurological diseases, and its symptoms are manifested in various aspects such as cognition, behavior, sensation, language and emotion, including memory loss, logical thinking disorder, tremors in hands and feet, etc. Brain dysfunction will seriously affect the patient's life and even lead to the patient's death.

[0003] Non-invasive neuromodulation technologies such as transcranial magnetic stimulation (TMS) and transcranial magnetic and electrical stimulation are often used in clinical intervention of brain dysfunction due to their advantages of being non-invasive, having few side effects, and being effective quickly. The principle of TMS is to use a pulsed magnetic field that can penetrate the skull to directly act on the patient's brain, change the membrane potential of the cerebral cortex nerve cells, generate induced currents, affect brain metabolism and neural electrical activity, and thus cause a series of physiological and biochemical reactions. The target point where the stimulation device that generates a pulsed magnetic field fits the patient's skull is called the target point. Since the abnormal areas of the brains of different patients are different, the target point needs to be selected according to the actual situation of the patient. During treatment, it is necessary to ensure that the stimulation device is accurately positioned to the target point and fits the target point throughout the process. However, due to the invisibility of the target point in the brain, the placement of the stimulation device in clinical practice is often determined subjectively by medical staff, and the accurate positioning of the target point cannot be guaranteed; the target point offset caused by the patient's accidental movement during the treatment process is difficult to judge by the naked eye, and the full fit of the target point cannot be guaranteed (brain map navigation transcranial magnetic stimulation coil posture positioning device, method, and equipment (CN114073820A)). Summary of the invention

[0004] In order to solve the above problems, the patient's reconstructed head model is often used for target positioning in current research. sMRI (Structural Magnetic Resonance Imaging) technology uses the spin of hydrogen protons in the human body in a strong magnetic field and the excitation of radio frequency pulses to obtain the structural information of the tissue by detecting the signal changes of protons in the magnetic field, and is often stored in DICOM format. The acquired sMRI images are preprocessed and the image segmentation algorithm is used to obtain the head contour and brain details. Based on the segmented image data, the patient's head model is reconstructed using a three-dimensional modeling tool, and smoothed using an interpolation algorithm to improve the model's fidelity and usability. The head model established in this way can accurately reflect the patient's head structure and provide strong support for neuroscience research, head surgery planning and other fields.

[0005] However, the target selected on the head model can only be observed on the screen and cannot be accurately located to the actual position. Consider using other technologies to achieve surface visualization of the target. The present invention provides two alternative solutions. The first solution is to use laser to project the target onto the body surface. The second solution is to display a virtual coil at the target on the patient's body surface through augmented reality technology (AR). The doctor wears a head-mounted augmented reality device to observe and adjust the real coil until it coincides with the virtual coil.

[0006] For the first solution, the target's surface is visualized by a laser gimbal that can generate visible light. The laser gimbal is a device that integrates high-precision laser technology and gimbal control technology. It can quickly adjust the direction of the laser beam through built-in precision sensors and control systems, and ensure the stability and accuracy of the laser beam during use. This allows for precise positioning of the target.

[0007] For the second solution, AR technology is used to display virtual coils to achieve surface visualization of the target. AR technology can seamlessly superimpose virtual elements such as text, images, and 3D models onto images of the real environment through 3D modeling and real-time rendering, and can enable users to interact with virtual information. It allows doctors to intuitively and clearly observe the coil position calculated by the system, and can also adjust the target through actual observation.

[0008] Accordingly, in order to capture the patient's movements during treatment, face recognition technology is used to obtain the patient's coordinates in three-dimensional space and calculate its position in real time. A commonly used face recognition network is PFLD, which has the advantages of strong real-time performance and high precision. Some targeted improvements to PFLD enable PFLD to correctly identify face occlusion, and the 3D coordinates of the feature points in the camera space can be obtained by passing the identified 2D face feature points into the Kinect camera API. Finally, the mutual conversion between the reconstructed head model target, face recognition target, and laser gimbal landing point is achieved through coordinate transformation, so that the laser gimbal is always accurately projected at the patient's target target, allowing the patient to adjust the position autonomously according to the guidance of the visual interface.

[0009] The present invention provides a transcranial magnetic navigation body surface visualization and autonomous adjustment system, which reconstructs the patient's head model through sMRI technology to obtain the reconstructed head model coordinate system, establishes the real head coordinate system under the camera's perspective through face recognition technology, realizes the virtual-real mapping of the target through the laser pan-tilt platform, and finally guides the patient to adjust the position autonomously through the visualization interface to ensure the target fits. The accurate fit of the target can significantly improve the effectiveness of transcranial magnetic and other technologies, thereby delaying the deterioration of neurological diseases and maintaining the health level and quality of life of patients.

[0010] The present invention focuses on the visualization and autonomous adjustment of the target surface of transcranial magnetic technology. It reconstructs the patient's head model through sMRI technology to obtain the reconstructed head model coordinate system, and establishes the real head coordinate system under the camera's perspective through face recognition technology. It also provides two intuitive visualization solutions: using a laser gimbal to achieve virtual-real mapping of the target; using augmented reality technology to guide physicians to accurately position the transcranial magnetic coil, while supporting a visualization interface to guide patients to autonomously adjust their positions to ensure target fit. Accurate fit of the target can significantly improve the effectiveness of transcranial magnetic and other technologies, thereby delaying the deterioration of neurological diseases and maintaining the health level and quality of life of patients.

[0011] The purpose of the present invention is achieved by at least one of the following technical solutions.

[0012] A transcranial magnetic navigation body surface visualization and autonomous adjustment system, including a head model three-dimensional reconstruction module, a binocular camera face recognition module, a laser gimbal navigation module, an augmented reality module, a coordinate conversion module and a visualization autonomous adjustment interface;

[0013] The head model three-dimensional reconstruction module is used to process the patient's sMRI data and establish a three-dimensional head model for subsequent target selection;

[0014] The binocular camera face recognition module recognizes the patient's facial feature points through the binocular camera image and establishes a real head model coordinate system for subsequent target registration and monitoring;

[0015] The laser gimbal navigation module uses infrared laser to illuminate the target position on the patient's head to make the target visible, and the medical staff adjusts the coil according to the visualized target;

[0016] The augmented reality module directly superimposes the coil model of the optimal posture corresponding to the selected target on the patient's body surface through AR technology, and the medical staff observes the coil model corresponding to the target through AR glasses to adjust the coil;

[0017] The coordinate conversion module calculates the conversion matrix of the coordinate systems established in the head model 3D reconstruction module, the binocular camera face recognition module, the laser gimbal navigation module and the augmented reality module, thereby realizing information interconnection between the modules;

[0018] The visual self-adjustment interface is used to display the patient's three-dimensional head modeling and mark the correct position, so that the patient can self-check at any time during the treatment and adjust his or her own position following the guidance of the interface.

[0019] Furthermore, in the head model 3D reconstruction module, the input patient sMRI data is first preprocessed to improve image quality and ensure data consistency, including denoising, smoothing, and intensity normalization, as follows:

[0020] Reducing noise in sMRI data through filtering or other denoising algorithms to obtain clearer images;

[0021] Smoothing technology is used to reduce texture or irregular shapes in the image to ensure the smoothness of the surface of the 3D head model;

[0022] Adjust the intensity value range of sMRI data to make it consistent under different scanning conditions;

[0023] Then, an image segmentation algorithm is used to obtain the patient's skull contour and brain details. The image segmentation algorithm divides the image into different regions or objects according to pixel or voxel features in the image; the image segmentation algorithm includes a threshold segmentation method or a region growing algorithm;

[0024] After using the image segmentation algorithm, the Marching Cubes (MC) algorithm was used to perform three-dimensional modeling of the patient's head based on the preprocessed sMRI data;

[0025] Finally, a three-dimensional head model of the patient with a smooth surface can be obtained.

[0026] Furthermore, in the head model three-dimensional reconstruction module, a reconstruction head model coordinate system of the patient's head is established by selecting feature points of the three-dimensional head model;

[0027] A transition coordinate system {V} belonging to the three-dimensional model space is established in the three-dimensional model space as a connecting bridge from the model space to the real space head coordinate system. The transition coordinate system {V} is used to represent the established three-dimensional head model; the origin of the transition coordinate system {V} is defined as the position of the nose tip on the three-dimensional head model, and the definitions of the x-axis, y-axis and z-axis of the transition coordinate system {V} are consistent with the definitions of the real space head coordinate system; the three-dimensional head model is overlapped with the actual human head, and the overlap basis is each other's left eye, right eye and nose tip feature points, thereby realizing the model space being put on the real human head.

[0028] Furthermore, in the face recognition module of the binocular camera, an existing lightweight classification convolutional network CNN is added after the third convolutional layer of the existing known convolutional neural network MobileNet-V2, and the output end of the convolutional neural network MobileNet-V2 is connected to a branch CNN network for predicting occlusion, so as to obtain an improved network model, and the improved network model inputs the face image obtained by the binocular camera, and the classification convolutional network CNN outputs multiple feature points of the face image and their classification, and the branch CNN network predicts the occluded area, the probability distribution of the occlusion and the category of the occlusion;

[0029] After the improved network model recognizes the feature points of the face image, the recognized 2D feature point coordinates are passed to the camera API to obtain the 3D coordinates of the feature point in the camera space; and the real space head coordinate system {Head} is established according to the 3D coordinates of the relevant feature points to dynamically track the head; assuming that the coordinates of the left eye center, right eye center and nose tip of the feature points in the camera coordinate system {Camera} are represented by and Indicates; the origin of the real space head coordinate system {Head} is defined as the position of the nose tip. In order to symbolize the morphological characteristics of the human body as much as possible, the x-axis is defined as the direction from right to left of the head, and the y-axis is defined as from bottom to top; according to the rule of the right-hand coordinate system, the three coordinate axes of the real space head coordinate system are determined in the camera coordinate system as follows:

[0030]

[0031] Z Head =X Head ×Y Head

[0032] Among them, X Head The x-axis of the head coordinate system {Head} is a vector pointing from the right eye feature point to the left eye feature point; Head The y-axis of {Head} is obtained by multiplying the vector from the nose tip feature point to the left eye feature point by the vector from the nose tip feature point to the right eye feature point. This ensures that Y Head and X Head Vertical, and the direction is close to the head from bottom to top; Z Head The z-axis of {Head} is defined by X Head and Y Head The cross product is obtained from the back to the front towards the head; finally, for X Head , Y Head and Z Head All are unitized.

[0033] Furthermore, in the binocular camera face recognition module, the coordinate position information of the positioning target in the visual space can be determined through the head coordinate system in the real space and the position of the treatment target found according to the doctor's experience. However, in the case of dynamic movement of the patient's head, the measurement will have a large error. The use of the Kalman filter algorithm can improve the measurement accuracy. Kalman filtering can make the best estimate of the state of the system and has extremely wide applications in navigation, tracking and other fields;

[0034] Therefore, the Kalman filter is used to optimally estimate the actual position of the target, as follows:

[0035] First, define the state vector of the target as:

[0036] x k =[P x,k ,P y,k ,P z,k ,V x,k ,V y,k ,V z,k ] T

[0037] Among them, P ik , V ik , i∈(x,y,z) respectively represent the target position at t k The position component and velocity component of the i-axis direction in the camera coordinate system at each moment; the conversion relationship between the state vectors of the target at adjacent moments is:

[0038]

[0039] Among them, F is the transfer matrix of two adjacent state vectors, dt is the interval between two adjacent moments, and w k is the state transfer error, subject to N(0,Q k )’s Gaussian distribution, Q k Variance set for continuous debugging;

[0040] The observation vector y of the target k is the coordinate value measured by the camera visual positioning, then:

[0041]

[0042] Among them, tv k is the observation error, subject to N(0,R k ), H is the measurement matrix;

[0043] Finally, the optimal estimate of the state vector can be continuously updated through the five formulas of Kalman filtering:

[0044]

[0045] in, represents the prior state estimate at time k, Represents the posterior state estimate at time k-1; represents the covariance matrix of the posterior estimate at time k, Represents the prior estimated covariance matrix at time k-1; It is the result of Kalman filtering, that is, the optimal estimate.

[0046] The binocular camera face recognition module can more accurately and reliably identify and track the patient's facial feature points in a complex environment, and construct the coordinate system of the real head model according to the coordinate system modeling method described above, thereby realizing subsequent target registration and monitoring.

[0047] Furthermore, in the laser gimbal navigation module, a two-axis gimbal is used to control the movement of the infrared laser, and the two-axis gimbal includes a horizontal axis and a pitch axis, which are used for horizontal movement and pitch movement;

[0048] A right-hand rectangular coordinate system {Infrared} is established with the center point of the infrared laser as the origin. The ray emitted when the laser device is at zero position is the x-axis, and the positive direction of the x-axis is consistent with the direction of the ray. The xoy plane is the plane swept by the infrared laser when the two-axis gimbal moves horizontally at the laser zero position, and the xoz plane is the plane swept by the infrared laser when the two-axis gimbal moves in pitch at the laser zero position. Among them, θ1 is the rotation angle of the horizontal axis relative to the zero position, and θ2 is the rotation angle of the pitch axis relative to the zero position; when the gimbal rotates horizontally by θ1 and pitches by θ2, the measured distance from the laser center to the ray blocking point is l, then the coordinates (x, y, z) in the infrared laser rectangular coordinate system can be obtained, and the mathematical expression is as follows:

[0049]

[0050] Furthermore, in the laser gimbal navigation module, since the movement control of the two-axis gimbal is limited by the rotation angles of the horizontal axis and the pitch axis, after obtaining the target position, it is necessary to solve the angle between the projection of the target on the xoy plane and the x-axis. and the angle between the projection on the xoz plane and the x-axis That is, the rotation angle of the horizontal axis and the pitch axis. The mathematical expression is as follows:

[0051]

[0052] By determining the required rotation angles of the horizontal and pitch axes of the two-axis gimbal, the movement of the gimbal is controlled so that the laser can adjust its position so that its rays irradiate the target position on the patient's head and make the target visible, thereby better assisting the doctor to manually adjust the treatment position and achieve precise target treatment.

[0053] Furthermore, in the augmented reality module, in order to display the virtual coil in the correct position, it is necessary to position it through the spatial anchor point technology, as follows:

[0054] First, the depth camera of the AR glasses is used to scan and analyze the environment to create a 3D scan map of the real environment;

[0055] Then, the coordinate conversion module imports the converted target coordinates into the AR glasses, which is a point on the surface of the patient's skull, where the AR glasses will establish an anchor point;

[0056] Finally, the three-dimensional model of the coil is associated with the anchor point, so that the coil can be fixed at the target point and the images rendered at different angles are updated in real time as the medical staff moves their perspective.

[0057] Through the visual expression of the virtual model and unrestricted gesture interaction, the doctor's operation process is made more immersive, assisting the doctor in adjusting the position of the treatment coil device to achieve accurate treatment.

[0058] Furthermore, the head model 3D reconstruction module and the binocular camera face recognition module established three coordinate systems, including the transition coordinate system {V} established by the 3D head model reconstructed from sMRI data, the binocular camera coordinate system {Camera} and the head coordinate system {Head} acquired by the PFLD network; in the coordinate conversion module, the conversion relationship between the three coordinate systems is as follows:

[0059] According to the expressions of the three coordinate axes of {Head} under {Camera}, the rotation matrix from {Head} to {Camera} can be determined Combined with the origin definition of the {Head} coordinate system, the origin of {Head} in {Camera} is represented as P Nose , therefore, the rotation and translation matrix from {Head} to {Camera} is:

[0060]

[0061] During the process of camera visual positioning, the patient's head will move continuously, and the rotation and translation matrix from {Head} to {Camera} will change in real time according to the position changes of the feature points of the patient's facial recognition;

[0062] In order to realize the conversion from the camera coordinate system {Camera} to the infrared laser coordinate system {Infrared}, it is necessary to read the corresponding points in the two coordinate systems and use the coordinate conversion formula to obtain the rotation matrix and translation matrix; through Zhang's chessboard calibration method, the camera is used to identify the chessboard corner points, and the host computer is used to control the two-axis gimbal to point the laser to the corresponding corner point to obtain the homogeneous coordinate conversion matrix.

[0063] After completing the calibration of the camera and infrared laser, the target position in the camera can be converted to the infrared laser coordinate system by the following formula:

[0064]

[0065] in, is the coordinate of the target point in the infrared laser coordinate system, is the coordinate of the target point in the camera coordinate system. The conversion from the camera coordinate system {Camera} to the AR glasses coordinate system {AR} can be calibrated through the AR glasses anchor point technology, directly transmitting the data to the AR glasses, and the AR glasses are calibrated by themselves through the AR glasses anchor point technology.

[0066] Furthermore, in the visual autonomous adjustment interface, the purpose of this interface is to display the patient's three-dimensional head modeling and mark the correct position, so that the patient can self-check at any time during the treatment and adjust his or her own position following the guidance of the interface.

[0067] The visual self-adjustment interface displays the position of the patient's head relative to the target stimulation area, providing a visual interface for the patient to see the position of his or her head relative to the target point, thereby accurately and effectively adjusting the TMS coil;

[0068] When patients undergo long-term treatment, the laser may map a target point that deviates from the actual treatment area because the patient may move his head or body unconsciously; the binocular camera monitors the patient's head position in real time, and once a deviation between the actual head position and the predetermined treatment area is detected, the visual autonomous adjustment interface will prompt the patient to make appropriate adjustments; the patient follows the prompts on the visual autonomous adjustment interface and slowly moves the head or body in the specified direction until the laser target is realigned with the predetermined treatment position; during this process, the visual autonomous adjustment interface will update the display content in real time to ensure that the patient returns to the correct position; once the laser target is realigned with the treatment area, the system will confirm the accuracy of the position; at this time, the visual autonomous adjustment interface will display "correct position" or similar confirmation information, indicating that the patient can maintain the current position and continue treatment.

[0069] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0070] 1. Traditional transcranial magnetic target location methods, such as the "5cm rule" or the international EEG 10-20 location method, are simple and economical, but they rely too much on the operator's experience and have technical differences, which may lead to inaccurate location. The present invention reconstructs a head model through the sMRI data of each participant to confirm the target, which can better match the subject's actual head shape, thereby significantly reducing the location heterogeneity caused by individual anatomical differences.

[0071] 2. The present invention uses face recognition technology to monitor the target deviation caused by the patient's movement during transcranial magnetic therapy. The real-time and precision of the network enables the system to quickly and accurately capture the patient's subtle movements, and then guide the patient to adjust the position to keep the stimulation target accurately aligned with the predetermined treatment area. In addition, the module reduces the potential risks caused by the deviation of the stimulation position, such as the side effects that may be caused by stimulating non-target brain areas, and provides strong technical support to ensure the safety and effectiveness of the treatment.

[0072] 3. The present invention uses a laser gimbal to control the laser device to project the target onto the patient's body surface, ensuring the target positioning of transcranial magnetic stimulation, and also enabling the patient to quickly discover and correct possible deviations during the treatment process, thereby ensuring the accuracy and safety of the treatment.

[0073] 4. The present invention uses AR glasses to directly superimpose the target position of the coil in the form of a three-dimensional model in the user's field of view, making the target position of transcranial magnetic stimulation clear at a glance. It not only reduces the difficulty of operation, but also improves the accuracy of positioning, which helps to achieve more accurate treatment. In addition, the real-time update function of AR glasses can ensure the synchronization of information during the treatment process, and medical staff can observe the position of the coil in real time.

[0074] 5. The present invention realizes module integration and information communication between modules by mutual conversion between multiple coordinate systems. The coordinate system conversion ensures the consistency between the sMRI image coordinates and the actual coordinates, and the conversion with the laser equipment ensures the accuracy and real-time performance of the target adjustment.

[0075] 6. The present invention prompts the patient to notice the target deviation through a visual interface and guides the patient to adjust it by himself. In the absence of self-adjustment by the patient, the therapist needs to continuously monitor the patient's head position and manually adjust the position of the TMS device when necessary. By guiding the patient to adjust the target by himself through the interface, the therapist can pay more attention to the patient's reaction and treatment effect, thereby reducing their workload. In addition, by guiding the patient to adjust the target by himself through the interface, the patient can participate more actively in the treatment process, enhancing their sense of participation and self-control ability. This autonomous adjustment method not only improves the treatment effect, but also enhances the patient's treatment experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a structural schematic diagram of a transcranial magnetic navigation body surface visualization and autonomous adjustment system in an embodiment of the present invention;

[0077] Figure 2 It is a structural schematic diagram of an improved network model in an embodiment of the present invention;

[0078] Figure 3 It is a schematic diagram of the process of visualizing the autonomous adjustment interface in an embodiment of the present invention. DETAILED DESCRIPTION

[0079] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the specific implementation of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0080] Example:

[0081] A transcranial magnetic navigation body surface visualization and autonomous adjustment system, such as Figure 1 As shown, it includes a head model 3D reconstruction module, a binocular camera face recognition module, a laser gimbal navigation module, an augmented reality module, a coordinate conversion module, and a visual autonomous adjustment interface;

[0082] The head model three-dimensional reconstruction module is used to process the patient's sMRI data and establish a three-dimensional head model for subsequent target selection;

[0083] The binocular camera face recognition module recognizes the patient's facial feature points through the binocular camera image and establishes a real head model coordinate system for subsequent target registration and monitoring;

[0084] The laser gimbal navigation module uses infrared laser to illuminate the target position on the patient's head to make the target visible, and the medical staff adjusts the coil according to the visualized target;

[0085] The augmented reality module directly superimposes the coil model of the optimal posture corresponding to the selected target on the patient's body surface through AR technology, and the medical staff observes the coil model corresponding to the target through AR glasses to adjust the coil;

[0086] The coordinate conversion module calculates the conversion matrix of the coordinate systems established in the head model 3D reconstruction module, the binocular camera face recognition module, the laser gimbal navigation module and the augmented reality module, thereby realizing information interconnection between the modules;

[0087] The visual self-adjustment interface is used to display the patient's three-dimensional head modeling and mark the correct position, so that the patient can self-check at any time during the treatment and adjust his or her own position following the guidance of the interface.

[0088] Furthermore, in the head model 3D reconstruction module, the input patient sMRI data is first preprocessed to improve image quality and ensure data consistency, including denoising, smoothing, and intensity normalization, as follows:

[0089] Reducing noise in sMRI data through filtering or other denoising algorithms to obtain clearer images;

[0090] Smoothing technology is used to reduce texture or irregular shapes in the image to ensure the smoothness of the surface of the 3D head model;

[0091] Adjust the intensity value range of sMRI data to make it consistent under different scanning conditions;

[0092] Then, an image segmentation algorithm is used to obtain the patient's skull contour and brain details. The image segmentation algorithm divides the image into different regions or objects according to pixel or voxel features in the image; the image segmentation algorithm includes a threshold segmentation method or a region growing algorithm;

[0093] After using the image segmentation algorithm, the Marching Cubes (MC) algorithm was used to perform three-dimensional modeling of the patient's head based on the preprocessed sMRI data;

[0094] Finally, a three-dimensional head model of the patient with a smooth surface can be obtained. In one embodiment, the accuracy requirement for establishing the model is 0.1 mm.

[0095] Furthermore, in the head model three-dimensional reconstruction module, a reconstruction head model coordinate system of the patient's head is established by selecting feature points of the three-dimensional head model;

[0096] A transition coordinate system {V} belonging to the three-dimensional model space is established in the three-dimensional model space as a connecting bridge from the model space to the real space head coordinate system. The transition coordinate system {V} is used to represent the established three-dimensional head model; the origin of the transition coordinate system {V} is defined as the position of the nose tip on the three-dimensional head model, and the definitions of the x-axis, y-axis and z-axis of the transition coordinate system {V} are consistent with the definitions of the real space head coordinate system; the three-dimensional head model is overlapped with the actual human head, and the overlap basis is each other's left eye, right eye and nose tip feature points, thereby realizing the model space being put on the real human head.

[0097] Furthermore, in the face recognition module of the binocular camera, an existing lightweight classification convolutional network CNN is added after the third convolutional layer of the existing known convolutional neural network MobileNet-V2, and the output end of the convolutional neural network MobileNet-V2 is connected to a branch CNN network for predicting occlusion, so as to obtain an improved network model, and the improved network model inputs the face image obtained by the binocular camera, and the classification convolutional network CNN outputs multiple feature points of the face image and their classification, and the branch CNN network predicts the occluded area, the probability distribution of the occlusion and the category of the occlusion;

[0098] After the improved network model recognizes the feature points of the face image, the recognized 2D feature point coordinates are passed to the camera API to obtain the 3D coordinates of the feature point in the camera space; and the real space head coordinate system {Head} is established according to the 3D coordinates of the relevant feature points to dynamically track the head; assuming that the coordinates of the left eye center, right eye center and nose tip of the feature points in the camera coordinate system {Camera} are represented by and Indicates; the origin of the real space head coordinate system {Head} is defined as the position of the nose tip. In order to symbolize the morphological characteristics of the human body as much as possible, the x-axis is defined as the direction from right to left of the head, and the y-axis is defined as from bottom to top; according to the rule of the right-hand coordinate system, the three coordinate axes of the real space head coordinate system are determined in the camera coordinate system as follows:

[0099]

[0100] Z Head =X Head ×Y Head

[0101] Among them, X Head The x-axis of the head coordinate system {Head} is a vector pointing from the right eye feature point to the left eye feature point; Head The y-axis of {Head} is obtained by multiplying the vector from the nose tip feature point to the left eye feature point by the vector from the nose tip feature point to the right eye feature point. This ensures that Y Head and X Head Vertical, and the direction is close to the head from bottom to top; Z Head The z-axis of {Head} is defined by X Head and Y Head The cross product is obtained from the back to the front towards the head; finally, for X Head , Y Head and Z Head All are unitized.

[0102] Furthermore, in the binocular camera face recognition module, the coordinate position information of the positioning target in the visual space can be determined through the head coordinate system in the real space and the position of the treatment target found according to the doctor's experience. However, in the case of dynamic movement of the patient's head, the measurement will have a large error. The use of the Kalman filter algorithm can improve the measurement accuracy. Kalman filtering can make the best estimate of the state of the system and has extremely wide applications in navigation, tracking and other fields;

[0103] Therefore, the Kalman filter is used to optimally estimate the actual position of the target, as follows:

[0104] First, define the state vector of the target as:

[0105] x k =[P x,k ,P y,k ,P z,k ,V x,k ,V y,k ,V z,k ] T

[0106] Among them, P ik , V ik , i∈(x,y,z) respectively represent the target position at t k The position component and velocity component of the i-axis direction in the camera coordinate system at each moment; the conversion relationship between the state vectors of the target at adjacent moments is:

[0107]

[0108] Among them, F is the transfer matrix of two adjacent state vectors, dt is the interval between two adjacent moments, and w k is the state transfer error, subject to N(0,Q k )’s Gaussian distribution, Q k Variance set for continuous debugging;

[0109] The observation vector y of the target k is the coordinate value measured by the camera visual positioning, then:

[0110]

[0111] Among them, tv k is the observation error, subject to N(0,R k ), H is the measurement matrix;

[0112] Finally, the optimal estimate of the state vector can be continuously updated through the five formulas of Kalman filtering:

[0113]

[0114] in, represents the prior state estimate at time k, Represents the posterior state estimate at time k-1; represents the covariance matrix of the posterior estimate at time k, Represents the prior estimated covariance matrix at time k-1; It is the result of Kalman filtering, that is, the optimal estimate.

[0115] The binocular camera face recognition module can more accurately and reliably identify and track the patient's facial feature points in a complex environment, and construct the coordinate system of the real head model according to the coordinate system modeling method described above, thereby realizing subsequent target registration and monitoring.

[0116] Furthermore, in the laser gimbal navigation module, a two-axis gimbal is used to control the movement of the infrared laser, and the two-axis gimbal includes a horizontal axis and a pitch axis, which are used for horizontal movement and pitch movement;

[0117] A right-hand rectangular coordinate system {Infrared} is established with the center point of the infrared laser as the origin. The ray emitted when the laser device is at zero position is the x-axis, and the positive direction of the x-axis is consistent with the direction of the ray. The xoy plane is the plane swept by the infrared laser when the two-axis gimbal moves horizontally at the laser zero position, and the xoz plane is the plane swept by the infrared laser when the two-axis gimbal moves in pitch at the laser zero position. Among them, θ1 is the rotation angle of the horizontal axis relative to the zero position, and θ2 is the rotation angle of the pitch axis relative to the zero position; when the gimbal rotates horizontally by θ1 and pitches by θ2, the measured distance from the laser center to the ray blocking point is l, then the coordinates (x, y, z) in the infrared laser rectangular coordinate system can be obtained, and the mathematical expression is as follows:

[0118]

[0119] Furthermore, in the laser gimbal navigation module, since the movement control of the two-axis gimbal is limited by the rotation angles of the horizontal axis and the pitch axis, after obtaining the target position, it is necessary to solve the angle between the projection of the target on the xoy plane and the x-axis. and the angle between the projection on the xoz plane and the x-axis That is, the rotation angle of the horizontal axis and the pitch axis. The mathematical expression is as follows:

[0120]

[0121] By determining the required rotation angles of the horizontal and pitch axes of the two-axis gimbal, the movement of the gimbal is controlled so that the laser can adjust its position so that its rays irradiate the target position on the patient's head and make the target visible, thereby better assisting the doctor to manually adjust the treatment position and achieve precise target treatment.

[0122] Furthermore, in the augmented reality module, in order to display the virtual coil in the correct position, it is necessary to position it through the spatial anchor point technology, as follows:

[0123] First, in one embodiment, the environment is scanned and analyzed using the depth camera of the HoloLens 2 AR glasses to create a 3D scan map of the real environment;

[0124] Then, the coordinate conversion module imports the converted target coordinates into the AR glasses, which is a point on the surface of the patient's skull, where the AR glasses will establish an anchor point;

[0125] Finally, the three-dimensional model of the coil is associated with the anchor point, so that the coil can be fixed at the target point and the images rendered at different angles are updated in real time as the medical staff moves their perspective.

[0126] Through the visual expression of the virtual model and unrestricted gesture interaction, the doctor's operation process is made more immersive, assisting the doctor in adjusting the position of the treatment coil device to achieve accurate treatment.

[0127] Furthermore, the head model 3D reconstruction module and the binocular camera face recognition module established three coordinate systems, including the transition coordinate system {V} established by the 3D head model reconstructed from sMRI data, the binocular camera coordinate system {Camera} and the head coordinate system {Head} acquired by the PFLD network; in the coordinate conversion module, the conversion relationship between the three coordinate systems is as follows:

[0128] According to the expressions of the three coordinate axes of {Head} under {Camera}, the rotation matrix from {Head} to {Camera} can be determined Combined with the origin definition of the {Head} coordinate system, the origin of {Head} in {Camera} is represented as P Nose , therefore, the rotation and translation matrix from {Head} to {Camera} is:

[0129]

[0130] During the process of camera visual positioning, the patient's head will move continuously, and the rotation and translation matrix from {Head} to {Camera} will change in real time according to the position changes of the feature points of the patient's facial recognition;

[0131] In order to realize the conversion from the camera coordinate system {Camera} to the infrared laser coordinate system {Infrared}, it is necessary to read the corresponding points in the two coordinate systems and use the coordinate conversion formula to obtain the rotation matrix and translation matrix; through Zhang's chessboard calibration method, the camera is used to identify the chessboard corner points, and the host computer is used to control the two-axis gimbal to point the laser to the corresponding corner point to obtain the homogeneous coordinate conversion matrix.

[0132] After completing the calibration of the camera and infrared laser, the target position in the camera can be converted to the infrared laser coordinate system by the following formula:

[0133]

[0134] in, is the coordinate of the target point in the infrared laser coordinate system, is the coordinate of the target in the camera coordinate system. The conversion from the camera coordinate system {Camera} to the AR glasses coordinate system {AR} can be calibrated through the AR glasses anchor point technology, directly transmitting the data to the HoloLens 2AR glasses, and the AR glasses are calibrated by themselves through the AR glasses anchor point technology.

[0135] Furthermore, in the visual self-adjustment interface, the purpose of this interface is to display the patient's 3D head model and mark the correct position, so that the patient can self-check at any time during the treatment and follow the interface to adjust his or her position. The specific interface is as follows: Figure 3 shown.

[0136] The visual self-adjustment interface displays the position of the patient's head relative to the target stimulation area, providing a visual interface for the patient to see the position of his or her head relative to the target point, thereby accurately and effectively adjusting the TMS coil;

[0137] When patients undergo long-term treatment, the laser may map a target point that deviates from the actual treatment area because the patient may move his head or body unconsciously; the binocular camera monitors the patient's head position in real time, and once a deviation between the actual head position and the predetermined treatment area is detected, the visual autonomous adjustment interface will prompt the patient to make appropriate adjustments; the patient follows the prompts on the visual autonomous adjustment interface and slowly moves the head or body in the specified direction until the laser target is realigned with the predetermined treatment position; during this process, the visual autonomous adjustment interface will update the display content in real time to ensure that the patient returns to the correct position; once the laser target is realigned with the treatment area, the system will confirm the accuracy of the position; at this time, the visual autonomous adjustment interface will display "correct position" or similar confirmation information, indicating that the patient can maintain the current position and continue treatment.

Claims

1. A transcranial magnetic navigation body surface visualization and autonomous adjustment system, characterized in that: It includes a head model 3D reconstruction module, a binocular camera face recognition module, a laser gimbal navigation module, an augmented reality module, a coordinate conversion module, and a visual autonomous adjustment interface; The head model three-dimensional reconstruction module is used to process the patient's sMRI data and establish a three-dimensional head model; The binocular camera face recognition module recognizes the facial feature points of the patient through the binocular camera image and establishes a real head model coordinate system; The laser gimbal navigation module uses infrared laser to illuminate the target position on the patient's head to make the target visible, so that medical staff can adjust the coil according to the visualized target; The augmented reality module directly superimposes the coil model of the optimal posture corresponding to the selected target on the patient's body surface through AR technology, so that medical personnel can observe the coil model corresponding to the target through AR glasses to adjust the coil; The coordinate conversion module calculates the conversion matrix of the coordinate systems established in the head model 3D reconstruction module, the binocular camera face recognition module, the laser gimbal navigation module and the augmented reality module, thereby realizing information interconnection between the modules; The visual self-adjustment interface is used to display the patient's three-dimensional head modeling and mark the correct position, so that the patient can self-check at any time during the treatment process and adjust his or her own position following the guidance of the interface.

2. A transcranial magnetic navigation body surface visualization and autonomous adjustment system according to claim 1, characterized in that: In the head model 3D reconstruction module, the input patient sMRI data is first preprocessed to improve image quality and ensure data consistency, including denoising, smoothing, and intensity normalization; Then, an image segmentation algorithm is used to obtain the patient's skull contour and brain details. The image segmentation algorithm divides the image into different regions or objects according to pixel or voxel features in the image; the image segmentation algorithm includes a threshold segmentation method or a region growing algorithm; After using the image segmentation algorithm, the marching cubes algorithm was used to perform three-dimensional modeling of the patient's head based on the preprocessed sMRI data; Finally, a three-dimensional head model of the patient with a smooth surface is obtained.

3. A transcranial magnetic navigation body surface visualization and autonomous adjustment system according to claim 2, characterized in that: In the head model 3D reconstruction module, the reconstructed head model coordinate system of the patient's head is established by selecting the feature points of the 3D head model; A transition coordinate system {V} belonging to the three-dimensional model space is established in the three-dimensional model space as a connecting bridge from the model space to the real space head coordinate system. The transition coordinate system {V} is used to represent the established three-dimensional head model; the origin of the transition coordinate system {V} is defined as the position of the nose tip on the three-dimensional head model, and the definitions of the x-axis, y-axis and z-axis of the transition coordinate system {V} are consistent with the definitions of the real space head coordinate system; the three-dimensional head model is overlapped with the actual human head, and the overlap basis is each other's left eye, right eye and nose tip feature points, thereby realizing the model space being put on the real human head.

4. The transcranial magnetic navigation body surface visualization and autonomous adjustment system according to claim 1, characterized in that: In the face recognition module of the binocular camera, an existing known classification convolutional network CNN is added after the third convolutional layer of the existing known convolutional neural network MobileNet-V2, and the output end of the convolutional neural network MobileNet-V2 is connected to a branch CNN network for predicting occlusion to obtain an improved network model. The improved network model inputs a face image obtained by the binocular camera, and the classification convolutional network CNN outputs multiple feature points of the face image and their classification, and the branch CNN network predicts the occluded area, the probability distribution of the occlusion, and the category of the occlusion; After the improved network model recognizes the feature points of the face image, the recognized 2D feature point coordinates are passed to the camera API to obtain the 3D coordinates of the feature point in the camera space; and the real space head coordinate system {Head} is established according to the 3D coordinates of the relevant feature points to dynamically track the head; assuming that the coordinates of the left eye center, right eye center and nose tip of the feature points in the camera coordinate system {Camera} are represented by and Indicates; define the origin of the real space head coordinate system {Head} as the position of the nose tip, define the x-axis as the direction from right to left of the head, and the y-axis as from bottom to top; according to the rule of the right-hand coordinate system, determine the calculation of the three coordinate axes of the real space head coordinate system in the camera coordinate system as: Z Head =X Head ×Y Head Among them, X Head The x-axis of the head coordinate system {Head} is a vector pointing from the right eye feature point to the left eye feature point; Head Represents the y-axis of {Head}, which is obtained by multiplying a vector from the nose tip feature point to the left eye feature point by a vector from the nose tip feature point to the right eye feature point; Z Head The z-axis of {Head} is defined by X Head and Y Head The cross product is obtained from the back to the front towards the head; finally, for X Head , Y Head and Z Head All are unitized.

5. The transcranial magnetic navigation body surface visualization and autonomous adjustment system according to claim 4, characterized in that: In the binocular camera face recognition module, the coordinate position information of the positioning target in the visual space is determined through the head coordinate system in the real space and the position of the treatment target found according to the doctor's experience. The Kalman filter is used to optimally estimate the actual position of the target, as follows: First, define the state vector of the target as: x k =[P x,k ,P y,k ,P z,k ,V x,k ,V y,k ,V z,k ] T Among them, P ik , V ik , i∈(x,y,z) respectively represent the target position at t k The position component and velocity component of the i-axis direction in the camera coordinate system at each moment; the conversion relationship between the state vectors of the target at adjacent moments is: Among them, F is the transfer matrix of two adjacent state vectors, dt is the interval between two adjacent moments, and w k is the state transfer error, subject to N(0,Q k )’s Gaussian distribution, Q k Variance set for continuous debugging; The observation vector y of the target k is the coordinate value measured by the camera visual positioning, then: Among them, tv k is the observation error, subject to N(0,R k ), H is the measurement matrix; Finally, the optimal estimate of the state vector can be continuously updated through the five formulas of Kalman filtering: in, represents the prior state estimate at time k, Represents the posterior state estimate at time k-1; represents the covariance matrix of the posterior estimate at time k, Represents the prior estimated covariance matrix at time k-1; It is the result of Kalman filtering, that is, the optimal estimate.

6. The transcranial magnetic navigation body surface visualization and autonomous adjustment system according to claim 1, characterized in that: In the laser gimbal navigation module, a two-axis gimbal is used to control the movement of the infrared laser. The two-axis gimbal includes a horizontal axis and a pitch axis, which are used for horizontal movement and pitch movement. A right-hand rectangular coordinate system {Infrared} is established with the center point of the infrared laser as the origin. The ray emitted when the laser device is at zero position is the x-axis, and the positive direction of the x-axis is consistent with the direction of the ray. The xoy plane is the plane swept by the infrared laser when the two-axis gimbal moves horizontally at the laser zero position, and the xoz plane is the plane swept by the infrared laser when the two-axis gimbal moves in pitch at the laser zero position. Among them, θ1 is the rotation angle of the horizontal axis relative to the zero position, and θ2 is the rotation angle of the pitch axis relative to the zero position. When the gimbal rotates horizontally by θ1 and pitches by θ2, the measured distance from the laser center to the ray blocking point is l, and the mathematical expression of the coordinates (x, y, z) in the infrared laser rectangular coordinate system is as follows:

7. The transcranial magnetic navigation body surface visualization and autonomous adjustment system according to claim 9, characterized in that: In the laser gimbal navigation module, after obtaining the target position, it is necessary to solve the angle between the target projection on the xoy plane and the x-axis. and the angle between the projection on the xoz plane and the x-axis That is, the rotation angle of the horizontal axis and the pitch axis. The mathematical expression is as follows: By determining the required rotation angles of the horizontal axis and the pitch axis of the two-axis gimbal, the movement of the gimbal is controlled so that the laser can be adjusted to illuminate the target position on the patient's head so that the target is visible.

8. The transcranial magnetic navigation body surface visualization and autonomous adjustment system according to claim 1, characterized in that: In the augmented reality module, in order to display the virtual coil in the correct position, it is necessary to use the spatial anchor technology for positioning, as follows: First, the depth camera of the AR glasses is used to scan and analyze the environment to create a 3D scan map of the real environment; Then, the coordinate conversion module imports the converted target coordinates into the AR glasses, which is a point on the surface of the patient's skull, where the AR glasses will establish an anchor point; Finally, the three-dimensional model of the coil is associated with the anchor point, so that the coil can be fixed at the target point and the images rendered at different angles are updated in real time as the medical staff moves their perspective. Through the visual expression of the virtual model and unrestricted gesture interaction, the doctor's operation process is made more immersive, assisting the doctor in adjusting the position of the treatment coil device to achieve accurate treatment.

9. A transcranial magnetic navigation body surface visualization and autonomous adjustment system according to any one of claims 1 to 8, characterized in that: The head model 3D reconstruction module and the binocular camera face recognition module established three coordinate systems, including the transition coordinate system {V} established by the 3D head model reconstructed from sMRI data, the binocular camera coordinate system {Camera} and the head coordinate system {Head} acquired by the PFLD network; in the coordinate conversion module, the conversion relationship between the three coordinate systems is as follows: According to the expressions of the three coordinate axes of {Head} under {Camera}, determine the rotation matrix from {Head} to {Camera} Combined with the origin definition of the {Head} coordinate system, the origin of {Head} in {Camera} is represented as P Nose , therefore, the rotation and translation matrix from {Head} to {Camera} is: During the process of camera visual positioning, the patient's head will move continuously, and the rotation and translation matrix from {Head} to {Camera} will change in real time according to the position changes of the feature points of the patient's facial recognition; In order to realize the conversion from the camera coordinate system {Camera} to the infrared laser coordinate system {Infrared}, it is necessary to read the corresponding points in the two coordinate systems and use the coordinate conversion formula to obtain the rotation matrix and translation matrix; through Zhang's chessboard calibration method, the camera is used to identify the chessboard corner points, and the host computer is used to control the two-axis gimbal to point the laser to the corresponding corner point to obtain the homogeneous coordinate conversion matrix. After completing the calibration of the camera and infrared laser, the target position in the camera can be converted to the infrared laser coordinate system by the following formula: in, is the coordinate of the target point in the infrared laser coordinate system, is the coordinate of the target point in the camera coordinate system; the conversion from the camera coordinate system {Camera} to the AR glasses coordinate system {AR} is calibrated through the AR glasses anchor point technology, and the data is directly transmitted to the AR glasses, and the AR glasses are calibrated by themselves through the AR glasses anchor point technology.

10. The transcranial magnetic navigation body surface visualization and autonomous adjustment system according to claim 9, characterized in that: The visual self-adjustment interface displays the position of the patient's head relative to the target stimulation area, providing the patient with a visual interface that enables the patient to see the position of his or her head relative to the target point; When patients undergo long-term treatment, the laser may map a target point that deviates from the actual treatment area because the patient may move his head or body unconsciously; the binocular camera monitors the patient's head position in real time. Once a deviation between the actual head position and the predetermined treatment area is detected, the visual autonomous adjustment interface will prompt the patient to make appropriate adjustments; the patient follows the prompts on the visual autonomous adjustment interface to move the head or body in the specified direction until the laser target is realigned with the predetermined treatment position; during this process, the visual autonomous adjustment interface will update the display content in real time to ensure that the patient returns to the correct position; once the laser target is realigned with the treatment area, the system will confirm the accuracy of the position; at this time, the visual autonomous adjustment interface will display a confirmation message, indicating that the patient can maintain the current position and continue treatment.

Images