Transcranial magnetic stimulation navigation method, device, equipment and medium based on dynamic tracking

Abstract

The embodiment of the present invention discloses a transcranial magnetic stimulation navigation method, device, equipment and medium based on dynamic tracking. The present invention relates to the field of transcranial magnetic stimulation medical technology, which includes: acquiring a depth image taken by a depth camera, and extracting facial feature points of the depth image to obtain image facial feature points; extracting model facial feature points of a three-dimensional head model created based on a nuclear magnetic resonance image; performing coordinate system registration on the image facial feature points and the model facial feature points to obtain a coordinate conversion matrix, and performing coordinate conversion on the preset calibration model target coordinates according to the coordinate conversion matrix to obtain the real target coordinates; estimating the real target coordinates to obtain the target target coordinates, and planning the best path according to the target target coordinates through an improved artificial potential field method, and automatically navigating the movement of the manipulator according to the best path. The embodiment of the present application can improve the accuracy of target coordinate positioning and the safety during transcranial magnetic stimulation navigation.

Description

Technical Field

[0001] The present invention relates to the field of transcranial magnetic stimulation medical technology, and in particular to a transcranial magnetic stimulation navigation method, device, equipment and medium based on dynamic tracking. Background Art

[0002] Transcranial Magnetic Stimulation (TMS) is a non-invasive, painless brain stimulation method that affects the excitement of brain neurons by acting on the human nerve center through a transcranial magnetic coil. In the prior art, many researchers have proposed applying robots and visual positioning systems to transcranial magnetic stimulation systems. For example, the position of the patient's head and the transcranial magnetic coil is located by an infrared optical camera plus a marker positioning marker. However, attaching optical positioning markers to the patient's head will also make the patient feel uncomfortable, and there are problems such as cumbersome calibration process and high cost. In addition, there is also a method of locating the patient's head position by using a camera, but the position information obtained is not accurate. The end of the manipulator carrying the TMS coil is easy to collide with the patient's head when it reaches the patient's head for treatment according to the guidance of the planned path. Summary of the invention

[0003] The embodiments of the present invention provide a transcranial magnetic stimulation navigation method, device, equipment and medium based on dynamic tracking, aiming to solve the problems of low target coordinate accuracy and low safety in existing transcranial magnetic stimulation navigation.

[0004] In a first aspect, an embodiment of the present invention provides a transcranial magnetic stimulation navigation method based on dynamic tracking, which includes:

[0005] Acquire a depth image taken by a depth camera, and extract facial feature points from the depth image using a facial feature point detection model to obtain image facial feature points, wherein the facial feature point detection model is obtained by training an improved PFLD network;

[0006] Extracting facial feature points of a model of a three-dimensional head model created based on a nuclear magnetic resonance image through the PFLD network;

[0007] Performing coordinate system registration on the facial feature points of the image and the facial feature points of the model to obtain a coordinate conversion matrix, and performing coordinate conversion on the preset calibration model target point coordinates according to the coordinate conversion matrix to obtain the real target point coordinates;

[0008] The real target coordinates are estimated by an optimal estimation algorithm to obtain the target target coordinates, and the optimal path is planned according to the target target coordinates by an improved artificial potential field method. The movement of the manipulator is automatically navigated according to the optimal path to move the TMS coil to a position corresponding to the target target coordinates.

[0009] In a second aspect, an embodiment of the present invention further provides a transcranial magnetic stimulation navigation device based on dynamic tracking, which includes:

[0010] A first extraction unit is used to obtain a depth image taken by a depth camera, and extract facial feature points from the depth image through a facial feature point detection model to obtain image facial feature points, wherein the facial feature point detection model is obtained by training an improved PFLD network;

[0011] A second extraction unit is used to extract model face feature points of a three-dimensional head model created based on a nuclear magnetic resonance image through the PFLD network;

[0012] A registration conversion unit, used for performing coordinate system registration on the image face feature points and the model face feature points to obtain a coordinate conversion matrix, and performing coordinate conversion on the preset calibration model target point coordinates according to the coordinate conversion matrix to obtain the real target point coordinates;

[0013] A planning and navigation unit is used to estimate the real target point coordinates through an optimal estimation algorithm to obtain the target target point coordinates, and to plan an optimal path according to the target target point coordinates through an improved artificial potential field method, and to automatically navigate the movement of the manipulator according to the optimal path to move the TMS coil to a position corresponding to the target target point coordinates.

[0014] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0016] The embodiment of the present invention provides a transcranial magnetic stimulation navigation method, device, equipment and medium based on dynamic tracking. The method includes: obtaining a depth image taken by a depth camera, and extracting facial feature points from the depth image through a facial feature point detection model to obtain image facial feature points, wherein the facial feature point detection model is obtained by training an improved PFLD network; extracting model facial feature points of a three-dimensional head model created based on a nuclear magnetic resonance image through the PFLD network; performing coordinate system registration on the image facial feature points and the model facial feature points to obtain a coordinate conversion matrix, and performing coordinate conversion on the preset calibration model target point coordinates according to the coordinate conversion matrix to obtain real target point coordinates; estimating the real target point coordinates through an optimal estimation algorithm to obtain target target point coordinates, and planning an optimal path through an improved artificial potential field method according to the target target point coordinates, and automatically navigating the movement of the manipulator according to the optimal path to move the TMS coil to a position corresponding to the target target point coordinates. The technical solution of the embodiment of the present invention performs coordinate transformation on the preset calibration model target coordinates through a coordinate transformation matrix to dynamically track the target coordinates, thereby improving the accuracy of target coordinate positioning; the optimal path is planned according to the target target coordinates through an improved artificial potential field method, and the movement of the manipulator is automatically navigated according to the optimal path, thereby preventing the TMS coil carried by the end of the manipulator from colliding with the patient's head, and improving the safety of transcranial magnetic stimulation navigation to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0018] Figure 1 is a schematic diagram of a scenario of a transcranial magnetic stimulation navigation method based on dynamic tracking provided by an embodiment of the present invention;

[0019] Figure 2 A schematic flow chart of a transcranial magnetic stimulation navigation method based on dynamic tracking provided by an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of a sub-process of a transcranial magnetic stimulation navigation method based on dynamic tracking provided by an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of a sub-process of a transcranial magnetic stimulation navigation method based on dynamic tracking provided by an embodiment of the present invention;

[0022] Figure 5A schematic block diagram of a transcranial magnetic stimulation navigation device based on dynamic tracking provided by an embodiment of the present invention; and

[0023] Figure 6 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0026] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0027] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0028] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0029] See also Figure 1 , Figure 1: is a scene diagram of a transcranial magnetic stimulation navigation method based on dynamic tracking provided by an embodiment of the present invention. The transcranial magnetic stimulation navigation method based on dynamic tracking of an embodiment of the present invention can be applied to a terminal. For example, the transcranial magnetic stimulation navigation method based on dynamic tracking can be implemented by a software program configured on the terminal, thereby improving the accuracy of target coordinate positioning and the safety of transcranial magnetic stimulation navigation. Figure 1 The transcranial magnetic stimulation navigation system in the embodiment of the present invention includes a control module, a visual positioning module, a manipulator control module and a transcranial magnetic stimulation module, wherein the visual positioning includes a depth camera; the transcranial magnetic stimulation module includes a transcranial magnetic therapy device and a TMS coil; the manipulator control module includes a force sensor and a manipulator; the control module is a terminal. It should be noted that in the embodiment of the present invention, a force sensor is connected to the end of the manipulator, a transcranial magnetic therapy device is arranged below the force sensor, and the TMS coil is automatically plugged into the transcranial magnetic therapy device through the transcranial magnetic coil interface; the manipulator is a six-degree-of-freedom manipulator; the depth camera is used for facial recognition and positioning of the patient; common TMS coils include circular coils, "8"-shaped coils and "H"-shaped coils. It should also be noted that, in an embodiment of the present invention, the software program configured on the terminal is implemented through a model-view-control architecture, which includes a View object, a Model object, and a Controller class, wherein the View object is responsible for realizing image visualization and interface interaction; the Model object includes a camera control class, a robot control class, and a force sensor control class, etc., wherein the camera control class is responsible for maintaining the state of the depth camera and obtaining the depth image of the depth camera; the robot control class is responsible for encapsulating the control interface of the robot, realizing continuous motion control and inching control, pause, teaching, and higher-level trajectory tracking planning of each axis of the robot; the force sensor control class is responsible for maintaining the state of the force sensor and reading the value of the force sensor in real time; the Controller class is responsible for coordinating data interaction between the Model object and the View object, including responding to events of the View object, sending drawing data to the View object, calling the corresponding processing function of the Model object, and receiving feedback information from the Model object.

[0030] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of a transcranial magnetic stimulation navigation method based on dynamic tracking provided by an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S100-S130.

[0031] S100, obtaining a depth image taken by a depth camera, and extracting facial feature points from the depth image through a facial feature point detection model to obtain image facial feature points, wherein the facial feature point detection model is obtained by training an improved PFLD network.

[0032] In an embodiment of the present invention, when a patient is undergoing transcranial magnetic stimulation, a depth camera shoots the patient's face to obtain a depth image, the camera control class obtains the depth image, and extracts facial feature points from the depth image through a facial feature point detection model to obtain image facial feature points, wherein the facial feature point detection model is obtained by training an improved PFLD network. Understandably, the PFLD (Practical Facial LandmarkDetector) network includes a MobileNet-V2 backbone network and a multi-scale fully connected layer. It should be noted that in an embodiment of the present invention, a depth camera is used to shoot the patient's face because the depth camera has the advantages of precise positioning, stability and reliability compared to ordinary cameras. It should also be noted that in an embodiment of the present invention, there is no specific limitation on the number of depth cameras, which can be determined according to actual conditions.

[0033] See also Figure 3 In one embodiment, for example, in an embodiment of the present invention, the step S100 includes the following steps S101-S106.

[0034] S101, adding a convolutional neural network for classification in the backbone network of the PFLD network to improve the PFLD network to obtain a target PFLD network;

[0035] S102, labeling the training sample set to obtain a Ground Truth label image, and training the target PFLD network using the Ground Truth label image;

[0036] S103, using the test sample set to test the trained target PFLD network to obtain an average classification accuracy;

[0037] S104, judging whether the average classification accuracy is greater than a preset classification accuracy, if the average classification accuracy is greater than the preset classification accuracy, executing step S105, otherwise executing step S106;

[0038] S105, using the trained target PFLD network as the facial feature point detection model;

[0039] S106, returning to the step of training the target PFLD network using the Ground Truth label image until the facial feature point detection model is obtained.

[0040] In an embodiment of the present invention, a convolutional neural network for classification is added to the MobileNet-V2 backbone network of the PFLD network to improve the PFLD network to obtain a target PFLD network; the training sample set is labeled to obtain a Ground Truth label image, and the specific process of labeling is as follows: the label of the image not containing a face is (1,0,0), the label of the image containing a face but the face is blocked is (0,1,0), and the label of the face not blocked is (0,0,1), and the Ground Truth label image is obtained by the Ground Truth label image. The target PFLD network is trained using the Ground Truth label image. It is understandable that the number of training times can be set during training, and the training is stopped when the number of training times is reached. After the training is completed, the trained target PFLD network is tested using the test sample set to obtain the face classification accuracy, occlusion classification accuracy and unobstructed face classification accuracy, and the average of the face classification accuracy, occlusion classification accuracy and unobstructed face classification accuracy is calculated to obtain the average classification accuracy. It is determined whether the average classification accuracy is greater than the preset classification accuracy. If the average classification accuracy is greater than the preset classification accuracy, it indicates that the classification accuracy of the model is high, and the trained target PFLD network is used as the face feature point detection model. Otherwise, it indicates that the classification accuracy of the model is low, and the step of training the target PFLD network using the Ground Truth label image is returned to the step of training the target PFLD network until the face feature point detection model is obtained. It should be noted that in an embodiment of the present invention, during the training of the target PFLD network, the cross entropy loss function is used for training, the training environment is NVIDIA RTX3090, the batch size is set to 256, the weight decay is 1e-6, the maximum number of iterations is 64,000, and the initial learning rate is 1e-4.

[0041] S110, extracting model facial feature points of a three-dimensional head model created based on a nuclear magnetic resonance image through the PFLD network.

[0042] In an embodiment of the present invention, a nuclear magnetic resonance image is an image formed by performing layer-by-layer cross-sectional scanning on a target part of a patient, such as the head, using a nuclear magnetic resonance apparatus, that is, the nuclear magnetic resonance image is an image formed during nuclear magnetic resonance. First, a three-dimensional head model is created by a moving cube algorithm based on the nuclear magnetic resonance image, and then the model facial feature points of the three-dimensional head model are extracted by the PFLD network. It should be noted that in an embodiment of the present invention, the PFLD network is used to extract facial feature points because the PFLD network has high accuracy and high speed, and the moving cube algorithm is an existing algorithm and will not be described in detail here. It should also be noted that in an embodiment of the present invention, the VTK visualization interaction library is used to visualize the three-dimensional model and obtain a nuclear magnetic resonance image.

[0043] S120, performing coordinate system registration on the facial feature points of the image and the facial feature points of the model to obtain a coordinate transformation matrix, and performing coordinate transformation on the preset calibration model target point coordinates according to the coordinate transformation matrix to obtain the real target point coordinates.

[0044] In the embodiment of the present invention, a real head coordinate system {Head}, a camera head coordinate system {Camera}, a model coordinate system {Model} and a model space head coordinate system {V} are defined. Based on the above defined coordinate systems, the image face feature points are mapped from the real head coordinate system to the camera head coordinate system to obtain a first coordinate transformation matrix: The model face feature points are mapped from the model coordinate system to the model space head coordinate system to obtain a second coordinate transformation matrix The first coordinate transformation matrix With the second coordinate transformation matrix Multiply to get the coordinate transformation matrix Get the coordinate transformation matrix that transforms the model coordinate system to the camera head coordinate system According to the coordinate transformation matrix The preset calibration model target coordinates are converted to obtain the real target coordinates, wherein the preset calibration model target coordinates are the coordinates of the area to be stimulated in the patient's head. It should be noted that in the embodiment of the present invention, the process of converting the preset calibration model target coordinates to obtain the real target coordinates is to overlap the created three-dimensional head model with the actual head, and the overlap basis is the overlap of the left eye feature points, the right eye feature points and the nose feature points.

[0045] S130, estimating the real target coordinates by an optimal estimation algorithm to obtain the target target coordinates, and planning an optimal path according to the target target coordinates by an improved artificial potential field method, and automatically navigating the movement of the manipulator according to the optimal path to move the TMS coil to a position corresponding to the target target coordinates.

[0046] In an embodiment of the present invention, the coordinates of the target target are estimated by estimating the real target coordinates by an optimal estimation algorithm, wherein the optimal estimation algorithm is a Kalman filter estimation, and it can be understood that the optimal position of the real target coordinates can be estimated by the Kalman filter estimation; the optimal path is planned by an improved artificial potential field method according to the target target coordinates, wherein the artificial potential field method (Artificial Potential Fields, referred to as APF) is a potential field guided method proposed by Khatib et al., the principle of which is to allow obstacles to generate a repulsive field on the robot, and the target point to generate an attractive force on the obstacle, and the path planning trajectory of the robot is obtained by deriving the gradient descent of the overall potential field. According to the optimal path, the movement of the manipulator is automatically navigated to move the TMS coil to a position corresponding to the target target coordinates. It should be noted that the artificial potential field method plans the robot path by superimposing two potential fields, including the attractive field (Attractive Field) formed by the target point and the repulsive field (Repulsive Field) formed by the obstacle.

[0047] See also Figure 4 In one embodiment, for example, in an embodiment of the present invention, the step S130 includes the following steps S131-S132.

[0048] S131, obtaining the current coordinates of the TMS coil, and calculating the artificial potential field through the improved artificial potential field method according to the current coordinates, the target point coordinates and the preset obstacle point coordinates, wherein the improved artificial potential field method adds a custom height field on the basis of the artificial potential field method;

[0049] S132, using a gradient descent method to solve the trajectory points of the artificial potential field to obtain a plurality of path planning points, and obtaining an optimal path according to the plurality of path planning points.

[0050] In the embodiment of the present invention, firstly, a custom height field is added on the basis of the artificial potential field, that is, the improved artificial potential field method includes the gravitational field U att (p), repulsive field U rep (p) and the height field U lift (p), where the custom height field is as shown in formula (1), where η is the adjustment factor of the height field, which is used to adjust the size of the height field; d 0 is the preset distance, d(p, p target ) represents the distance between the current coordinate position and the target point position. When d(p, p target) is less than the preset distance, indicating that the manipulator does not need to raise its head, then the height field is equal to 0, otherwise the height field needs to be calculated. The terminal robot control class obtains the current coordinates of the TMS coil, and calculates the gravitational field, force field and height field according to the current coordinates, the target point coordinates and the preset obstacle point coordinates through the improved artificial potential field method, and adds the gravitational field, the force field and the height field to obtain the artificial potential field, wherein the formula of the artificial potential field is U(p)=U att (p)+U rep (p)+U lift (p). The gradient descent method is used to solve the trajectory points of the artificial potential field to obtain multiple path planning points, and the optimal path is obtained according to the multiple path planning points. It should be noted that in the embodiment of the present invention, during the automatic navigation of the movement of the manipulator according to the optimal path, the force sensor control class will detect whether the force value of the force sensor exceeds the preset force value; if the force value exceeds the preset force value, indicating that the TMS coil is too close to the patient's head or a collision occurs, a retraction instruction is sent to the manipulator to make the TMS coil retreat a preset distance along the direction of the normal vector, wherein the direction of the normal vector is a direction perpendicular to the center point of the TMS coil;

[0051] The current coordinates of the TMS coil are calculated according to the preset distance of retreat, and the step of calculating the artificial potential field by the improved artificial potential field method according to the current coordinates, the target point coordinates and the preset obstacle point coordinates is returned to perform, so as to move the TMS coil to a position corresponding to the target point coordinates to perform magnetic stimulation on the patient. It can be understood that if the force value does not exceed the preset force value, indicating that the TMS coil is not in close contact with the patient's head or has not collided, the manipulator continues to be navigated according to the optimal path until the TMS coil is moved to a position corresponding to the target point coordinates.

[0052]

[0053] Figure 5 is a schematic block diagram of a transcranial magnetic stimulation navigation device 200 based on dynamic tracking provided by an embodiment of the present invention. Figure 5 As shown, corresponding to the above transcranial magnetic stimulation navigation method based on dynamic tracking, the present invention also provides a transcranial magnetic stimulation navigation device 200 based on dynamic tracking. The transcranial magnetic stimulation navigation device 200 based on dynamic tracking includes a unit for executing the above transcranial magnetic stimulation navigation method based on dynamic tracking, and the device can be configured in a terminal. Specifically, please refer to Figure 5The dynamic tracking-based transcranial magnetic stimulation navigation device 200 includes a first extraction unit 201 , a second extraction unit 202 , a registration conversion unit 203 , and a planning navigation unit 204 .

[0054] Among them, the first extraction unit 201 is used to obtain a depth image taken by a depth camera, and extract facial feature points from the depth image through a facial feature point detection model to obtain image facial feature points, wherein the facial feature point detection model is obtained by training the improved PFLD network; the second extraction unit 202 is used to extract model facial feature points of a three-dimensional head model created based on a nuclear magnetic resonance image through the PFLD network; the registration and conversion unit 203 is used to perform coordinate system registration on the image facial feature points and the model facial feature points to obtain a coordinate conversion matrix, and perform coordinate conversion on the preset calibration model target point coordinates according to the coordinate conversion matrix to obtain real target point coordinates; the planning and navigation unit 204 is used to estimate the real target point coordinates through an optimal estimation algorithm to obtain target target point coordinates, and plan an optimal path through an improved artificial potential field method according to the target target point coordinates, and automatically navigate the movement of the manipulator according to the optimal path to move the TMS coil to a position corresponding to the target target point coordinates.

[0055] In some embodiments, such as the present embodiment, the first extraction unit 201 includes a first detection unit, a feature extraction unit, an improvement unit, a training unit, a testing unit, a judgment unit, and an as unit.

[0056] Among them, the first detection unit is used to classify and detect the depth image through a facial feature point detection model to obtain an image classification result; the feature extraction unit is used to extract facial feature points from the depth image to obtain image facial feature points if the classification result is that the face is not obstructed; the improvement unit is used to add a convolutional neural network for classification in the backbone network of the PFLD network to improve the PFLD network to obtain a target PFLD network; the training unit is used to label the training sample set to obtain a Ground Truth label image, and train the target PFLD network through the Ground Truth label image; the testing unit is used to test the trained target PFLD network using the test sample set to obtain an average classification accuracy; the judgment unit is used to judge whether the average classification accuracy is greater than a preset classification accuracy; the acting unit is used to use the trained target PFLD network as the facial feature point detection model if the average classification accuracy is greater than the preset classification accuracy.

[0057] In some embodiments, such as this embodiment, the registration conversion unit 203 includes a first conversion unit, a second conversion unit, and a first calculation unit.

[0058] Among them, the first conversion unit is used to map the image facial feature points from the real head coordinate system to the camera head coordinate system to obtain a first coordinate conversion matrix; the second conversion unit is used to map the model facial feature points from the model coordinate system to the model space head coordinate system to obtain a second coordinate conversion matrix; the first calculation unit is used to multiply the first coordinate conversion matrix with the second coordinate conversion matrix to obtain a coordinate conversion matrix.

[0059] In some embodiments, such as the present embodiment, the planning and navigation unit 204 includes a second calculation unit, a third calculation unit, a second detection unit, a fallback unit, and a return execution unit.

[0060] The second calculation unit is used to obtain the current coordinates of the TMS coil, and calculate the artificial potential field according to the current coordinates, the target point coordinates and the preset obstacle point coordinates through the improved artificial potential field method, wherein the improved artificial potential field method adds a custom height field on the basis of the artificial potential field method; the third calculation unit is used to solve the trajectory points of the artificial potential field by the gradient descent method to obtain multiple path planning points, and obtain the optimal path according to the multiple path planning points; the second detection unit is used to detect whether the force value of the force sensor exceeds the preset force value during the automatic navigation of the movement of the manipulator according to the optimal path; the retraction unit is used to send a retraction instruction to the manipulator if the force value exceeds the preset force value, so that the TMS coil retracts a preset distance along the direction of the normal vector; the return execution unit is used to calculate the current coordinates of the TMS coil according to the preset retraction distance, and return to execute the step of calculating the artificial potential field according to the current coordinates, the target point coordinates and the preset obstacle point coordinates through the improved artificial potential field method, so as to move the TMS coil to a position corresponding to the target point coordinates.

[0061] In some embodiments, such as this embodiment, the second computing unit includes a fourth computing unit and a fifth computing unit.

[0062] Among them, the fourth calculation unit is used to calculate the gravitational field, the force field and the height field according to the current coordinates, the target point coordinates and the preset obstacle point coordinates; the fifth calculation unit is used to add the gravitational field, the force field and the height field to obtain the artificial potential field.

[0063] The specific implementation of the dynamic tracking-based transcranial magnetic stimulation navigation device 200 of the embodiment of the present invention corresponds to the above-mentioned dynamic tracking-based transcranial magnetic stimulation navigation method, which will not be described in detail here.

[0064] The above-mentioned dynamic tracking-based transcranial magnetic stimulation navigation device can be implemented in the form of a computer program. The computer program can be used in Figure 6 Runs on the computer device shown.

[0065] See also Figure 6 , Figure 6 300 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 300 is a terminal.

[0066] See also Figure 6 The computer device 300 includes a processor 302 , a memory and a network interface 305 connected via a system bus 301 , wherein the memory may include a storage medium 303 and an internal memory 304 .

[0067] The storage medium 303 may store an operating system 3031 and a computer program 3032. When the computer program 3032 is executed, the processor 302 may execute a transcranial magnetic stimulation navigation method based on dynamic tracking.

[0068] The processor 302 is used to provide computing and control capabilities to support the operation of the entire computer device 300 .

[0069] The internal memory 304 provides an environment for the operation of the computer program 3032 in the storage medium 303. When the computer program 3032 is executed by the processor 302, the processor 302 can execute a transcranial magnetic stimulation navigation method based on dynamic tracking.

[0070] The network interface 305 is used to communicate with other devices over the network. Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 300 to which the solution of the present application is applied. The specific computer device 300 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0071] The processor 302 is used to run a computer program 3032 stored in a memory to implement the following steps: acquiring a depth image taken by a depth camera, and extracting facial feature points from the depth image through a facial feature point detection model to obtain image facial feature points, wherein the facial feature point detection model is obtained by training an improved PFLD network; extracting model facial feature points of a three-dimensional head model created based on a nuclear magnetic resonance image through the PFLD network; performing coordinate system registration on the image facial feature points and the model facial feature points to obtain a coordinate transformation matrix, and performing coordinate transformation on the preset calibration model target point coordinates according to the coordinate transformation matrix to obtain real target point coordinates; estimating the real target point coordinates through an optimal estimation algorithm to obtain target target point coordinates, and planning an optimal path through an improved artificial potential field method based on the target target point coordinates, and automatically navigating the movement of the manipulator according to the optimal path to move the TMS coil to a position corresponding to the target target point coordinates.

[0072] In certain embodiments, such as the present embodiment, the processor 302, when implementing the step of planning the optimal path according to the target point coordinates by using the improved artificial potential field method, specifically implements the following steps: obtaining the current coordinates of the TMS coil, and calculating the artificial potential field according to the current coordinates, the target point coordinates and the preset obstacle point coordinates by using the improved artificial potential field method, wherein the improved artificial potential field method adds a custom height field on the basis of the artificial potential field method; using the gradient descent method to solve the trajectory points of the artificial potential field to obtain multiple path planning points, and obtaining the optimal path according to the multiple path planning points.

[0073] In certain embodiments, such as the present embodiment, when the processor 302 implements the step of calculating the artificial potential field by the improved artificial potential field method according to the current coordinates, the target point coordinates and the preset obstacle point coordinates, it specifically implements the following steps: calculating the gravitational field, the force field and the height field according to the current coordinates, the target point coordinates and the preset obstacle point coordinates; adding the gravitational field, the force field and the height field to obtain the artificial potential field.

[0074] In certain embodiments, such as the present embodiment, the processor 302, when implementing the step of automatically navigating the movement of the manipulator according to the optimal path to move the TMS coil to a position corresponding to the target target point coordinates, specifically implements the following steps: in the process of automatically navigating the movement of the manipulator according to the optimal path, detecting whether the force value of the force sensor exceeds a preset force value; if the force value exceeds the preset force value, sending a retraction instruction to the manipulator to make the TMS coil retract a preset distance along the direction of the normal vector; calculating the current coordinates of the TMS coil according to the preset retraction distance, and returning to execute the step of calculating the artificial potential field by the improved artificial potential field method according to the current coordinates, the target target point coordinates and the preset obstacle point coordinates, so as to move the TMS coil to a position corresponding to the target target point coordinates.

[0075] In certain embodiments, such as the present embodiment, when the processor 302 implements the step of extracting facial feature points from the depth image through a facial feature point detection model to obtain image facial feature points, the processor 302 specifically implements the following steps: performing classification detection on the depth image through a facial feature point detection model to obtain an image classification result; if the classification result is that the face is not obstructed, extracting facial feature points from the depth image to obtain image facial feature points.

[0076] In certain embodiments, such as the present embodiment, the processor 302, when implementing the step of obtaining the facial feature point detection model by training the improved PFLD network, specifically implements the following steps: adding a convolutional neural network for classification in the backbone network of the PFLD network to improve the PFLD network to obtain a target PFLD network; labeling the training sample set to obtain a Ground Truth label image, and training the target PFLD network with the Ground Truth label image; testing the trained target PFLD network with a test sample set to obtain an average classification accuracy; determining whether the average classification accuracy is greater than a preset classification accuracy; if the average classification accuracy is greater than the preset classification accuracy, using the trained target PFLD network as the facial feature point detection model.

[0077] In certain embodiments, such as the present embodiment, when the processor 302 implements the step of performing coordinate system registration on the image facial feature points and the model facial feature points to obtain a coordinate transformation matrix, it specifically implements the following steps: mapping the image facial feature points from the real head coordinate system to the camera head coordinate system to obtain a first coordinate transformation matrix; mapping the model facial feature points from the model coordinate system to the model space head coordinate system to obtain a second coordinate transformation matrix; and multiplying the first coordinate transformation matrix by the second coordinate transformation matrix to obtain a coordinate transformation matrix.

[0078] It should be understood that in the embodiment of the present application, the processor 302 may be a central processing unit (CPU), and the processor 302 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0079] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0080] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program. When the computer program is executed by a processor, the processor executes any embodiment of the above-mentioned transcranial magnetic stimulation navigation method based on dynamic tracking.

[0081] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0082] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0083] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0084] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0085] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0086] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0087] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0088] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A transcranial magnetic stimulation navigation method based on dynamic tracking, It is characterized in that include: Acquire a depth image taken by a depth camera, and extract facial feature points from the depth image using a facial feature point detection model to obtain image facial feature points, wherein the facial feature point detection model is obtained by training an improved PFLD network; Extracting facial feature points of a model of a three-dimensional head model created based on a nuclear magnetic resonance image through the PFLD network; Performing coordinate system registration on the facial feature points of the image and the facial feature points of the model to obtain a coordinate conversion matrix, and performing coordinate conversion on the preset calibration model target point coordinates according to the coordinate conversion matrix to obtain the real target point coordinates; The real target coordinates are estimated by an optimal estimation algorithm to obtain the target target coordinates, and the optimal path is planned by an improved artificial potential field method according to the target target coordinates, and the movement of the manipulator is automatically navigated according to the optimal path to move the TMS coil to a position corresponding to the target target coordinates; wherein, the height field is calculated by a preset formula, and the improved artificial potential field is obtained by summing the gravitational field, the potential field and the height field, and a plurality of path planning points and the optimal path are obtained by solving the gradient descent method, and the preset formula is: Wherein, η is the adjustment factor of the height field, which is used to adjust the height field size; d 0 is the preset distance, d(p,p target ) represents the distance between the current coordinate position and the target point position. When d(p,p target ) is less than the preset distance, indicating that there is no need to lift the robot arm, then the height field is equal to 0, otherwise the height field needs to be calculated.

2. The method according to claim 1, It is characterized in that The step of planning the best path according to the target point coordinates by using an improved artificial potential field method comprises: Obtaining the current coordinates of the TMS coil, and calculating the artificial potential field through the improved artificial potential field method according to the current coordinates, the target point coordinates and the preset obstacle point coordinates, wherein the improved artificial potential field method adds a custom height field on the basis of the artificial potential field method; A gradient descent method is used to solve the trajectory points of the artificial potential energy field to obtain multiple path planning points, and an optimal path is obtained according to the multiple path planning points.

3. The method according to claim 2, It is characterized in that The step of calculating the artificial potential field by the improved artificial potential field method according to the current coordinates, the target point coordinates and the preset obstacle point coordinates comprises: Calculate the gravitational field, force field and height field according to the current coordinates, the target point coordinates and the preset obstacle point coordinates; The gravitational field, the force field and the height field are added together to obtain an artificial potential energy field.

4. The method according to claim 1, It is characterized in that The step of automatically navigating the movement of the manipulator according to the optimal path to move the TMS coil to a position corresponding to the coordinates of the target point includes: During the automatic navigation of the movement of the manipulator according to the optimal path, detecting whether the force value of the force sensor exceeds a preset force value; If the force value exceeds the preset force value, a retraction instruction is sent to the manipulator to make the TMS coil retract a preset distance along the direction of the normal vector; The current coordinates of the TMS coil are calculated according to the preset distance of retreat, and the step of calculating the artificial potential field by the improved artificial potential field method according to the current coordinates, the target point coordinates and the preset obstacle point coordinates is returned to execute, so as to move the TMS coil to a position corresponding to the target point coordinates.

5. The method according to claim 1, It is characterized in that The step of extracting facial feature points from the depth image using a facial feature point detection model to obtain image facial feature points includes: Performing classification detection on the depth image through a facial feature point detection model to obtain an image classification result; If the classification result is that the face is not blocked, facial feature points are extracted from the depth image to obtain image facial feature points.

6. The method according to claim 1, It is characterized in that The step of obtaining the facial feature point detection model by training the improved PFLD network includes: Adding a convolutional neural network for classification in the backbone network of the PFLD network to improve the PFLD network to obtain a target PFLD network; Labeling the training sample set to obtain a Ground Truth label image, and training the target PFLD network using the Ground Truth label image; Using the test sample set to test the trained target PFLD network to obtain an average classification accuracy; Determining whether the average classification accuracy is greater than a preset classification accuracy; If the average classification accuracy is greater than the preset classification accuracy, the trained target PFLD network is used as the facial feature point detection model.

7. The method according to claim 1, It is characterized in that The step of performing coordinate system registration on the image face feature points and the model face feature points to obtain a coordinate conversion matrix comprises: Mapping the facial feature points of the image from the real head coordinate system to the camera head coordinate system to obtain a first coordinate transformation matrix; Mapping the model face feature points from the model coordinate system to the model space head coordinate system to obtain a second coordinate transformation matrix; The first coordinate transformation matrix is ​​multiplied by the second coordinate transformation matrix to obtain a coordinate transformation matrix.

8. A transcranial magnetic stimulation navigation device based on dynamic tracking, It is characterized in that include: A first extraction unit is used to obtain a depth image taken by a depth camera, and extract facial feature points from the depth image through a facial feature point detection model to obtain image facial feature points, wherein the facial feature point detection model is obtained by training an improved PFLD network; A second extraction unit is used to extract model face feature points of a three-dimensional head model created based on a nuclear magnetic resonance image through the PFLD network; A registration conversion unit, used for performing coordinate system registration on the image face feature points and the model face feature points to obtain a coordinate conversion matrix, and performing coordinate conversion on the preset calibration model target point coordinates according to the coordinate conversion matrix to obtain the real target point coordinates; A planning and navigation unit is used to estimate the real target coordinates by an optimal estimation algorithm to obtain the target target coordinates, and plan the best path according to the target target coordinates by an improved artificial potential field method, and automatically navigate the movement of the manipulator according to the best path to move the TMS coil to a position corresponding to the target target coordinates; wherein, the height field is calculated by a preset formula, and the improved artificial potential field is obtained by summing the gravitational field, the potential field and the height field, and a plurality of path planning points and the best path are obtained by solving the gradient descent method, and the preset formula is: Wherein, η is the adjustment factor of the height field, which is used to adjust the height field size; d 0 is the preset distance, d(p,p target ) represents the distance between the current coordinate position and the target point position. When d(p,p target ) is less than the preset distance, indicating that there is no need to lift the robot arm, then the height field is equal to 0, otherwise the height field needs to be calculated.

9. A computer device, It is characterized in that The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, It is characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images