Neuronavigation assistance method, device, electronic device and storage medium
Through facial and stereo model feature point registration, the position of neuronavigation equipment is adjusted, and the problem of low neuronavigation accuracy in the existing technology is solved, and high-precision neuronavigation assistance is achieved.
Patent Information
- Application Number
- CN202111499263.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-09
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-12-09
AI Technical Summary
There is a problem of low accuracy in existing neuronavigation technologies, making it difficult to accurately locate the lesion.
By obtaining the set of feature points in the face and diorama, perform feature points registration, adjust the position of the neural navigation auxiliary device until the registration result value is met, and an accurate stop position is achieved.
Improves the accuracy of neuronavigation assistance, enables accurate positioning of targets, and avoids the need to wear external devices and use probe tools.
Smart Images

Figure CN114187339B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of machine vision technology, and in particular to a neuronavigation auxiliary method, device, electronic equipment and storage medium. Background Art
[0002] With the development of machine vision, it has been applied to many fields. For example, in the medical field, machine vision can be used to accurately locate the target position. For example, it can be applied to neuronavigation. Neuronavigation refers to the use of computer-aided technology to assist doctors in quickly determining the location of lesions.
[0003] In the prior art, neuronavigation is often performed by wearing external equipment and using probe tools to mark on the patient's face, which has the problem of low accuracy. Summary of the invention
[0004] The present invention provides a neuronavigation assistance method, device, electronic device and storage medium, which are used to solve the defect of low accuracy of neuronavigation assistance in the prior art and achieve the purpose of improving the accuracy of neuronavigation assistance.
[0005] The present invention provides a neuronavigation assistance method, comprising: obtaining a first target feature point set, the first target feature point set includes at least one first target feature point, and the first target feature point is a feature point set located on a face; obtaining a second target feature point set, the second target feature point set includes at least one second target feature point, and the second target feature point is a feature point set located in a three-dimensional model; aligning a first candidate feature point with a second candidate feature point to obtain a first alignment result value, the first candidate feature point is a feature point in the first target feature point set, the second candidate feature point is a feature point in the second target feature point set, and the feature types of the first candidate feature point and the second candidate feature point are the same; and adjusting the moving position of a neuronavigation assistance device until the first alignment result value is met.
[0006] According to a neuronavigation assistance method provided by the present invention, the method also includes: when the first registration result value is greater than an error threshold, in the stereo model, with the second candidate feature point as the center, based on the area expansion radius, obtaining a target area; obtaining a set of feature points in the target area; using a third candidate feature point in the feature point set to align with the first candidate feature point to obtain a second registration result value; when the second registration result value is greater than the first registration result value, replacing the second candidate feature point with the third candidate feature point.
[0007] According to a neuronavigation assistance method provided by the present invention, the method also includes: acquiring a target image sequence, the target image sequence includes at least one target image, and the target image includes the first target feature point set; in the target image sequence, acquiring a first distance of feature points between a candidate target image and a forward image, and acquiring a second distance of feature points between a candidate target image and a backward image; when each of the first distances is greater than or equal to a first distance threshold, and each of the second distances is greater than or equal to a second distance threshold, updating the acquired first target feature point set.
[0008] According to a neuronavigation assistance method provided by the present invention, obtaining the second target feature point set includes: obtaining a medical image; performing three-dimensional reconstruction processing on the medical image to obtain a three-dimensional model corresponding to the medical image; and obtaining the second target feature point set in the three-dimensional model according to the feature type.
[0009] According to a neuronavigation auxiliary method provided by the present invention, obtaining the first target feature point set includes: obtaining an initial image, wherein the initial image includes a target object; obtaining an initial feature point set in the target object; determining the coordinates of each initial feature point in the initial feature point set in a pixel coordinate system to obtain an intermediate feature point set; performing a coordinate system conversion on each intermediate feature point in the intermediate feature point set to obtain the first target feature point set, wherein the coordinates of each target feature point in the first target feature point set are coordinates in a camera coordinate system.
[0010] According to a neuronavigation assistance method provided by the present invention, the method also includes: obtaining a calibration parameter set of an image acquisition device in the camera coordinate system; correcting the coordinates of the initial image using candidate calibration parameters in the calibration parameter set to obtain an intermediate image; performing a coordinate system transformation on the intermediate image to obtain a target image; the coordinates of the target image are coordinates in a pixel coordinate system.
[0011] The present invention also provides a neuronavigation auxiliary device, comprising: a first processing module, used to obtain a first target feature point set, the first target feature point set includes at least one first target feature point, and the first target feature point is a feature point set located on the face; a second processing module, used to obtain a second target feature point set, the second target feature point set includes at least one second target feature point, and the second target feature point is a feature point set located in a three-dimensional model; a third processing module, used to align the first candidate feature point with the second candidate feature point to obtain a first alignment result value, the first candidate feature point is a feature point in the first target feature point set, the second candidate feature point is a feature point in the second target feature point set, and the feature types of the first candidate feature point and the second candidate feature point are the same; a fourth processing module, used to adjust the moving position of the neuronavigation auxiliary device until the first alignment result value is met.
[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the above-mentioned neuronavigation assistance methods are implemented.
[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned neuronavigation-assisted methods are implemented.
[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any one of the above-mentioned neuronavigation assisted methods are implemented.
[0015] The neuronavigation assistance method, device, electronic device and storage medium provided by the present invention are as follows: obtaining a first target feature point set, the first target feature point set includes at least one first target feature point, the first target feature point is a feature point set located on the face; obtaining a second target feature point set, the second target feature point set includes at least one second target feature point, the second target feature point is a feature point set located in a three-dimensional model; registering the first candidate feature point with the second candidate feature point to obtain a first registration result value, the first candidate feature point is a feature point in the first target feature point set, the second candidate feature point is a feature point in the second target feature point set, and the first candidate feature point and the second candidate feature point have the same feature type; adjusting the moving position of the neuronavigation assistance device until the first registration result value is satisfied. By registering the first candidate feature point in the first target feature point set with the second candidate feature point of the same feature type in the second target feature point set, the moving position of the neuronavigation assistance device can be adjusted according to the first registration result value, thereby obtaining an accurate stop position of the neuronavigation assistance device, thereby improving the accuracy of neuronavigation assistance. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0017] Figure 1 This is one of the flow charts of the neuronavigation assistance method provided by the present invention;
[0018] Figure 2 This is the second flow chart of the neuronavigation assistance method provided by the present invention;
[0019] Figure 3 This is the third flow chart of the neuronavigation assistance method provided by the present invention;
[0020] Figure 4 This is the fourth flow chart of the neuronavigation assistance method provided by the present invention;
[0021] Figure 5 This is the fifth flow chart of the neuronavigation assistance method provided by the present invention;
[0022] Figure 6 This is the sixth flow chart of the neuronavigation assistance method provided by the present invention;
[0023] Figure 7 This is the seventh flow chart of the neuronavigation auxiliary method provided by the present invention;
[0024] Figure 8 It is one of the first target feature point set schematic diagrams of the neuro-navigation auxiliary method provided by the present invention;
[0025] Fig. 9 It is one of the schematic diagrams of the second target feature point set of the neuro-navigation auxiliary method provided by the present invention;
[0026] Fig.10 is a schematic diagram of the structure of the neuronavigation auxiliary device provided by the present invention;
[0027] Fig.11 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0029] Combine the following Figure 1-Figure 7 The neuronavigation-assisted method of the present invention is described.
[0030] In one embodiment, Figure 1 As shown, a neuronavigation assistance method is provided, which is described by taking the method applied to a server as an example, and includes the following steps:
[0031] Step 102: Acquire a first target feature point set, where the first target feature point set includes at least one first target feature point, and the first target feature point is a feature point set located on the face.
[0032] The feature point refers to a point that can reflect the essential features of an object. For example, in an acquired image including a face, the nose tip of the face can be considered as a feature point that can reflect the face.
[0033] Specifically, the server may acquire an image including the target object through an image acquisition device, and acquire a set of feature points in the image that can reflect essential characteristics of the target object.
[0034] In one embodiment, the server may be connected to an image acquisition device, which may be a binocular camera. An image containing a human face is captured by the binocular camera, and a first target feature point set is extracted using a facial feature point detection algorithm.
[0035] In one embodiment, a method for acquiring an image containing a human face may be to use the left camera in the above-mentioned binocular camera to obtain a first image, use the right camera in the above-mentioned binocular camera to obtain a second image, and select the right 60% area in the first image and the left 60% area in the second image respectively to obtain an image containing a human face, thereby improving the accuracy of face recognition.
[0036] Step 104 : acquiring a second target feature point set, where the second target feature point set includes at least one second target feature point, and the second target feature point is a feature point set located in the three-dimensional model.
[0037] The three-dimensional model refers to a model obtained by three-dimensionally reconstructing an image. For example, a three-dimensional reconstruction of an MRI (Magnetic Resonance Imaging) medical image is performed to obtain an STL (Standard Tessellation Language) model.
[0038] Specifically, the server may utilize a three-dimensional reconstruction method to perform three-dimensional reconstruction on an image for which the second target feature point set needs to be obtained, to obtain a three-dimensional model, and in the three-dimensional model, the second target feature point set is selected.
[0039] In one embodiment, taking an MRI medical image as an example, the server selects a target area on the MRI medical image to obtain two-dimensional segmentation data corresponding to the target area, uses an image processing plug-in to modify the above two-dimensional segmentation data, and reconstructs the modified two-dimensional segmentation data in three dimensions to obtain a corresponding three-dimensional image. In the three-dimensional image, feature points are screened to obtain a second target feature point set.
[0040] Step 106, aligning the first candidate feature point and the second candidate feature point to obtain a first alignment result value, wherein the first candidate feature point is a feature point in the first target feature point set, the second candidate feature point is a feature point in the second target feature point set, and the first candidate feature point and the second candidate feature point have the same feature type.
[0041] Among them, the feature type refers to the type of feature point. For example, if the first candidate feature point in the first target feature point set is the left inner corner of the eye, then the second candidate feature point in the second target feature point set is also the left inner corner of the eye, and the two feature types are the same. Registration refers to matching the first candidate feature point with the second candidate feature point. Through this matching, the similarity between the first candidate feature point and the second candidate feature point can be obtained. A large similarity indicates a large registration result value, and a small similarity indicates a small registration result value.
[0042] Specifically, after the server obtains the first target feature point set and the second target feature point set, it can arbitrarily select a feature point in the first target feature point set as the first candidate feature point. At the same time, in the second target feature point set, select a feature point with the same feature type as the first candidate feature point as the second candidate feature point, align the first candidate feature point with the second candidate feature point, and obtain a first alignment result value.
[0043] In one embodiment, the server may obtain a corresponding first registration result value set by traversing each first candidate feature point in the first target feature point set.
[0044] Step 108: adjusting the moving position of the neuro-navigation auxiliary device until the first registration result value is satisfied.
[0045] Among them, the neuronavigation auxiliary equipment refers to equipment that plays an auxiliary role in neuronavigation, for example, a TMS (whole-brain transcranial magnetic stimulation) coil, which can determine whether the first candidate feature point and the second candidate feature point are feature points that meet the alignment criteria by adjusting the position of the TMS coil.
[0046] Specifically, after obtaining the first registration result value, the server continuously adjusts the moving position of the neuro-navigation auxiliary device until the first registration result value is satisfied.
[0047] In one embodiment, the server can send a control signal to the robotic arm to control the robotic arm and adjust the moving position of the neuronavigation auxiliary device until the first alignment result value is met, and then send a position determination signal to stop adjusting the moving position of the neuronavigation auxiliary device. The position where the movement stops is the determined target position.
[0048] In one embodiment, the server can use TCP (Tool Center Point) calibration method to calibrate the coordinates of the TMS coil in the robot coordinate system, and then use SVD (Singular Value Decomposition) algorithm to transform the robot coordinate system into the camera coordinate system, so that the TMS coil and the acquired image containing the human face are in the same coordinate system.
[0049] In the above neuro-navigation assistance method, a first target feature point set is obtained, the first target feature point set includes at least one first target feature point, and the first target feature point is a feature point set located on the face; a second target feature point set is obtained, the second target feature point set includes at least one second target feature point, and the second target feature point is a feature point set located in a three-dimensional model; the first candidate feature point and the second candidate feature point are registered to obtain a first registration result value, the first candidate feature point is a feature point in the first target feature point set, the second candidate feature point is a feature point in the second target feature point set, and the first candidate feature point and the second candidate feature point have the same feature type; the moving position of the neuro-navigation assistance device is adjusted until the first registration result value is satisfied. By registering the first candidate feature point in the first target feature point set with the second candidate feature point of the same feature type in the second target feature point set, the moving position of the neuro-navigation assistance device can be adjusted according to the first registration result value, thereby obtaining an accurate stop position of the neuro-navigation assistance device, thereby improving the accuracy of neuro-navigation assistance.
[0050] In one embodiment, Figure 2 As shown, the neuronavigation-assisted method further includes:
[0051] Step 202: When the first registration result value is greater than the error threshold, in the stereo model, the target area is obtained based on the area expansion radius with the second candidate feature point as the center.
[0052] The error threshold refers to the critical value of the error. If it is greater than or equal to the critical value, it means that the similarity between the first candidate feature point and the second candidate feature point is small, and the corresponding registration result value is small; if it is less than the critical value, it means that the similarity between the first candidate feature point and the second candidate feature point is large, and the corresponding registration result value is large. The area expansion radius refers to the radius of the area expansion with the second candidate feature point as the center. For example, assuming that the target area is spherical, the target area can be obtained with the second candidate feature point as the center and 10 mm as the radius.
[0053] Specifically, when the first alignment result value is greater than the error threshold, the server determines that the alignment value between the first candidate feature point and the second candidate feature point is not ideal, and needs to find a feature point in the neighborhood of the second candidate feature point that has a more ideal alignment result value with the first candidate feature point. Therefore, the target area can be obtained based on the area expansion radius, and the search can be performed in the target area.
[0054] Step 204: Acquire a set of feature points in the target area.
[0055] Specifically, after obtaining the target area, the server detects the feature points in the target area through a facial key point detection algorithm, such as SIFT (Scale-invariant feature transform), SURF (Speeded-Up Robust Features) or HOG (Histogram of Oriented Gradient), and obtains a set of feature points in the target area.
[0056] Step 206: align the third candidate feature point in the feature point set with the first candidate feature point to obtain a second alignment result value.
[0057] Specifically, after obtaining the feature point set in the target area, the server arbitrarily selects a feature point from the feature point set in the target area as a third candidate feature point, and aligns it with the first candidate feature point to obtain a second alignment result value.
[0058] Specifically, the server obtains a second registration result value set by traversing all feature points in the feature point set, screens each second registration result value in the second registration result value set to obtain a registration result value with the smallest error, and determines the largest registration result value as the second registration result value.
[0059] Step 208: When the second registration result value is greater than the first registration result value, the second candidate feature point is replaced by the third candidate feature point.
[0060] Specifically, after the server obtains the second registration result value, if the second registration result value is greater than the first registration result value, it indicates that the newly selected third candidate feature point has a greater similarity with the first candidate feature point when being registered, and is more accurate, and the second candidate feature point is replaced with the third candidate feature point. For example, the first registration result value is 60%, which is the registration result value obtained by registering the second candidate feature point B with the first candidate feature point A. In order to improve the registration accuracy, the third candidate feature point C in a spherical neighborhood with a radius of 10 mm and the second candidate feature point B as the center is 80%. Then the third candidate feature point C is used to replace the second candidate feature point B, and the third candidate feature point C is determined as the required feature point.
[0061] In this embodiment, when the first registration result value is greater than the error threshold, in the stereo model, with the second candidate feature point as the center, based on the area expansion radius, the target area is obtained, a set of feature points in the target area is obtained, and the third candidate feature point in the feature point set is used to align with the first candidate feature point to obtain a second registration result value. When the second registration result value is greater than the first registration result value, the third candidate feature point is used to replace the second candidate feature point, thereby achieving the purpose of accurately obtaining the third candidate feature point with high registration accuracy.
[0062] In one embodiment, Figure 3 As shown, the neuronavigation-assisted method further includes:
[0063] Step 302: Acquire a target image sequence, where the target image sequence includes at least one target image, and the target image includes a first target feature point set.
[0064] Specifically, the server is connected to an image acquisition device, and the image acquisition device collects images at a certain frequency to obtain a target image sequence of continuous frames.
[0065] In one embodiment, the server is connected to a binocular camera, which captures images at a frequency of 16 Hz to obtain a target image sequence. In each image in the target image sequence, facial feature points are extracted using a face recognition algorithm, and a Kalman filter is used to make the feature points of the obtained target image sequence stable, thereby obtaining a first target feature point set in each image.
[0066] In one embodiment, in order to improve the accuracy of human head motion detection, the target image includes not only the first target feature point set but also an additional feature point set. For example, assuming that the first target feature point set includes 7 feature points, the target image includes 22 feature points.
[0067] Step 304: in the target image sequence, obtain a first distance between feature points of the candidate target image and the forward image, and obtain a second distance between feature points of the candidate target image and the backward image.
[0068] The forward image refers to an image captured before the candidate target image, and the backward image refers to an image captured after the candidate target image.
[0069] Specifically, after the server obtains the target image sequence, it uses any target image in the target image sequence as a candidate target image, and performs Euclidean distance calculation between feature points of the same feature type between the candidate target image and the forward image to obtain a first distance, and performs Euclidean distance calculation between feature points of the same feature type between the candidate target image and the backward image to obtain a second distance.
[0070] Step 306: When each first distance and each second distance is greater than or equal to the distance threshold, the acquired first target feature point set is updated.
[0071] Specifically, after obtaining the first distance and the second distance, the server determines whether the head wearing the neuronavigation auxiliary device moves by comparing the first distance and the second distance with the distance threshold. When each first distance and each second distance are greater than or equal to the distance threshold, it is considered that the head has moved, and the acquired first target feature point set is updated; when each first distance and each second distance are less than the distance threshold, it is considered that the head has not moved, and the acquired first target feature point set is not updated.
[0072] In one embodiment, the distance threshold value of each target image in the target image sequence acquired by the server in two-dimensional space is 1.5 mm; when each first distance and each second distance is greater than or equal to 1.5 mm, it is considered that the head has moved, and the acquired first target feature point set is updated; when each first distance and each second distance is less than 1.5 mm, it is considered that the head has not moved, and the acquired first target feature point set is not updated. And it is necessary to meet the requirement that the distance threshold value of each target image in the target image sequence acquired by the server in three-dimensional space is 2.5 mm; when each first distance and each second distance is greater than or equal to 2.5 mm, it is considered that the head has moved, and the acquired first target feature point set is updated; when each first distance and each second distance is less than 2.5 mm, it is considered that the head has not moved, and the acquired first target feature point set is not updated.
[0073] In this embodiment, by acquiring a target image sequence, in the target image sequence, acquiring the first distance of the feature points between the candidate target image and the forward image, and acquiring the second distance of the feature points between the candidate target image and the backward image, when each first distance and each second distance is greater than or equal to the distance threshold, updating the acquired first target feature point set, the purpose of accurately obtaining the first target feature point set can be achieved.
[0074] In one embodiment, Figure 4 As shown, obtaining the second target feature point set includes:
[0075] Step 402: Acquire a medical image.
[0076] Among them, medical images refer to images used for medical treatment or medical research.
[0077] Specifically, the server is connected to a medical image acquisition device, which captures medical images and transmits the medical images to the server. For example, the medical image acquisition device is a magnetic resonance device, which acquires an MRI image and transmits the image to the server. Accordingly, the server receives the MRI image.
[0078] Step 404 , performing three-dimensional reconstruction processing on the medical image to obtain a three-dimensional model corresponding to the medical image.
[0079] Specifically, after the server acquires the medical image, it performs three-dimensional reconstruction processing on the medical image using a three-dimensional reconstruction method to obtain a three-dimensional model corresponding to the medical image.
[0080] Step 406: Acquire a second target feature point set in the three-dimensional model according to the feature type.
[0081] Specifically, after the server obtains the three-dimensional model and the first target feature point set, the server selects feature points in the three-dimensional model according to the feature types of the feature points in the set to obtain the second target feature point set. For example, if the first target feature point in the first target feature point set includes seven feature points, namely, the left inner corner of the eye, the left outer corner of the eye, the right inner corner of the eye, the right outer corner of the eye, the root of the nose, the tip of the nose, and the peak of the lip, then in the three-dimensional model, seven feature points, namely, the left inner corner of the eye, the left outer corner of the eye, the right inner corner of the eye, the right outer corner of the eye, the root of the nose, the tip of the nose, and the peak of the lip are selected as the second target feature point set.
[0082] In this embodiment, by acquiring a medical image, performing a three-dimensional reconstruction process on the medical image, obtaining a three-dimensional model corresponding to the medical image, and obtaining a second target feature point set in the three-dimensional model according to the feature type, the purpose of accurately obtaining the second target feature point set can be achieved.
[0083] In one embodiment, Figure 5 As shown, obtaining the first target feature point set includes:
[0084] Step 502: Acquire an initial image, where the initial image includes a target object.
[0085] The target object refers to an object in the initial image that needs to be processed in a targeted manner. For example, if the initial image includes a face and the face needs to be processed, the face is the target object.
[0086] Specifically, the server may be connected to an image acquisition device, and after the image acquisition device acquires an initial image, the image is transmitted to the server. The image acquisition device may be a binocular camera.
[0087] In one embodiment, the left camera of the binocular camera is selected as the main camera, and a camera coordinate system is established; for the left and right cameras of the binocular camera, the image is corrected using the internal parameters of the camera to remove the distortion of the captured image and obtain an initial image.
[0088] Step 504: Acquire an initial feature point set in the target object.
[0089] Specifically, after acquiring the initial image, the server uses a feature point extraction method to extract an initial feature point set in the target object.
[0090] In one embodiment, the server uses the open source library OpenCV facial feature point detection algorithm and the LBF algorithm to extract facial feature points of the face in the region of interest, that is, an initial feature point set.
[0091] Step 506: In the pixel coordinate system, coordinates of each initial feature point in the initial feature point set are determined to obtain an intermediate feature point set.
[0092] Specifically, after obtaining the initial feature point set, the server extracts the coordinates of each initial feature point in the initial feature point set to obtain an intermediate feature point set consisting of each intermediate feature point carrying the coordinates in the pixel coordinate system.
[0093] Step 508 , performing coordinate system conversion on each intermediate feature point in the intermediate feature point set to obtain a first target feature point set, wherein the coordinates of each target feature point in the first target feature point set are coordinates in the camera coordinate system.
[0094] Specifically, after obtaining the intermediate feature point set, the server performs coordinate transformation on the candidate intermediate feature points in the set to obtain a first target feature point set, where the coordinates of each intermediate feature point in the set are coordinates in the camera coordinate system.
[0095] In one embodiment, assuming that the image acquisition device is a binocular camera, the coordinates of the candidate intermediate feature point in the camera pixel coordinate system of the left camera of the binocular camera are expressed as (u1, v1), and the coordinates in the camera pixel coordinate system of the right camera of the binocular camera are expressed as (u2, v2). The three-dimensional coordinates of the candidate intermediate feature point in the camera coordinate system are expressed as (X w , Y w , Z w ), the coordinate system can be transformed using the least squares method, expressed as the formula:
[0096]
[0097]
[0098]
[0099]
[0100] in, and They represent the elements in the binocular camera projection matrices M1 and M2 respectively, where M1=K1[R1 t1], M2=K2[R2 t2], where K1 and K2 represent the internal parameters of the binocular camera, and R1, R2, t1 and t2 represent the external parameters.
[0101] It can be understood that the method in formula (1) can be used to transform the coordinate system of each candidate intermediate feature point in the intermediate feature point set to obtain the first target feature point set.
[0102] In this embodiment, by acquiring an initial image, an initial feature point set in the target object is obtained, and in a pixel coordinate system, the coordinates of each initial feature point in the above initial feature point set are determined to obtain an intermediate feature point set, and the coordinate system of each intermediate feature point in the intermediate feature point set is transformed to obtain a first target feature point set, thereby achieving the purpose of accurately obtaining the first target feature point set in the camera coordinate system.
[0103] In one embodiment, Figure 6 As shown, the neuronavigation-assisted method further includes:
[0104] Step 602: Obtain a set of calibration parameters of an image acquisition device in a camera coordinate system.
[0105] Specifically, the server obtains a calibration parameter set of an image acquisition device through a parameter acquisition plug-in or script program, wherein the image acquisition device includes a binocular camera, a mobile phone or other image acquisition cards.
[0106] Step 604: In the calibration parameter set, the coordinates of the initial image are corrected using the candidate calibration parameters to obtain an intermediate image.
[0107] Specifically, after obtaining the calibration parameter set, the server uses one or more calibration parameters therein to correct the coordinates of the initial image to obtain an intermediate image.
[0108] In one embodiment, assuming that the image acquisition device is a binocular camera, the calibration parameter set includes the radial distortion coefficient k 11 , k 12 , k 13 , k 21 , k 22 and k 23 , tangential distortion coefficient p 11 、p 12 、p 21 and p22 , the coordinates of the image taken by the left camera in the binocular camera are expressed as (x1, y1) before coordinate correction, the coordinates of the image taken by the right camera in the binocular camera are expressed as (x2, y2) before coordinate correction, the coordinates of the image taken by the left camera in the binocular camera are expressed as (x1′, y1′) after coordinate correction, and the coordinates of the image taken by the right camera in the binocular camera are expressed as (x2′, y2′) after coordinate correction, which can be expressed as the formula:
[0109]
[0110]
[0111] Where r1 = x1 2 +y1 2 , r2=x2 2 +y2 2 .
[0112] Step 606, coordinate system conversion is performed on the intermediate image to obtain a target image; the coordinates of the target image are coordinates in the pixel coordinate system.
[0113] Specifically, after the server obtains the intermediate image, since the coordinates of the intermediate image are in the camera coordinate system, it can be converted into coordinates in the pixel coordinate system, and the image with coordinates in the pixel coordinate system is determined as the target image.
[0114] In one embodiment, the coordinates of the left camera of the binocular camera are expressed as (u1, v1) in the camera pixel coordinate system, and the coordinates of the right camera of the binocular camera are expressed as (u2, v2) in the camera pixel coordinate system. The coordinates of the image captured by the left camera in the binocular camera before coordinate correction are expressed as (x1, y1), and the coordinates of the image captured by the right camera in the binocular camera before coordinate correction are expressed as (x2, y2), and the coordinates of the image captured by the left camera in the binocular camera after coordinate correction are expressed as (x1′, y1′), and the coordinates of the image captured by the right camera in the binocular camera after coordinate correction are expressed as (x2′, y2′), which can be expressed as:
[0115] u1=x1′*fx1+cx1
[0116] v1=y1′*fy1+cy1
[0117] u2=x2′*fx2+cx2
[0118] v2=y2′*fy2+cy2 (3)
[0119] Among them, fx1 and fy1 represent the focal length of the left camera in the binocular camera, fx2 and fy2 represent the focal length of the right camera in the binocular camera, cx1 and cy1 represent the principal point coordinates of the left camera in the binocular camera, and cx2 and cy2 represent the principal point coordinates of the right camera in the binocular camera.
[0120] In this embodiment, by obtaining a calibration parameter set of the image acquisition device in the camera coordinate system, using candidate calibration parameters in the calibration parameter set to correct the coordinates of the initial image to obtain an intermediate image, and performing coordinate system transformation on the intermediate image to obtain a target image, the purpose of accurately obtaining the target image can be achieved.
[0121] In one embodiment, Figure 7 As shown, a binocular camera collects binocular images as the source of the first target feature point set, and an MRI image is used as the source of the second target feature point set. The binocular image is collected by the binocular camera, and after the image is distorted, the facial feature points are identified by using the face recognition detection algorithm; the three-dimensional coordinates of the facial feature points in the camera coordinate system are calculated by coordinate system conversion, so as to obtain the point cloud in the camera coordinate system, that is, the first target feature point set. At the same time, an MRI image is obtained, and feature points are collected on the MRI image to obtain a point cloud in the MRI coordinate system, that is, the second target feature point set. The point cloud in the camera coordinate system and the point cloud in the MRI coordinate system are registered or aligned, and according to the alignment result, the TMS coil is guided to move to the target position. According to whether the point cloud of the continuous video frames, that is, the continuous images, obtained by the binocular camera changes, the three-dimensional coordinates of the facial features are updated. Specifically, the left camera in the binocular camera is selected as the main camera, and the camera coordinate system is established; the binocular camera is calibrated using the Zhang Zhengyou calibration method to obtain the camera's internal parameters K1 and K2, external parameters R1, R2, t1 and t2, and radial distortion coefficient k 11 , k 12 , k 13 , k 21 , k 22 and k 23 , tangential distortion coefficient p 11 、p 12 、p 21 and p 22. The internal parameters K1 and K2 are composed of fx1, fy1, fx2, fy2, cx1, cy1, cx2 and cy2. fx1, fy1, fx2 and fy2 are the focal lengths of the two cameras. cx1, cy1, cx2 and cy2 are the main point coordinates of the two cameras. The external parameters R1 and R2 are the rotation matrices of the world coordinate system converted to the left and right camera coordinate systems of the binocular camera, which describe the direction of the coordinate axis of the world coordinate system relative to the camera coordinate axis. t1 and t2 are the translation matrices of the world coordinate system converted to the left and right camera coordinate systems, which describe the position of the spatial origin in the camera coordinate system. For the left and right cameras of the binocular camera, the internal parameters of the camera are used to correct the image, and the above formula (2) is used to remove image distortion. Use the above formula (3) to convert the image coordinate system to the pixel coordinate system. As Figure 8 As shown in the figure, the portraits taken by the binocular camera usually appear in the right half of the left camera and the left half of the right camera. Therefore, the right 60% area of the left image and the left 60% area of the right image are selected as the region of interest to improve the accuracy of face recognition. The open source library OpenCV face feature point detection algorithm FacemarkLBF is used to extract the coordinates of the facial feature points in the region of interest in the pixel coordinate system. The left inner corner of the eye, the left outer corner of the eye, the right inner corner of the eye, the right outer corner of the eye, the root of the nose, the tip of the nose and the peak of the lip are selected. For each feature point, the least squares method is used to solve the following formula, such as formula (1), so as to obtain the point cloud in the camera coordinate system. At the same time, specifically, the MRI medical image three-dimensional reconstruction obtains the STL model, such as Fig. 9 As shown, seven points, namely, the left inner canthus, the left outer canthus, the right inner canthus, the right outer canthus, the root of the nose, the tip of the nose and the peak of the lip, are selected in the STL model as the point cloud in the medical imaging coordinate system, that is, the point cloud in the MRI coordinate system.
[0122] In one embodiment, the specific process of the above registration or alignment includes: using the marker-based matching algorithm vtkLandmarkTransform in VTK (Visualization Toolkit, visualization tool library) to perform initial registration on the point cloud in the camera coordinate system and the point cloud in the image coordinate system, and for the point with the largest error in the initial registration result, searching for all points in the spherical neighborhood with a radius of 10 mm in the STL model, if there is a feature point with a smaller registration error, use the feature point with the smallest error as the new registration point, and update the registration error until the registration error no longer decreases, and then search the 7 feature points in turn using the same method, re-register, and use the result as the final registration result.
[0123] In one embodiment, the method for performing assisted neuronavigation specifically includes: using the TCP calibration method to calibrate the coordinates of the TMS coil in the robot coordinate system; using the SVD algorithm to convert the robot coordinate system to the camera coordinate system; using a binocular camera to collect images at a frequency of 16HZ, using a face recognition algorithm to extract facial feature points for each frame of the image, using a Kalman filter to stabilize the feature points of consecutive frame images, obtaining a sparse three-dimensional point cloud of facial feature points in each frame of the image, selecting a surgical target in the image coordinate system, and determining the coordinates of the target in the camera coordinate system after the facial feature point cloud of each frame of the image has undergone the above registration process; controlling the robotic arm to move the TMS coil to the target, and when the force sensor at the end of the robotic arm detects a force of 2 Newtons, the coil is in contact with the target. During the navigation process, in order to ensure accurate detection of head movement, 22 feature points in the center of the face can be selected. The 22 feature points are relatively stable in face recognition during the navigation process, and will not cause large errors due to occlusion caused by head rotation. The Euclidean distance of the coordinate changes of the 22 feature points in continuous frames is calculated. When the Euclidean distance of the coordinate changes of the 22 feature points in the two-dimensional space is less than 1.5 mm, and the Euclidean distance of the coordinate changes in the three-dimensional space is less than 2.5 mm, it is considered that the head has not moved, and the point cloud coordinates of the feature points are not updated; when the Euclidean distance of the coordinate change of any feature point in the two-dimensional space is greater than or equal to 1.5 mm, or the Euclidean distance of the coordinate change in the three-dimensional space is greater than or equal to 2.5 mm, it is considered that the head has moved, the point cloud coordinates of the feature points are updated, and the above registration process is continued to obtain the new coordinates of the target in the camera coordinate system, and control the TMS coil to move to the target position under the new coordinates.
[0124] The above method uses binocular stereo vision and face recognition technology based on facial feature points. The binocular camera captures facial images in real time, obtains the three-dimensional point cloud of facial feature points, integrates the information of the patient's STL model reconstructed by MRI medical images, and completes high-precision spatial registration and neuronavigation. The registration accuracy of this method can reach 2.16mm, and the navigation accuracy is 4.35mm. In addition, there is no need to wear any external equipment, nor to use probe tools to mark the patient's face in advance. It runs in real time and efficiently, improving the application value.
[0125] The neuronavigation assisting device provided by the present invention is described below. The neuronavigation assisting device described below and the neuronavigation assisting method described above can be referenced to each other.
[0126] In one embodiment, Fig.10As shown, a neuronavigation auxiliary device 1000 is provided, comprising: a first processing module 1002, a second processing module 1004, a third processing module 1006 and a fourth processing module 1008, wherein: the first processing module 1002 is used to obtain a first target feature point set, the first target feature point set includes at least one first target feature point, and the first target feature point is a feature point set located on the face; the second processing module 1004 is used to obtain a second target feature point set, the second target feature point set includes at least one second target feature point, and the second target feature point is a feature point set located in a three-dimensional model; the third processing module 1006 is used to align the first candidate feature point with the second candidate feature point to obtain a first alignment result value, the first candidate feature point is a feature point in the first target feature point set, the second candidate feature point is a feature point in the second target feature point set, and the feature types of the first candidate feature point and the second candidate feature point are the same; the fourth processing module 1008 is used to adjust the moving position of the neuronavigation auxiliary device until the first alignment result value is met.
[0127] In one embodiment, a neuronavigation auxiliary device is used to obtain a target area in a three-dimensional model based on an area expansion radius with a second candidate feature point as the center when a first registration result value is greater than an error threshold; obtain a set of feature points in the target area; use a third candidate feature point in the feature point set to align with the first candidate feature point to obtain a second registration result value; and replace the second candidate feature point with the third candidate feature point when the second registration result value is greater than the first registration result value.
[0128] In one embodiment, a neuronavigation auxiliary device is used to obtain a target image sequence, the target image sequence includes at least one target image, and the target image includes a first target feature point set; in the target image sequence, a first distance of feature points between a candidate target image and a forward image is obtained, and a second distance of feature points between the candidate target image and a backward image is obtained; when each first distance and each second distance are greater than or equal to a distance threshold, the obtained first target feature point set is updated.
[0129] In one embodiment, the second processing module 1004 is used to acquire a medical image; perform three-dimensional reconstruction on the medical image to obtain a three-dimensional model corresponding to the medical image; and acquire a second target feature point set in the three-dimensional model according to the feature type.
[0130] In one embodiment, the first processing module 1002 is used to obtain an initial image, which includes a target object; obtain an initial feature point set in the target object; determine the coordinates of each initial feature point in the initial feature point set in a pixel coordinate system to obtain an intermediate feature point set; perform coordinate system conversion on each intermediate feature point in the intermediate feature point set to obtain a first target feature point set, and the coordinates of each target feature point in the first target feature point set are coordinates in a camera coordinate system.
[0131] In one embodiment, a neuronavigation auxiliary device is used to obtain a calibration parameter set of an image acquisition device in a camera coordinate system; in the calibration parameter set, the coordinates of an initial image are corrected using candidate calibration parameters to obtain an intermediate image; the coordinate system of the intermediate image is converted to obtain a target image; the coordinates of the target image are coordinates in a pixel coordinate system.
[0132] Fig.11 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.11 As shown, the electronic device may include: a processor (processor) 1110, a communication interface (Communications Interface) 1120, a memory (memory) 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140. The processor 1110 can call the logic instructions in the memory 1130 to execute the neuronavigation assistance method, which includes: obtaining a first target feature point set, the first target feature point set includes at least one first target feature point, and the first target feature point is a feature point set located on the face; obtaining a second target feature point set, the second target feature point set includes at least one second target feature point, and the second target feature point is a feature point set located in a three-dimensional model; aligning the first candidate feature point with the second candidate feature point to obtain a first alignment result value, the first candidate feature point is a feature point in the first target feature point set, the second candidate feature point is a feature point in the second target feature point set, and the feature types of the first candidate feature point and the second candidate feature point are the same; adjusting the moving position of the neuronavigation assistance device until the first alignment result value is met.
[0133] In addition, the logic instructions in the above-mentioned memory 1130 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0134] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the neuronavigation assistance method provided by the above methods, which includes: obtaining a first target feature point set, the first target feature point set includes at least one first target feature point, and the first target feature point is a feature point set located on the face; obtaining a second target feature point set, the second target feature point set includes at least one second target feature point, and the second target feature point is a feature point set located in a three-dimensional model; aligning the first candidate feature point with the second candidate feature point to obtain a first alignment result value, the first candidate feature point is a feature point in the first target feature point set, the second candidate feature point is a feature point in the second target feature point set, and the feature types of the first candidate feature point and the second candidate feature point are the same; adjusting the moving position of the neuronavigation assistance device until the first alignment result value is met.
[0135] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the neuronavigation assistance method provided by the above-mentioned methods, the method comprising: obtaining a first target feature point set, the first target feature point set including at least one first target feature point, the first target feature point being a feature point set located on the face; obtaining a second target feature point set, the second target feature point set including at least one second target feature point, the second target feature point being a feature point set located in a three-dimensional model; aligning the first candidate feature point with the second candidate feature point to obtain a first alignment result value, the first candidate feature point being a feature point in the first target feature point set, the second candidate feature point being a feature point in the second target feature point set, the feature types of the first candidate feature point and the second candidate feature point being the same; adjusting the moving position of the neuronavigation assistance device until the first alignment result value is met.
[0136] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0137] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A neuronavigation-assisted method, characterized in that: include: Acquire a first target feature point set, where the first target feature point set includes at least one first target feature point, and the first target feature point is a feature point set located on the face; Acquire a second target feature point set, where the second target feature point set includes at least one second target feature point, and the second target feature point is a feature point set located in the three-dimensional model; Aligning a first candidate feature point with a second candidate feature point to obtain a first alignment result value, wherein the first candidate feature point is a feature point in the first target feature point set, the second candidate feature point is a feature point in the second target feature point set, and the first candidate feature point and the second candidate feature point have the same feature type; The moving position of the neuro-navigation auxiliary device is adjusted until the first registration result value is satisfied.
2. The neuronavigation assisted method according to claim 1, characterized in that: The method further comprises: When the first registration result value is greater than an error threshold, in the stereo model, taking the second candidate feature point as the center and based on the area expansion radius, obtaining a target area; Acquire a set of feature points in the target area; Using a third candidate feature point in the feature point set to align with the first candidate feature point to obtain a second alignment result value; When the second registration result value is greater than the first registration result value, the second candidate feature point is replaced by the third candidate feature point.
3. The neuronavigation assisted method according to claim 1, characterized in that: The method further comprises: Acquire a target image sequence, wherein the target image sequence includes at least one target image, and the target image includes the first target feature point set; In the target image sequence, obtaining a first distance between feature points of a candidate target image and a forward image, and obtaining a second distance between feature points of a candidate target image and a backward image; When each of the first distances and each of the second distances is greater than or equal to a distance threshold, the acquired first target feature point set is updated.
4. The neuronavigation assisted method according to claim 1, characterized in that: The obtaining of the second target feature point set comprises: Acquiring medical images; Performing three-dimensional reconstruction on the medical image to obtain a three-dimensional model corresponding to the medical image; According to the feature type, the second target feature point set is obtained in the three-dimensional model.
5. The neuronavigation assisted method according to any one of claims 1 to 4, characterized in that: The obtaining of the first target feature point set comprises: Acquire an initial image, wherein the initial image includes a target object; Acquire an initial feature point set in the target object; In a pixel coordinate system, determining the coordinates of each initial feature point in the initial feature point set to obtain an intermediate feature point set; A coordinate system conversion is performed on each intermediate feature point in the intermediate feature point set to obtain the first target feature point set, wherein the coordinates of each target feature point in the first target feature point set are coordinates in the camera coordinate system.
6. The neuronavigation assistance method according to claim 5, characterized in that: The method further comprises: In the camera coordinate system, obtaining a set of calibration parameters of an image acquisition device; In the calibration parameter set, the coordinates of the initial image are corrected using candidate calibration parameters to obtain an intermediate image; The intermediate image is subjected to coordinate system transformation to obtain a target image; the coordinates of the target image are coordinates in a pixel coordinate system.
7. A neuronavigation auxiliary device, characterized in that: include: A first processing module is used to obtain a first target feature point set, where the first target feature point set includes at least one first target feature point, and the first target feature point is a feature point set located on the face; A second processing module is used to obtain a second target feature point set, where the second target feature point set includes at least one second target feature point, and the second target feature point is a feature point set located in the three-dimensional model; A third processing module is used to align the first candidate feature point with the second candidate feature point to obtain a first alignment result value, wherein the first candidate feature point is a feature point in the first target feature point set, the second candidate feature point is a feature point in the second target feature point set, and the first candidate feature point and the second candidate feature point have the same feature type; The fourth processing module is used to adjust the moving position of the neuro-navigation auxiliary device until the first registration result value is met.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the neuronavigation assistance method according to any one of claims 1 to 6 are implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the neuronavigation assistance method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the neuronavigation assistance method according to any one of claims 1 to 6 are implemented.
