Brain tissue positioning method based on facial point cloud registration

Abstract

The invention discloses a brain tissue positioning method based on facial point cloud registration. The problems that in the prior art, operation is complex, treatment time is long, a patient needs to keep the head still for a long time, and diagnosis and treatment efficiency is low are mainly solved. According to the implementation scheme, brain tissue data B of a patient is obtained through nuclear magnetic resonance imaging, and a head contour point cloud M of the patient is obtained from B; capturing a patient face RGB image and a depth point cloud by using an RGB-D camera, segmenting the point cloud by using key points of the patient face RGB image, and generating a patient face point cloud D under an RGB-D camera coordinate system; the head contour point cloud M and the face point cloud D of the patient are registered, a rotation matrix and a translation vector between the two point clouds are obtained through registration, the brain tissue data B of the patient under the MRI coordinate system are converted to be under the RGB-D camera coordinate system, and brain tissue positioning is achieved. Sampling time can be shortened, time for keeping the head of a patient stationary is shortened, operation of a doctor is simplified, overall diagnosis and treatment efficiency is improved, and the method can be used for assisting the doctor in diagnosis and treatment of brain tissue.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a brain tissue positioning method, which can be used to assist doctors in diagnosing and treating brain tissue. Background Art

[0002] With the rapid development and widespread application of computer imaging technology in the medical field, medical imaging technology has become one of the most intensively and rapidly researched and developed areas in the medical field.

[0003] During certain treatments or surgeries, the brain tissue information obtained by medical imaging technology needs to be compared with the patient's actual brain condition in real time to analyze the patient's brain condition. The premise for real-time comparison is that the brain tissue needs to be accurately located.

[0004] Brain tissue localization can assist doctors in diagnosing and treating brain tissue. Usually, the head contour point cloud obtained by visual medical imaging technologies such as MRI and CT before treatment is matched with the patient's facial point cloud obtained by other methods during treatment to complete the positioning of brain tissue.

[0005] The patent document with application number 202210203187.X discloses a method for visual positioning of brain tissue, which uses the OptiTrack camera to combine magnetic resonance imaging technology and augmented reality technology to complete real-time positioning and visualization of brain tissue. This method has three shortcomings, which will reduce the overall diagnosis and treatment efficiency: First, when collecting head point clouds during patient treatment, doctors need to use a positioning pen to collect point by point on the patient's head to form a head point cloud, which increases the complexity of the doctor's operation; second, using a positioning pen to collect head point clouds point by point will inevitably increase the treatment time, and the patient needs to keep his head still during this period, which requires a high level of cooperation from the patient; third, the number of point clouds collected with the positioning pen is small, which affects the accuracy of subsequent point cloud alignment.

[0006] Patent document with application number 202311218381.6 discloses a surgical navigation space transformation method and system, which samples the patient's face during surgery through a non-contact RGB-D camera and aligns the sampling results with the patient's preoperative CT image. Although this method uses a non-contact RGB-D camera to sample the patient's face during surgery, it does not involve obtaining the patient's brain tissue data or the positioning of the brain tissue through MRI technology. Summary of the invention

[0007] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a brain tissue localization method based on facial point cloud registration, so as to reduce sampling time, simplify the doctor's operation, reduce the time the patient keeps his head still, and improve the overall diagnosis and treatment efficiency.

[0008] The technical solution to achieve the purpose of the present invention is: using a non-contact RGB-D camera instead of a positioning pen to obtain the facial point cloud of the patient during treatment, that is, only the RGB-D camera needs to be used to capture data once in front of the patient's face as required, and the facial key point detection algorithm can be combined to obtain the patient's facial point cloud, and then it is aligned with the patient's head contour point cloud obtained by using MRI technology and edge detection algorithm to obtain the patient's brain tissue data in the RGB-D camera coordinate system, so as to achieve fast and accurate brain tissue positioning. The specific steps include the following:

[0009] (1) Obtain the patient's brain tissue data B = {b1, b2, b3, ..., b i ,...}, where b i is any voxel grid in the brain tissue data B, and the patient's head contour point cloud M is obtained from the brain tissue data B using an edge detection algorithm to obtain the brain tissue data B and the head contour point cloud M in the MRI coordinate system;

[0010] (2) Use an RGB-D camera to capture the RGB image and depth point cloud of the patient's face. Use the facial key point detection algorithm to extract the key points of the RGB image of the patient's face captured by the RGB-D camera, and combine it with the depth point cloud for face segmentation to generate the patient's facial point cloud D in the RGB-D camera coordinate system;

[0011] (3) Use the point cloud registration algorithm to register the patient's head contour point cloud M in the MRI coordinate system with the facial point cloud D in the RGB-D camera coordinate system to obtain the rotation matrix R and translation vector t between the two point clouds;

[0012] (4) Using the rotation matrix R and translation vector t obtained by point cloud registration, the patient’s brain tissue data B in the MRI coordinate system is converted to the RGB-D camera coordinate system to obtain the patient’s brain tissue data E in the RGB-D camera coordinate system, thereby realizing the positioning of the brain tissue.

[0013] Compared with the prior art, the present invention has the following advantages:

[0014] The brain tissue localization method based on facial point cloud registration of the present invention uses a non-contact RGB-D camera to quickly and easily obtain the patient's facial point cloud data, and registers it with the patient's head contour point cloud obtained by using MRI technology and edge detection algorithm to obtain the patient's brain tissue data in the RGB-D camera coordinate system, thereby achieving fast and accurate brain tissue localization, which not only greatly reduces the sampling time and the time that the patient keeps his head still, but also simplifies the doctor's operation and improves the overall diagnosis and treatment efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the implementation of the present invention;

[0016] Figure 2 is a schematic diagram of the patient's brain tissue obtained using MRI technology in the present invention;

[0017] Figure 3 is a schematic diagram of a head contour point cloud obtained by using an edge detection algorithm in the present invention;

[0018] Figure 4 is a schematic diagram of a patient's facial point cloud acquired by an RGB-D camera in the present invention;

[0019] Figure 5 It is a schematic diagram of the result of registering the head contour point cloud and the facial point cloud using the present invention;

[0020] Figure 6 The present invention utilizes Figure 5 Schematic diagram of the brain tissue localization results based on the registration results. DETAILED DESCRIPTION

[0021] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that the following description is only used to explain the present invention and is not used to limit the present invention.

[0022] Reference Figure 1 , the implementation steps of this example are as follows:

[0023] Step 1: Acquire MRI data and preprocess them.

[0024] 1.1) Obtain the patient's brain tissue data B = {b1, b2, b3, ..., b i ,...}, where b i is any voxel grid in the brain tissue data B, such as Figure 2 As shown;

[0025] 1.2) setting an extraction threshold according to the grayscale value of the head contour in the brain tissue data B, extracting the brain tissue data within the threshold range, and obtaining the head contour;

[0026] 1.3) Compare the extraction results under different thresholds, convert the most complete head contour in the extraction results into a head contour point cloud M, and obtain the head contour point cloud M in the MRI coordinate system:

[0027] M={m1,m2,m3,...,m j ,...},

[0028] Among them, m j is any point in the head contour point cloud M, such as Figure 3 shown.

[0029] Step 2: Obtain the patient’s facial point cloud D using an RGB-D camera.

[0030] 2.1) Use an RGB-D camera to capture RGB images and depth point cloud data of the patient’s face;

[0031] 2.2) Set up the corresponding deep learning network according to the face key point detection algorithm;

[0032] The facial key point detection algorithm includes MediaPipe Face Mesh, Dilb68, etc. Each algorithm has its corresponding deep learning network. This example uses but is not limited to MediaPipe Face Mesh as the facial key point detection algorithm. Its corresponding deep learning network includes a feature extraction module, a key point regression module and a post-processing module.

[0033] 2.3) Input the patient's facial RGB image captured by the RGB-D camera into the selected deep learning face key point detection network to obtain N face key points F, N ≥ 68;

[0034] 2.4) Align the RGB image of the patient's face captured by the RGB-D camera with the depth point cloud it captured, extract the face area in the depth point cloud according to the position of the N key points F in the patient's face RGB image, realize face segmentation, and generate the patient's face point cloud D in the RGB-D camera coordinate system:

[0035] D={d1,d2,d3,...,d k ,...}

[0036] Among them, d k is any point in the patient's facial point cloud D, such as Figure 4 shown.

[0037] Step 3: perform point cloud registration.

[0038] 3.1) Extract the feature points X of the head contour point cloud M and the facial point cloud D respectively M and X D ;

[0039] 3.2) Use the positions of the corresponding feature points of the two point clouds to project the head contour point cloud M to the RGB-D camera coordinate system to obtain the projected head contour point cloud M′, and calculate the distance S between the corresponding feature points of the two point clouds after projection C :

[0040]

[0041] in, is any pair of feature points of the two point clouds after projection;

[0042] 3.3) Set the threshold H for coarse registration C , and its distance to the corresponding feature point For comparison:

[0043] If satisfied If the number of feature point pairs is greater than the preset number N, the projection result is used as the result of rough registration, and the rotation matrix R of the two point clouds after rough registration is obtained. C With the translation vector t C , execute step 3.4);

[0044] Otherwise, return to step 3.2);

[0045] 3.4) Using the point cloud registration algorithm, the head contour point cloud M and the facial point cloud D are precisely registered multiple times based on the rough registration;

[0046] 3.5) Calculate the distance between the corresponding points of the two point clouds after each fine registration

[0047]

[0048] in, is any pair of corresponding points of the two point clouds after precise registration;

[0049] 3.6) Set the threshold H for fine registration F , and the distance between the corresponding points of the two point clouds For comparison:

[0050] like Then point p is set as an inlier point, and the registration result with the largest number of inliers is used as the registration result of the two point clouds, and the rotation matrix R of the two point clouds after precise registration is obtained. F With the translation vector t F ;

[0051] Otherwise, return to step 3.4);

[0052] 3.7) Using the rotation matrix R obtained by coarse registration C With the translation vector t C And the rotation matrix R obtained by precise alignment F With the translation vector t F Calculate the rotation matrix R and translation vector t between two point clouds:

[0053] R=R F R c

[0054] t=R F t C +t F .

[0055] Step 4: Brain tissue localization.

[0056] 4.1) Using the rotation matrix R and translation vector t obtained by point cloud registration, the patient brain tissue data B in the MRI coordinate system is converted to the RGB-D camera coordinate system to obtain the position information of the patient brain tissue data E in the RGB-D camera coordinate system:

[0057] E = RB + t;

[0058] 4.2) Based on the calculated position information, the patient's brain tissue data is displayed in the RGB-D camera coordinate system to achieve the positioning of the brain tissue.

[0059] The effect of the present invention can be further illustrated by the following simulation experiment:

[0060] 1. Simulation experiment conditions:

[0061] Acquire the patient's brain tissue data B through magnetic resonance imaging (MRI);

[0062] The patient's head contour point cloud M is obtained from the brain tissue data B using an edge detection algorithm;

[0063] Use an RGB-D camera to capture RGB images and depth point cloud data of the patient’s face;

[0064] Using the facial key point detection algorithm, the key points of the patient's facial RGB image captured by the RGB-D camera are extracted;

[0065] The key points of the patient's facial RGB image are combined with the depth point cloud for face segmentation to generate the patient's facial point cloud D in the RGB-D camera coordinate system.

[0066] 2. Simulation experiment content

[0067] Simulation experiment 1: The present invention uses the point cloud registration algorithm to register the patient's head contour point cloud M in the MRI coordinate system with the facial point cloud D in the RGB-D camera coordinate system, and obtains the rotation matrix R and translation vector t between the two point clouds, as shown in Figure 5 As shown. Figure 5 It can be seen that the registered head contour point cloud M (blue) can be completely fitted on the facial point cloud D (yellow), indicating that the present invention has a good registration effect.

[0068] Simulation experiment 2: The present invention uses the rotation matrix R and translation vector t obtained by point cloud registration in simulation experiment 1 to transform the patient brain tissue data B in the MRI coordinate system into the RGB-D camera coordinate system, and obtains the patient brain tissue data E in the RGB-D camera coordinate system to realize the positioning of the brain tissue. The results are shown in FIG. Figure 6 As shown. Figure 6 It can be seen that in the RGB-D camera coordinate system, the patient's brain tissue data E represented by gray matches the facial point cloud D represented by yellow, indicating that the present invention has a more accurate positioning effect.

[0069] The above description is only a specific example of the present invention and does not constitute any limitation to the present invention. It is obvious that for professionals in this field, after understanding the content and principles of the present invention, it is possible to make various modifications and changes in form and details without departing from the principles and structures of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

[0070] It should be noted that the step numbers in the specification and claims of the present invention are only for a clear description of the implementation scheme of the present invention to facilitate understanding, and the order of the step numbers is not limited.

Claims

1. A brain tissue localization method based on facial point cloud registration, characterized in that: The following steps are involved: (1) Obtain the patient's brain tissue data B = {b1, b2, b3, ..., b i }, where b i is any voxel grid in the brain tissue data B, and the patient's head contour point cloud M is obtained from the brain tissue data B using an edge detection algorithm to obtain the brain tissue data B and the head contour point cloud M in the MRI coordinate system; (2) Use an RGB-D camera to capture the RGB image and depth point cloud of the patient's face. Use the facial key point detection algorithm to extract the key points of the RGB image of the patient's face captured by the RGB-D camera, and combine it with the depth point cloud for face segmentation to generate the patient's facial point cloud D in the RGB-D camera coordinate system; (3) Use the point cloud registration algorithm to register the patient's head contour point cloud M in the MRI coordinate system with the facial point cloud D in the RGB-D camera coordinate system to obtain the rotation matrix R and translation vector t between the two point clouds; (4) Using the rotation matrix R and translation vector t obtained by point cloud registration, the patient’s brain tissue data B in the MRI coordinate system is converted to the RGB-D camera coordinate system to obtain the patient’s brain tissue data E in the RGB-D camera coordinate system, thereby realizing the positioning of the brain tissue.

2. The method according to claim 1, characterized in that Step (1) uses an edge detection algorithm to obtain the patient's head contour point cloud M from the brain tissue data B. The implementation steps include the following: 1a) setting an extraction threshold according to the grayscale value of the head contour in the brain tissue data B, extracting the brain tissue data within the threshold range, and obtaining the head contour; 1b) Compare the extraction results under different thresholds, convert the most complete head contour in the extraction results into a head contour point cloud M, and obtain the head contour point cloud M in the MRI coordinate system: M={m1,m2,m3,...,m j } Among them, m j is any point in the head contour point cloud M.

3. The method according to claim 1, characterized in that: Step (2) uses a facial key point detection algorithm to extract key points of the patient's facial RGB image captured by the RGB-D camera. The implementation steps include the following: 2a) Setting up a corresponding deep learning network based on the facial key point detection algorithm; 2b) Input the patient's facial RGB image captured by the RGB-D camera into the facial key point detection network to obtain N facial key points F, N ≥ 68.

4. The method according to claim 1, characterized in that: Step (2) combines the key points of the patient's facial RGB image with the depth point cloud data to perform face segmentation, and the implementation steps include the following: 2c) aligning the RGB image of the patient's face captured by the RGB-D camera with the depth point cloud captured by it; 2d) According to the positions of N key points F in the patient's facial RGB image, the face area in the depth point cloud is extracted to achieve face segmentation and generate the patient's facial point cloud D in the RGB-D camera coordinate system. <h2 style=";text-align:left;direction:ltr">D = {d1,d2,d3,...,d<h2 style=";text-align:left;direction:ltr"> k <h2 style=";text-align:left;direction:ltr"> ,...} Among them, d k is any point in the patient's facial point cloud D.

5. The method according to claim 1, characterized in that Step (3) uses a point cloud registration algorithm to register the patient's head contour point cloud M in the MRI coordinate system with the facial point cloud D in the RGB-D camera coordinate system. The implementation steps include the following: 3a) Extract feature points X of head contour point cloud M and facial point cloud D respectively M and X D ; 3b) Use the positions of the corresponding feature points of the two point clouds to project the head contour point cloud M to the RGB-D camera coordinate system to obtain the projected head contour point cloud M′, and calculate the distance S between the corresponding feature points of the two point clouds after projection C : in, and is any pair of feature points of the two point clouds after projection; 3c) Set the threshold H for rough registration C , and its distance to the corresponding feature point For comparison: If satisfied If the number of feature point pairs is greater than the preset number N, the projection result is used as the result of rough registration, and the rotation matrix R of the two point clouds after rough registration is obtained. C With the translation vector t C , execute step 3d); Otherwise, return to step 3b); 3d) Using the point cloud registration algorithm, the head contour point cloud M and the facial point cloud D are precisely registered multiple times based on the rough registration; 3e) Calculate the distance between the corresponding points of the two point clouds after each fine registration in, and is any pair of corresponding points of the two point clouds after precise registration; 3f) Set the threshold H for fine registration F , and the distance between the corresponding points of the two point clouds For comparison: like Then point p is set as an inlier point, and the registration result with the largest number of inliers is used as the registration result of the two point clouds, and the rotation matrix R of the two point clouds after precise registration is obtained. F With the translation vector t F ; Otherwise, return to step 3d); 3f) Using the rotation matrix R obtained by coarse registration C With the translation vector t C And the rotation matrix R obtained by precise alignment F With the translation vector t F Calculate the result of the registration of the two point clouds and obtain the rotation matrix R and translation vector t between the two point clouds: R=R F R c t=R F t C +t F 。 6. The method according to claim 1, characterized in that In step (4), the patient brain tissue data B in the MRI coordinate system is converted to the RGB-D camera coordinate system. The implementation steps include the following: 4a) Using the rotation matrix R and translation vector t obtained by point cloud registration, the patient brain tissue data B in the MRI coordinate system is converted to the RGB-D camera coordinate system to obtain the position information of the patient brain tissue data E in the RGB-D camera coordinate system: E=RB+t 4b) Based on the calculated position information, the patient's brain tissue data is displayed in the RGB-D camera coordinate system.

Images