A method and system for presenting the head of a patient undergoing brain surgery based on pose estimation

By using deep learning algorithms and posture estimation algorithms, the posture of the head of the patient with brain surgery is adjusted in real time, and the problem of mismatch between the surgical path and the actual posture in the existing technology is solved, and the accuracy of the surgical path and the actual posture is achieved. Matching reduces the risk of surgery.

CN118799505BActive Publication Date: 2024-11-22JILIN UNIV FIRST HOSPITAL
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411285162.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-13
Publication Date
2024-11-22
Estimated Expiration
2044-09-13

AI Technical Summary

Technical Problem

The existing technology is difficult to obtain and adjust the posture of the patient's head in real time, resulting in the mismatch of the surgical path and the actual posture, which increases the risk of surgery. The lack of pre-trained posture estimation network model provides accurate posture data, resulting in the surgical process. The control and adjustments in the process are not accurate enough.

Method used

By using high-resolution MRI imaging equipment to obtain multi-plane two-dimensional image data of the patient's head, reconstructing a three-dimensional image model, combining deep learning algorithms to automatically label the skull, brain tissue, cerebral artery, brain veins and lesion structures, to obtain the best The surgical path of the patient's head is adjusted in real time using the posture estimation algorithm, and the pre-trained posture estimation network model is input to obtain real-time head posture data, and the consistency index of surgical path and head status is generated. Assess the risk of surgery.

Benefits of technology

Real-time matching of the surgical path and actual position is achieved, precise control of the surgical process, reducing the risk of surgery and improving the success rate of surgery.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118799505B_ABST
    Figure CN118799505B_ABST
Patent Text Reader

Abstract

The present invention provides a method and system for presenting the head of a patient undergoing brain surgery based on posture estimation, which relates to the technical field of medical image processing and surgical application. The specific steps include: reconstructing a three-dimensional image model including a skull, brain tissue, brain arteries, brain veins and lesion structures; using a deep learning algorithm to automatically annotate the three-dimensional image model, and using a path planning function based on the annotated structure to obtain the optimal surgical path and path matching coefficient; matching the three-dimensional image corresponding to the optimal surgical path with the patient's actual head posture image; obtaining and processing real-time head posture data, generating a head state coefficient, and combining it with the path matching coefficient to obtain a consistency index between the surgical path and the head state; and evaluating the surgical risk level according to the interval range of the consistency index. The present invention can obtain and adjust the posture of the patient's head in real time, so that the surgical path can be consistent with the actual posture, accurately control the surgical process, and reduce surgical risks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing and surgical applications, and in particular to a method and system for presenting a head of a patient undergoing brain surgery based on posture estimation. Background Art

[0002] Brain surgery, also known as neurosurgery, is a complex and delicate surgical procedure that is primarily used to treat diseases involving the brain, spinal cord, and nervous system. Brain surgery often involves high-risk and high-precision operations, which requires the surgical team to have a high degree of professional skills and experience, while also requiring the use of advanced medical equipment and technology to ensure the success rate of the operation and the safety of the patient.

[0003] In the prior art, a method and system for presenting a patient's head during brain surgery based on posture estimation is disclosed with publication number CN117557724 A. The method comprises: collecting two-dimensional head image data of the patient before surgery and reconstructing a brain model of the patient; pre-cropping the brain vein sub-model through the skull sub-model using Boolean operations, and completing the cropping of the brain vein model after determining the bone window position and size of the brain model after selecting the target surgical path; creating a brain vein posture estimation network model and a posture estimation training data set, and iteratively training the brain vein posture estimation network model; obtaining an RGB image of the patient's craniotomy area during surgery and inputting it into the brain vein posture estimation network model to obtain the patient's brain vein posture, fusing the brain vein posture with the patient's head posture for virtual reality presentation, and controlling the real-time update of the virtual reality presentation through preset gestures.

[0004] However, there are still the following shortcomings: As can be seen from the above statements, it is difficult for the existing technology to obtain and adjust the posture of the patient's head in real time, resulting in a mismatch between the surgical path and the actual posture, increasing the surgical risk, and there is no pre-trained posture estimation network model to provide accurate posture data, resulting in inaccurate control and adjustment during the operation.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0006] The purpose of the present invention is to provide a method and system for presenting the head of a patient undergoing brain surgery based on posture estimation, so as to solve the problems raised in the above-mentioned background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A method for presenting a head of a patient undergoing brain surgery based on pose estimation, the specific steps comprising:

[0009] S1. Use high-resolution MRI imaging equipment to obtain multi-plane 2D image data of the patient's head before surgery, and use reconstruction algorithms to reconstruct the multi-plane 2D image data into a 3D image model. The 3D image model includes the skull, brain tissue, brain arteries, brain veins and lesion structures;

[0010] S2. Use deep learning algorithms to automatically annotate the skull, brain tissue, brain arteries, brain veins and lesion structures in the three-dimensional image model, use path planning functions to obtain the optimal surgical path based on the annotated skull, brain tissue, brain arteries, brain veins and lesion structures, and obtain the path matching coefficient based on the optimal surgical path;

[0011] S3. Matching the three-dimensional image corresponding to the optimal surgical path with the actual head posture image of the patient; when the three-dimensional image corresponding to the optimal surgical path is inconsistent with the actual head posture image of the patient, using a posture estimation algorithm to adjust the posture of the patient's head in real time, and obtaining an adjusted RGB image of the patient's head;

[0012] S4. Inputting the adjusted RGB image of the patient's head into the pre-trained posture estimation network model to obtain real-time head posture data, processing the real-time head posture data to generate a state coefficient of the head, performing data processing on the state coefficient of the head and the path matching coefficient, and obtaining a consistency index between the surgical path and the head state;

[0013] S5. Assess the surgical risk level of patients undergoing brain surgery based on the interval range of the consistency index between the surgical path and the head status.

[0014] Furthermore, a deep learning algorithm is used to automatically annotate the skull, brain tissue, brain arteries, brain veins and lesion structures in the three-dimensional image model. Based on the annotated skull, brain tissue, brain arteries, brain veins and lesion structures, the process of using the path planning function to find the optimal surgical path is as follows:

[0015] Align the 3D image data to the standard reference coordinate system, manually annotate the skull, brain tissue, brain arteries, brain veins, and lesion structures, generate label data for training, format the data into a 3D voxel grid for deep learning model input, select 3D U-Net as the convolutional neural network, and use the path planning function to obtain the optimal surgical path. , based on the following formula:

[0016]

[0017] in, For path middle The cost coefficient of the point, For path The index of the previous point, They are the indicator functions of the skull, brain tissue, brain arteries, brain veins and lesions, are the weight coefficients of skull, brain tissue, brain artery, brain vein and lesion respectively;

[0018] If point In the skull, ,otherwise ;

[0019] If point In brain tissue, ,otherwise ;

[0020] If point In the cerebral arteries, ,otherwise ;

[0021] If point In the cerebral veins, ,otherwise ;

[0022] If point In the cerebral veins, ,otherwise ;

[0023] By calculating the cost of each point, the path planning function selects the path with the minimum total cost. Through the cost coefficient, the risk of the path passing through each point is effectively evaluated to select the optimal path and avoid high-risk areas.

[0024] Furthermore, according to the optimal surgical path, the process of obtaining the path matching coefficient is as follows:

[0025]

[0026] in, is the path matching coefficient, For path Length, For path A three-dimensional coordinate point on For the path The sum of all points on Point on the path The risk-weighted sum of For the The weight coefficient of the anatomical structure, For the Class anatomy at point The binary representation at Point belongs to Anatomical structure, ,otherwise , , When the anatomical structure is the skull, When the anatomical structure is brain tissue, When the anatomical structure is the brain artery, When the anatomical structure is the brain vein, When, the anatomical structure is the lesion.

[0027] Furthermore, the three-dimensional image corresponding to the optimal surgical path is matched with the actual head posture image of the patient. When the three-dimensional image corresponding to the optimal surgical path is inconsistent with the actual head posture image of the patient, the posture estimation algorithm is used to adjust the posture of the patient's head in real time. The process of obtaining the adjusted RGB image of the patient's head is as follows:

[0028] The three-dimensional image corresponding to the optimal surgical path is calibrated as the optimal three-dimensional image. When calculating the posture difference between the actual head posture image and the optimal three-dimensional image, the difference between the two in rotation and translation is considered;

[0029] The rotation difference is quantified by calculating the norm of the rotation matrix between the actual head pose and the optimal 3D image pose, according to the following formula:

[0030]

[0031] in, A quantized value representing the rotation difference, The rotation matrix representing the actual head pose, The rotation matrix representing the optimal 3D image pose, represents the Frobenius norm;

[0032] The translation difference is quantified by calculating the Euclidean distance between the actual head pose and the optimal 3D image pose, according to the following formula:

[0033]

[0034] in, represents the Euclidean distance between the actual head pose and the optimal 3D image pose, The translation vector representing the actual head pose, represents the translation vector of the optimal 3D image pose, is the Euclidean norm of the vector;

[0035] Quantized value based on rotation difference and Euclidean distance , adjust the posture of the patient's head, and obtain the adjusted RGB image of the patient's head.

[0036] Furthermore, the adjusted RGB image of the patient's head is input into the pre-trained posture estimation network model to obtain real-time head posture data, and the real-time head posture data is processed to generate the state coefficient of the head as follows:

[0037] The preprocessed image is input into the pre-trained pose estimation network model. The pose estimation network is based on a deep learning method to predict the spatial pose of the head, that is, the position and orientation. The head pose data output from the model includes rotation and displacement parameters, which are converted into Euler angles. The Euler angles include pitch angle, yaw angle and roll angle. The pitch angle, yaw angle and roll angle are processed to generate the state coefficient of the head. The formula is as follows:

[0038]

[0039] in, represents the state coefficient of the head, represents the pitch angle of the head, represents the yaw angle of the head, represents the roll angle of the head, represents the weight coefficient of the head's pitch angle, , represents the weight coefficient of the head's yaw angle, , represents the weight coefficient of the head roll angle, , .

[0040] Furthermore, the state coefficient of the head and the path matching coefficient are processed to obtain the consistency index of the surgical path and the head state, according to the following formula:

[0041]

[0042] in, Indicates the consistency index of surgical path and head status, represents the normalization coefficient, represents the weight coefficient of the path matching coefficient, , represents the weight coefficient of the head state coefficient, , .

[0043] Furthermore, according to the interval range of the consistency index of the surgical path and the head state, the process of evaluating the surgical risk level of the patient undergoing brain surgery is as follows:

[0044] Set the normalization coefficient The consistency index is 1. The value range of is [-1,1];

[0045] when exist When it is between, it means that the matching degree between the surgical path and the head status is low, the risk of the surgery is high, and the success rate of the surgery is low;

[0046] when exist When it is between, it means that the matching degree between the surgical path and the head status is moderate, but there is still a moderate risk;

[0047] when exist When it is between, it indicates that the surgical path and the head status are highly matched, the surgical risk is small, and the patient's surgical success rate is high.

[0048] To achieve the above object, the present invention also provides the following technical solutions:

[0049] A head presentation system for a patient undergoing brain surgery based on posture estimation, the system being used to execute any of the above-mentioned head presentation methods for a patient undergoing brain surgery based on posture estimation, comprising:

[0050] A data acquisition module is used to use a high-resolution MRI imaging device before surgery to obtain two-dimensional image data of multiple planes of the patient's head, and to use a reconstruction algorithm to reconstruct the two-dimensional image data of multiple planes into a three-dimensional image model. The three-dimensional image model includes the skull, brain tissue, brain arteries, brain veins, and lesion structures;

[0051] A data processing module is used to automatically annotate the skull, brain tissue, brain artery, brain vein and lesion structure in the three-dimensional image model using a deep learning algorithm, and to obtain the optimal surgical path using a path planning function based on the annotated skull, brain tissue, brain artery, brain vein and lesion structure, and to obtain a path matching coefficient based on the optimal surgical path;

[0052] An adjustment module is used to match the three-dimensional image corresponding to the optimal surgical path with the actual head posture image of the patient. When the three-dimensional image corresponding to the optimal surgical path is inconsistent with the actual head posture image of the patient, a posture estimation algorithm is used to adjust the posture of the patient's head in real time to obtain an adjusted RGB image of the patient's head.

[0053] An index acquisition module is used to input the adjusted RGB image of the patient's head into a pre-trained posture estimation network model to obtain real-time head posture data, process the real-time head posture data to generate a state coefficient of the head, perform data processing on the state coefficient of the head and the path matching coefficient, and obtain a consistency index between the surgical path and the head state;

[0054] The risk assessment module is used to assess the surgical risk level of patients undergoing brain surgery based on the interval range of the consistency index between the surgical path and the head status.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] The present invention uses a deep learning algorithm to automatically annotate the skull, brain tissue, brain arteries, brain veins and lesion structures in a three-dimensional image model, uses a path planning function to obtain the optimal surgical path based on the annotated five anatomical structures, obtains a path matching coefficient based on the optimal surgical path, matches the three-dimensional image corresponding to the optimal surgical path with the patient's actual head posture image, and when the three-dimensional image corresponding to the optimal surgical path is inconsistent with the patient's actual head posture image, uses a posture estimation algorithm to adjust the patient's head posture in real time, obtains the adjusted RGB image of the patient's head, inputs the adjusted RGB image of the patient's head into a pre-trained posture estimation network model, obtains real-time head posture data, and then can obtain and adjust the patient's head posture in real time, so that the surgical path can be consistent with the actual posture, accurately control the surgical process, and reduce surgical risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the overall method flow of the present invention;

[0058] Figure 2 This is a block diagram of the module composition of the present invention. DETAILED DESCRIPTION

[0059] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0060] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0061] Example:

[0062] See also Figure 1 , the present invention provides a technical solution:

[0063] A method for presenting the head of a patient undergoing brain surgery based on pose estimation, such as Figure 1 As shown, the specific steps include:

[0064] S1. Use high-resolution MRI imaging equipment to obtain multi-plane 2D image data of the patient's head before surgery, and use reconstruction algorithms to reconstruct the multi-plane 2D image data into a 3D image model. The 3D image model includes the skull, brain tissue, brain arteries, brain veins and lesion structures;

[0065] S2. Use deep learning algorithms to automatically annotate the skull, brain tissue, brain arteries, brain veins and lesion structures in the three-dimensional image model, use path planning functions to obtain the optimal surgical path based on the annotated skull, brain tissue, brain arteries, brain veins and lesion structures, and obtain the path matching coefficient based on the optimal surgical path;

[0066] S3. Matching the three-dimensional image corresponding to the optimal surgical path with the actual head posture image of the patient; when the three-dimensional image corresponding to the optimal surgical path is inconsistent with the actual head posture image of the patient, using a posture estimation algorithm to adjust the posture of the patient's head in real time, and obtaining an adjusted RGB image of the patient's head;

[0067] S4. Inputting the adjusted RGB image of the patient's head into the pre-trained posture estimation network model to obtain real-time head posture data, processing the real-time head posture data to generate a state coefficient of the head, performing data processing on the state coefficient of the head and the path matching coefficient, and obtaining a consistency index between the surgical path and the head state;

[0068] S5. Assess the surgical risk level of patients undergoing brain surgery based on the interval range of the consistency index between the surgical path and the head status.

[0069] On the basis of the above embodiment, a high-resolution MRI imaging device is used before the operation to obtain two-dimensional image data of multiple planes of the patient's head, and a reconstruction algorithm is used to reconstruct the two-dimensional image data of multiple planes into a three-dimensional image model. The three-dimensional image model includes the skull, brain tissue, brain arteries, brain veins and lesion structure. The process is as follows:

[0070] Use high-resolution MRI imaging equipment to obtain two-dimensional image data of the patient's head in multiple planes (including axial, sagittal, and coronal planes). The image data of each plane should cover the entire head area as much as possible.

[0071] The skull, brain tissue, brain arteries, brain veins and lesion structures have different functions. Among them, the skull provides the external shape of the head and serves as the boundary of the three-dimensional image model. Brain tissue includes gray matter and white matter, which are the functional areas of the brain. Brain arteries and veins are vascular structures that require special attention to avoid damage during surgery. Lesions are the target areas of surgery, such as tumors or lesions.

[0072] The collected two-dimensional image data is divided into training set, validation set and test set. A deep learning model suitable for three-dimensional image reconstruction is selected, such as a 3D convolutional neural network. The trained CNN model is used to input new two-dimensional image data for reconstruction. After preprocessing the new multi-plane two-dimensional image data, it is stacked into three-dimensional data blocks. The three-dimensional image models reconstructed from different planes are integrated to generate a complete three-dimensional body.

[0073] On the basis of the above embodiment, the process of automatically annotating the skull, brain tissue, brain artery, brain vein and lesion structure in the three-dimensional image model using a deep learning algorithm, and planning the optimal surgical path using a path planning function based on the annotated skull, brain tissue, brain artery, brain vein and lesion structure is as follows:

[0074] The three-dimensional image data is aligned to a standard reference coordinate system, and the skull, brain tissue, brain arteries, brain veins, and lesion structures are manually annotated to generate labeled data for training. The data is formatted into a format suitable for deep learning model input, such as a 3D voxel grid.

[0075] Choose a deep learning model suitable for 3D image data, such as 3D U-Net. 3D U-Net is a convolutional neural network used for medical image segmentation.

[0076] The convolutional neural network model architecture includes:

[0077] Input layer: receives 3D image data;

[0078] Encoder part: gradually reduce the spatial resolution and extract high-level features;

[0079] Decoder part: gradually restores the spatial resolution and generates an output of the same size as the input;

[0080] Output layer: Generates a 3D image containing the classification of each voxel (skull, brain tissue, brain arteries, brain veins, and lesion structures).

[0081] Use path planning functions to obtain the optimal surgical path , based on the following formula:

[0082]

[0083] in, For path middle The cost coefficient of the point, For path The index of the previous point, They are the indicator functions of the skull, brain tissue, brain arteries, brain veins and lesions, are the weight coefficients of skull, brain tissue, brain artery, brain vein and lesion respectively;

[0084] If point In the skull, ,otherwise ;

[0085] If point In brain tissue, ,otherwise ;

[0086] If point In the cerebral arteries, ,otherwise ;

[0087] If point In the cerebral veins, ,otherwise ;

[0088] If point In the cerebral veins, ,otherwise ;

[0089] By calculating the cost of each point, the path planning function selects the path with the minimum total cost. Through the cost coefficient, the risk of the path passing through each point is effectively evaluated to select the optimal path and avoid high-risk areas.

[0090] When the surgical environment or conditions change, the weights of each structure can be dynamically adjusted to update the cost function in real time, which enables path planning to adapt to new conditions and ensure the best choice of surgical path.

[0091] On the basis of the above embodiment, according to the optimal surgical path, the process of obtaining the path matching coefficient is as follows:

[0092]

[0093] in, is the path matching coefficient, For path Length, For path A three-dimensional coordinate point on For the path The sum of all points on Point on the path The risk-weighted sum of For the The weight coefficient of the anatomical structure, For the Class anatomy at point The binary representation at Point belongs to Anatomical structure, ,otherwise , , When the anatomical structure is the skull, When the anatomical structure is brain tissue, When the anatomical structure is the brain artery, When the anatomical structure is the brain vein, When, the anatomical structure is the lesion.

[0094] On the basis of the above embodiment, the three-dimensional image corresponding to the optimal surgical path is matched with the actual head posture image of the patient. When the three-dimensional image corresponding to the optimal surgical path is inconsistent with the actual head posture image of the patient, the posture estimation algorithm is used to adjust the posture of the patient's head in real time. The process of obtaining the adjusted RGB image of the patient's head is as follows:

[0095] The three-dimensional image corresponding to the optimal surgical path is calibrated as the optimal three-dimensional image. When calculating the posture difference between the actual head posture image and the optimal three-dimensional image, the difference between the two in spatial position (translation) and direction (rotation) is considered;

[0096] The rotation difference is quantified by calculating the norm of the rotation matrix between the actual head pose and the optimal 3D image pose, according to the following formula:

[0097]

[0098] in, A quantized value representing the rotation difference, The rotation matrix representing the actual head pose, The rotation matrix representing the optimal 3D image pose, represents the Frobenius norm;

[0099] The translation difference is quantified by calculating the Euclidean distance between the actual head pose and the optimal 3D image pose, according to the following formula:

[0100]

[0101] in, represents the Euclidean distance between the actual head pose and the optimal 3D image pose, The translation vector representing the actual head pose, represents the translation vector of the optimal 3D image pose, is the Euclidean norm of the vector;

[0102] Quantized value based on rotation difference and Euclidean distance , adjust the posture of the patient's head, and obtain the adjusted RGB image of the patient's head. Input the preprocessed image into the pre-trained posture estimation network model. The posture estimation network is usually based on a deep learning method, which can predict the spatial posture (i.e., position and orientation) of the head. The head posture data output from the model includes rotation and displacement parameters, which are converted into Euler angles. The conversion of rotation and displacement parameters into Euler angles is an existing technology and will not be described in detail here.

[0103] Euler angles include pitch angle, yaw angle and roll angle. The pitch angle, yaw angle and roll angle are processed to generate the state coefficient of the head. The formula is as follows:

[0104]

[0105] in, represents the state coefficient of the head, represents the pitch angle of the head, represents the yaw angle of the head, represents the roll angle of the head, represents the weight coefficient of the head's pitch angle, , represents the weight coefficient of the head's yaw angle, , represents the weight coefficient of the head roll angle, , .

[0106] On the basis of the above embodiment, the state coefficient of the head and the path matching coefficient are processed to obtain the consistency index of the surgical path and the head state, according to the following formula:

[0107]

[0108] in, Indicates the consistency index of surgical path and head status, represents the normalization coefficient, represents the weight coefficient of the path matching coefficient, , represents the weight coefficient of the head state coefficient, , .

[0109] On the basis of the above embodiment, the process of evaluating the surgical risk level of a patient undergoing brain surgery according to the interval range of the consistency index of the surgical path and the head state is as follows:

[0110] Set the normalization coefficient The consistency index is 1. The value range of is [-1,1];

[0111] when exist When it is between, it means that the matching degree between the surgical path and the head status is low, the risk of the surgery is high, and the success rate of the surgery is low;

[0112] when exist When it is between, it means that the matching degree between the surgical path and the head status is moderate, but there is still a moderate risk;

[0113] when exist When it is between, it indicates that the surgical path and the head status are highly matched, the surgical risk is small, and the patient's surgical success rate is high.

[0114] In the formula , , , , , , and The specific value of is generally determined by those skilled in the art according to actual conditions. The essence of this formula is a comprehensive analysis by weighted summation. The technicians in this field collect multiple groups of sample data, and set corresponding weight coefficients or adjustment coefficients for each group of sample data. The set weight coefficients or adjustment coefficients and the collected sample data are substituted into the formula. Through repeated experiments and parameter adjustments, the accuracy of the model output and the rationality of the results are observed, and these factor coefficients are gradually adjusted. The performance and effect of the model under different parameter settings are compared to find the optimal coefficient combination. The calculated weight coefficients or adjustment coefficients are screened and averaged to obtain , , , , , , and The value of .

[0115] See also Figure 2 , the present invention also provides a technical solution:

[0116] A head presentation system for a patient undergoing brain surgery based on posture estimation, the system being used to execute any of the above-mentioned head presentation methods for a patient undergoing brain surgery based on posture estimation, comprising:

[0117] A data acquisition module is used to use a high-resolution MRI imaging device before surgery to obtain two-dimensional image data of multiple planes of the patient's head, and to use a reconstruction algorithm to reconstruct the two-dimensional image data of multiple planes into a three-dimensional image model. The three-dimensional image model includes the skull, brain tissue, brain arteries, brain veins, and lesion structures;

[0118] A data processing module is used to automatically annotate the skull, brain tissue, brain artery, brain vein and lesion structure in the three-dimensional image model using a deep learning algorithm, and to obtain the optimal surgical path using a path planning function based on the annotated skull, brain tissue, brain artery, brain vein and lesion structure, and to obtain a path matching coefficient based on the optimal surgical path;

[0119] An adjustment module is used to match the three-dimensional image corresponding to the optimal surgical path with the actual head posture image of the patient. When the three-dimensional image corresponding to the optimal surgical path is inconsistent with the actual head posture image of the patient, a posture estimation algorithm is used to adjust the posture of the patient's head in real time to obtain an adjusted RGB image of the patient's head.

[0120] An index acquisition module is used to input the adjusted RGB image of the patient's head into a pre-trained posture estimation network model to obtain real-time head posture data, process the real-time head posture data to generate a state coefficient of the head, perform data processing on the state coefficient of the head and the path matching coefficient, and obtain a consistency index between the surgical path and the head state;

[0121] The risk assessment module is used to assess the surgical risk level of patients undergoing brain surgery based on the interval range of the consistency index between the surgical path and the head status.

[0122] In addition, the size of the weight coefficient or adjustment coefficient is a specific value obtained by quantifying each parameter. In order to facilitate subsequent comparison, the size of the coefficient depends on the amount of sample data and the technical personnel in this field preliminarily set the corresponding weight coefficient or adjustment coefficient for each set of sample data. It is not unique as long as it does not affect the proportional relationship between the parameter and the quantized value.

[0123] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0124] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. Those skilled in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software methods depends on the specific application and design constraints of the technical solution.

[0125] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0126] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for presenting a patient's head during brain surgery based on pose estimation, characterized in that: The specific steps include: S1. Use high-resolution MRI imaging equipment to obtain multi-plane 2D image data of the patient's head before surgery, and use reconstruction algorithms to reconstruct the multi-plane 2D image data into a 3D image model. The 3D image model includes the skull, brain tissue, brain arteries, brain veins and lesion structures; S2. Use deep learning algorithms to automatically annotate the skull, brain tissue, brain arteries, brain veins and lesion structures in the three-dimensional image model, use path planning functions to obtain the optimal surgical path based on the annotated skull, brain tissue, brain arteries, brain veins and lesion structures, and obtain the path matching coefficient based on the optimal surgical path; S3. Matching the three-dimensional image corresponding to the optimal surgical path with the actual head posture image of the patient; when the three-dimensional image corresponding to the optimal surgical path is inconsistent with the actual head posture image of the patient, using a posture estimation algorithm to adjust the posture of the patient's head in real time, and obtaining an adjusted RGB image of the patient's head; S4. Inputting the adjusted RGB image of the patient's head into the pre-trained posture estimation network model to obtain real-time head posture data, processing the real-time head posture data to generate a state coefficient of the head, performing data processing on the state coefficient of the head and the path matching coefficient, and obtaining a consistency index between the surgical path and the head state; S5. Evaluate the surgical risk level of patients undergoing brain surgery based on the interval range of the consistency index between the surgical path and the head status; The deep learning algorithm is used to automatically annotate the skull, brain tissue, brain arteries, brain veins and lesion structures in the three-dimensional image model. Based on the annotated skull, brain tissue, brain arteries, brain veins and lesion structures, the process of using the path planning function to find the optimal surgical path is as follows: The three-dimensional image data is aligned to the standard reference coordinate system. The skull, brain tissue, brain arteries, brain veins and lesion structures are manually annotated to generate label data for training. The data is formatted into a 3D voxel grid for deep learning model input. 3D U-Net is selected as the convolutional neural network. The path planning function is used to obtain the optimal surgical path L. The formula is as follows: C(p)=w bone ·I bone (p)+w tissue ·I tissue (p)+w artery ·I artery (p)+w vein ·I vein (p)+w tumor ·I tumor( p) Where C(p) is the cost coefficient of point p in path L, p is the index of the point on path L, and I bone (p), I tissue (p), I artery (p), I vein (p), I tumor (p) are the indicator functions of the skull, brain tissue, brain artery, brain vein and lesion, respectively, and w bone 、w tissue 、w artery 、w vein 、w tumor are the weight coefficients of skull, brain tissue, brain artery, brain vein and lesion respectively; If point p is in the skull, then I bone (p)=1, otherwise I bone (p) = 0; If point p is in the brain tissue, then I tissue (p)=1, otherwise I tissue (p) = 0; If point p is in a cerebral artery, then I artery (p)=1, otherwise I artery (p) = 0; If point p is in a cerebral vein, then I vein (p)=1, otherwise I vein (p) = 0; If point p is in a cerebral vein, then I tumor (p)=1, otherwise I tumor (p) = 0; By calculating the cost of each point, the path planning function selects the path with the minimum total cost. Through the cost coefficient, the risk of each point on the path is effectively evaluated to select the optimal path and avoid high-risk areas. According to the optimal surgical path, the process of obtaining the path matching coefficient is as follows: Among them, XS lj is the path matching coefficient, L C is the length of path L, (x, y, z) is a three-dimensional coordinate point on path L, ∑ (x,y,z)∈L is the sum of all points on the path L, ∑ k γ k ·B k (x, y, z) is the risk-weighted sum of the points (x, y, z) on the path, γ k is the weight coefficient of the kth type of anatomical structure, B k (x, y, z) is the binary representation of the k-th anatomical structure at the point (x, y, z). If the point (x, y, z) belongs to the k-th anatomical structure, then B k (x, y, z) = 1, otherwise B k (x, y, z) = 0, k = 1, 2, K, 5, when k = 1, the anatomical structure is the skull, when k = 2, the anatomical structure is the brain tissue, when k = 3, the anatomical structure is the cerebral artery, when k = 4, the anatomical structure is the cerebral vein, and when k = 5, the anatomical structure is the lesion.

2. The method for presenting a patient's head during brain surgery based on pose estimation according to claim 1, characterized in that: The three-dimensional image corresponding to the optimal surgical path is matched with the actual head posture image of the patient. When the three-dimensional image corresponding to the optimal surgical path is inconsistent with the actual head posture image of the patient, the posture estimation algorithm is used to adjust the posture of the patient's head in real time. The process of obtaining the adjusted RGB image of the patient's head is as follows: The three-dimensional image corresponding to the optimal surgical path is calibrated as the optimal three-dimensional image. When calculating the posture difference between the actual head posture image and the optimal three-dimensional image, the difference between the two in rotation and translation is considered; The rotation difference is quantified by calculating the norm of the rotation matrix between the actual head pose and the optimal 3D image pose, according to the following formula: ΔR=PR-AP F Where ΔR represents the quantized value of the rotation difference, R represents the rotation matrix of the actual head pose, A represents the rotation matrix of the optimal 3D image pose, and P·P F represents the Frobenius norm; The translation difference is quantified by calculating the Euclidean distance between the actual head pose and the optimal 3D image pose, according to the following formula: Δt=Pt1-t2P Where Δt represents the Euclidean distance between the actual head pose and the optimal 3D image pose, t1 represents the translation vector of the actual head pose, t2 represents the translation vector of the optimal 3D image pose, and P·P represents the Euclidean norm of the vector. According to the quantized value ΔR of the rotation difference and the Euclidean distance Δt, the posture of the patient's head is adjusted, and an RGB image of the adjusted patient's head is obtained.

3. The method for presenting a patient's head during brain surgery based on pose estimation according to claim 2, characterized in that: The process of inputting the adjusted RGB image of the patient's head into the pre-trained posture estimation network model to obtain real-time head posture data, processing the real-time head posture data, and generating the state coefficient of the head is as follows: The preprocessed image is input into the pre-trained pose estimation network model. The pose estimation network is based on a deep learning method to predict the spatial pose of the head, that is, the position and orientation. The head pose data output from the model includes rotation and displacement parameters, which are converted into Euler angles. The Euler angles include pitch angle, yaw angle and roll angle. The pitch angle, yaw angle and roll angle are processed to generate the state coefficient of the head. The formula is as follows: XS zt =ω1·fj+ω2·pj+ω3·gj Among them, XS zt represents the state coefficient of the head, fj represents the pitch angle of the head, pj represents the yaw angle of the head, gj represents the roll angle of the head, ω1 represents the weight coefficient of the pitch angle of the head, 0.2≤ω1≤0.4, ω2 represents the weight coefficient of the yaw angle of the head, 0.2≤ω2≤0.4, ω3 represents the weight coefficient of the roll angle of the head, 0.2≤ω3≤0.4, ω1+ω2+ω3=1.

4. The method for presenting a patient's head during brain surgery based on posture estimation according to claim 3, characterized in that: The head state coefficient and path matching coefficient are processed to obtain the consistency index of the surgical path and the head state, based on the following formula: Wherein, ZS represents the consistency index between the surgical path and the head state, λ represents the normalization coefficient, ε represents the weight coefficient of the path matching coefficient, 0≤ε≤1, β represents the weight coefficient of the head state coefficient, 0≤β≤1, ε+β=1.

5. The method for presenting a patient's head during brain surgery based on pose estimation according to claim 4, characterized in that: Based on the interval range of the consistency index of the surgical path and head status, the process of assessing the surgical risk level of patients undergoing brain surgery is as follows: The normalization coefficient λ is set to 1, and the range of the consistency index ZS is [-1,1]; When ZS is between [-1, -0.5], it means that the matching degree between the surgical path and the head state is low, the risk of surgery is high, and the success rate of the patient's surgery is low; When ZS is between [-0.5, 0.5], it means that the matching degree between the surgical path and the head status is moderate, but there is still a moderate risk; When ZS is between [0.5, 1], it indicates that the surgical path and the head state are highly matched, the surgical risk is low, and the patient's surgical success rate is high.

6. A system for presenting a head of a patient undergoing a brain surgery based on posture estimation, the system being used to perform the method for presenting a head of a patient undergoing a brain surgery based on posture estimation according to any one of claims 1 to 5, comprising: A data acquisition module is used to use a high-resolution MRI imaging device before surgery to obtain two-dimensional image data of multiple planes of the patient's head, and to use a reconstruction algorithm to reconstruct the two-dimensional image data of multiple planes into a three-dimensional image model. The three-dimensional image model includes the skull, brain tissue, brain arteries, brain veins, and lesion structures; A data processing module is used to automatically annotate the skull, brain tissue, brain artery, brain vein and lesion structure in the three-dimensional image model using a deep learning algorithm, and to obtain the optimal surgical path using a path planning function based on the annotated skull, brain tissue, brain artery, brain vein and lesion structure, and to obtain a path matching coefficient based on the optimal surgical path; An adjustment module is used to match the three-dimensional image corresponding to the optimal surgical path with the actual head posture image of the patient. When the three-dimensional image corresponding to the optimal surgical path is inconsistent with the actual head posture image of the patient, a posture estimation algorithm is used to adjust the posture of the patient's head in real time to obtain an adjusted RGB image of the patient's head. An index acquisition module is used to input the adjusted RGB image of the patient's head into a pre-trained posture estimation network model to obtain real-time head posture data, process the real-time head posture data to generate a state coefficient of the head, perform data processing on the state coefficient of the head and the path matching coefficient, and obtain a consistency index between the surgical path and the head state; A risk assessment module is used to assess the surgical risk level of patients undergoing brain surgery based on the interval range of the consistency index of the surgical path and the head status; The deep learning algorithm is used to automatically annotate the skull, brain tissue, brain arteries, brain veins and lesion structures in the three-dimensional image model. Based on the annotated skull, brain tissue, brain arteries, brain veins and lesion structures, the process of using the path planning function to find the optimal surgical path is as follows: The three-dimensional image data is aligned to the standard reference coordinate system. The skull, brain tissue, brain arteries, brain veins and lesion structures are manually annotated to generate label data for training. The data is formatted into a 3D voxel grid for deep learning model input. 3D U-Net is selected as the convolutional neural network. The path planning function is used to obtain the optimal surgical path L. The formula is as follows: C(p)=w bone ·I bone (p)+w tissue ·I tissue (p)+w artery ·I artery (p)+w vein ·I vein (p)+w tumor ·I tumor (p) Where C(p) is the cost coefficient of point p in path L, p is the index of the point on path L, and I bone (p), I tissue (p), I artery (p), I vein (p), I tumor (p) are the indicator functions of the skull, brain tissue, brain artery, brain vein and lesion, respectively, and w bone 、w tissue 、w artery 、w vein 、w tumor are the weight coefficients of skull, brain tissue, brain artery, brain vein and lesion respectively; If point p is in the skull, then I bone (p)=1, otherwise I bone (p) = 0; If point p is in the brain tissue, then I tissue (p)=1, otherwise I tissue (p) = 0; If point p is in a cerebral artery, then I artery (p)=1, otherwise I artery (p) = 0; If point p is in a cerebral vein, then I vein (p)=1, otherwise I vein (p) = 0; If point p is in a cerebral vein, then I tumor (p)=1, otherwise I tumor (p) = 0; By calculating the cost of each point, the path planning function selects the path with the minimum total cost. Through the cost coefficient, the risk of each point on the path is effectively evaluated to select the optimal path and avoid high-risk areas. According to the optimal surgical path, the process of obtaining the path matching coefficient is as follows: Among them, XS lj is the path matching coefficient, L C is the length of path L, (x, y, z) is a three-dimensional coordinate point on path L, ∑ (x,y,z)∈L is the sum of all points on the path L, ∑ k γ k ·B k (x, y, z) is the risk-weighted sum of the points (x, y, z) on the path, γ k is the weight coefficient of the kth type of anatomical structure, B k (x, y, z) is the binary representation of the k-th anatomical structure at the point (x, y, z). If the point (x, y, z) belongs to the k-th anatomical structure, then B k (x, y, z) = 1, otherwise B k (x, y, z) = 0, k = 1, 2, K, 5, when k = 1, the anatomical structure is the skull, when k = 2, the anatomical structure is the brain tissue, when k = 3, the anatomical structure is the cerebral artery, when k = 4, the anatomical structure is the cerebral vein, and when k = 5, the anatomical structure is the lesion.

Citation Information

Patent Citations

  • Method and system for presenting head of brain surgery patient based on pose estimation

    CN117557724A

  • Autonomous navigation method for tracheal intubation robot based on self-supervised monocular depth estimation

    CN116966381A

  • Real-time surgical instrument pose estimation method and system based on deep learning and stereoscopic vision

    CN117011381A

  • Unmarked surgical navigation method and system based on 3D structured light

    CN117618110A

  • Image guiding device for brain surgery

    US20160022171A1