Surgical navigation image real-time registration method and system based on multi-mode body surface mark tracking

Abstract

The invention discloses a surgical navigation image real-time registration method and system based on multi-modal body surface mark tracking. The registration method comprises the following steps: S1, constructing an A-type three-dimensional model of the head of a patient before an operation; s2, constructing a B-type three-dimensional model of the head of the patient in real time during an operation; s3, in the operation, at the current time point, registration of a depth mode, registration of a texture mode and registration of a multi-view tracking mode are carried out at the same time; and S4, repeating the step S2 to the step S3 at the next time point, and simultaneously realizing the registration of the depth mode, the registration of the texture mode and the registration of the multi-view tracking mode, thereby realizing the real-time registration of the surgical navigation image. According to the registration method, real-time registration of the surgical navigation image and the body position of the patient can be achieved, the registration operation is simple and convenient, the registration time is shorter than 30 seconds, and compared with an existing registration method, the time needed for registration is greatly shortened.

Description

Technical Field

[0001] The present invention relates to the field of medical devices, and in particular to a real-time registration method and system for surgical navigation images based on multimodal tracking of body surface landmarks. Background Art

[0002] With the development of precision medicine, surgical navigation imaging systems play a vital role. They are used to provide doctors with real-time and precise information about the patient's internal structures, helping doctors make more accurate judgments and operations during surgery.

[0003] However, existing surgical navigation systems have several issues with the registration process. For example, the invasive nature of the system due to its reliance on external markers, along with the complexity and time-consuming nature of the registration process, require re-registration if the patient moves during surgery. This process is time-consuming, and the user (e.g., surgeon) must wait for re-registration to complete before continuing the procedure. These issues, of course, can reduce surgical efficiency and accuracy.

[0004] Therefore, how to develop a method and system that can align surgical navigation images with patient positions in real time, quickly and accurately, thereby improving the efficiency and accuracy of surgical navigation, has become an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a real-time registration method and system for surgical navigation images based on multimodal tracking of body surface landmarks, so as to overcome the above-mentioned defects of the prior art.

[0006] In order to solve the above technical problems, the invention adopts the following technical solutions:

[0007] A real-time registration method for surgical navigation images based on multimodal body surface landmark tracking is provided, which includes the following steps:

[0008] S1: constructing a type A three-dimensional model of the patient's head before surgery;

[0009] S2: Real-time construction of a B-type 3D model of the patient’s head during surgery;

[0010] S3: During the operation, at the current moment, the depth modality registration, texture modality registration, and multi-view tracking modality registration are performed simultaneously;

[0011] S4: at the next time point, repeating steps S2 to S3, and simultaneously achieving registration of the depth modality, registration of the texture modality, and registration of the multi-view tracking modality, thereby achieving real-time registration of the surgical navigation image;

[0012] Step S1 specifically comprises: performing a medical imaging scan on the patient's head to obtain an A-type two-dimensional image of the patient's head; constructing an A-type three-dimensional model of the patient's head in a three-dimensional space using the A-type two-dimensional image of the patient's head; selecting a plurality of A-type body surface landmark regions from the A-type three-dimensional model of the patient's head, and obtaining depth information, texture information, and contour information of each A-type body surface landmark region; the A-type body surface landmark regions include but are not limited to the auricle region, the external nose region, and the forehead region;

[0013] Step S2 specifically includes: using an RGB-D camera to take a real-time photo of the patient's head and obtain real-time depth information, texture information, and contour information of the patient's head; the RGB-D camera uses the obtained depth information of the patient's head to construct a B-type three-dimensional model of the patient's head in real-time in three-dimensional space; the B-type three-dimensional model includes the depth information, texture information, and contour information of the patient's head.

[0014] Preferably, in step S3, the registration of the depth modality is specifically as follows: at the current time point, for any A-type body surface marker area, a plurality of A-type position feature points are set in the A-type body surface marker area using the depth information of the A-type body surface marker area and an A-type position feature point arrangement pattern of the A-type body surface marker area is obtained, the A-type position feature point arrangement pattern including the three-dimensional spatial coordinates of all A-type position feature points in the A-type body surface marker area and their mutual positional relationships, and then an ICP algorithm is used to search from the current B-type three-dimensional model for a group of B-type position feature points that have the best overall matching degree with the A-type position feature point arrangement pattern of the A-type body surface marker area; if the B-type position feature point arrangement pattern composed of a group of B-type position feature points of the current B-type three-dimensional model matches the A-type position feature point arrangement pattern of the A-type body surface marker area, If the overall matching deviation of the type-A position feature point arrangement pattern in the type-A body surface landmark area is the smallest, then the group of type-B position feature points of the current type-B three-dimensional model is the group of type-B position feature points that best matches the type-A position feature point arrangement pattern of the type-A body surface landmark area; if the ICP algorithm searches for a group of type-B position feature points that best matches the type-A position feature point arrangement pattern of the type-A body surface landmark area from the current type-B three-dimensional model, then it is considered that the depth modality registration of the type-A body surface landmark area is completed; otherwise, it is considered that the depth modality registration of the type-A body surface landmark area is incomplete and needs to be continued; for other type-A body surface landmark areas, repeat the above steps, and at the current moment, the depth modality registration of all type-A body surface landmark areas is performed simultaneously.

[0015] Preferably, in step S3, the registration of the texture modality is specifically as follows: at the current time point, for any A-type body surface marker area, a plurality of A-type texture feature points are set in the A-type body surface marker area using the texture information of the A-type body surface marker area and an A-type texture feature point arrangement pattern of the A-type body surface marker area is obtained, the A-type texture feature point arrangement pattern includes the texture information of all A-type texture feature points in the A-type body surface marker area, and then an ORB algorithm, a FREAK algorithm or a SIFT algorithm is used to search from the current B-type three-dimensional model for a group of B-type texture feature points that have the best overall matching degree with the A-type texture feature point arrangement pattern of the A-type body surface marker area; if the B-type texture feature point arrangement pattern composed of a group of B-type texture feature points in the B-type body surface marker area matches the A-type body surface marker area, If the overall matching deviation of the A-type texture feature point arrangement pattern within the domain is the smallest, then the group of B-type texture feature points of the current B-type three-dimensional model is the group of B-type texture feature points with the best overall matching degree with the A-type texture feature point arrangement pattern of the A-type body surface landmark area; if the ORB algorithm, FREAK algorithm or SIFT algorithm searches for a group of B-type texture feature points with the best overall matching degree with the A-type texture feature point arrangement pattern of the A-type body surface landmark area from the current B-type three-dimensional model, then it is considered that the texture modality registration of the A-type body surface landmark area is completed; otherwise, it is considered that the texture modality registration of the A-type body surface landmark area is incomplete and needs to be continued; for other A-type body surface landmark areas, repeat the above steps, and at the current time point, the texture modality registration of all A-type body surface landmark areas is performed simultaneously.

[0016] Preferably, in step S3, the registration of the multi-view tracking modality is specifically as follows: at the current moment, for any A-type body surface marker area, the contour information of the A-type body surface marker area is used to obtain the A-type contour line pattern of the A-type body surface marker area at different perspectives; the registration of the depth modality and the registration of the texture modality are used to select and determine an A-type contour line pattern of a specific perspective from the A-type contour line patterns of the A-type body surface marker area at different perspectives, and then the current B-type three-dimensional model is searched for the B-type contour line with the best overall matching degree with the A-type contour line pattern of the specific perspective of the A-type body surface marker area; specifically, the A-type contour line pattern of the A-type body surface marker area is divided into a plurality of line segments, and for any one of the line segments, the current B-type three-dimensional model is searched for the line segment that matches the line segment. The line segment with the smallest matching deviation is taken as the B-type contour line segment with the best matching degree with the line segment; for the remaining line segments of the A-type contour line pattern, the above steps are repeated, and the registration of all line segments of the A-type contour line pattern of the A-type body surface landmark area is performed simultaneously; when the B-type contour line segment with the best matching degree with all line segments of the A-type contour line pattern of each perspective of the A-type body surface landmark area is searched from the current B-type three-dimensional model, it is considered that the registration of the multi-view tracking mode of the A-type body surface landmark area is completed, otherwise, it is considered that the registration of the multi-view tracking mode of the A-type body surface landmark area is incomplete and needs to be continued; for other A-type body surface landmark areas, the above steps are repeated, and at the current moment, the registration of the multi-view tracking mode of all A-type body surface landmark areas is performed simultaneously.

[0017] Preferably, in step S3, the depth modality registration probability, the texture modality registration probability, and the multi-view tracking modality registration probability are calculated, and the comprehensive registration probability is calculated using the depth modality registration probability, the texture modality registration probability, and the multi-view tracking modality registration probability, and the registration with the highest comprehensive registration probability is taken as the best registration; in step S3, the comprehensive registration probability is calculated according to the following comprehensive registration probability formula:

[0018]

[0019] Among them, p(θ|D) represents the comprehensive registration probability, represents the depth modality registration probability, represents the texture modality registration probability, represents the registration probability of multi-view tracking modality, represents the depth modality data, Represents texture modal data, Represents multi-view tracking modal data; θ represents the posture change vector, which is composed of the rotation vector θ r and the translation vector θ t composition.

[0020] Preferably, in step S3, the depth modality registration probability is calculated according to the following depth modality probability formula:

[0021]

[0022] Among them, p(θ|D d ) represents the depth modality registration probability, represents the depth value, σ d represents the user-defined standard deviation, X i Represents the point on the surface of the B-type 3D model, P i (θ) represents the closest point, N i Represents the normal vector of the model point.

[0023] Preferably, in step S3, the texture modality registration probability is calculated according to the following texture modality probability formula:

[0024]

[0025] Among them, p(θ|D t ) represents the depth modality registration probability, X i ′ represents the monitoring point in the current frame, X i (θ) represents the projection point, σ t Represents the user-defined standard deviation.

[0026] Preferably, in step S3, the multi-view tracking modality registration probability is calculated according to the following multi-view tracking modality probability formula:

[0027]

[0028] Among them, p(θ|D r ) represents the depth modality registration probability, θ represents the pose change vector, n represents the number of viewing angles, d i (θ) represents the contour distance of the i-th view, ω i Represents the set of pixels on the corresponding line, lll i Indicates the corresponding line.

[0029] Preferably, in step S3, the registration of the depth modality, the registration of the texture modality, and the registration of the multi-view tracking modality all use the Newton optimization method to estimate the posture change vector:

[0030]

[0031] Among them, θ represents the posture change vector, g represents the gradient vector, H represents the Hessian matrix, λ r represents the regularization parameter of the rotation, λ tRepresents the regularization parameter of the rotation, and I3 represents a 3*3 matrix.

[0032] Provided is a real-time registration system for surgical navigation images based on multimodal body surface landmark tracking. The registration system adopts and is capable of executing the real-time registration method for surgical navigation images based on multimodal body surface landmark tracking. The registration system includes an A-type 3D model construction module, a B-type 3D model construction module, a registration module, and a display module.

[0033] The A-type three-dimensional model construction module can execute the step S1; the A-type three-dimensional model construction module can construct an A-type three-dimensional model of the patient's head before surgery, specifically: performing a medical imaging scan on the patient's head to obtain an A-type two-dimensional image of the patient's head; using the A-type two-dimensional image of the patient's head to construct an A-type three-dimensional model of the patient's head in a three-dimensional space; selecting a plurality of A-type body surface landmark areas from the A-type three-dimensional model of the patient's head, and obtaining depth information, texture information, and contour information of each A-type body surface landmark area; the A-type body surface landmark areas include but are not limited to the auricle area, the external nose area, and the forehead area;

[0034] The B-type 3D model construction module is capable of executing step S2; the B-type 3D model construction module is capable of constructing a B-type 3D model of the patient's head in real time during surgery, specifically by: using an RGB-D camera to photograph the patient's head in real time, and obtaining depth information, texture information, and contour information of the patient's head in real time; the RGB-D camera uses the obtained depth information of the patient's head to construct a B-type 3D model of the patient's head in real time in a 3D space; the B-type 3D model includes the depth information, texture information, and contour information of the patient's head;

[0035] The registration module can execute step S3; the registration module can realize the following functions: during the operation, at the current time point, simultaneously perform the registration of the depth modality, the registration of the texture modality, and the registration of the multi-view tracking modality;

[0036] The registration module can realize the registration of depth modality, which is specifically as follows: at the current moment, for any A-type body surface marker area, a number of A-type position feature points are set in the A-type body surface marker area using the depth information of the A-type body surface marker area and the A-type position feature point arrangement pattern of the A-type body surface marker area is obtained, the A-type position feature point arrangement pattern includes the three-dimensional spatial coordinates of all A-type position feature points in the A-type body surface marker area and their mutual positional relationship, and then the ICP algorithm is used to search from the current B-type three-dimensional model for a group of B-type position feature points with the best overall matching degree with the A-type position feature point arrangement pattern of the A-type body surface marker area; if the B-type position feature point arrangement pattern composed of a group of B-type position feature points of the current B-type three-dimensional model is consistent with the A-type If the overall matching deviation of the A-type position feature point arrangement pattern in the body surface landmark region is the smallest, then the group of B-type position feature points of the current B-type three-dimensional model is the group of B-type position feature points that best matches the A-type position feature point arrangement pattern of the A-type body surface landmark region; if the ICP algorithm searches for a group of B-type position feature points that best matches the A-type position feature point arrangement pattern of the A-type body surface landmark region from the current B-type three-dimensional model, then it is considered that the depth modality registration of the A-type body surface landmark region is completed; otherwise, it is considered that the depth modality registration of the A-type body surface landmark region is incomplete and needs to be continued; the above steps are repeated for other A-type body surface landmark regions, and at the current time point, the depth modality registration of all A-type body surface landmark regions is performed simultaneously;

[0037] The registration module can also realize the registration of texture modalities, which is specifically as follows: at the current moment, for any A-type body surface marker area, a number of A-type texture feature points are set in the A-type body surface marker area using the texture information of the A-type body surface marker area and the A-type texture feature point arrangement pattern of the A-type body surface marker area is obtained, the A-type texture feature point arrangement pattern includes the texture information of all A-type texture feature points in the A-type body surface marker area, and then the ORB algorithm, FREAK algorithm or SIFT algorithm is used to search from the current B-type three-dimensional model for a group of B-type texture feature points that have the best overall matching degree with the A-type texture feature point arrangement pattern of the A-type body surface marker area; if the B-type texture feature point arrangement diagram composed of a group of B-type texture feature points in the B-type body surface marker area is consistent with the A-type body surface marker area If the overall matching deviation of the A-type texture feature point arrangement pattern within the current B-type three-dimensional model is the smallest, then the group of B-type texture feature points of the current B-type three-dimensional model is the group of B-type texture feature points with the best overall matching degree with the A-type texture feature point arrangement pattern of the A-type body surface landmark area; if the ORB algorithm, the FREAK algorithm or the SIFT algorithm searches for a group of B-type texture feature points with the best overall matching degree with the A-type texture feature point arrangement pattern of the A-type body surface landmark area from the current B-type three-dimensional model, then it is considered that the registration of the texture modality of the A-type body surface landmark area is completed; otherwise, it is considered that the registration of the texture modality of the A-type body surface landmark area is incomplete and needs to be continued; for other A-type body surface landmark areas, repeat the above steps, and at the current time point, the registration of the texture modalities of all A-type body surface landmark areas is performed simultaneously;

[0038] The registration module can also realize the registration of multi-view tracking modalities, which is specifically as follows: at the current moment, for any A-type body surface marker area, the contour information of the A-type body surface marker area is used to obtain the A-type contour line pattern of different perspectives of the A-type body surface marker area; the above-mentioned depth modality registration and the above-mentioned texture modality registration are used to select and determine an A-type contour line pattern of a specific perspective from the A-type contour line patterns of different perspectives of the A-type body surface marker area, and then search the current B-type three-dimensional model for the B-type contour line with the best overall matching degree with the A-type contour line pattern of the specific perspective of the A-type body surface marker area; specifically, the A-type contour line pattern of the A-type body surface marker area is divided into several line segments, and for any one of the line segments, the current B-type three-dimensional model is searched for the line segment that matches the line segment. The line segment with the smallest matching deviation is used as the B-type contour line segment with the best matching degree with the line segment; the above steps are repeated for the remaining line segments of the A-type contour line pattern, and the registration of all line segments of the A-type contour line pattern of the A-type body surface landmark area is performed simultaneously; when the B-type contour line segment with the best matching degree with all line segments of the A-type contour line pattern of each view of the A-type body surface landmark area is searched from the current B-type three-dimensional model, it is considered that the registration of the multi-view tracking modality of the A-type body surface landmark area is completed; otherwise, it is considered that the registration of the multi-view tracking modality of the A-type body surface landmark area is incomplete and needs to be continued; the above steps are repeated for other A-type body surface landmark areas, and at the current time point, the registration of the multi-view tracking modality of all A-type body surface landmark areas is performed simultaneously;

[0039] The registration module can also perform the above step S4; the registration module can also realize the following functions: at the next time point, repeat the above functions, and simultaneously realize the registration of the depth modality, the texture modality, and the multi-view tracking modality, thereby realizing real-time registration of the surgical navigation image;

[0040] The display module is used to display the real-time registration process and the registered image.

[0041] Any range described in the present invention includes the end value and any numerical value between the end values ​​and any sub-range formed by the end value or any numerical value between the end values.

[0042] Unless otherwise specified, all raw materials in the present invention can be purchased commercially, and the equipment used in the present invention can adopt conventional equipment in the relevant field or refer to the existing technology in the relevant field.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] (1) The present invention provides a real-time registration method for surgical navigation images based on multimodal body surface landmark tracking. Before surgery, a medical image scan (such as a CT scan) is performed on the patient's head, and a type A three-dimensional model of the patient's head is constructed. During surgery, an RGB-D camera is used to take real-time photos of the patient's head and a type B three-dimensional model of the patient's head is constructed in real time. During the surgery, depth modality registration, texture modality registration, and multi-view tracking modality registration are performed simultaneously, thereby achieving real-time registration of the surgical navigation image with the patient's body position. The registration operation is simple and convenient, and the registration time is less than 30 seconds, which greatly shortens the time required for registration compared with existing registration methods.

[0045] (2) The real-time registration method for surgical navigation images based on multimodal body surface landmark tracking provided by the present invention can realize real-time tracking of the patient's body position. When the patient's body position moves during the operation, there is no need to re-register the operation, which can avoid the user (such as the surgeon) from waiting, thereby improving the efficiency of the operation.

[0046] (3) The real-time registration method for surgical navigation images based on multimodal body surface landmark tracking provided by the present invention can achieve overall control of the registration accuracy within 1.5 mm, which helps to improve the accuracy of surgical navigation.

[0047] (4) The real-time registration method for surgical navigation images based on multimodal body surface landmark tracking provided by the present invention can be achieved by taking real-time photos of the patient's head using an RGB-D camera during the operation. The entire process is a non-invasive operation, so the patient's body position does not have any trauma. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0049] Figure 1 A flowchart of a real-time registration method for surgical navigation images based on multimodal body surface landmark tracking provided by an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of the structure of real-time registration of surgical navigation images based on multimodal body surface landmark tracking provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to explain the present invention more clearly, the present invention is further described below in conjunction with preferred embodiments. Those skilled in the art should understand that the following specific description is illustrative rather than restrictive and should not be used to limit the scope of protection of the present invention.

[0052] like Figure 1 As shown, this embodiment provides a real-time registration method for surgical navigation images based on multimodal body surface landmark tracking, the method comprising the following steps:

[0053] S1: constructing a type A three-dimensional model of the patient's head before surgery;

[0054] The above-mentioned step S1 specifically comprises: performing a medical imaging scan (e.g., a CT scan) on the patient's head to obtain an A-type two-dimensional image of the patient's head; constructing an A-type three-dimensional model of the patient's head in a three-dimensional space using the A-type two-dimensional image of the patient's head; selecting a plurality of A-type surface landmark regions from the A-type three-dimensional model of the patient's head, and obtaining depth information (i.e., three-dimensional spatial coordinates), texture information, and contour information of each A-type surface landmark region; the A-type surface landmark regions include but are not limited to the auricle region, the external nose region, and the forehead region;

[0055] S2: Real-time construction of a B-type 3D model of the patient’s head during surgery;

[0056] Step S2 is specifically as follows: using an RGB-D camera (Red-Green-Blue Depth Camera) to take a real-time photo of the patient's head and obtain the depth information (i.e., three-dimensional spatial coordinates), texture information, and contour information of the patient's head in real time; the RGB-D camera uses the obtained depth information of the patient's head to construct a B-type three-dimensional model of the patient's head in real time in three-dimensional space; the B-type three-dimensional model includes the depth information, texture information, and contour information of the patient's head;

[0057] S3: During the operation, at the current time point (e.g., T1), the depth modality registration, texture modality registration, and multi-view tracking modality registration are performed simultaneously.

[0058] In the above step S3, the registration of the depth modality is specifically as follows: at the current time point (for example, time point T1), for any A-type body surface landmark area (for example, A-type external nose area), using the depth information of the A-type body surface landmark area, a number of A-type position feature points (for example, nose tip point, nose bridge middle point, etc.) are set in the A-type body surface landmark area and the A-type position feature point arrangement pattern of the A-type body surface landmark area (for example, A-type external nose area) is obtained. The A-type position feature point arrangement pattern includes the three-dimensional spatial coordinates of all A-type position feature points in the A-type body surface landmark area (for example, A-type external nose area) and their mutual positional relationships, and then uses ICP (Iterative Closest The algorithm searches for a group of B-type position feature points with the best overall matching degree with the A-type position feature point arrangement pattern of the A-type body surface landmark area (for example, the A-type external nose area) from the current B-type three-dimensional model; if the B-type position feature point arrangement pattern composed of a group of B-type position feature points of the current B-type three-dimensional model has the smallest overall matching deviation with the A-type position feature point arrangement pattern in the A-type body surface landmark area (the ideal state is that the two are completely matched, but it is difficult to achieve complete matching between the two in actual applications, and the minimum matching deviation between the two can be achieved within the allowable error range), then the group of B-type position feature points of the current B-type three-dimensional model is the best overall matching degree with the A-type body surface landmark area (for example, the A-type external nose area). a set of B-type position feature points with the best matching degree of the A-type position feature point arrangement pattern of the A-type surface landmark region (e.g., the A-type external nose region) are searched for by the ICP algorithm from the current B-type 3D model; if the set of B-type position feature points with the best matching degree of the A-type position feature point arrangement pattern of the A-type surface landmark region (e.g., the A-type external nose region) are searched for from the current B-type 3D model, then the depth modality registration of the A-type surface landmark region is considered to be completed; otherwise, the depth modality registration of the A-type surface landmark region is considered to be incomplete and needs to be continued; for other A-type surface landmark regions, the above steps are repeated, and at the current time point (e.g., time point T1), the depth modality registration of all A-type surface landmark regions is performed simultaneously;

[0059] In the above step S3, the registration of the texture modality is specifically as follows: at the current time point (for example, time point T1), for any A-type body surface landmark area (for example, A-type external nose area), a number of A-type texture feature points are set in the A-type body surface landmark area using the texture information of the A-type body surface landmark area and an A-type texture feature point arrangement pattern of the A-type body surface landmark area (for example, A-type external nose area) is obtained. The A-type texture feature point arrangement pattern includes the texture information of all A-type texture feature points in the A-type body surface landmark area (for example, A-type external nose area), and then the ORB (Oriented FAST and RotatedBRIEF) algorithm, the FREAK (Fast Retina Keypoint) algorithm or the SIFT (Scale-Invariant FeatureTransform) algorithm (select any one of the three) searches for a group of B-type texture feature points with the best overall matching degree with the A-type texture feature point arrangement pattern of the A-type body surface landmark area (for example, the A-type external nose area) from the current B-type three-dimensional model; if the B-type texture feature point arrangement pattern composed of a group of B-type texture feature points in the B-type body surface landmark area and the A-type texture feature point arrangement pattern in the A-type body surface landmark area have the smallest overall matching deviation (the ideal state is that the two are completely matched, but it is difficult to achieve complete matching between the two in actual applications, and the minimum matching deviation between the two can be achieved within the allowable error range), then the group of B-type texture feature points of the current B-type three-dimensional model is the best matching degree with the A-type body surface landmark area ( For example, a group of B-type texture feature points with the best overall matching degree of the A-type texture feature point arrangement pattern of the A-type body surface landmark area (for example, the A-type external nose area); if the ORB algorithm, the FREAK algorithm or the SIFT algorithm searches for a group of B-type texture feature points with the best overall matching degree with the A-type texture feature point arrangement pattern of the A-type body surface landmark area (for example, the A-type external nose area) from the current B-type three-dimensional model, then it is considered that the registration of the texture modality of the A-type body surface landmark area is completed; otherwise, it is considered that the registration of the texture modality of the A-type body surface landmark area is incomplete and needs to be continued; for other A-type body surface landmark areas, repeat the above steps, and at the current time point (for example, time point T1), the registration of the texture modalities of all A-type body surface landmark areas is performed simultaneously;

[0060] In the above-mentioned step S3, the registration of the multi-view tracking modality is specifically as follows: at the current time point (for example, time point T1), for any A-type body surface landmark region (for example, the A-type external nose region), using the contour information of the A-type body surface landmark region to obtain the A-type contour line pattern of the A-type body surface landmark region (for example, the A-type external nose region) at different perspectives (for example, for the A-type external nose region, obtaining the front A-type contour line pattern and the left side A-type contour line pattern of the A-type external nose region); using the above-mentioned depth modality registration and the above-mentioned texture modality registration to select and determine the A-type contour line pattern of a specific perspective (for example, the front A-type contour line pattern of the A-type external nose region) from the A-type contour line patterns of different perspectives of the A-type body surface landmark region, and then searching from the current B-type three-dimensional model for the B-type contour line that has the best overall match with the A-type contour line pattern of the A-type body surface landmark region (for example, the A-type external nose region) at the specific perspective;

[0061] Specifically, the A-type contour pattern of the A-type body surface landmark area is divided into several line segments. For any line segment, a line segment with the smallest matching deviation with the line segment is searched from the current B-type three-dimensional model (the ideal state is that the two line segments are completely matched, that is, the spacing between the two line segments is zero. In actual applications, it is difficult to achieve complete matching between the two line segments. The minimum matching deviation between the two line segments can be achieved within the allowable error range). This is used as the B-type contour line segment with the best matching degree with the line segment. The above steps are repeated for the remaining line segments of the A-type contour pattern, and all the A-type contour patterns of the A-type body surface landmark area are searched for from the current B-type three-dimensional model. The registration of the line segments is performed simultaneously; when the B-type contour line segment with the best matching degree with all the line segments of the A-type contour line pattern of each view of the A-type body surface landmark area is searched from the current B-type 3D model, the registration of the multi-view tracking modality of the A-type body surface landmark area is considered to be completed; otherwise, the registration of the multi-view tracking modality of the A-type body surface landmark area is considered to be incomplete and needs to be continued; the above steps are repeated for other A-type body surface landmark areas, and at the current time point (for example, time point T1), the registration of the multi-view tracking modality of all A-type body surface landmark areas is performed simultaneously;

[0062] S4: At the next time point (for example, time point T2), repeat the above steps S2 to S3, and simultaneously realize the alignment of depth modality, texture modality, and multi-view tracking modality, thereby realizing real-time alignment of surgical navigation images.

[0063] In actual application, the above step S1 is performed before the operation, and the above steps S2 to S4 are performed during the operation.

[0064] In the above step S3, the depth modality registration probability, texture modality registration probability, and multi-view tracking modality registration probability are calculated, and the comprehensive registration probability is calculated using the depth modality registration probability, texture modality registration probability, and multi-view tracking modality registration probability, and the registration with the highest comprehensive registration probability is taken as the best registration.

[0065] In the above step S3, the comprehensive registration probability is calculated according to the following comprehensive registration probability formula (1):

[0066]

[0067] Among them, p(θ|D) represents the comprehensive registration probability, represents the depth modality registration probability, represents the texture modality registration probability, represents the registration probability of multi-view tracking modality, represents the depth modality data, Represents texture modal data, Represents multi-view tracking modal data; θ represents the posture change vector, which is composed of the rotation vector θ r and the translation vector θ t composition.

[0068] In the above step S3, the depth modality registration probability is calculated according to the following depth modality probability formula (2):

[0069]

[0070] Among them, p(θ|D d ) represents the depth modality registration probability, represents the depth value, σ d represents the user-defined standard deviation, X i Represents the point on the surface of the B-type 3D model, P i (θ) represents the closest point, N i Represents the normal vector of the model point.

[0071] In the above step S3, the texture modal registration probability is calculated according to the following texture modal probability formula (3):

[0072]

[0073] Among them, p(θ|D t ) represents the depth modality registration probability, X i ′ represents the monitoring point in the current frame, X i (θ) represents the projection point, σ t Represents the user-defined standard deviation.

[0074] In the above step S3, the multi-view tracking modal registration probability is calculated according to the following multi-view tracking modal probability formula (4):

[0075]

[0076] Among them, p(θ|D r ) represents the depth modality registration probability, θ represents the pose change vector, n represents the number of viewing angles, d i (θ) represents the contour distance of the i-th view, ω i Represents the set of pixels on the corresponding line, lll i Indicates the corresponding line.

[0077] In step S3 above, the registration of the depth modality, the registration of the texture modality, and the registration of the multi-view tracking modality all use the Newton optimization method to estimate the pose change vector:

[0078]

[0079] Among them, θ represents the posture change vector, g represents the gradient vector, H represents the Hessian matrix, λ r represents the regularization parameter of the rotation, λ t Represents the regularization parameter of the rotation, and I3 represents a 3*3 matrix.

[0080] like Figure 2 As shown, this embodiment provides a real-time registration system for surgical navigation images based on multimodal body surface landmark tracking. The registration system includes a type A three-dimensional model construction module, a type B three-dimensional model construction module, a registration module, and a display module.

[0081] The A-type three-dimensional model construction module can execute the above-mentioned step S1; the A-type three-dimensional model construction module can construct a type A three-dimensional model of the patient's head before surgery, specifically: performing a medical imaging scan (such as a CT scan) on the patient's head to obtain a type A two-dimensional image of the patient's head; using the type A two-dimensional image of the patient's head to construct a type A three-dimensional model of the patient's head in three-dimensional space; selecting a plurality of type A body surface landmark areas from the type A three-dimensional model of the patient's head, and obtaining depth information (i.e., three-dimensional spatial coordinates), texture information, and contour information of each type A body surface landmark area; the type A body surface landmark areas include but are not limited to the auricle area, the external nose area, and the forehead area;

[0082] The B-type 3D model construction module can execute the above step S2; the B-type 3D model construction module can construct a B-type 3D model of the patient's head in real time during the operation, specifically: using an RGB-D camera to take a real-time photo of the patient's head and obtain in real time the depth information (i.e., 3D spatial coordinates), texture information, and contour information of the patient's head; the RGB-D camera uses the obtained depth information of the patient's head to construct a B-type 3D model of the patient's head in real time in 3D space; the B-type 3D model includes the depth information, texture information, and contour information of the patient's head;

[0083] The registration module can perform the above step S3; the registration module can achieve the following functions: during the operation, at the current time point (for example, time point T1), simultaneously perform depth modality registration, texture modality registration, and multi-view tracking modality registration;

[0084] The registration module can realize the registration of depth modality, which is specifically as follows: at the current time point (for example, time point T1), for any A-type body surface landmark area (for example, A-type external nose area), the depth information of the A-type body surface landmark area is used to set a number of A-type position feature points (for example, nose tip point, nose bridge middle point, etc.) in the A-type body surface landmark area and obtain the A-type position feature point arrangement pattern of the A-type body surface landmark area (for example, A-type external nose area). The A-type position feature point arrangement pattern includes the three-dimensional spatial coordinates of all A-type position feature points in the A-type body surface landmark area (for example, A-type external nose area) and their mutual positional relationship, and then uses ICP (Iterative Closest The algorithm searches for a group of B-type position feature points with the best overall matching degree with the A-type position feature point arrangement pattern of the A-type body surface landmark area (for example, the A-type external nose area) from the current B-type three-dimensional model; if the B-type position feature point arrangement pattern composed of a group of B-type position feature points of the current B-type three-dimensional model has the smallest overall matching deviation with the A-type position feature point arrangement pattern in the A-type body surface landmark area (the ideal state is that the two are completely matched, but it is difficult to achieve complete matching between the two in actual applications, and the minimum matching deviation between the two can be achieved within the allowable error range), then the group of B-type position feature points of the current B-type three-dimensional model is the best overall matching degree with the A-type body surface landmark area (for example, the A-type external nose area). a set of B-type position feature points with the best matching degree of the A-type position feature point arrangement pattern of the A-type surface landmark region (e.g., the A-type external nose region) are searched for by the ICP algorithm from the current B-type 3D model; if the set of B-type position feature points with the best matching degree of the A-type position feature point arrangement pattern of the A-type surface landmark region (e.g., the A-type external nose region) are searched for from the current B-type 3D model, then the depth modality registration of the A-type surface landmark region is considered to be completed; otherwise, the depth modality registration of the A-type surface landmark region is considered to be incomplete and needs to be continued; for other A-type surface landmark regions, the above steps are repeated, and at the current time point (e.g., time point T1), the depth modality registration of all A-type surface landmark regions is performed simultaneously;

[0085] The registration module can also realize the registration of texture modalities, which is specifically as follows: at the current time point (for example, time point T1), for any A-type body surface landmark area (for example, A-type external nose area), a number of A-type texture feature points are set in the A-type body surface landmark area using the texture information of the A-type body surface landmark area and the A-type texture feature point arrangement pattern of the A-type body surface landmark area (for example, A-type external nose area) is obtained, the A-type texture feature point arrangement pattern includes the texture information of all A-type texture feature points in the A-type body surface landmark area (for example, A-type external nose area), and then the ORB algorithm, FREAK algorithm or SIFT algorithm (select any one of the three) is used to search from the current B-type three-dimensional model for a group of B-type texture feature points with the best overall matching degree with the A-type texture feature point arrangement pattern of the A-type body surface landmark area (for example, A-type external nose area); if the B-type texture feature point arrangement pattern composed of a group of B-type texture feature points in the B-type body surface landmark area is consistent with the A-type texture feature points in the A-type body surface landmark area, The overall matching deviation of the arrangement pattern is minimized (the ideal state is that the two are completely matched. In actual applications, it is difficult to achieve complete matching between the two. It is sufficient to achieve the minimum matching deviation between the two within the allowable error range). Then, the group of B-type texture feature points of the current B-type three-dimensional model is a group of B-type texture feature points with the best overall matching degree with the A-type texture feature point arrangement pattern of the A-type body surface landmark area (for example, the A-type external nose area); if the ORB algorithm, the FREAK algorithm or the SIFT algorithm searches for a group of B-type texture feature points with the best overall matching degree with the A-type texture feature point arrangement pattern of the A-type body surface landmark area (for example, the A-type external nose area) from the current B-type three-dimensional model, it is considered that the registration of the texture mode of the A-type body surface landmark area is completed; otherwise, it is considered that the registration of the texture mode of the A-type body surface landmark area is incomplete and needs to be continued; for other A-type body surface landmark areas, repeat the above steps, and at the current time point (for example, time point T1), the registration of the texture modes of all A-type body surface landmark areas is performed simultaneously;

[0086] The registration module can also realize the registration of multi-view tracking modalities, which is specifically as follows: at the current time point (for example, time point T1), for any A-type body surface landmark area (for example, A-type external nose area), the contour information of the A-type body surface landmark area is used to obtain the A-type contour line pattern of the A-type body surface landmark area (for example, A-type external nose area) at different perspectives (for example, for the A-type external nose area, the front A-type contour line pattern and the left side A-type contour line pattern of the A-type external nose area are obtained); using the above-mentioned depth modality registration and the above-mentioned The texture modality registration selects an A-type contour pattern of a specific perspective (for example, the frontal A-type contour pattern of the A-type external nose region) from the A-type contour patterns of different perspectives of the A-type body surface landmark region, and then searches the current B-type 3D model for a B-type contour line that has the best overall match with the A-type contour pattern of the specific perspective of the A-type body surface landmark region (for example, the A-type external nose region); specifically, the A-type contour pattern of the A-type body surface landmark region is divided into several line segments, and for any of the line segments, A line segment with the smallest matching deviation from the line segment is searched from the current B-type 3D model (ideally, the two line segments completely match, i.e., the spacing between the two line segments is zero. In actual applications, it is difficult to achieve a complete match between the two line segments. It is sufficient to achieve the smallest matching deviation between the two line segments within the allowable error range) as the B-type contour line segment with the best matching degree with the line segment; the above steps are repeated for the remaining line segments of the A-type contour line pattern, and the registration of all line segments of the A-type contour line pattern of the A-type body surface landmark region is performed simultaneously; when the B-type contour line segment with the best matching degree with all line segments of the A-type contour line pattern of each view of the A-type body surface landmark region is searched from the current B-type 3D model, it is considered that the registration of the multi-view tracking modality of the A-type body surface landmark region is completed; otherwise, it is considered that the registration of the multi-view tracking modality of the A-type body surface landmark region is incomplete and needs to be continued; the above steps are repeated for other A-type body surface landmark regions, and at the current time point (e.g., time point T1), the registration of the multi-view tracking modality of all A-type body surface landmark regions is performed simultaneously;

[0087] The registration module can also perform the above step S4; the registration module can also achieve the following functions: at the next time point (for example, time point T2), repeat the above functions, and simultaneously achieve depth modality registration, texture modality registration, and multi-view tracking modality registration, thereby achieving real-time registration of surgical navigation images;

[0088] The display module is used to display the real-time registration process and the registered image.

[0089] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications based on the above description are possible. It is not possible to enumerate all embodiments here. Any obvious variations or modifications arising from the technical solution of the present invention remain within the scope of protection of the present invention.

Claims

1. A real-time registration method for surgical navigation images based on multimodal body surface landmark tracking, characterized in that: The registration method includes the following steps: S1: constructing a type A three-dimensional model of the patient's head before surgery; S2: Real-time construction of a B-type 3D model of the patient’s head during surgery; S3: During the operation, at the current moment, the depth modality registration, texture modality registration, and multi-view tracking modality registration are performed simultaneously; S4: at the next time point, repeating steps S2 to S3, and simultaneously achieving registration of the depth modality, registration of the texture modality, and registration of the multi-view tracking modality, thereby achieving real-time registration of the surgical navigation image; Step S1 specifically comprises: performing a medical imaging scan on the patient's head to obtain an A-type two-dimensional image of the patient's head; constructing an A-type three-dimensional model of the patient's head in a three-dimensional space using the A-type two-dimensional image of the patient's head; selecting a plurality of A-type body surface landmark regions from the A-type three-dimensional model of the patient's head, and obtaining depth information, texture information, and contour information of each A-type body surface landmark region; the A-type body surface landmark regions include but are not limited to the auricle region, the external nose region, and the forehead region; Step S2 specifically includes: using an RGB-D camera to take a real-time photo of the patient's head and obtain real-time depth information, texture information, and contour information of the patient's head; the RGB-D camera uses the obtained depth information of the patient's head to construct a B-type three-dimensional model of the patient's head in real-time in three-dimensional space; the B-type three-dimensional model includes the depth information, texture information, and contour information of the patient's head.

2. The real-time registration method for surgical navigation images based on multimodal body surface landmark tracking according to claim 1, characterized in that: In step S3, the registration of the depth modality is specifically as follows: at the current time point, for any A-type body surface landmark region, using the depth information of the A-type body surface landmark region, setting a number of A-type position feature points in the A-type body surface landmark region and obtaining an A-type position feature point arrangement pattern of the A-type body surface landmark region, the A-type position feature point arrangement pattern including the three-dimensional spatial coordinates of all A-type position feature points in the A-type body surface landmark region and their mutual positional relationships, and then using the ICP algorithm to search from the current B-type three-dimensional model for a group of B-type position feature points that have the best overall matching degree with the A-type position feature point arrangement pattern of the A-type body surface landmark region; If the overall matching deviation between a B-type position feature point arrangement pattern formed by a group of B-type position feature points of the current B-type 3D model and the A-type position feature point arrangement pattern in the A-type body surface landmark region is the smallest, then the group of B-type position feature points of the current B-type 3D model is the group of B-type position feature points that best matches the A-type position feature point arrangement pattern in the A-type body surface landmark region; If the ICP algorithm searches for a set of B-type position feature points that best match the A-type position feature point arrangement pattern of the A-type body surface landmark area from the current B-type three-dimensional model, then the depth modality registration of the A-type body surface landmark area is considered to be completed; otherwise, the depth modality registration of the A-type body surface landmark area is considered to be incomplete and needs to be continued; for other A-type body surface landmark areas, repeat the above steps, and at the current moment, the depth modality registration of all A-type body surface landmark areas is performed simultaneously.

3. The real-time registration method for surgical navigation images based on multimodal body surface landmark tracking according to claim 1, characterized in that: In the step S3, the registration of the texture modality is specifically as follows: at the current time point, for any A-type body surface marker area, a number of A-type texture feature points are set in the A-type body surface marker area using the texture information of the A-type body surface marker area and an A-type texture feature point arrangement pattern of the A-type body surface marker area is obtained, the A-type texture feature point arrangement pattern includes the texture information of all A-type texture feature points in the A-type body surface marker area, and then an ORB algorithm, a FREAK algorithm or a SIFT algorithm is used to search from the current B-type three-dimensional model for a group of B-type texture feature points that have the best overall matching degree with the A-type texture feature point arrangement pattern of the A-type body surface marker area; if the B-type texture feature point arrangement pattern composed of a group of B-type texture feature points in the B-type body surface marker area is consistent with the A-type texture feature point arrangement pattern in the A-type body surface marker area, If the overall matching deviation of the A-type texture feature point arrangement pattern is the smallest, then the group of B-type texture feature points of the current B-type three-dimensional model is the group of B-type texture feature points with the best overall matching degree with the A-type texture feature point arrangement pattern of the A-type body surface landmark area; if the ORB algorithm, FREAK algorithm or SIFT algorithm searches for a group of B-type texture feature points with the best overall matching degree with the A-type texture feature point arrangement pattern of the A-type body surface landmark area from the current B-type three-dimensional model, then it is considered that the texture modality registration of the A-type body surface landmark area is completed; otherwise, it is considered that the texture modality registration of the A-type body surface landmark area is incomplete and needs to be continued; for other A-type body surface landmark areas, repeat the above steps, and at the current moment, the texture modality registration of all A-type body surface landmark areas is performed simultaneously.

4. The real-time registration method for surgical navigation images based on multimodal body surface landmark tracking according to claim 1, characterized in that: In step S3, the registration of the multi-view tracking modality is specifically as follows: at the current time point, for any A-type body surface landmark region, using the contour information of the A-type body surface landmark region to obtain the A-type contour line pattern of the A-type body surface landmark region at different perspectives; using the depth modality registration and the texture modality registration to select and determine an A-type contour line pattern of a specific perspective from the A-type contour line patterns of the A-type body surface landmark region at different perspectives, and then searching from the current B-type three-dimensional model for a B-type contour line that has the best overall match with the A-type contour line pattern of the A-type body surface landmark region at the specific perspective; The A-type contour line pattern of the A-type body surface landmark area is divided into several line segments. For any line segment, a line segment with the smallest matching deviation with the line segment is searched from the current B-type three-dimensional model as the B-type contour line segment with the best matching degree with the line segment; the above steps are repeated for the remaining line segments of the A-type contour line pattern, and the registration of all line segments of the A-type contour line pattern of the A-type body surface landmark area is performed simultaneously; when the B-type contour line segment with the best matching degree with all line segments of the A-type contour line pattern of each perspective of the A-type body surface landmark area is searched from the current B-type three-dimensional model, it is considered that the registration of the multi-view tracking modality of the A-type body surface landmark area is completed; otherwise, it is considered that the registration of the multi-view tracking modality of the A-type body surface landmark area is incomplete and needs to be continued; the above steps are repeated for other A-type body surface landmark areas, and at the current time point, the registration of the multi-view tracking modality of all A-type body surface landmark areas is performed simultaneously.

5. The real-time registration method for surgical navigation images based on multimodal body surface landmark tracking according to claim 1, characterized in that: In step S3, the depth modality registration probability, the texture modality registration probability, and the multi-view tracking modality registration probability are calculated, and the comprehensive registration probability is calculated using the depth modality registration probability, the texture modality registration probability, and the multi-view tracking modality registration probability, and the registration with the highest comprehensive registration probability is taken as the best registration; In step S3, the comprehensive registration probability is calculated according to the following comprehensive registration probability formula: Among them, p(θ|D) represents the comprehensive registration probability, represents the depth modality registration probability, represents the texture modality registration probability, represents the registration probability of multi-view tracking modality, represents the depth modality data, Represents texture modal data, Represents multi-view tracking modal data; θ represents the posture change vector, which is composed of the rotation vector θ r and the translation vector θ t composition.

6. The real-time registration method for surgical navigation images based on multimodal body surface landmark tracking according to claim 5, characterized in that: In step S3, the depth modality registration probability is calculated according to the following depth modality probability formula: Where p(θ|D d ) represents the depth modality registration probability, represents the depth value, σ d represents the user-defined standard deviation, X i Represents the point on the surface of the B-type 3D model, P i (θ) represents the closest point, N i Represents the normal vector of the model point.

7. The real-time registration method for surgical navigation images based on multimodal body surface landmark tracking according to claim 5, characterized in that: In step S3, the texture modal registration probability is calculated according to the following texture modal probability formula: Where p(θ|D t ) represents the depth modality registration probability, X i ′ represents the monitoring point in the current frame, X i (θ) represents the projection point, σ t Represents the user-defined standard deviation.

8. The real-time registration method for surgical navigation images based on multimodal body surface landmark tracking according to claim 5, characterized in that: In step S3, the multi-view tracking modality registration probability is calculated according to the following multi-view tracking modality probability formula: Where p(θ|D r ) represents the depth modality registration probability, θ represents the pose change vector, n represents the number of viewing angles, d i (θ) represents the contour distance of the i-th view, ω i Represents the set of pixels on the corresponding line, lll i Indicates the corresponding line.

9. The real-time registration method for surgical navigation images based on multimodal body surface landmark tracking according to claim 5, characterized in that: In step S3, the registration of the depth modality, the registration of the texture modality, and the registration of the multi-view tracking modality all use the Newton optimization method to estimate the pose change vector: Among them, θ represents the posture change vector, g represents the gradient vector, H represents the Hessian matrix, λ r represents the regularization parameter of the rotation, λ t represents the regularization parameter of the rotation, and I3 represents a 3*3 matrix.

10. A real-time registration system for surgical navigation images based on multimodal body surface landmark tracking, the registration system adopts and can execute the real-time registration method for surgical navigation images based on multimodal body surface landmark tracking according to any one of claims 1 to 9, characterized in that: The registration system includes an A-type 3D model construction module, a B-type 3D model construction module, a registration module, and a display module; The A-type three-dimensional model construction module can execute the step S1; the A-type three-dimensional model construction module can construct an A-type three-dimensional model of the patient's head before surgery, specifically: performing a medical imaging scan on the patient's head to obtain an A-type two-dimensional image of the patient's head; using the A-type two-dimensional image of the patient's head to construct an A-type three-dimensional model of the patient's head in a three-dimensional space; selecting a plurality of A-type body surface landmark areas from the A-type three-dimensional model of the patient's head, and obtaining depth information, texture information, and contour information of each A-type body surface landmark area; the A-type body surface landmark areas include but are not limited to the auricle area, the external nose area, and the forehead area; The B-type 3D model construction module is capable of executing step S2; the B-type 3D model construction module is capable of constructing a B-type 3D model of the patient's head in real time during surgery, specifically by: using an RGB-D camera to photograph the patient's head in real time, and obtaining depth information, texture information, and contour information of the patient's head in real time; the RGB-D camera uses the obtained depth information of the patient's head to construct a B-type 3D model of the patient's head in real time in a 3D space; the B-type 3D model includes the depth information, texture information, and contour information of the patient's head; The registration module can execute step S3; the registration module can realize the following functions: during the operation, at the current time point, simultaneously perform the registration of the depth modality, the registration of the texture modality, and the registration of the multi-view tracking modality; The registration module can realize the registration of the depth modality, which is specifically as follows: at the current time point, for any A-type body surface landmark area, using the depth information of the A-type body surface landmark area, a number of A-type position feature points are set in the A-type body surface landmark area and an A-type position feature point arrangement pattern of the A-type body surface landmark area is obtained. The A-type position feature point arrangement pattern includes the three-dimensional spatial coordinates of all A-type position feature points in the A-type body surface landmark area and their mutual positional relationships. Then, the ICP algorithm is used to search from the current B-type three-dimensional model for a group of B-type position feature points that have the best overall match with the A-type position feature point arrangement pattern of the A-type body surface landmark area. If the overall matching deviation between a B-type position feature point arrangement pattern formed by a group of B-type position feature points of the current B-type 3D model and the A-type position feature point arrangement pattern in the A-type body surface landmark region is the smallest, then the group of B-type position feature points of the current B-type 3D model is the group of B-type position feature points that best matches the A-type position feature point arrangement pattern in the A-type body surface landmark region; If the ICP algorithm searches for a set of B-type position feature points from the current B-type 3D model that best matches the A-type position feature point arrangement pattern of the A-type body surface landmark region, the depth modality registration of the A-type body surface landmark region is considered complete. Otherwise, the depth modality registration of the A-type body surface landmark region is considered incomplete and needs to be continued. Repeat the above steps for other A-type body surface landmark regions, and at the current time point, the depth modality registration of all A-type body surface landmark regions is performed simultaneously. The registration module can also realize the registration of texture modalities, which is specifically as follows: at the current moment, for any A-type body surface marker area, a number of A-type texture feature points are set in the A-type body surface marker area using the texture information of the A-type body surface marker area and the A-type texture feature point arrangement pattern of the A-type body surface marker area is obtained, the A-type texture feature point arrangement pattern includes the texture information of all A-type texture feature points in the A-type body surface marker area, and then the ORB algorithm, FREAK algorithm or SIFT algorithm is used to search from the current B-type three-dimensional model for a group of B-type texture feature points that have the best overall matching degree with the A-type texture feature point arrangement pattern of the A-type body surface marker area; if the B-type texture feature point arrangement diagram composed of a group of B-type texture feature points in the B-type body surface marker area is consistent with the A-type body surface marker area If the overall matching deviation of the A-type texture feature point arrangement pattern within the current B-type three-dimensional model is the smallest, then the group of B-type texture feature points of the current B-type three-dimensional model is the group of B-type texture feature points with the best overall matching degree with the A-type texture feature point arrangement pattern of the A-type body surface landmark area; if the ORB algorithm, the FREAK algorithm or the SIFT algorithm searches for a group of B-type texture feature points with the best overall matching degree with the A-type texture feature point arrangement pattern of the A-type body surface landmark area from the current B-type three-dimensional model, then it is considered that the registration of the texture modality of the A-type body surface landmark area is completed; otherwise, it is considered that the registration of the texture modality of the A-type body surface landmark area is incomplete and needs to be continued; for other A-type body surface landmark areas, repeat the above steps, and at the current time point, the registration of the texture modalities of all A-type body surface landmark areas is performed simultaneously; The registration module can also realize the registration of multi-view tracking modalities, which is specifically as follows: at the current moment, for any A-type body surface marker area, the contour information of the A-type body surface marker area is used to obtain the A-type contour line pattern of different perspectives of the A-type body surface marker area; the above-mentioned depth modality registration and the above-mentioned texture modality registration are used to select and determine a specific perspective A-type contour line pattern from the A-type body surface marker area's A-type contour line patterns of different perspectives, and then search the current B-type three-dimensional model for the B-type contour line with the best overall matching degree with the A-type contour line pattern of the specific perspective of the A-type body surface marker area; the A-type contour line pattern of the A-type body surface marker area is divided into several line segments, and for any one of the line segments, the current B-type three-dimensional model is searched for the line segment matching the line segment. The line segment with the smallest difference is taken as the B-type contour line segment with the best matching degree with the line segment; the above steps are repeated for the remaining line segments of the A-type contour line pattern, and the registration of all line segments of the A-type contour line pattern of the A-type body surface landmark area is performed simultaneously; when the B-type contour line segment with the best matching degree with all line segments of the A-type contour line pattern of each view of the A-type body surface landmark area is searched from the current B-type 3D model, it is considered that the registration of the multi-view tracking modality of the A-type body surface landmark area is completed; otherwise, it is considered that the registration of the multi-view tracking modality of the A-type body surface landmark area is incomplete and needs to be continued; the above steps are repeated for other A-type body surface landmark areas, and at the current time point, the registration of the multi-view tracking modality of all A-type body surface landmark areas is performed simultaneously; The registration module can also perform the above step S4; the registration module can also realize the following functions: at the next time point, repeat the above functions, and simultaneously realize the registration of the depth modality, the texture modality, and the multi-view tracking modality, thereby realizing real-time registration of the surgical navigation image; The display module is used to display the real-time registration process and the registered image.

Images