Target real-time tracking method and readable storage medium

Abstract

The present invention discloses a real-time target tracking method and a readable storage medium. The real-time target tracking method includes the following steps: obtaining a scalp point cloud based on a medical imaging image of a subject's head, and obtaining an initial facial point cloud based on an initial facial image of the subject; registering the scalp point cloud with the initial facial point cloud based on a first registration method to obtain a heterogeneous point cloud spatial transformation relationship between the scalp point cloud and the initial facial point cloud; obtaining real-time position information of the target on the initial facial point cloud based on the target's initial position information and the heterogeneous point cloud spatial transformation relationship; and obtaining a real-time position relationship of the target in each frame of the facial point cloud based on the target's real-time position information on the initial facial point cloud and the homogeneous point cloud spatial transformation relationship between two adjacent frames of facial point cloud.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a real-time target tracking method and a readable storage medium. Background Art

[0002] Target navigation is a surgical assistance system that reconstructs the lesion condition based on preoperative imaging data such as CT and MRI. During surgery, it uses ultrasonic, electromagnetic, optical and other tracking systems to accurately track the real target and display the relative position of surgical instruments and the real target in real time, thereby assisting in surgery.

[0003] The optical tracking system is mainly based on point cloud registration. By reconstructing the point cloud of the preoperative image, a scalp point cloud is obtained. During the operation, the patient's facial point cloud is obtained in real time through optical photography. The real-time registration of the two point clouds can achieve the purpose of real-time tracking and positioning of the target. Registration can generally be divided into two steps: coarse registration and fine registration. Coarse registration is performed without knowing the initial pose of the two parts of the point cloud. After completion, a rough registration matrix is ​​provided. Fine registration is performed on the basis of coarse registration. It relies on the coarse registration matrix provided by the coarse registration and optimizes the matrix to obtain a more accurate registration matrix. ICP is a commonly used algorithm in real-time registration. Its accuracy and speed are more dependent on the initial registration matrix, that is, the position of the point cloud after the initial registration matrix. The closer the position of the two point clouds after registration with the initial registration matrix, the higher the performance and accuracy of ICP.

[0004] The typical real-time registration process involves coarsely and finely registering each frame of optical camera point cloud (e.g., facial point cloud) with the scalp point cloud to calculate the real-time coordinates of the target. This process is time-consuming and results in poor real-time performance. Furthermore, the continuous registration of the scalp point cloud and facial point cloud within the target system presents the additional difficulty of registering heterogeneous point clouds, which can easily lead to deviations in target positioning. Summary of the Invention

[0005] In order to solve at least one aspect of the above-mentioned problems and defects in the prior art, the present invention provides a real-time target tracking method and a readable storage medium. The technical method is as follows:

[0006] According to one aspect of the present invention, a method for real-time tracking of a target is provided, comprising the following steps:

[0007] obtaining a scalp point cloud based on a medical imaging image of the subject's head, and obtaining an initial facial point cloud based on an initial facial image of the subject;

[0008] registering the scalp point cloud with the initial facial point cloud based on a first registration method to obtain a heterogeneous point cloud spatial transformation relationship between the scalp point cloud and the initial facial point cloud;

[0009] Obtaining real-time position information of the target on the initial facial point cloud based on the initial position information of the target and the spatial transformation relationship of the heterogeneous point cloud;

[0010] The real-time position relationship of the target point in each frame of the facial point cloud is obtained based on the real-time position information of the target point on the initial facial point cloud and the homologous point cloud spatial transformation relationship between two adjacent frames of facial point clouds.

[0011] Specifically, the method for obtaining the real-time position relationship of the target point in each frame of the facial point cloud based on the real-time position information of the target point on the initial facial point cloud and the homologous point cloud spatial transformation relationship between two adjacent frames of facial point cloud comprises the following steps:

[0012] Obtaining a real-time position relationship of the target point on the facial point cloud of the adjacent frame to the initial facial point cloud based on a homologous point cloud spatial transformation relationship between the facial point cloud of the adjacent frame to the initial facial point cloud and the initial facial point cloud and real-time position information of the target point on the initial facial point cloud;

[0013] Iterate the previous step to obtain the real-time position relationship of the target in each frame of the facial point cloud.

[0014] Furthermore, the method for obtaining the spatial transformation relationship of homologous point clouds between two adjacent frames of facial point clouds is to align the surface features of the facial point cloud of the previous frame with the surface features of the facial point cloud of the current frame based on the second registration method to obtain the spatial transformation relationship of homologous point clouds between the facial point cloud of the previous frame and the facial point cloud of the current frame.

[0015] Specifically, the method for registering the scalp point cloud with the initial facial point cloud based on the first registration method includes the following steps:

[0016] Obtaining a facial model point cloud based on the scalp point cloud, and obtaining facial model normal vectors of all points in the facial model point cloud based on the facial model point cloud;

[0017] Obtaining initial facial normal vectors for all points in the initial facial point cloud based on the initial facial point cloud;

[0018] Optimize all facial model normal vectors and all initial facial normal vectors based on the normal vector optimization method;

[0019] Obtaining surface features of the facial model point cloud and the surface features of the initial facial point cloud based on the optimized facial model normal vector and the optimized initial facial normal vector;

[0020] Feature matching is performed based on the surface features of the facial model point cloud and the surface features of the initial facial point cloud using the first registration method to obtain an optimized spatial transformation relationship for converting the scalp point cloud into the space of the initial facial point cloud.

[0021] Specifically, the method for obtaining facial model normal vectors of all points in the facial model point cloud based on the facial model point cloud comprises the following steps:

[0022] Determine all neighboring points within a k1 neighborhood of each point in the facial model point cloud using a kd Tree search method, where the k1 neighborhood is a sphere with a length of a first predetermined multiple of a voxel in the medical image of the subject as a radius;

[0023] Based on all neighboring points in the k1 neighborhood of each point, the tangent plane of each point is obtained by least squares fitting and the normal vector of the tangent plane of each point in the facial model point cloud is obtained. The normal vector of the tangent plane of each point in the facial model point cloud is the facial model normal vector.

[0024] Specifically, the method for obtaining initial facial normal vectors of all points in the initial facial point cloud based on the initial facial point cloud comprises the following steps:

[0025] Determine all neighboring points within a k2 neighborhood of each point in the initial facial point cloud using a kd Tree search method, where the k2 neighborhood is a sphere with a length of a second predetermined multiple of a voxel in the medical image of the subject as a radius;

[0026] Based on all neighboring points in the k2 neighborhood of each point, the least squares method is used to fit the tangent surface of each point and the normal vector of the tangent surface of each point in the initial facial point cloud is obtained. The normal vector of the tangent surface of each point in the initial facial point cloud is the initial facial normal vector.

[0027] Preferably, the method for optimizing the normal vector of the facial model comprises the following steps:

[0028] Reset the coordinates of the center point of the scalp and facial point cloud to the coordinate origin O and establish an XYZ coordinate system, in which the Y axis is parallel to the ground and its positive direction points to the front of the subject, the X axis is perpendicular to the Y axis and its positive direction points to the right of the subject, and the Z axis is perpendicular to the X and Y axes and its positive direction points to the top of the subject;

[0029] Obtaining a vector from a coordinate origin O to each point in the facial model point cloud based on the facial model point cloud;

[0030] Based on the inner product of the vector from the coordinate origin O to each point and the facial model normal vector of each point itself, it is determined whether to adjust the direction of the facial model normal vector of the point based on the value of the inner product. When the value of the inner product is greater than 0, the direction of the normal vector of the point is reversed.

[0031] Preferably, the method for optimizing the initial facial normal vector comprises the following steps:

[0032] The optical center of the camera is the coordinate origin O c Construct the camera coordinate system;

[0033] Obtain the coordinate origin O based on the initial facial point cloud c A vector to each point in the initial facial point cloud;

[0034] Based on the coordinate origin O c Inner product of the vector to each point in the initial facial point cloud with the initial facial normal vector of each point itself to obtain a solid inner product value;

[0035] Based on the entity inner product value, it is determined whether to adjust the direction of the initial facial normal vector of the point corresponding to the entity inner product value. When the entity inner product value is greater than 0, the direction of the normal vector of the point corresponding to the entity inner product value is reversed.

[0036] Furthermore, the facial model point cloud and the initial facial point cloud are downsampled to obtain a downsampled facial model point cloud and a downsampled initial facial point cloud, and then the facial model normal vector and the initial facial normal vector are obtained based on the downsampled facial model point cloud and the downsampled initial facial point cloud.

[0037] The downsampling method comprises the following steps:

[0038] rasterizing the spatial voxels of the facial model point cloud and the initial facial point cloud;

[0039] Determining whether each voxel grid after the facial model point cloud is rasterized contains a point in the facial model point cloud; if the current voxel grid contains a point in the facial model point cloud, selecting a point closest to the center point of the current voxel grid from the facial model point cloud as a sampling point, traversing the facial model point cloud to obtain a first sampling point set, where the first sampling point set is the downsampled facial model point cloud;

[0040] Determine whether each voxel grid after the initial facial point cloud is rasterized contains a point in the initial facial point cloud. When the current voxel grid contains a point in the initial facial point cloud, select the point closest to the center point of the current voxel grid from the initial facial point cloud as a sampling point, traverse the initial facial point cloud to obtain a second sampling point set, and the second sampling point set is the downsampled initial facial point cloud.

[0041] According to another aspect of the present invention, a readable storage medium is provided, wherein:

[0042] The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the above-mentioned target real-time tracking method is executed.

[0043] The target real-time tracking method and the readable storage medium according to the embodiments of the present invention have at least one of the following advantages:

[0044] (1) The target real-time tracking method and readable storage medium provided by the present invention realizes the real-time output of the changes between frames during the process of real-time tracking of the target by collecting the first frame and performing heterogeneous point cloud registration with the point cloud to be registered, and then performing homologous point cloud registration on each subsequent frame with the previous frame, thereby achieving the purpose of real-time tracking of the target;

[0045] (2) The target real-time tracking method and readable storage medium provided by the present invention achieve real-time tracking of the target through the method of initial heterogeneous point cloud registration and subsequent homologous point cloud real-time registration;

[0046] (3) The real-time target tracking method and readable storage medium provided by the present invention track targets by combining heterogeneous point cloud registration with homologous point cloud registration, thereby improving the target tracking speed while ensuring the target tracking accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] These and / or other aspects and advantages of the present invention will become apparent and readily understood from the following description of the preferred embodiments taken in conjunction with the accompanying drawings, in which:

[0048] Figure 1 is a flow chart of a target real-time tracking method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The technical solution of the present invention will be further described in detail below through examples and in conjunction with the accompanying drawings. In the specification, the same or similar reference numerals indicate the same or similar components. The following description of the embodiments of the present invention with reference to the accompanying drawings is intended to explain the overall inventive concept of the present invention and should not be construed as limiting the present invention.

[0050] See also Figure 1 , which shows a target real-time tracking method according to an embodiment of the present invention. The target real-time tracking method comprises the following steps:

[0051] obtaining a scalp point cloud based on a medical imaging image of the subject's head, and obtaining an initial facial point cloud based on an initial facial image of the subject;

[0052] registering the scalp point cloud with the initial facial point cloud based on a first registration method to obtain a heterogeneous point cloud spatial transformation relationship between the scalp point cloud and the initial facial point cloud;

[0053] Obtaining real-time position information of the target on the initial facial point cloud based on the initial position information of the target and the spatial transformation relationship of the heterogeneous point cloud;

[0054] The real-time position relationship of the target point in each frame of the facial point cloud is obtained based on the real-time position information of the target point on the initial facial point cloud and the homologous point cloud spatial transformation relationship between two adjacent frames of facial point clouds.

[0055] A point cloud is a dataset of points in space that can represent a three-dimensional shape or object, typically acquired using a 3D scanner. The location of each point in a point cloud is described by a set of Cartesian coordinates (x, y, z), some of which may also include color information (R, G, B) or the intensity of an object's reflective surface.

[0056] Heterogeneous point cloud registration, also known as cross-source point cloud registration, refers to the process of transforming point clouds collected by different 3D sensors into the same space to achieve alignment between point clouds.

[0057] Homologous point cloud registration refers to the process of aligning point clouds collected by the same type of 3D sensors in the same space.

[0058] The medical imaging image of the subject may be an MRI image, a CT imaging image, or an ultrasound imaging image, preferably an MRI image, and more preferably an MRI image with an accuracy of 1 mm.

[0059] In one example, a point cloud belonging to the scalp portion of the subject's head is extracted from a medical image of the subject's head using a ray method, thereby obtaining the subject's scalp point cloud, which no longer contains characteristic attributes of the interior of the brain.

[0060] The following is an example of the optimization method for registration using an MRI image of a subject's head as an example of a medical image. The principle of the optimization method for registration based on CT imaging images or ultrasound imaging images is completely consistent with the principle of the optimization method for registration based on MRI images, and will not be repeated here.

[0061] The ray method specifically includes the following steps:

[0062] An O'-RAS coordinate system was established based on the MRI image of the subject's head, where the origin O' is the lower right corner of the MRI image, the R axis is parallel to the horizontal plane and its positive direction points to the right of the subject, the S axis is perpendicular to the R axis and its positive direction points to the top of the subject, and the A axis is perpendicular to the RO'S plane and its positive direction points to the front of the subject;

[0063] Recursively iterate the A axis and S axis to extract the one-dimensional sequence S of MRI images a＝v,s＝bFor example, in the O'-RAS coordinate system, we obtain a plane parallel to the RO'S plane with a straight line a=v, where v represents the value on the A axis; then we extract a straight line s=b in the plane, and obtain a one-dimensional sequence S a＝v,s＝b , where b represents the value on the S axis; traverse the A axis and S axis to obtain the set of all straight lines, obtain the coordinates of the first and last non-zero points at both ends of each straight line and store them, and then obtain the scalp coordinate set, which is the scalp point cloud C1.

[0064] Then, the facial model point cloud of the subject is obtained based on the scalp point cloud. The specific method includes the following steps:

[0065] Obtaining the center point of the scalp point cloud by using a mean method based on the coordinates of all points in the scalp point cloud. For example, calculating the average value of the coordinates of all points in the scalp point cloud, and the average value is the coordinate of the center point of the scalp point cloud;

[0066] Subtracting the coordinate value of the center point of the point cloud from the coordinate values ​​of all points in the scalp point cloud to obtain reset scalp coordinates of all points and a reset scalp coordinate set;

[0067] The coordinates of the center point of the point cloud are reset to the coordinate origin O and an XYZ coordinate system is established. In the XYZ coordinate system, the Y axis is parallel to the ground and its positive direction points to the front of the subject, the X axis is perpendicular to the Y axis and its positive direction points to the right of the subject, and the Z axis is perpendicular to the XOY plane and its positive direction points to the top of the subject. Points with Y axis coordinate values ​​greater than 0 (i.e., y>0) are selected from the reset scalp coordinate set and the point set F is obtained. The point set F is the facial model point cloud S of the subject. mri_anterior .

[0068] Data standardization is achieved by resetting the scalp coordinates and the center point of the point cloud, which is conducive to the stability of the subsequent FPFH feature extraction results and thus optimizes the registration process.

[0069] The initial facial image of the subject is obtained using a stereo camera, and the initial facial image is an image with depth information. A stereo camera is a device that uses stereo imaging technology to perform stereoscopic imaging, and includes structured light stereo cameras, time-of-flight 3D cameras, binocular stereo vision cameras, and the like. Those skilled in the art can select a camera based on actual needs, as long as it can produce a stereoscopic image of the subject.

[0070] In one example, the subject's initial facial image D is based on the stereo camera intrinsic parameter matrix M i The initial facial point cloud C2 is obtained by calculation. In one example, the stereo camera internal parameter matrix M i It can be obtained by calibrating the stereo camera. The expression of the initial facial point cloud C2 is:

[0071]

[0072] After obtaining the subject’s initial facial point cloud C2, the head point cloud C1 (e.g., the facial model point cloud S mri_anterior ) is downsampled with the initial facial point cloud C2 to obtain a downsampled head point cloud C1 and a downsampled initial facial point cloud C2.

[0073] The method and principle of downsampling the scalp point cloud C1 are exactly the same as those of downsampling the initial facial point cloud C2. The method of downsampling the scalp point cloud C1 will be used as an example for an exemplary description below, and the method of downsampling the initial facial point cloud C2 will not be repeated here.

[0074] The following steps are used to downsample the scalp point cloud:

[0075] Rasterize the voxels in the space where the scalp point cloud C1 is located;

[0076] Determine whether each voxel grid contains a point in the scalp point cloud C1. If the current voxel grid contains a point in the scalp point cloud C1, select the point in the scalp point cloud C1 that is closest to the center point of the current voxel grid as a sampling point.

[0077] The scalp point cloud C1 is traversed to obtain a sampling point set, where the sampling point set is the downsampled scalp point cloud C1.

[0078] In one example, a method for determining whether each voxel grid contains a point in the scalp point cloud C1 is to determine whether the coordinates of the point are within the coordinate range of the current voxel grid. When the coordinates of the point are within the coordinate range, it is determined that the current voxel grid contains the point.

[0079] For example, the facial model point cloud S mri_anterior The space voxels are rasterized, and the length, width and height of each grid are 1 cm. Then, it is determined whether each voxel grid contains the facial model point cloud S. mri_anterior When the current voxel grid contains the facial model point cloud S mri_anterior The points in the facial model point cloud S mri_anterior The point closest to the center point of the current voxel grid is selected as the sampling point.

[0080] For example, the coordinate range of the current voxel grid is (1,1,1) to (2,2,2), and the facial model point cloud S mri_anterior There is at least one point in the voxel grid, for example, the coordinates of one point (1.5, 1.7, 2) fall within the above coordinate range, then it is determined that the current voxel grid contains the facial model point cloud S mri_anteriorThen, from the points falling within the coordinate range, select a point closest to the center of the current voxel grid as the sampling point. If only one point falls within the coordinate range of the current voxel grid, then this point is the sampling point. mri_anterior , thus obtaining all sampling points, which form the facial model point cloud S after downsampling mri_anterior .

[0081] At the same time, the initial facial point cloud C2 is downsampled according to the scalp point cloud C1 downsampling method, and all the downsampled sampling points are obtained, thereby obtaining the downsampled initial facial point cloud C2.

[0082] The downsampling method can also be to use Open3D to convert the scalp point cloud C1 (for example, the facial model point cloud S mri_anterior ) is downsampled to 10mm resolution, where the side length of the voxel grid is set to 10mm. Similarly, the initial facial point cloud C2 is downsampled to 10mm resolution using the voxel grid using Open3D.

[0083] In one embodiment, the downsampled initial facial point cloud C2 is filtered to remove noise generated by the stereo camera, such as outliers. The filtering method used is a k-nearest neighbor point cloud filtering method, a statistical outlier removal method, or a radius outlier removal method.

[0084] For example, the statistical outlier removal method is as follows: First, the average distance between each point in the downsampled initial facial point cloud and its 20 (parameter, adjustable) neighboring points is calculated, and the average distance is recorded as the neighbor distance of the point; then, the average and standard deviation of the neighbor distances of all points in the initial facial point cloud are calculated, and the average of the neighbor distances of all points is recorded as the average neighbor distance, and the standard deviation is recorded as the neighbor distance standard deviation. When the absolute value of the difference between the neighbor distance of the point and the average neighbor distance is greater than 2 times the neighbor distance standard deviation, the point is determined to be an outlier and is finally removed.

[0085] For example, the radius outlier removal method works as follows: for each point in the initial facial point cloud, the distances from other points in the initial facial point cloud to the current point are counted, and the number of points with a distance less than 5mm to the current point is recorded. These points with a distance less than 5mm are the current neighboring points, and the number of points with a distance less than 5mm is the number of neighbors of the current point. If the number of neighbors of the current point is less than 20, the current point is determined to be an outlier and is then removed.

[0086] In one example, the filtered initial facial point cloud C2 can be further pre-processed using an image segmentation method. Image segmentation methods include the nose tip method and a deep learning-based facial point cloud segmentation method, wherein the nose tip method is specifically as follows: first, the nose tip point in the initial facial point cloud C2 is estimated separately using the RANSAC algorithm, and then all points in the point cloud are traversed to obtain all points within 8 cm of the nose tip point, which form the segmented initial facial point cloud C2. Through image segmentation, the point cloud image only retains the facial information of the subject and further filters out more redundant information, thereby shortening the time to obtain the real-time location information of the target point. Of course, those skilled in the art can also implement facial point cloud segmentation based on existing deep learning methods, specifically using a trained image segmentation model to perform facial segmentation on the initial facial point cloud C2. This example is only an illustrative example, and those skilled in the art should not be understood as a limitation of the present invention.

[0087] In one example, the scalp point cloud C1 (eg, the facial model point cloud S mri_anterior ) with the initial facial point cloud C2 comprises the following steps:

[0088] Based on the downsampled facial model point cloud S mri_anterior Get the facial model point cloud S mri_anterior The facial model normal vectors of all points in ;

[0089] Obtaining initial facial normal vectors for all points in the initial facial point cloud C2 based on the downsampled initial facial point cloud C2;

[0090] Optimize all facial model normal vectors and all initial facial normal vectors based on the normal vector optimization method;

[0091] Obtain the facial model point cloud S based on the optimized facial model normal vector and the optimized initial facial normal vector mri_anterior The surface features of and the surface features of the initial facial point cloud C2;

[0092] Based on the facial model point cloud S mri_anterior The surface features of the scalp point cloud C1 and the surface features of the initial facial point cloud C2 are feature matched by the first registration method to obtain a heterogeneous point cloud spatial transformation relationship in the space of the initial facial point cloud C2 to convert the scalp point cloud C1.

[0093] Based on the subject's facial model point cloud S mri_anterior Get the facial model point cloud S mri_anterior The facial model normal vectors of all points in . The specific method includes the following steps:

[0094] Determine the facial model point cloud S by kd Tree search method mri_anterior all neighboring points within a k1 neighborhood of each point in , where the k1 neighborhood is a sphere with a length of a first predetermined multiple of a voxel of the MRI image of the subject's head as a radius;

[0095] The tangent plane of each point is obtained by fitting all the neighboring points in the k1 neighborhood of each point through the least squares method. The tangent plane is the plane with the minimum sum of the distances from all neighboring points to the fitted plane. The normal vector of the tangent plane of each point is the normal vector of the facial model of each point, denoted as n i (h,j,l).

[0096] For example, the facial model point cloud S is determined by the kd Tree search method mri_anterior All the neighboring points in the k1 neighborhood of any point k' in the face are searched. The k1 neighborhood is a sphere with point k' as the center and 2 times the voxel length as the radius. The neighboring points are all the points in the sphere that form a point pair with point k'. 20 neighboring points are obtained by searching. Then, the 20 neighboring points are fitted by the least squares method, and the tangent surface of point k' is obtained. The normal vector of the tangent surface is calculated, and the facial model normal vector of point k' is obtained, which is recorded as n k’ (h', j', l'), the facial model normal vector of point k' can also be called the normal line of point k'.

[0097] Since the number of neighboring points directly determines the quality of the fitted tangent surface, it is preferred to use a neighborhood with a radius of 2 times the voxel length. The tangent surface fitted by the number of neighboring points it covers is better, for example, making the facial model normal vector obtained later better, and then when performing FPFH feature extraction, features (i.e., descriptors) that are closer to or more matched to the subject's physical facial point cloud can be extracted.

[0098] The method for obtaining the initial facial normal vectors of all points in the initial facial point cloud C2 based on the initial facial point cloud C2 comprises the following steps:

[0099] Determine all neighboring points within a k2 neighborhood of each point in the entity facial point cloud using a kd Tree search method, where the k2 neighborhood is a sphere with a length of a second predetermined multiple of a voxel in the MRI image of the subject's head as a radius;

[0100] Based on the least squares fitting of all the neighboring points in the k2 neighborhood of each point, the tangent plane of each point is obtained and the normal vector of the tangent plane of each point in the entity facial point cloud is obtained. The normal vector of the tangent plane of each point in the initial facial point cloud C2 is the initial facial normal vector N cam , the initial facial normal vector N for each point camIt can also be called the normal of each point.

[0101] It can be seen that the method and principle of obtaining the initial facial normal vectors of all points in the initial facial point cloud C2 based on the initial facial point cloud C2 are similar to those based on the facial model point cloud S of the subject. mri_anterior Get the facial model point cloud S mri_anterior The method and principle of obtaining the facial model normal vectors of all points in are exactly the same and will not be repeated here.

[0102] In one example, the facial model point cloud S is obtained respectively mri_anterior The facial model normal vector n for each point i (h, j, l) and the initial facial normal vector N of each point of the initial facial point cloud C2 cam Afterwards, the normal vector n of the facial model is further adjusted by the normal vector optimization method. i (h,j,l) and the initial facial normal vector N cam , to ensure that the facial model point cloud S mri_anterior and the stability of the C2 features of the initial facial point cloud, thereby improving the quality of their respective FPFH descriptors.

[0103] The optimization method of the normal vector of the facial model includes the following steps:

[0104] Based on the facial model point cloud S mri_anterior Get the coordinate origin O to the facial model point cloud S mri_anterior The vector of each point in ;

[0105] Based on the inner product of the vector from the coordinate origin O to each point and the facial model normal vector of each point itself, and based on the value of the inner product, it is determined whether to adjust the direction of the facial model normal vector of the point. When the value of the inner product is greater than 0, the direction of the normal vector of the point is reversed, and when the value of the inner product is less than or equal to 0, the direction of the normal vector of the point is not adjusted.

[0106] This results in the facial model point cloud S mri_anterior The direction of the facial model normal vectors of all points is toward the coordinate origin O, that is, toward the direction of the subject’s skull. mri_anterior After the directions of the normal vectors of all points of the facial model are all facing the subject's skull, the surface features composed of the facial model point cloud can be more uniform and can be better distinguished from other noise point clouds, thereby improving the quality of the FPFH features of the facial model point cloud.

[0107] For example, for any point k', the vector from the origin O to the point k' is calculate and the facial model normal vector n of the point k' k’The inner product between (h', j', l'). When the inner product value is greater than 0, the facial model normal vector n of point k' is k’ (h', j', l') is negated to get the optimized facial model normal vector -n k’ (-h',-j',-l'). For example, the normal vector n of the facial model at any point k' k” When the value of (h', j', l') is ≤ 0, the facial model normal vector n of point k' is k” (h”, j”, l”) is the optimized normal vector of the facial model.

[0108] In one example, the initial facial normal vector N cam The optimization method includes the following steps:

[0109] The optical center of the camera is the coordinate origin O c Construct the camera coordinate system;

[0110] Obtain the coordinate origin O based on the initial facial point cloud C2 c A vector to each point in the initial facial point cloud C2;

[0111] Based on the coordinate origin O c The vector to each point in the initial facial point cloud C2 is respectively inner-producted with the initial facial normal vector of each point itself to obtain a solid inner-product value;

[0112] Based on the entity inner product value, it is determined whether to adjust the direction of the initial facial normal vector of the point corresponding to the entity inner product value. When the entity inner product value is greater than 0, the direction of the normal vector of the point corresponding to the entity inner product value is reversed.

[0113] The principle of the optimization method for the initial facial normal vector is exactly the same as that of the optimization method for the facial model normal vector, and will not be described in detail here.

[0114] When the directions of the initial facial normal vectors of all points in the initial facial point cloud C2 are all facing the subject's skull, the surface features formed by the initial facial point cloud C2 can be more uniform and can be better distinguished from other noise point clouds, thereby improving the quality of the FPFH features of the initial facial point cloud C2.

[0115] For example, in the camera coordinate system, its coordinate origin O c Set as the optical center of the camera. When the reconstructed initial facial point cloud C2 is located in the camera coordinate system, the point cloud of the nose tip is located close to the coordinate origin O c On one side, the initial facial normal vector N of each point in the initial facial point cloud C2 is cam The direction is adjusted to face the coordinate origin O c direction, which is equivalent to the initial facial normal vector N of all pointscam The direction of is adjusted to point from the subject's face to the optical center of the camera, and the adjusted initial facial normal vector N' is obtained for each point cam , and then the adjusted initial facial normal vector N′ of all points cam Take the inverse and get the optimized initial facial normal vector -N′ for each point cam , that is, the initial facial normal vectors of all points -N′ cam The direction is from the optical center of the camera to the subject's face, which is equivalent to pointing from the subject's face to the subject's skull.

[0116] After obtaining the optimized facial model normal vector and the optimized initial facial normal vector, surface features are extracted based on their respective normal vectors. In one example, the surface features are preferably FPFH features.

[0117] In one example, a method for obtaining FPFH features of a facial model point cloud based on an optimized facial model normal vector includes the following steps:

[0118] Determine the facial model point cloud S by kd Tree search method mri_anterior all pairs of points within a k3 neighborhood of each point in , where the k3 neighborhood is a sphere with a length of a third predetermined multiple of a voxel in the MRI image of the subject's head as a radius;

[0119] Based on the facial model point cloud S mri_anterior The FPFH feature of each point is obtained by combining all point pairs in the k3 neighborhood of each point in and the optimized facial model normal vector of each point in all point pairs.

[0120] For example, the facial model point cloud S is determined by the kd Tree search method mri_anterior All point pairs within the k3 neighborhood of any point k' in the image, where the k3 neighborhood is a sphere with point k' as the center and a radius of 5 times the length of the voxel in the MRI image of the subject's head, are constructed for point k' and the optimized facial model normal vector -n of point k' is obtained k’ (-h', -j', -l') and the optimized facial model normal vector n of point k' k” (h”, j”, l”); based on the point pair (k’, k”) and the optimized facial model normal vector of point k’ - n k’ (-h', -j', -l') and the optimized facial model normal vector n of point k' k”(h', j', l') respectively calculate the simplified point feature histogram SPFH(k') of point k' and the simplified point feature histogram SPFH(k") of point k', and then use the adjacent SPFH values ​​to calculate the FPFH feature (i.e., FPFH descriptor) FPFH(k'). Among them, the expression of FPFH(k') is:

[0121]

[0122] In formula (2), k represents the number of nearest neighbor points of point k', ω i Represents the weight, and the weight value is the distance from point k' to its neighboring point k".

[0123] Traverse the facial model point cloud S mri_anterior All points in , get the FPFH features of each point.

[0124] The method for obtaining the FPFH feature of the initial facial point cloud C2 based on the optimized initial facial normal vector includes the following steps:

[0125] Determine all point pairs within a k4 neighborhood of each point in the entity face point cloud by a kd Tree search method, where the k4 neighborhood is a sphere with a length of a fourth predetermined multiple (e.g., 5 times) of a voxel in the medical image of the subject as a radius;

[0126] The FPFH feature of each point is obtained based on all point pairs within the k4 neighborhood of each point in the entity facial point cloud and the optimized initial facial normal vector of each point in all point pairs.

[0127] That is to say, the method and principle of obtaining the FPFH features of the initial facial point cloud C2 based on the optimized initial facial normal vector are exactly the same as the method and principle of obtaining the FPFH features of the facial model point cloud based on the optimized facial model normal vector, so they will not be repeated here.

[0128] In one example, the facial model point cloud S based on each point mri_anterior The FPFH features of the scalp point cloud C1 and the FPFH features of the initial facial point cloud C2 at each point are matched by a coarse registration method (e.g., a RANSAC algorithm) to obtain an optimized spatial transformation relationship M0 in the space of the scalp point cloud C1 to the initial facial point cloud C2. The optimized spatial transformation relationship M0 includes an optimized translation transformation matrix and an optimized rotation transformation matrix.

[0129] Then, based on the optimized spatial transformation relationship M0, the optimized translation transformation matrix and the preliminary rotation transformation matrix (i.e., the optimized spatial transformation relationship M0) are further optimized through a precise registration method (e.g., an ICP algorithm) to obtain a heterogeneous spatial transformation relationship M1. Then, the scalp point cloud C1 and the initial facial point cloud C2 are spatially transformed based on the heterogeneous spatial transformation relationship M1, so as to subsequently calculate the real-time position information of the target.

[0130] For example, scalp point cloud C1 / face model point cloud S mri_anterior ) is registered to the initial facial point cloud C2 based on the optimized spatial matrix M0 (i.e., the optimized translation transformation matrix and the optimized rotation transformation matrix), and the coarsely registered point cloud C2 is obtained t , then roughly align the point cloud C2 t Use Point2Plane ICP algorithm to calculate scalp point cloud C1 / face model point cloud S mri_anterior Coarse registration point cloud C2 t The root mean square error of the registration between the two and the new registration matrix M0′ (new translation transformation matrix and new rotation transformation matrix), then the new registration matrix M0′ is iteratively coarsely registered with the registration matrix M0, and the current registration root mean square error is calculated again. When the root mean square error is less than 1mm, the iteration is stopped, and the registration matrix M0′ is the heterogeneous point cloud space transformation matrix M1 (including translation transformation matrix and rotation transformation matrix). When the point cloud C2′ obtained based on the transformation of the registration matrix M0′ t Scalp point cloud C1 / face model point cloud S mri_anterior If the root mean square error between them is still greater than 1mm, the registration matrix M0″ is obtained by the Point2Plane ICP algorithm based on the registration matrix M0′, and the registration matrix M0″ is iteratively registered with the matrix M0′ until a new point cloud is obtained to the scalp point cloud C1 / facial model point cloud S mri_anterior When the root mean square error is less than 1 mm, the iteration is stopped.

[0131] In one example, the convergence error criterion of the ICP algorithm can also be determined based on the error function. For example, when the error function M of two adjacent iterations is opt When the difference is less than 1.1 times the voxel length of the MRI image (three-dimensional image), the iteration is stopped, and the registration matrix obtained at this time is the heterogeneous point cloud registration matrix.

[0132] M opt =argmin R,t ∑ i ((R·x i +ty i )·n i ) 2 (3)

[0133] In formula (3), xi represents the i-th point in the point cloud obtained based on the new registration matrix, y i Represents scalp point cloud C1 / face model point cloud S mri_anterior Zhong and x i The closest point, R represents the rotation transformation matrix, t represents the translation transformation matrix, n i represents y i The normal vector of .

[0134] Those skilled in the art will appreciate that the coarse registration method can also be the Sampled Consistent Initial Registration (SAC-IA) algorithm, the Random Sampling Maximum Likelihood Algorithm (MLESAC), or the Progressive Consistent Sampling Algorithm (PROSAC) algorithm, which matches the two point clouds in the feature space to obtain the spatial transformation relationship between corresponding points in the two point clouds. Those skilled in the art can also use deep learning algorithms such as PointNet, PerfectMatch, or FCGF to extract surface features from the scalp point cloud C1 and the initial facial point cloud C2. This example is merely illustrative and should not be construed as limiting the present invention.

[0135] For example, a pre-trained deep learning model is used to extract FCGF (Fully Convolutional Geometric Features) features from the scalp point cloud C1 and the initial facial point cloud C2, respectively, or a pre-trained deep learning model is used to use a multi-layer perceptron (MLP) to extract local features of each point on the scalp point cloud, and to extract local features of each point on the initial facial point cloud C2. Then, the DGP (Deep Global Registration) method is used to align the scalp point cloud C1 and the initial facial point cloud C2 based on the extracted features, and the optimized spatial transformation relationship M0 is output.

[0136] For example, the MLESAC algorithm is used for registration, which specifically includes the following steps: First, randomly select the facial model point cloud S mri_anterior and the initial facial point cloud C2 as inliers; then, the scalp point cloud model and the initial point cloud model are fitted according to the inliers; then, the remaining subsets in the sample are taken as non-inliers, and the fitting errors between the respective non-inliers and the respective models are calculated; then, the probability distribution of the respective non-inliers belonging to their own models is estimated using the respective fitting errors; then, these non-inliers are recalculated according to the probability distribution as weights to obtain their respective new inliers; the above steps are repeated until the termination condition is met (such as reaching a certain number of iterations or the number of inliers has reached a certain proportion, etc.); finally, all the respective inliers are used to re-estimate the spatial transformation relationship between the respective new models and the two new models, and this spatial transformation relationship is the optimized spatial transformation relationship M0.

[0137] For example, the process of using the PROSAC algorithm is as follows: Initialization: From the facial model point cloud S mri_anterior Randomly extract some points from their respective data sets of the initial facial point cloud C2 as their respective initial inlier sets, and construct corresponding models at the same time; local sampling: for each inlier point, sample from its surrounding neighborhood to obtain a set of candidate inliers; estimate parameters: for each candidate inlier point in their respective candidate inliers, use the parameter estimation method to calculate their own model parameters and corresponding inliers; preliminary screening: sort all candidate models in their respective models according to their own number of inliers, and select the first part of the models as the candidate set of inlier models; ultimate judgment: for each candidate set of inlier models, use the model verification method to verify and obtain their respective final inlier sets; update the inlier set and model: use the obtained final inlier set to update the corresponding inlier set and the estimated model; iterative loop: if the obtained number of inliers is greater than the threshold, repeat the local sampling step to the step of updating the inlier set and model, otherwise output their respective final inlier set, their respective estimated models and the optimized spatial transformation relationship M0.

[0138] After the spatial transformation relationship M1 of the heterogeneous point cloud, the real-time position information v of the target on the initial facial point cloud C2 is obtained based on the initial position information of the target (for example, the real-time coordinate v1 of the target) and the spatial transformation relationship M1 of the heterogeneous point cloud. The expression of the real-time position information v of the target on the initial facial point cloud C2 is:

[0139] v=M1×v1 (4)

[0140] Then, a facial image of a frame adjacent to the initial facial image of the subject is obtained and a corresponding facial point cloud C3 is reconstructed. The facial point cloud C3 obtains a homologous point cloud spatial transformation relationship M2 based on a second registration method (eg, Point2Plane ICP algorithm).

[0141] The facial point cloud C3 is based on the unit matrix as the initial matrix, and is iteratively optimized by the Point2Plane ICP algorithm to finally obtain the homologous point cloud spatial transformation relationship M2. The method of optimizing the spatial transformation relationship between the facial point cloud C3 and the initial facial point cloud C2 by the Point2Plane ICP algorithm to obtain the homologous point cloud spatial transformation relationship M2 is completely consistent with the method and principle of further optimizing the above-mentioned optimized spatial transformation relationship M0 based on the optimized spatial transformation relationship M0 and obtaining the heterologous spatial transformation relationship M1 by the ICP algorithm, and will not be repeated here. Therefore, based on the real-time position information v of the target and the homologous point cloud spatial transformation relationship between the facial point cloud C3 and the initial facial point cloud C2, the real-time position information v′ of the target in the facial point cloud C3 can be obtained. The expression of the real-time position information v′ of the target in the facial point cloud C3 is:

[0142] v′=M2×v (5)

[0143] Similarly, the facial point cloud C4 of the adjacent frame of the facial point cloud C3 and the homologous point cloud spatial transformation relationship between the facial point cloud C4 and the facial point cloud C3 can also be obtained. Then, based on the real-time position information of the target in the facial point cloud C3 and the homologous point cloud spatial transformation relationship between the facial point cloud C4 and the facial point cloud C3, the real-time position information of the target in the facial point cloud C4 is obtained.

[0144] Thus, by obtaining the target's real-time position information in the previous frame and the spatial transformation relationship between the facial point cloud of the previous frame and the facial point cloud of the current frame, the target's real-time position information in the facial point cloud of the current frame can be derived, thus achieving real-time target tracking. In other words, it is only necessary to initially align the scalp point cloud with the initial facial point cloud, and then only track the point cloud displacement changes in adjacent frames to achieve real-time target tracking.

[0145] Those skilled in the art will appreciate that the technical term "initial facial image" herein should be broadly understood as the facial image first used, and should not be understood as the earliest facial image captured by the stereo camera. For example, the initial facial image can be the first facial image captured by the stereo camera, or the third facial image captured, or even the tenth facial image captured. When the initial facial image is the third facial image, the adjacent facial image is the fourth facial image, and so on, and no further details will be given here.

[0146] In one instance, two sub-threads can be designed. For example, thread 1 completes stereo camera shooting and point cloud generation, and thread 2 is responsible for registration. Then, the message queue Q is used to complete the collaborative work. First, Q is initialized to an empty value, and the tasks in the message queue Q are executed based on the first-in-first-out principle.

[0147] Among them, the workflow of thread 1 is:

[0148] (1) Send a request to the stereo camera to obtain the current RGBD depth facial image;

[0149] (2) Generate an initial facial point cloud based on the RGBD depth facial image;

[0150] (3) Preprocessing the point cloud (removing outliers and segmenting faces);

[0151] (4) Put the processed point cloud into queue Q.

[0152] Among them, the workflow of thread 2 is:

[0153] (1) Initialize the point cloud Pc to be registered as C2t in the previous step (coarse registration stage);

[0154] (2) Continuously monitor the queue Q and start alignment when the queue Q is not empty;

[0155] (3) Take out the point cloud Pj in the queue Q, use it to complete one ICP iteration, and check whether there is a point cloud in the queue Q.

[0156] In one example, a readable storage medium is provided according to another embodiment of the present invention. The "readable storage medium" of an embodiment of the present invention refers to any medium that participates in providing a program or instruction to a processor for execution. The medium can take a variety of forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks, such as storage devices. Volatile media include dynamic memory, such as main memory. Transmission media include coaxial cables, copper wires, and optical fibers, including wires comprising buses. Transmission media can also take the form of sound waves or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of readable storage media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic medium, CD-ROMs, DVDs, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cartridges, carrier waves as described below, or any other medium from which a computer can read.

[0157] The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the above-mentioned target real-time tracking method is executed.

[0158] In one example, the readable storage medium is arranged on a server in the form of a memory, such as a cloud server. The cloud server is also provided with a processor, and the processor executes the program or instruction stored in the memory. The processor may be a central processing unit (CPU). In one example, the cloud server may be a virtual server formed by mapping a physical server through virtualization technology. Among them, there may be one or more physical servers. When there are multiple physical servers, the cloud server may be a virtual server formed by mapping a server cluster through virtualization technology. The virtual server may also be one or more. In one example, the cloud server may be provided to users through a cloud platform.

[0159] The target real-time tracking method and the readable storage medium according to the embodiments of the present invention have at least one of the following advantages:

[0160] (1) The target real-time tracking method and readable storage medium provided by the present invention realizes the real-time output of the changes between frames during the process of real-time tracking of the target by collecting the first frame and performing heterogeneous point cloud registration with the point cloud to be registered, and then performing homologous point cloud registration on each subsequent frame with the previous frame, thereby achieving the purpose of real-time tracking of the target;

[0161] (2) The target real-time tracking method and readable storage medium provided by the present invention achieve real-time tracking of the target through the method of initial heterogeneous point cloud registration and subsequent homologous point cloud real-time registration;

[0162] (3) The real-time target tracking method and readable storage medium provided by the present invention track targets by combining heterogeneous point cloud registration with homologous point cloud registration, thereby improving the target tracking speed while ensuring the target tracking accuracy.

[0163] Although some embodiments of the present general inventive concept have been shown and described, it will be appreciated by those skilled in the art that changes may be made to these embodiments without departing from the principles and spirit of the present general inventive concept, the scope of which is defined in the claims and their equivalents.

Claims

1. A method for real-time tracking of a target, comprising the following steps: obtaining a scalp point cloud based on a medical imaging image of the subject's head, and obtaining an initial facial point cloud based on an initial facial image of the subject; registering the scalp point cloud with the initial facial point cloud based on a first registration method to obtain a heterogeneous point cloud spatial transformation relationship between the scalp point cloud and the initial facial point cloud; Obtaining real-time position information of the target on the initial facial point cloud based on the initial position information of the target and the spatial transformation relationship of the heterogeneous point cloud; The real-time position relationship of the target point in each frame of the facial point cloud is obtained based on the real-time position information of the target point on the initial facial point cloud and the homologous point cloud spatial transformation relationship between two adjacent frames of facial point clouds.

2. The target real-time tracking method according to claim 1, wherein: The method for obtaining the real-time position relationship of the target point in each frame of facial point cloud based on the real-time position information of the target point on the initial facial point cloud and the homologous point cloud spatial transformation relationship between two adjacent frames of facial point cloud comprises the following steps: Obtaining a real-time position relationship of the target point on the facial point cloud of the adjacent frame to the initial facial point cloud based on a homologous point cloud spatial transformation relationship between the facial point cloud of the adjacent frame to the initial facial point cloud and the initial facial point cloud and real-time position information of the target point on the initial facial point cloud; Iterate the previous step to obtain the real-time position relationship of the target in each frame of the facial point cloud.

3. The target real-time tracking method according to claim 2, wherein: The method for obtaining the spatial transformation relationship of homologous point clouds between two adjacent frames of facial point clouds is to align the surface features of the facial point cloud of the previous frame with the surface features of the facial point cloud of the current frame based on the second alignment method to obtain the spatial transformation relationship of homologous point clouds between the facial point cloud of the previous frame and the facial point cloud of the current frame.

4. The method for real-time target tracking according to any one of claims 1 to 3, wherein: The method for registering the scalp point cloud with the initial facial point cloud based on the first registration method comprises the following steps: Obtaining a facial model point cloud based on the scalp point cloud, and obtaining facial model normal vectors of all points in the facial model point cloud based on the facial model point cloud; Obtaining initial facial normal vectors for all points in the initial facial point cloud based on the initial facial point cloud; Optimize all facial model normal vectors and all initial facial normal vectors based on the normal vector optimization method; Obtaining surface features of the facial model point cloud and the surface features of the initial facial point cloud based on the optimized facial model normal vector and the optimized initial facial normal vector; Feature matching is performed based on the surface features of the facial model point cloud and the surface features of the initial facial point cloud using the first registration method to obtain the heterogeneous point cloud spatial transformation relationship for converting the scalp point cloud into the space of the initial facial point cloud.

5. The target real-time tracking method according to claim 4, wherein: The method for obtaining facial model normal vectors of all points in the facial model point cloud based on the facial model point cloud comprises the following steps: Determine all neighboring points within a k1 neighborhood of each point in the facial model point cloud using a kd Tree search method, where the k1 neighborhood is a sphere with a length of a first predetermined multiple of a voxel in the medical image of the subject as a radius; Based on all neighboring points in the k1 neighborhood of each point, the tangent plane of each point is obtained by least squares fitting and the normal vector of the tangent plane of each point in the facial model point cloud is obtained. The normal vector of the tangent plane of each point in the facial model point cloud is the facial model normal vector.

6. The target real-time tracking method according to claim 4, wherein: The method for obtaining initial facial normal vectors of all points in the initial facial point cloud based on the initial facial point cloud comprises the following steps: Determine all neighboring points within a k2 neighborhood of each point in the initial facial point cloud using a kd Tree search method, where the k2 neighborhood is a sphere with a length of a second predetermined multiple of a voxel in the medical image of the subject as a radius; Based on all neighboring points in the k2 neighborhood of each point, the least squares method is used to fit the tangent surface of each point and the normal vector of the tangent surface of each point in the initial facial point cloud is obtained. The normal vector of the tangent surface of each point in the initial facial point cloud is the initial facial normal vector.

7. The target real-time tracking method according to claim 4, wherein: The optimization method of the normal vector of the facial model includes the following steps: Reset the coordinates of the center point of the scalp and facial point cloud to the coordinate origin O and establish an XYZ coordinate system, in which the Y axis is parallel to the ground and its positive direction points to the front of the subject, the X axis is perpendicular to the Y axis and its positive direction points to the right of the subject, and the Z axis is perpendicular to the X and Y axes and its positive direction points to the top of the subject; Obtaining a vector from a coordinate origin O to each point in the facial model point cloud based on the facial model point cloud; Based on the inner product of the vector from the coordinate origin O to each point and the facial model normal vector of each point itself, it is determined whether to adjust the direction of the facial model normal vector of the point based on the value of the inner product. When the value of the inner product is greater than 0, the direction of the normal vector of the point is reversed.

8. The target real-time tracking method according to claim 7, wherein: The optimization method of the initial facial normal vector includes the following steps: The optical center of the camera is the coordinate origin O c Construct the camera coordinate system; Obtain the coordinate origin O based on the initial facial point cloud c A vector to each point in the initial facial point cloud; Based on the coordinate origin O c Inner product of the vector to each point in the initial facial point cloud with the initial facial normal vector of each point itself to obtain a solid inner product value; Based on the entity inner product value, it is determined whether to adjust the direction of the initial facial normal vector of the point corresponding to the entity inner product value. When the entity inner product value is greater than 0, the direction of the normal vector of the point corresponding to the entity inner product value is reversed.

9. The target real-time tracking method according to claim 4, wherein: The facial model point cloud and the initial facial point cloud are downsampled to obtain a downsampled facial model point cloud and a downsampled initial facial point cloud, and then the facial model normal vector and the initial facial normal vector are obtained based on the downsampled facial model point cloud and the downsampled initial facial point cloud. The downsampling method comprises the following steps: rasterizing the spatial voxels of the facial model point cloud and the initial facial point cloud; Determining whether each voxel grid after the facial model point cloud is rasterized contains a point in the facial model point cloud; if the current voxel grid contains a point in the facial model point cloud, selecting a point closest to the center point of the current voxel grid from the facial model point cloud as a sampling point, traversing the facial model point cloud to obtain a first sampling point set, where the first sampling point set is the downsampled facial model point cloud; Determine whether each voxel grid after the initial facial point cloud is rasterized contains a point in the initial facial point cloud. When the current voxel grid contains a point in the initial facial point cloud, select the point closest to the center point of the current voxel grid from the initial facial point cloud as a sampling point, traverse the initial facial point cloud to obtain a second sampling point set, and the second sampling point set is the downsampled initial facial point cloud.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the real-time target tracking method according to any one of claims 1 to 9 is executed.

Images