Target point position tracking method and apparatus

Abstract

The application provides a target position tracking method and device, comprising: registering an initial face point cloud image of a target person and a face three-dimensional model to obtain a first transformation matrix, the initial face point cloud image being obtained by a structured light camera, and the face three-dimensional model being obtained by reconstructing a face medical image of the target person; transforming a coordinate of a target point in the face three-dimensional model according to the first transformation matrix to obtain a first physical position of the target point; in the case that the target person is detected to move, registering a current face point cloud image of the target person with the initial face point cloud image to obtain a second transformation matrix, the current face point cloud image being obtained by the structured light camera; and transforming the first physical position according to the second transformation matrix to obtain a second physical position of the target point. The application realizes real-time tracking of the physical position of the target point according to the movement of the target person, and the method is simple and has high accuracy.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a target location tracking method and apparatus. Background Technology

[0002] Neuronavigation technology combines various advanced technologies such as digital image processing and surgical instrument tracking and positioning. In the field of neuronavigation, the precise and rapid localization and tracking of target points is a primary research objective.

[0003] Surgical registration, a core component of neuronavigation, aims to precisely match medical imaging data with the patient's anatomy. Neuronavigation utilizes real-time imaging and tracking data to provide real-time feedback on target location and surgical instrument placement, thereby achieving high-precision navigation.

[0004] The most accurate neuronavigation method for surgical registration currently available uses a structured light camera to capture the geometric information of object surfaces in a scene, enabling precise measurement and reconstruction of the scene's three-dimensional shape. During surgery, this method performs a structured light scan on the patient's head to obtain the geometric information of the head surface, and then precisely registers this information with a pre-acquired medical image model, thereby achieving a match between physical space and image space.

[0005] The advantage of this neuronavigation method lies in its highly accurate registration process and the fact that it requires no additional markers. However, it requires the patient's head to remain relatively still during the measurement process to ensure the accuracy of the results. This means that if the patient moves during the procedure, the target location cannot be tracked in real time, thus making it impossible to complete the neuronavigation task. Summary of the Invention

[0006] This invention provides a target position tracking method and apparatus to overcome the shortcomings of existing technologies that cannot track target positions in real time, thereby enabling real-time tracking of target positions.

[0007] This invention provides a target location tracking method, comprising:

[0008] The initial facial point cloud image and the 3D facial model of the target person are registered to obtain the first transformation matrix. The initial facial point cloud image is acquired by a structured light camera, and the 3D facial model is reconstructed from the facial medical image of the target person.

[0009] The coordinates of the target point in the three-dimensional face model are transformed according to the first transformation matrix to obtain the first physical position of the target point;

[0010] When the movement of the target person is detected, the current face point cloud image of the target person is registered with the initial face point cloud image to obtain a second transformation matrix. The current face point cloud image is acquired by the structured light camera.

[0011] The second physical position of the target point is obtained by transforming the first physical position according to the second transformation matrix.

[0012] According to a target location tracking method provided by the present invention, before registering the current face point cloud image of the target person with the initial face point cloud image to obtain the second transformation matrix when the movement of the target person is detected, the method further includes:

[0013] The face image of the target person and the corresponding face point cloud image are acquired in real time. The face image and the corresponding face point cloud image are acquired by a time-of-flight camera.

[0014] Detect a first facial key point from the face image, and obtain the first facial key point point cloud corresponding to the first facial key point from the face point cloud image corresponding to the face image;

[0015] Determine the distance between the first facial key point cloud in the face point cloud images captured by the time-of-flight camera at the current moment and the previous moment;

[0016] If the distance is greater than a preset threshold, the movement of the target person is determined.

[0017] According to a target location tracking method provided by the present invention, when movement of the target person is detected, the current face point cloud image of the target person is registered with the initial face point cloud image to obtain a second transformation matrix, including:

[0018] When the movement of the target person is detected, the current face point cloud image is segmented to obtain the current face point cloud, and the initial face point cloud image is segmented to obtain the initial face point cloud.

[0019] The current face point cloud and the initial face point cloud are registered to obtain the second transformation matrix.

[0020] According to a target location tracking method provided by the present invention, the step of registering the current face point cloud and the initial face point cloud to obtain a second transformation matrix includes:

[0021] The current face point cloud and the initial face point cloud are registered based on the iterative nearest point algorithm to obtain the second transformation matrix.

[0022] According to a target location tracking method provided by the present invention, the method involves registering an initial facial point cloud image and a 3D facial model of the target person to obtain a first transformation matrix, including:

[0023] The initial face point cloud image of the target person is segmented to obtain the initial face point cloud;

[0024] A first frontal face planar image is obtained based on the initial face point cloud; a second face key point is detected from the first frontal face planar image; and a second face key point cloud corresponding to the second face key point is obtained from the initial face point cloud.

[0025] Determine the second frontal face plane image corresponding to the three-dimensional face model, detect the third facial key points from the second frontal face plane image, and obtain the third facial key point point cloud corresponding to the third facial key points from the three-dimensional face model;

[0026] The second facial key point cloud and the third facial key point cloud are registered to obtain the first transformation matrix.

[0027] According to a target location tracking method provided by the present invention, the registration of the second facial key point cloud and the third facial key point cloud to obtain the first transformation matrix includes:

[0028] The second facial key point cloud and the third facial key point cloud are registered;

[0029] Determine the first target registration error between the registered second facial key point cloud and the registered third facial key point cloud;

[0030] From the registered third face key point cloud obtained from the face 3D model, select points multiple times within a preset neighborhood range of each point as new registration points;

[0031] The registered second facial key point cloud and the newly selected registration point are registered together, and the second target registration error between the registered second facial key point cloud and the newly selected registration point is determined.

[0032] The minimum value is selected from the second target registration errors corresponding to the newly selected registration points. If the minimum value is less than the first target registration error, the transformation matrix generated in the registration corresponding to the minimum value is used as the first transformation matrix.

[0033] The present invention also provides a target position tracking device, comprising:

[0034] The first registration module is used to register the initial face point cloud image and the face 3D model of the target person to obtain the first transformation matrix. The initial face point cloud image is acquired by a structured light camera, and the face 3D model is obtained by reconstructing the face medical image of the target person.

[0035] The first transformation module is used to transform the coordinates of the target point in the three-dimensional face model according to the first transformation matrix to obtain the first physical position of the target point;

[0036] The second registration module is used to register the current face point cloud image of the target person with the initial face point cloud image when the movement of the target person is detected, so as to obtain a second transformation matrix. The current face point cloud image is acquired by the structured light camera.

[0037] The second transformation module is used to transform the first physical position according to the second transformation matrix to obtain the second physical position of the target point.

[0038] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target position tracking method as described above.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the target position tracking method as described above.

[0040] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the target position tracking method as described above.

[0041] The target location tracking method and apparatus provided by this invention first registers an initial face point cloud image captured by a structured light camera with a 3D face model reconstructed from a face medical image. The first physical position of the target is obtained using a first transformation matrix obtained from the registration. When the target moves, the face point cloud image is captured again using the structured light camera. The current face point cloud image is then re-registered with the initial face point cloud image. The first physical position of the target is adjusted using a second transformation matrix obtained from the re-registration. This enables real-time tracking of the target's physical position based on the target's movement, without the need for external markers. The method is simple and highly accurate. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1 This is one of the flowcharts illustrating the target location tracking method provided by the present invention;

[0044] Figure 2 This is the second flowchart illustrating the target location tracking method provided by the present invention;

[0045] Figure 3 This is a schematic diagram of the target position tracking device provided by the present invention;

[0046] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0048] The following is combined Figure 1 A target location tracking method according to the present invention includes:

[0049] Step 101: Register the initial face point cloud image and the face 3D model of the target person to obtain the first transformation matrix. The initial face point cloud image is acquired by a structured light camera, and the face 3D model is obtained by reconstructing the face medical image of the target person.

[0050] The target person can be a patient who needs to be tracked for the location of the target.

[0051] A structured light camera was used to capture a point cloud image of the target person's face. The first captured point cloud image of the target person's face was used as the initial point cloud image. The initial point cloud image is a three-dimensional point cloud in a physical space coordinate system.

[0052] Structured light cameras can quickly acquire 3D shape information of an object's surface with high precision, exceeding that of other types of sensors. Compared to laser navigation systems, structured light cameras can obtain large amounts of point cloud data in a shorter time, demonstrating their superior speed and accuracy. Furthermore, structured light cameras can operate in low-light conditions, while other optical navigation systems suffer from light interference problems.

[0053] Facial medical images can be MRI (Magnetic Resonance Imaging) images. A 3D facial model is obtained by modeling the face using these medical images of the target person. The 3D facial model is a 3D point cloud in the medical image coordinate system.

[0054] The initial face point cloud image captured by the structured light camera is registered with the 3D face model obtained from face medical image modeling. During the registration process, the 3D face model is spatially transformed so that the spatially transformed 3D face model overlaps with the initial face point cloud image as much as possible.

[0055] The transformation matrix used in the final spatial transformation of the 3D face model is used as the first transformation matrix, enabling the target person to complete spatial registration without relying on external tools.

[0056] Step 102: Transform the coordinates of the target point in the three-dimensional face model according to the first transformation matrix to obtain the first physical position of the target point;

[0057] In the 3D face model, the target point represents the location of the lesion on the subject. The coordinates of the target point in the 3D face model are in the medical image coordinate system. By performing a spatial transformation on the target point coordinates according to the first transformation matrix, the coordinates of the target point in the physical space coordinate system are obtained, which is the first physical position of the target point.

[0058] Step 103: When the movement of the target person is detected, the current face point cloud image of the target person is registered with the initial face point cloud image to obtain a second transformation matrix. The current face point cloud image is acquired by the structured light camera.

[0059] The method of monitoring whether the target person moves in real time is not limited in the embodiments of this application.

[0060] If movement of the target person is detected, a structured light camera is used to re-acquire the current facial point cloud image of the target person.

[0061] The current face point cloud image is registered with the initial face point cloud image. During the registration process, the initial face point cloud image is spatially transformed so that the spatially transformed initial face point cloud image overlaps with the current face point cloud image as much as possible.

[0062] The transformation matrix used for the final spatial transformation of the initial face point cloud image is used as the second transformation matrix.

[0063] Step 104: Transform the first physical position according to the second transformation matrix to obtain the second physical position of the target point.

[0064] The first physical position of the target point is spatially transformed according to the second transformation matrix to obtain the second physical position of the target point in the physical space coordinate system, so as to apply the registration result to neural navigation.

[0065] This embodiment first registers an initial face point cloud image captured by a structured light camera with a 3D face model reconstructed from a face medical image. The first transformation matrix obtained from the registration is used to obtain the first physical position of the target point. When the target moves, the structured light camera captures another face point cloud image, and the current face point cloud image is re-registered with the initial face point cloud image. The second transformation matrix obtained from the re-registration is used to adjust the first physical position of the target point. This achieves real-time tracking of the physical position of the target point based on the movement of the target, without the need for external markers. The method is simple and highly accurate.

[0066] Based on the above embodiments, before registering the current face point cloud image of the target person with the initial face point cloud image to obtain the second transformation matrix when the target person's movement is detected, this embodiment further includes:

[0067] The face image of the target person and the corresponding face point cloud image are acquired in real time. The face image and the corresponding face point cloud image are acquired by a TOF (Time of Flight) camera.

[0068] The TOF camera employs stereo vision imaging technology. It continuously captures RGB images and point cloud images of the target person's face.

[0069] Detect a first facial key point from the face image, and obtain the first facial key point point cloud corresponding to the first facial key point from the face point cloud image corresponding to the face image;

[0070] PFLD (A Practical Facial Landmark Detector) can be used to extract facial landmarks from face images.

[0071] Key facial features can include seven points: the inner left corner of the eye, the outer left corner of the eye, the inner right corner of the eye, the outer right corner of the eye, the root of the nose, the tip of the nose, and the cupid's bow.

[0072] The two-dimensional coordinates (x, y) of each facial key point are compared with the corresponding two-dimensional coordinates (x, y) in each three-dimensional coordinate of the facial point cloud image. The three-dimensional coordinates that are closest to the facial key points are selected from the three-dimensional coordinates of the facial point cloud image and used as the facial key point point cloud.

[0073] The 3D coordinates of the 7 points can be updated using Kalman filtering to obtain a point cloud of facial key points for neural navigation.

[0074] Determine the distance between the first facial key point cloud in the face point cloud images acquired by the TOF camera at the current moment and the previous moment;

[0075] The distance d between the first facial landmarks in the face point cloud images acquired by the TOF camera at the current time and the previous time can be calculated using the following formula:

[0076]

[0077] Where n is the number of first key points corresponding to the first facial key point cloud, (x i2 ,yi2,z i2 (x) represents the three-dimensional coordinates of the i-th first keypoint at the current time. i1 ,yi1,z i1 ) represents the three-dimensional coordinates of the i-th first key point at the previous moment.

[0078] If the distance is greater than a preset threshold, the movement of the target person is determined.

[0079] If the range of the jitter of the first facial key point cloud is less than or equal to a preset threshold, that is, if the distance between the first facial key point clouds at two consecutive moments is less than or equal to the preset threshold, it is impossible to determine whether the target person has moved, and this situation is ignored.

[0080] The preset threshold is set based on experience, such as 2mm.

[0081] When the distance between the first facial key point cloud at two different time points is greater than a preset threshold, it is determined that the target person has moved. The face point cloud image is then captured again using a structured light camera to update the physical position of the target point.

[0082] This application embodiment accurately determines whether the target person has moved by using the distance between the first facial key point cloud in the face point cloud image acquired by the TOF camera at the current time and the previous time. Once the target person has moved, the face point cloud image is captured again using a structured light camera to update the physical position of the target point. This achieves real-time tracking of the physical position of the target point based on the movement of the target person. The method is simple and has high tracking accuracy.

[0083] Based on the above embodiments, in this embodiment, when the movement of the target person is detected, the current face point cloud image of the target person is registered with the initial face point cloud image to obtain a second transformation matrix, including:

[0084] When the movement of the target person is detected, the current face point cloud image is segmented to obtain the current face point cloud, and the initial face point cloud image is segmented to obtain the initial face point cloud.

[0085] If movement of the target person is detected, the face point cloud image is captured again using a structured light camera, which is the current face point cloud image.

[0086] The PointNet++ point cloud segmentation algorithm can be used to segment the current face point cloud from the current face point cloud image, and to segment the initial face point cloud from the initial face point cloud image.

[0087] The current face point cloud and the initial face point cloud are registered to obtain the second transformation matrix.

[0088] The current face point cloud is registered with the initial face point cloud. During the registration process, the initial face point cloud is spatially transformed so that the spatially transformed initial face point cloud overlaps with the current face point cloud as much as possible.

[0089] The transformation matrix used in the final spatial transformation of the initial face point cloud is used as the second transformation matrix. The physical position of the target point is updated using the second transformation matrix, thereby completing neural navigation.

[0090] Based on the above embodiments, the registration of the current face point cloud and the initial face point cloud to obtain the second transformation matrix in this embodiment includes:

[0091] The current face point cloud and the initial face point cloud are registered using the ICP (Iterative Closest Point) algorithm to obtain the second transformation matrix.

[0092] Based on the above embodiments, the registration of the initial facial point cloud image and the 3D facial model of the target person to obtain the first transformation matrix in this embodiment includes:

[0093] The initial face point cloud image of the target person is segmented to obtain the initial face point cloud;

[0094] The initial face point cloud image is obtained by capturing the scene where the target person is located using a structured light camera. The initial face point cloud image includes the complete face point cloud of the target person.

[0095] The PointNet++ point cloud segmentation algorithm is used to segment the face point cloud from the initial face point cloud image to obtain the initial face point cloud.

[0096] A first frontal face planar image is obtained based on the initial face point cloud; a second face key point is detected from the first frontal face planar image; and a second face key point cloud corresponding to the second face key point is obtained from the initial face point cloud.

[0097] The initial face point cloud was read using the VTK (Visualization Toolkit) open-source software, and the 3D coordinate information of all points in the initial face point cloud was obtained. The initial face point cloud was rotated so that the target person's face was facing the display screen, and a screenshot was taken to obtain the first frontal face image.

[0098] The PFLD algorithm is used to extract the second facial key points from the first frontal face planar image.

[0099] The key points of the second face may include seven points: the left inner corner of the eye, the left outer corner of the eye, the right inner corner of the eye, the right outer corner of the eye, the root of the nose, the tip of the nose, and the cupid's bow.

[0100] The two-dimensional coordinates (x, y) of each second facial key point are compared with the corresponding two-dimensional coordinates (x, y) in each three-dimensional coordinate of the initial facial point cloud. The three-dimensional coordinates that are closest to the second facial key point are selected from the three-dimensional coordinates of the initial facial point cloud and used as the second facial key point point cloud.

[0101] Determine the second frontal face plane image corresponding to the three-dimensional face model, detect the third facial key points from the second frontal face plane image, and obtain the third facial key point point cloud corresponding to the third facial key points from the three-dimensional face model;

[0102] A 3D model of the face was obtained by reconstructing a 3D model from MRI medical images. The VTK open-source software was then used to read the 3D model and obtain the 3D coordinate information of all points in the 3D model.

[0103] Rotate the 3D model of the face to the display screen so that the face is facing the screen, and take a screenshot to obtain a second frontal plane image.

[0104] The PFLD algorithm is used to extract the key points of the third face from the second frontal face plane image.

[0105] The key points of the third face may include seven points: the left inner corner of the eye, the left outer corner of the eye, the right inner corner of the eye, the right outer corner of the eye, the root of the nose, the tip of the nose, and the cupid's bow.

[0106] The two-dimensional coordinates (x, y) of each third facial key point are compared with the corresponding two-dimensional coordinates (x, y) in each three-dimensional coordinate of the face 3D model. The three-dimensional coordinate point that is closest to the third facial key point is selected from the three-dimensional coordinates of the face 3D model and used as the point cloud of the third facial key point.

[0107] Since the 3D model of a face contains point cloud information of the entire head, there will be multiple 3D points with different z values ​​for the same x and y values. The z values ​​of the selected 3D coordinate points are sorted, and the point with the largest z value is selected as the facial key point cloud in the medical image space.

[0108] The second facial key point cloud and the third facial key point cloud are registered to obtain the first transformation matrix.

[0109] The second and third facial landmark point clouds are registered. During the registration process, the third facial landmark point cloud is spatially transformed so that the spatially transformed third facial landmark point cloud overlaps with the second facial landmark point cloud as much as possible.

[0110] This application embodiment improves the accuracy of target point location tracking by registering the second facial key point cloud in the initial facial point cloud image captured by the structured light camera with the third facial key point cloud in the three-dimensional facial model reconstructed from the facial medical image.

[0111] Based on the above embodiments, the registration of the second facial key point cloud and the third facial key point cloud to obtain the first transformation matrix in this embodiment includes:

[0112] The second facial landmark point cloud and the third facial landmark point cloud are registered;

[0113] SAC-IA (Sample Consensus Initial Alignment) was used to coarsely register the point cloud of the second facial landmark in the physical space coordinate system and the point cloud of the third facial landmark in the medical image coordinate system.

[0114] The second and third facial key point clouds were re-registered based on the ICP algorithm.

[0115] The ICP algorithm was used to perform fine registration on the coarsely registered second and third facial landmark point clouds.

[0116] Determine the first target registration error (TRE) between the registered second facial key point cloud and the registered third facial key point cloud;

[0117] The formula for target registration error is as follows:

[0118]

[0119] Where, p i Let q represent the i-th point in the reference point cloud. i This represents the nearest point on the target point cloud after spatial transformation, and N represents the total number of points in the reference point cloud.

[0120] From the registered third face key point cloud obtained from the face 3D model, select points multiple times within a preset neighborhood range of each point as new registration points;

[0121] For each point in the registered third-party facial key point cloud, search for all points within a preset neighborhood of that point in the 3D facial model, such as a spherical neighborhood with a radius of 2mm centered on that point.

[0122] Each time, select one point from the preset neighborhood of each point as a new registration point. Register all the newly selected registration points with the second facial key point cloud.

[0123] The registered second facial key point cloud and the newly selected registration point are registered together, and the second target registration error between the registered second facial key point cloud and the newly selected registration point is determined.

[0124] After registering the second facial landmark cloud with each newly selected registration point using the ICP algorithm, the second target registration error between the second facial landmark cloud and each newly selected registration point is calculated.

[0125] The minimum value is selected from the second target registration errors corresponding to the newly selected registration points. If the minimum value is less than the first target registration error, the transformation matrix generated in the registration corresponding to the minimum value is used as the first transformation matrix.

[0126] Assume that the minimum value among all second-target registration errors is less than the first-target registration error, and the corresponding transformation matrix is ​​T. T consists of a 3*3 rotation matrix R and a 3*1 translation vector t. For coordinate P1 in the medical influence coordinate system, the formula for calculating the corresponding coordinate P2 in the physical space coordinate system can be expressed as:

[0127] P2 = R * P1 + t.

[0128] Figure 2 A complete flowchart of target location tracking provided in the embodiments of this application is shown below. Figure 2As shown, the main steps of this process are as follows: 1. Acquire face point cloud images using a structured light camera; 2. Segment the face point cloud using the PointNet++ algorithm; 3. Detect face key points using the PFLD algorithm to obtain a 3D point cloud in the physical space coordinate system; 4. Adjust the angle of the 3D model built using the target person's image to obtain a 3D model planar view; 5. Detect face key points in the 3D model planar view using the PFLD algorithm; 6. Map the face key points in the planar view to the original 3D model space to obtain a 3D point cloud of feature points in the medical image coordinate system; 7. Perform spatial registration on the two sets of point clouds using a marker-based matching algorithm to locate the target point; 8. Acquire face images using a TOF camera and detect face key points; 9. Determine if the face key points have moved. If they have moved, re-capture the face point cloud using a structured light camera and re-register the spatial data. Continue detecting face movement until the task ends, completing the neural navigation.

[0129] In this embodiment of the invention, point cloud images of the target person were captured from different angles and distances, and spatial registration was performed. The results are shown in Table 1.

[0130] Table 1 Space Registration Results

[0131] SAC-IA coarse registration TRE / mm ICP precision matching TRE / mm After traversal, TRE / mm is finally registered. 1 4.214 3.533 0.152 2 4.124 3.631 0.159 3 4.126 3.623 0.150 4 5.015 3.957 0.314 5 3.495 2.854 0.232 6 5.178 4.521 0.167 7 6.342 5.103 0.370 8 4.018 3.560 0.066 9 5.167 4.713 0.094 10 4.388 3.826 0.133 11 4.962 4.393 0.245 12 5.783 5.688 0.243 13 5.645 5.361 0.115 14 5.963 5.400 0.078 15 5.547 4.327 0.172 16 5.890 5.405 0.158 17 6.185 5.661 0.161 18 5.723 5.560 0.097 19 4.635 4.017 0.134 20 6.121 5.607 0.063 mean 5.127 4.537 0.166

[0132] During neural navigation, a TOF camera was used to track facial key points. RGB images and depth maps were fused to obtain a 3D point cloud of key points for each frame, and the average distance was calculated. To simulate the need for re-registration during surgery, 500 consecutive frames of point cloud data were used to test segmentation speed and face registration performance. Face segmentation of 500 frames of point cloud data took a total of 284 seconds, with an average segmentation time of 0.568 seconds per frame, meeting the requirements for real-time segmentation. To test face registration performance, to simulate scenarios with movement at different time intervals, point cloud registration was tested at intervals of 1, 5, 10, and 30 frames. The test results are shown in Table 2. The average error of face registration after movement was 0.039 mm, with very little impact on target re-localization. The device parameters of the structured light camera and TOF camera used are shown in Table 3.

[0133] Table 2. Results of Neural Navigation Face Registration

[0134]

[0135] Table 3 Main performance parameters of the camera

[0136] Camera type 3D structured light camera TOF camera Measuring distance <1m <100m resolution 1680x1200 1024x1024 Frame rate 20fps 30fps Measurement accuracy <0.25mm <2mm

[0137] The target position tracking device provided by the present invention is described below. The target position tracking device described below can be referred to in correspondence with the target position tracking method described above.

[0138] like Figure 3 As shown, the device includes a first registration module 301, a first transformation module 302, a second registration module 303, and a second transformation module 304, wherein:

[0139] The first registration module 301 is used to register the initial face point cloud image and the face 3D model of the target person to obtain a first transformation matrix. The initial face point cloud image is acquired by a structured light camera, and the face 3D model is obtained by reconstructing the face medical image of the target person.

[0140] The first transformation module 302 is used to transform the coordinates of the target point in the three-dimensional face model according to the first transformation matrix to obtain the first physical position of the target point;

[0141] The second registration module 303 is used to register the current face point cloud image of the target person with the initial face point cloud image when the movement of the target person is detected, so as to obtain a second transformation matrix. The current face point cloud image is acquired by the structured light camera.

[0142] The second transformation module 304 is used to transform the first physical position according to the second transformation matrix to obtain the second physical position of the target point.

[0143] This invention first registers an initial face point cloud image captured by a structured light camera with a 3D face model reconstructed from a face medical image. The first transformation matrix obtained from the registration is used to determine the first physical position of the target point. If the target moves, the structured light camera captures another face point cloud image, which is then re-registered with the initial face point cloud image. The second transformation matrix obtained from this re-registration is used to adjust the first physical position of the target point. This allows for real-time tracking of the target's physical position based on its movement, without the need for external markers. The method is simple and highly accurate.

[0144] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a target location tracking method, which includes: registering an initial face point cloud image and a 3D face model of the target person to obtain a first transformation matrix, wherein the initial face point cloud image is acquired by a structured light camera, and the 3D face model is reconstructed from the target person's face medical image; transforming the coordinates of the target point in the 3D face model according to the first transformation matrix to obtain a first physical position of the target point; when movement of the target person is detected, registering the current face point cloud image of the target person with the initial face point cloud image to obtain a second transformation matrix, wherein the current face point cloud image is acquired by a structured light camera; and transforming the first physical position according to the second transformation matrix to obtain a second physical position of the target point.

[0145] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the target location tracking method provided by the above methods. The method includes: registering an initial face point cloud image and a three-dimensional face model of the target person to obtain a first transformation matrix. The initial face point cloud image is acquired by a structured light camera, and the three-dimensional face model is reconstructed from the face medical image of the target person; transforming the coordinates of the target point in the three-dimensional face model according to the first transformation matrix to obtain a first physical position of the target point; when the target person is detected to be moving, registering the current face point cloud image of the target person with the initial face point cloud image to obtain a second transformation matrix. The current face point cloud image is acquired by a structured light camera; transforming the first physical position according to the second transformation matrix to obtain a second physical position of the target point.

[0147] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the target location tracking method provided by the above methods. This method includes: registering an initial facial point cloud image and a three-dimensional facial model of a target person to obtain a first transformation matrix; the initial facial point cloud image is acquired using a structured light camera, and the three-dimensional facial model is reconstructed from a medical image of the target person's face; transforming the coordinates of a target point in the three-dimensional facial model according to the first transformation matrix to obtain a first physical position of the target point; when movement of the target person is detected, registering the current facial point cloud image of the target person with the initial facial point cloud image to obtain a second transformation matrix; the current facial point cloud image is acquired using a structured light camera; and transforming the first physical position according to the second transformation matrix to obtain a second physical position of the target point.

[0148] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target position tracking method, characterized in that, include: The initial facial point cloud image and the 3D facial model of the target person are registered to obtain the first transformation matrix. The initial facial point cloud image is acquired by a structured light camera, and the 3D facial model is reconstructed from the facial medical image of the target person. The coordinates of the target point in the three-dimensional face model are transformed according to the first transformation matrix to obtain the first physical position of the target point; When the movement of the target person is detected, the current face point cloud image of the target person is registered with the initial face point cloud image to obtain a second transformation matrix. The current face point cloud image is acquired by the structured light camera. The first physical position is transformed according to the second transformation matrix to obtain the second physical position of the target point; Before registering the current face point cloud image of the target person with the initial face point cloud image to obtain the second transformation matrix, the method further includes: The face image of the target person and the corresponding face point cloud image are acquired in real time. The face image and the corresponding face point cloud image are acquired by a time-of-flight camera. The time-of-flight camera employs stereoscopic vision imaging technology and is used to continuously capture facial images and facial point cloud images of the target person.

2. The target position tracking method according to claim 1, characterized in that, Before registering the current face point cloud image of the target person with the initial face point cloud image to obtain the second transformation matrix when the target person's movement is detected, the method further includes: Detect a first facial key point from the face image, and obtain the first facial key point point cloud corresponding to the first facial key point from the face point cloud image corresponding to the face image; Determine the distance between the first facial key point cloud in the face point cloud images captured by the time-of-flight camera at the current moment and the previous moment; If the distance is greater than a preset threshold, the movement of the target person is determined.

3. The target position tracking method according to claim 1, characterized in that, When movement of the target person is detected, the current facial point cloud image of the target person is registered with the initial facial point cloud image to obtain a second transformation matrix, including: When the movement of the target person is detected, the current face point cloud image is segmented to obtain the current face point cloud, and the initial face point cloud image is segmented to obtain the initial face point cloud. The current face point cloud and the initial face point cloud are registered to obtain the second transformation matrix.

4. The target position tracking method according to claim 3, characterized in that, The step of registering the current face point cloud and the initial face point cloud to obtain the second transformation matrix includes: The current face point cloud and the initial face point cloud are registered based on the iterative nearest point algorithm to obtain the second transformation matrix.

5. The target position tracking method according to any one of claims 1-4, characterized in that, The process of registering the initial facial point cloud image and the 3D facial model of the target person to obtain the first transformation matrix includes: The initial face point cloud image of the target person is segmented to obtain the initial face point cloud; A first frontal face planar image is obtained based on the initial face point cloud; a second face key point is detected from the first frontal face planar image; and a second face key point cloud corresponding to the second face key point is obtained from the initial face point cloud. Determine the second frontal face plane image corresponding to the three-dimensional face model, detect the third facial key points from the second frontal face plane image, and obtain the third facial key point point cloud corresponding to the third facial key points from the three-dimensional face model; The second facial key point cloud and the third facial key point cloud are registered to obtain the first transformation matrix.

6. The target position tracking method according to claim 5, characterized in that, The registration of the second facial key point cloud and the third facial key point cloud to obtain the first transformation matrix includes: The second facial key point cloud and the third facial key point cloud are registered; Determine the first target registration error between the registered second facial key point cloud and the registered third facial key point cloud; From the registered third face key point cloud obtained from the face 3D model, select points multiple times within a preset neighborhood range of each point as new registration points; The registered second facial key point cloud and the newly selected registration point are registered together, and the second target registration error between the registered second facial key point cloud and the newly selected registration point is determined. The minimum value is selected from the second target registration errors corresponding to the newly selected registration points. If the minimum value is less than the first target registration error, the transformation matrix generated in the registration corresponding to the minimum value is used as the first transformation matrix.

7. A target position tracking device, characterized in that, include: The first registration module is used to register the initial face point cloud image and the face 3D model of the target person to obtain the first transformation matrix. The initial face point cloud image is acquired by a structured light camera, and the face 3D model is obtained by reconstructing the face medical image of the target person. The first transformation module is used to transform the coordinates of the target point in the three-dimensional face model according to the first transformation matrix to obtain the first physical position of the target point; The second registration module is used to register the current face point cloud image of the target person with the initial face point cloud image when the movement of the target person is detected, so as to obtain a second transformation matrix. The current face point cloud image is acquired by the structured light camera. The second transformation module is used to transform the first physical position according to the second transformation matrix to obtain the second physical position of the target point; Before registering the current face point cloud image of the target person with the initial face point cloud image to obtain the second transformation matrix, upon detecting movement of the target person, the device is further configured to: The face image of the target person and the corresponding face point cloud image are acquired in real time. The face image and the corresponding face point cloud image are acquired by a time-of-flight camera. The time-of-flight camera employs stereoscopic vision imaging technology and is used to continuously capture facial images and facial point cloud images of the target person.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the target location tracking method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the target location tracking method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the target location tracking method as described in any one of claims 1 to 6.

Images