Visual tracking method and computing device

By establishing a three-plane spatial model and performing homography transformation, the problems of insufficient accuracy and poor stability of monocular camera visual tracking solutions in complex office environments were solved, achieving accurate positioning and stable tracking of the user's gaze on the display screen.

CN122289729APending Publication Date: 2026-06-26XFUSION DIGITAL TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610209381.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-02-12
Publication Date
2026-06-26

AI Technical Summary

Technical Problem

Visual tracking solutions based on monocular cameras suffer from insufficient tracking accuracy and poor stability in complex office environments.

Method used

By establishing a three-plane spatial model with the background image corresponding to the first plane, the face image corresponding to the second plane, and the display screen corresponding to the third plane as parallel planes, and by using homography transformation, the homography matrix of the face relative to the first plane is calculated to determine the physical coordinates of the user's gaze on the display screen.

Benefits of technology

It enables accurate tracking of users' gaze in complex office scenarios, avoiding tracking failures caused by changes in head posture and displacement, thus improving tracking accuracy and stability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122289729A_ABST
    Figure CN122289729A_ABST
Patent Text Reader

Abstract

This application provides a visual tracking method and computing device, relating to the field of computer technology, and applied to a visual tracking system. The visual tracking system includes a camera and a display screen. The method includes: determining a first plane and a second plane based on images captured by the camera, wherein the first plane is a plane corresponding to a background image in the image, and the second plane is a plane corresponding to a face in the image; determining the homography matrix of the face in the second plane relative to the first plane based on a first image and a second image captured by the camera in chronological order; determining a rotation matrix and a translation vector of the face based on the homography matrix; and determining the physical coordinates of the user's gaze on the display screen based on the rotation matrix and the translation vector. The technical solution provided by this application can solve the technical problems of insufficient tracking accuracy and poor stability in related technologies based on monocular cameras.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a visual tracking method and computing device. Background Technology

[0002] With the rapid growth in demand for digital office and intelligent interaction, the application value of vision tracking-based human-computer interaction technology in office scenarios is becoming increasingly prominent.

[0003] Among related technologies, visual tracking solutions based on monocular cameras lack sufficient tracking accuracy and are difficult to operate stably in complex office environments. Summary of the Invention

[0004] This application provides a visual tracking method and computing device, which can solve the technical problems of insufficient tracking accuracy and poor stability in visual tracking schemes based on monocular cameras in related technologies.

[0005] In a first aspect, embodiments of this application provide a visual tracking method applied to a visual tracking system, the visual tracking system including a camera and a display screen, the method comprising:

[0006] Based on the images captured by the camera, a first plane and a second plane are determined. The first plane is the plane corresponding to the background image in the image, and the second plane is the plane corresponding to the face in the image. The first plane, the second plane, and the third plane corresponding to the display screen are parallel to each other. Based on the first and second images captured by the camera in chronological order, the homography matrix of the face in the second plane relative to the first plane is determined. Based on the homography matrix, the rotation matrix and translation vector of the face are determined. The rotation matrix is ​​used to represent the rotation angle of the face and / or the displacement in the second plane, and the translation vector is used to represent the displacement of the face along a direction perpendicular to the third plane. Based on the rotation matrix and translation vector, the physical coordinates of the user's gaze on the display screen are determined.

[0007] The visual tracking method provided in this application embodiment can acquire images based on a monocular camera. By modeling the first plane corresponding to the background image, the second plane corresponding to the face image, and the third plane corresponding to the display screen into a three-plane spatial model that are parallel to each other, and by calculating the homography matrix of the face relative to the first plane through homography transformation, the translation, rotation, and displacement of the user's face perpendicular to the display screen can be determined based on the homography matrix. This allows for accurate tracking of the physical coordinates of the user's gaze on the display screen, avoiding tracking failures caused by changes in the user's head posture and / or displacement. It is applicable to complex office scenarios and solves the technical problems of insufficient tracking accuracy and poor stability in visual tracking schemes based on monocular cameras in related technologies.

[0008] In one possible implementation, determining the first plane and the second plane based on the image captured by the camera includes:

[0009] Acquire the initial image captured by the camera; identify the facial image and background image in the initial image; segment the facial image and background image, and determine the first plane and the second plane based on the segmented background image and facial image.

[0010] In this embodiment of the application, by determining the first plane and the second plane, a three-plane spatial model can be established based on the first plane, the second plane and the third plane corresponding to the display screen. This helps to determine the displacement information of the user's face relative to the display screen and eliminate the problem of gaze drift caused by the head moving closer to / away from the display screen.

[0011] In one possible implementation, the above method further includes:

[0012] A world coordinate system is established with the first plane as the reference plane, and the relative position of the third plane in the world coordinate system is determined according to the parameters of the camera.

[0013] In one possible implementation, determining the homography matrix of a face in the second plane relative to the first plane based on the first and second images captured by the camera in chronological order includes:

[0014] Determine a first feature array of the second plane in the first image relative to the first plane; determine a second feature array of the second plane in the second image relative to the first plane; and determine the homography matrix based on the first and second feature arrays.

[0015] In this embodiment of the application, by determining the above homography matrix, information such as the translation, rotation, and displacement of the face relative to the first plane can be determined, which can avoid tracking failure caused by changes in the user's head posture.

[0016] In one possible implementation, determining the physical coordinates of the user's gaze on the display screen based on the rotation matrix and translation vector includes:

[0017] Determine the user's first line of sight vector on the second plane; transform the first line of sight vector to the world coordinate system described above using the rotation matrix and translation vector to obtain the second line of sight vector; determine the physical coordinates of the user's line of sight on the display screen based on the second line of sight vector.

[0018] In one possible implementation, determining the physical coordinates of the user's gaze on the display screen based on the second gaze vector includes:

[0019] Determine the intersection point of the second line-of-sight vector and the third plane; based on the coordinates of this intersection point in the world coordinate system, determine the physical coordinates of the user's line of sight on the display screen.

[0020] In one possible implementation, determining the rotation matrix and translation vector of the face based on the homography matrix includes:

[0021] Based on the camera's intrinsic parameter matrix, the normal vector of the first plane, and the distance between the first plane and the camera, the homography matrix is ​​decomposed into a rotation matrix and a translation vector.

[0022] In this embodiment, by combining the camera intrinsic parameter matrix, the normal vector of the first plane, and the distance between the first plane and the camera, the homography matrix is ​​accurately decomposed, which can maintain tracking stability in complex dynamic scenes (such as rapid head movement of the user or changes in screen viewing angle) and reduce the gaze mapping drift problem caused by homography matrix decomposition error.

[0023] In one possible implementation, the above method further includes:

[0024] Every preset time interval, based on the images captured by the camera, the homography matrix of the face in the second plane relative to the first plane is updated; based on the updated homography matrix, the rotation matrix and translation vector of the face are updated; based on the updated rotation matrix and translation vector of the face, the physical coordinates of the user's gaze on the display screen are updated.

[0025] In this embodiment, by dynamically updating the homography matrix of the face in the second plane relative to the first plane, the physical coordinates of the user's gaze on the display screen can be dynamically updated, ensuring the accuracy of visual tracking.

[0026] In one possible implementation, the above method further includes:

[0027] Based on the physical coordinates of the user's line of sight on the display screen, a target display area is selected on the display screen; the content displayed in the target display area is subjected to a first process; wherein, the first process includes highlighting and / or refreshing.

[0028] In this embodiment of the application, by highlighting and / or refreshing the target display area, manual operation by the user can be reduced, thereby improving the interaction efficiency and immersion in the office scenario.

[0029] Secondly, embodiments of this application also provide a computing device, including: a memory and a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, causing the processor to perform the visual tracking method as provided in the first aspect.

[0030] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the visual tracking method provided in the first aspect above.

[0031] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the visual tracking method provided in the first aspect above.

[0032] The technical solutions provided in the second to fourth aspects of this application are similar to the beneficial effects achieved by the first aspect of this application and any corresponding feasible implementation method, and will not be described again. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating a visual tracking method provided in an embodiment of this application;

[0034] Figure 2 This is a schematic diagram of a three-plane spatial model provided in the embodiments of this application;

[0035] Figure 3 This is a schematic diagram illustrating an application scenario of a visual tracking method provided in the embodiments of this application;

[0036] Figure 4 This is a schematic diagram of the structure of a visual tracking device provided in an embodiment of this application;

[0037] Figure 5 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0038] Exemplary embodiments of this application will be described in detail below. When the following description relates to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0039] To facilitate a clear description of the technical solutions in the embodiments of this application, terms such as "exemplary" and "for example" are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0040] In this embodiment of the application, "and / or" is used to describe the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after it have an "or" relationship.

[0041] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application are briefly introduced below:

[0042] Human-Computer Interaction (HCI) refers to the process of information exchange between humans and computers using a certain dialogue language and interactive methods to complete a specific task.

[0043] Visual tracking refers to the technology of detecting, extracting, recognizing, and tracking moving targets in image sequences, obtaining parameters such as position, velocity, acceleration, and trajectory. It belongs to the field of computer vision and involves target detection, feature extraction, classification, and tracking filtering. Through parameter processing and analysis, it achieves behavioral understanding to complete advanced detection tasks, and can be applied to video surveillance, robot navigation, drone tracking, medical diagnosis, and sports broadcasting.

[0044] Homography transformation: refers to a two-dimensional projection transformation that maps points in one plane to points in another plane. It is applicable to scenarios such as image correction, stitching, and target tracking. Here, "plane" refers to a planar surface in an image or three-dimensional space.

[0045] With the rapid growth in demand for digital office and intelligent interaction, the application value of vision-tracking-based human-computer interaction technology in office scenarios is becoming increasingly prominent. For example, in scenarios such as code editing, data visualization, and remote collaboration, users can interact efficiently with the system through natural visual behaviors (such as gaze focus and head movements).

[0046] Visual tracking technology uses sensors such as cameras to capture and analyze a user's eye movements, gaze focus, or facial orientation in real time, thereby understanding the user's intentions and attention. This "what you see is what you get" interaction paradigm's core advantage lies in its effective alignment with natural human behavior patterns. Users do not need to learn complex commands or operate peripherals; they can complete interactions simply through natural gaze, blinking, and other instinctive actions, significantly reducing the cognitive load and technical barriers to interaction and improving the efficiency of information transmission and task execution.

[0047] In related technologies, visual tracking technology generally includes visual tracking schemes based on binocular cameras and visual tracking schemes based on monocular cameras.

[0048] Among them, the visual tracking scheme based on binocular cameras simulates the stereoscopic vision of human eyes by using two cameras and calculates the three-dimensional spatial position of the target using the principle of triangulation. Its advantage lies in its high theoretical accuracy, but it has the following drawbacks:

[0049] (1) High hardware cost: Two sets of cameras and calibration equipment are required, which significantly increases the hardware cost.

[0050] (2) High computational complexity: Real-time stereo matching algorithm has high computational power requirements and high power consumption.

[0051] (3) Strong environmental dependence: changes in lighting, occlusion or reflection will seriously affect the reliability of feature matching.

[0052] (4) Calibration sensitivity: Slight displacement or temperature difference of the equipment will cause calibration error and affect tracking stability.

[0053] Visual tracking schemes based on monocular cameras infer gaze direction by analyzing two-dimensional image changes in eye features. The core steps include:

[0054] (1) Feature extraction: Detect key features such as pupil and corneal reflective points.

[0055] (2) Model fitting: Matching features with a predefined eye geometry model.

[0056] (3) Mapping calculation: Establish the mapping relationship between features and screen gaze points (user calibration required).

[0057] While the aforementioned visual tracking solution based on a monocular camera reduces hardware costs, it still has the following drawbacks:

[0058] (1) Lack of depth information: The scale changes caused by the forward and backward movement of the head cannot be perceived, resulting in a decrease in tracking accuracy.

[0059] (2) Highly dependent on calibration: Frequent calibration is required to maintain the mapping model, resulting in a poor user experience.

[0060] (3) Sensitive to individual differences: The general model is difficult to adapt to the eye structure of different users (such as differences in corneal curvature).

[0061] (4) Weak dynamic adaptability: In scenarios where the head moves freely, the tracking stability is insufficient.

[0062] In summary, how to provide a low-cost, high-precision visual tracking solution that can also meet the need for stable operation in complex office environments is a technical problem that urgently needs to be solved.

[0063] To address the aforementioned technical problems, this application provides a visual tracking method that can acquire images using a monocular camera. It models a three-plane spatial model consisting of a first plane corresponding to the background image, a second plane corresponding to the face image, and a third plane corresponding to the display screen. Through homography transformation, it calculates the homography matrix of the face relative to the first plane. Based on this homography matrix, it can determine information such as the translation, rotation, and displacement of the user's head perpendicular to the display screen, thereby accurately tracking the physical coordinates of the user's gaze on the display screen. This avoids tracking failures caused by changes in the user's head posture and is applicable to complex office scenarios. It solves the technical problems of insufficient tracking accuracy and poor stability in related monocular camera-based visual tracking schemes.

[0064] The visual tracking method provided in this application embodiment will be described in detail below through specific implementation methods.

[0065] The visual tracking method provided in this application can be applied to office automation scenarios, such as code editor highlighting, dynamic display on intelligent operations center (IoC) large screens, and intelligent interactive scenarios (such as exhibitions and remote collaboration). For example, in office automation scenarios, when developers are writing code, their gaze is often focused on specific code segments or debugging information. By tracking the user's gaze trajectory in real time, the code area that the user is focusing on can be dynamically highlighted, helping the user quickly locate problems or jump to relevant modules, thereby significantly improving coding efficiency. In IoC large screen scenarios, report data can be partially refreshed based on the user's gaze focus to achieve dynamic interaction.

[0066] In some embodiments, the above-described visual tracking method can be applied to a visual tracking system, which includes a camera and a display screen. The camera, as a visual input device, can be used to acquire images or video streams of a target area; the display screen can be used to provide an interactive interface.

[0067] In some implementations, the camera described above is a monocular camera.

[0068] Optionally, the camera can be a standalone camera and connected to the display screen via cable or wireless connection. The optical center of the camera and the display screen can be on the same plane.

[0069] Alternatively, the aforementioned camera can be an integrated camera within the aforementioned display screen. For example, the aforementioned camera can be embedded in the bezel of the display screen (such as a camera built into a laptop or tablet).

[0070] In some implementations, the visual tracking method described above can be executed by the camera, by the display screen, or by a combination of the camera and the display screen. No limitation is made in the embodiments of this application.

[0071] Reference Figure 1 , Figure 1 This is a flowchart illustrating a visual tracking method provided in an embodiment of this application.

[0072] In some embodiments, the above visual tracking method includes:

[0073] S101. Based on the image captured by the camera, determine the first plane and the second plane; wherein, the first plane is the plane corresponding to the background image in the image, the second plane is the plane corresponding to the face in the image, and the first plane, the second plane and the third plane corresponding to the display screen are parallel to each other.

[0074] In some embodiments, the background image, facial image, and display screen in the image captured by the camera can be approximated as three parallel planes, establishing a three-plane spatial model. The positions of the plane corresponding to the background image and the plane corresponding to the display screen are fixed.

[0075] For example, refer to Figure 2 , Figure 2 This is a schematic diagram of a three-plane spatial model provided in the embodiments of this application.

[0076] The first plane is the plane containing the background image (such as a wall) captured by the camera, which can be regarded as a fixed reference plane; the second plane is the plane containing the facial image, which is parallel to the first plane; and the third plane is the plane containing the display screen, which is parallel to the first plane and the second plane, respectively.

[0077] In some implementations, a world coordinate system can be established with the first plane as the reference plane, and the relative position of the third plane in the world coordinate system can be determined according to the parameters of the camera (such as focal length), that is, the distance between the third plane and the first plane can be determined.

[0078] For example, a world coordinate system can be constructed using a key point on the first plane as the origin and the first plane as the XY plane, and the coordinates of the third plane on the Z-axis can be determined based on the focal length of the camera.

[0079] S102. Based on the first and second images captured by the camera, which are sequential in time, determine the homography matrix of the face in the second plane relative to the first plane.

[0080] Optionally, the second image can be the next frame of the first image; or, the second image can be the nth frame after the first image; n≥2, which is not limited in this embodiment.

[0081] S103. Based on the homography matrix, determine the rotation matrix and translation vector of the face.

[0082] The rotation matrix can be used to represent the rotation angle of the face and / or the displacement in the second plane, and the translation vector can be used to represent the displacement of the face along the direction perpendicular to the third plane.

[0083] Specifically, the rotation matrix can be used to represent the rotation angle of the face in the world coordinate system and / or the displacement along the X / Y axis, and the translation vector can be used to represent the displacement of the face along the Z axis.

[0084] S104. Determine the physical coordinates of the user's line of sight on the display screen based on the rotation matrix and translation vector.

[0085] In some implementations, the rotation matrix and translation vector described above, along with a preset gaze direction model (such as the relationship between head rotation angle and eye direction), can be combined to transform the gaze vector from the second plane to the world coordinate system described above, and calculate its intersection with the third plane to determine the physical coordinates of the gaze focus on the display screen.

[0086] In some implementations, interactive behaviors (such as code highlighting, report refreshing, etc.) can be triggered based on the physical coordinates of the gaze focus on the display screen, thereby achieving natural human-computer interaction.

[0087] In this embodiment, by establishing the above-mentioned three-plane spatial model and calculating the homography matrix of the face relative to the first plane, the user's facial posture changes (including rotation angle and / or displacement along the X / Y axis) and the user's displacement in the Z-axis direction can be calculated based on the homography matrix. This can avoid tracking failure caused by changes in the user's facial posture and eliminate the problem of gaze drift caused by the head moving closer to / away from the display screen; it is better suited for complex office scenarios.

[0088] The visual tracking method provided in this application embodiment can acquire images based on a monocular camera. By modeling the first plane corresponding to the background image, the second plane corresponding to the face image, and the third plane corresponding to the display screen into a three-plane spatial model that are parallel to each other, and by calculating the homography matrix of the face relative to the first plane through homography transformation, the translation, rotation, and displacement of the user's face perpendicular to the display screen can be determined based on the homography matrix. This allows for accurate tracking of the physical coordinates of the user's gaze on the display screen, avoiding tracking failures caused by changes in the user's head posture and / or displacement. It is applicable to complex office scenarios and solves the technical problems of insufficient tracking accuracy and poor stability in visual tracking schemes based on monocular cameras in related technologies.

[0089] In some embodiments, determining the first plane and the second plane based on the image captured by the camera includes:

[0090] Acquire the initial image captured by the camera; identify the facial image and background image in the initial image; segment the facial image and background image, and determine the first plane and the second plane based on the segmented background image and facial image.

[0091] In some implementations, taking a face image as an example, the initial image captured by the camera can be segmented to separate the face image from the background image; wherein the area where the background image is located is the non-face area in the initial image.

[0092] In some implementations, the face may be positioned in the center of the initial image by default.

[0093] For example, a deep learning model can be used to locate the face region in the initial image, determine the face bounding box or segmentation mask, and separate the face image from the background image based on the face bounding box or segmentation mask.

[0094] In this embodiment of the application, by determining the first plane and the second plane, a three-plane spatial model can be established based on the first plane, the second plane and the third plane corresponding to the display screen. This helps to determine the displacement information of the user's face relative to the display screen and eliminate the problem of gaze drift caused by the head moving closer to / away from the display screen.

[0095] In some embodiments, determining the homography matrix of a face in the second plane relative to the first plane based on a first image and a second image captured by a camera in chronological order includes:

[0096] Determine a first feature array of the second plane in the first image relative to the first plane; determine a second feature array of the second plane in the second image relative to the first plane; and determine the homography matrix based on the first and second feature arrays.

[0097] The first and second feature arrays mentioned above can be used to describe the position and pose data of certain key points on the face. For example, they can refer to the feature set of specific key points in a facial image, including the center position of the eyes, nose, mouth, etc., or the tilt angle of certain prominent points on the facial contour.

[0098] For example, after the camera captures the first image (which can be the first valid image at the current moment), the first image is processed to determine the first plane and the second plane in the first image, and then the first feature array p_{image} of the second plane in the first image relative to the first plane is determined.

[0099] After the camera captures the second image (which can be a valid image for the next frame), the second image is processed to determine the first plane and the second plane in the second image. Then, the second feature array p'_{image} of the second plane in the second image relative to the first plane is determined.

[0100] It is understandable that, since the first plane remains fixed, it can be determined that p_{image} and p'_{image} satisfy the homography transformation rules of computer graphics:

[0101] ;

[0102] In some implementations, the homography matrix H can be determined through homography transformation.

[0103] Here, H can be a 3×3 homography matrix.

[0104] In this embodiment of the application, the homography transformation described above can determine information such as the translation, rotation, and displacement of the face relative to the first plane, thereby avoiding tracking failure caused by changes in the user's head posture.

[0105] In some embodiments, determining the rotation matrix and translation vector of the face based on the homography matrix includes:

[0106] Based on the camera's intrinsic parameter matrix, the normal vector of the first plane, and the distance between the first plane and the camera, the homography matrix is ​​decomposed into a rotation matrix and a translation vector.

[0107] For example, the homography matrix H can be decomposed into a rotation matrix R and a translation vector t according to the following formula:

[0108] ;

[0109] Where K is the intrinsic parameter matrix of the camera (which has been calibrated at the time of manufacture), n is the normal vector of the first plane, and d is the distance between the first plane and the camera.

[0110] Understandably, since the position of the aforementioned camera is fixed, the rotation matrix R can represent the rotation angle of the face and / or the translation amount along the X / Y axis, and the translation vector t can represent the distance the face moves forward and backward along the Z axis.

[0111] In some embodiments, determining the physical coordinates of the user's line of sight on the display screen based on the rotation matrix and translation vector includes:

[0112] Determine the user's first line of sight vector on the second plane; transform the first line of sight vector to the world coordinate system described above using the rotation matrix and translation vector to obtain the second line of sight vector; determine the physical coordinates of the user's line of sight on the display screen based on the second line of sight vector.

[0113] In some implementations, determining the physical coordinates of the user's gaze on the display screen based on the second gaze vector includes:

[0114] Determine the intersection point of the second line-of-sight vector and the third plane; based on the coordinates of this intersection point in the world coordinate system, determine the physical coordinates of the user's line of sight on the display screen.

[0115] Optionally, based on the gaze estimation method, the user's first gaze vector g_{head} in the second plane can be determined. Then, combining only the rotation matrix R and disregarding the displacement between the user and the display screen, the first gaze vector g_{head} can be transformed to the aforementioned world coordinate system to obtain the second gaze vector g_{world}. .

[0116] Alternatively, based on the gaze estimation method, the user's first gaze vector g_{head} in the second plane can be determined. Then, combining the rotation matrix R and the translation vector t, the first gaze vector g_{head} can be transformed to the aforementioned world coordinate system to obtain the second gaze vector g_{world}. .

[0117] Furthermore, the intersection point of the ray emitted from the second line-of-sight vector g_{world} and the third plane is calculated. Based on the coordinates of this intersection point on the third plane, the physical coordinates of the user's line of sight on the display screen can be determined.

[0118] In this embodiment, the translation vector is determined by decomposing the homography matrix, which can determine the displacement of the user's face in the Z-axis direction. This solves the technical problem in related technologies where visual tracking schemes based on monocular cameras lack depth information, cannot perceive scale changes caused by the forward and backward movement of the user's head, and thus reduce tracking accuracy.

[0119] In some embodiments, the method further includes: selecting a target display area on the display screen based on the physical coordinates of the user's line of sight on the display screen; and performing a first processing on the content displayed within the target display area. Optionally, the first processing includes highlighting and / or refreshing.

[0120] For example, refer to Figure 3 , Figure 3 This is a schematic diagram illustrating an application scenario of a visual tracking method provided in the embodiments of this application.

[0121] like Figure 3 As shown, assuming the user's line of sight is at physical coordinates O(0,0) on the display screen, then an area with a width of n centimeters and a length of m centimeters can be selected as the target display area, with coordinates O(0,0) as the center point.

[0122] In some implementations, the content within the target display area can be highlighted.

[0123] For example, when a user is coding, the camera can fully track the user's gaze and perspective changes, thereby highlighting the information the user is interested in in a timely manner, enhancing the user's immersion, improving the interactive experience and coding efficiency.

[0124] In some implementations, the content within the target display area can be refreshed.

[0125] For example, in IoC large screen scenarios, report data can be partially refreshed based on the user's gaze focus to achieve dynamic interaction. This enhances the report data within the user's gaze focus by partially refreshing and dynamically displaying it, thereby improving the user experience while increasing the responsiveness and application efficiency of the large screen.

[0126] In this embodiment, user visual tracking can be achieved based on a monocular camera. By highlighting and / or refreshing the target display area, manual operation by the user can be reduced, thereby improving the interaction efficiency and immersion in the office environment.

[0127] In some embodiments, the method further includes: updating the homography matrix of the face in the second plane relative to the first plane according to the image captured by the camera at preset intervals; updating the rotation matrix and translation vector of the face according to the updated homography matrix; and updating the physical coordinates of the user's gaze on the display screen according to the updated rotation matrix and translation vector of the face.

[0128] In this embodiment, by dynamically updating the homography matrix of the face in the second plane relative to the first plane, the physical coordinates of the user's gaze on the display screen can be dynamically updated, ensuring the accuracy of visual tracking.

[0129] In some embodiments, this application also provides a visual tracking device. (See reference...) Figure 4 , Figure 4 This is a schematic diagram of the structure of a visual tracking device provided in an embodiment of this application. The visual tracking device 40 can be applied to the above-mentioned visual tracking system and includes:

[0130] The preprocessing module 401 is used to determine a first plane and a second plane based on the image captured by the camera; wherein the first plane is the plane corresponding to the background image in the image, the second plane is the plane corresponding to the face in the image, and the first plane, the second plane and the third plane corresponding to the display screen are parallel to each other.

[0131] The conversion module 402 is used to determine the homography matrix of the face in the second plane relative to the first plane based on the first image and the second image captured by the camera in chronological order.

[0132] The processing module 403 is used to determine the rotation matrix and translation vector of the face based on the homography matrix. The rotation matrix is ​​used to represent the rotation angle of the face and / or the displacement in the second plane, and the translation vector is used to represent the displacement of the face along the direction perpendicular to the third plane.

[0133] The positioning module 404 is used to determine the physical coordinates of the user's line of sight on the display screen based on the rotation matrix and translation vector.

[0134] In one possible implementation, the preprocessing module 401 is specifically used for:

[0135] Acquire the initial image captured by the camera; identify the facial image and background image in the initial image; segment the facial image and background image, and determine the first plane and the second plane based on the segmented background image and facial image.

[0136] In one possible implementation, the preprocessing module 401 is further configured to:

[0137] A world coordinate system is established with the first plane as the reference plane, and the relative position of the third plane in the world coordinate system is determined according to the parameters of the camera.

[0138] In one possible implementation, the conversion module 402 is specifically used for:

[0139] Determine a first feature array of the second plane in the first image relative to the first plane; determine a second feature array of the second plane in the second image relative to the first plane; and determine the homography matrix based on the first and second feature arrays.

[0140] In one possible implementation, the positioning module 404 is specifically used for:

[0141] Determine the user's first line of sight vector on the second plane; transform the first line of sight vector to the world coordinate system described above using the rotation matrix and translation vector to obtain the second line of sight vector; determine the physical coordinates of the user's line of sight on the display screen based on the second line of sight vector.

[0142] In one possible implementation, the positioning module 404 is specifically used for:

[0143] Determine the intersection point of the second line-of-sight vector and the display screen; based on the coordinates of this intersection point in the world coordinate system, determine the physical coordinates of the user's line of sight on the display screen.

[0144] In one possible implementation, the positioning module 404 is specifically used for:

[0145] Based on the camera's intrinsic parameter matrix, the normal vector of the first plane, and the distance between the first plane and the camera, the homography matrix is ​​decomposed into a rotation matrix and a translation vector.

[0146] In one possible implementation, it also includes:

[0147] The update module is used to update the homography matrix of the face in the second plane relative to the first plane at preset intervals, based on the images captured by the camera; update the rotation matrix and translation vector of the face based on the updated homography matrix; and update the physical coordinates of the user's gaze on the display screen based on the updated rotation matrix and translation vector of the face.

[0148] In one possible implementation, the positioning module 404 is further configured to:

[0149] Based on the physical coordinates of the user's line of sight on the display screen, a target display area is selected on the display screen; the content displayed in the target display area is subjected to a first process; wherein, the first process includes highlighting and / or refreshing.

[0150] It is understood that the specific implementation process of each module of the above-mentioned visual tracking device 40 can be found in the above-mentioned method embodiment, and the implementation principle and technical effect are similar. This embodiment will not be repeated here.

[0151] The above-described division of modules is merely a logical functional division. Each function can correspond to a single functional module, or two or more functions can be integrated into one functional module. In actual implementation, all or some modules can be integrated into a single physical entity, or they can be distributed across different physical entities. Furthermore, depending on the specific circumstances, the aforementioned functional modules may be implemented in hardware, software, or a combination of both. Whether a function is executed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0152] like Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. The computing device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the computing device 50 further includes a communication interface 503. The processor 501, memory 502, and communication interface 503 are connected via a bus.

[0153] In the specific implementation process, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to execute the above-mentioned visual tracking method.

[0154] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0155] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0156] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0157] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0158] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described visual tracking method.

[0159] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the aforementioned visual tracking method.

[0160] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0161] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0162] The above specific embodiments further illustrate the purpose, technical solution and beneficial effects of this application. It should be understood that the above are only specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this application.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of this application, and are not intended to limit them. Although the embodiments of this application have been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A visual tracking method, characterized in that, Applied to a visual tracking system, the visual tracking system including a camera and a display screen, the method includes: Based on the image captured by the camera, a first plane and a second plane are determined; the first plane is the plane corresponding to the background image in the image, and the second plane is the plane corresponding to the face in the image; the first plane, the second plane, and the third plane corresponding to the display screen are parallel to each other; Based on the first and second images captured by the camera in chronological order, determine the homography matrix of the face in the second plane relative to the first plane; Based on the homography matrix, the rotation matrix and translation vector of the face are determined. The rotation matrix represents the rotation angle of the face and / or the displacement in the second plane, and the translation vector represents the displacement of the face along a direction perpendicular to the third plane. The physical coordinates of the user's line of sight on the display screen are determined based on the rotation matrix and the translation vector.

2. The method according to claim 1, characterized in that, The step of determining the first plane and the second plane based on the images captured by the camera includes: Acquire the initial image captured by the camera; Identify the facial image and background image in the initial image; The facial image and the background image are segmented, and the first plane and the second plane are determined based on the segmented background image and facial image.

3. The method according to claim 1 or 2, characterized in that, The method further includes: A world coordinate system is established with the first plane as the reference plane, and the relative position of the third plane in the world coordinate system is determined according to the parameters of the camera.

4. The method according to claim 1, characterized in that, Determining the homography matrix of a face in the second plane relative to the first plane based on the first and second images captured by the camera in chronological order includes: Determine a first feature array of the second plane in the first image relative to the first plane; Determine a second feature array of the second plane in the second image relative to the first plane; The homography matrix is ​​determined based on the first feature array and the second feature array.

5. The method according to claim 3, characterized in that, Determining the physical coordinates of the user's line of sight on the display screen based on the rotation matrix and the translation vector includes: Determine the user's first gaze vector in the second plane; Based on the rotation matrix and the translation vector, the first line-of-sight vector is transformed to the world coordinate system to obtain the second line-of-sight vector; Based on the second gaze vector, the physical coordinates of the user's gaze on the display screen are determined.

6. The method according to claim 5, characterized in that, Determining the physical coordinates of the user's gaze on the display screen based on the second gaze vector includes: Determine the intersection point of the second line-of-sight vector and the third plane; Based on the coordinates of the intersection point in the world coordinate system, determine the physical coordinates of the user's line of sight on the display screen.

7. The method according to claim 5, characterized in that, Determining the rotation matrix and translation vector of the face based on the homography matrix includes: Based on the intrinsic parameter matrix of the camera, the normal vector of the first plane, and the distance between the first plane and the camera, the homography matrix is ​​decomposed into the rotation matrix and the translation vector.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: Every preset time interval, the homography matrix of the face in the second plane relative to the first plane is updated based on the images captured by the camera; Based on the updated homography matrix, update the rotation matrix and translation vector of the face; The user's gaze position on the display screen is updated based on the updated rotation matrix and translation vector of the face.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Based on the physical coordinates of the user's line of sight on the display screen, select the target display area on the display screen; The content displayed within the target display area undergoes a first process; the first process includes highlighting and / or refreshing.

10. A computing device, characterized in that, Including memory and processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the visual tracking method as described in any one of claims 1-9.