Method, system and terminal for realizing screen interaction in video call process

Abstract

The invention relates to the technical field of communication interaction, in particular to a method, a system and a terminal for realizing screen interaction in a video call process. The method comprises the following steps: arranging a near-infrared LED point light source at the edge of a display, collecting an eye image of a user through a camera, obtaining four reflection light spot images formed on the front surface of a cornea, and extracting the pixel coordinate position of each reflection light spot; in a system initialization stage, four light spot reference positions when a user directly views the center of a screen are recorded, and a bidirectional mapping relation between the light spot displacement and the horizontal rotation angle and the vertical rotation angle of an eyeball is established; in the real-time interaction process, the displacement of each light spot of the current frame relative to the previous frame is calculated, and the corresponding eyeball rotation angle is extracted based on the bidirectional mapping relation. Through three-dimensional space modeling and corneal curvature correction, adaptation to eyeball characteristics of different individuals is achieved, and the universality of the system is improved.

Description

Technical Field

[0001] This invention relates to the field of communication and interaction technology, and in particular to a method, system, and terminal for realizing screen interaction during video calls. Background Technology

[0002] Modern video conferencing systems employ Region of Interest (ROI) technology, allowing local users to select specific areas within a remote video feed for detailed viewing. The core mechanism of this technology involves the local device defining ROIs within the remote camera's field of view and mapping them to pan, tilt, zoom, and focus commands. This approach significantly enhances the interactivity and user experience of video conferencing. The system transmits ROI information to the remote device via Real-Time Transmission Control Protocol (RTCP) feedback messages, guiding it to capture and encode the video content of the specified area.

[0003] In traditional video calls and human-computer interaction scenarios, users primarily rely on mice, touchscreens, or keyboards for operation. This approach has certain limitations, especially in remote collaboration, accessible interaction, and virtual reality applications. Traditional interaction methods are not intuitive enough and may affect operational efficiency and comfort. Furthermore, early eye-tracking technologies typically relied on low-precision sensors or single optical features for gaze point recognition, making them susceptible to interference from changes in lighting, micro-eye movements, blinking, and individual differences. This resulted in inaccurate gaze point positioning and severe cursor drift, making it difficult to achieve high-precision, real-time gaze-based interactive operations. Summary of the Invention

[0004] Therefore, it is necessary for the present invention to provide a method, system and terminal for screen interaction during video calls, in order to solve at least one of the above-mentioned technical problems.

[0005] To achieve the above objectives, a method for enabling screen interaction during a video call includes the following steps: Step S1: Arrange near-infrared LED point light sources at the edge of the display, acquire images of the user's eyes through a camera, obtain images of four reflected light spots formed on the anterior surface of the cornea, and extract the pixel coordinate positions of each reflected light spot; Step S2: During the system initialization phase, record the four reference positions of the light spot when the user looks directly at the center of the screen, and establish a two-way mapping relationship between the light spot displacement and the horizontal and vertical rotation angles of the eyeball; Step S3: During real-time interaction, calculate the displacement of each light spot in the current frame relative to the previous frame, and extract the corresponding eye rotation angle based on the bidirectional mapping relationship. Step S4: Rotate the three-dimensional position vector of the pupil center relative to the eye rotation center by the eye rotation angle, and add it to the preset corneal curvature radius to obtain the real-time position of the pupil center in the camera coordinate system; Step S5: Map the real-time position to the display screen coordinate system to obtain the screen coordinates of the user's current gaze point; use the screen coordinates to make the cursor move with the user's gaze point, and trigger the selection or confirmation operation when the gaze point stays in the interactive area for more than a preset time threshold to realize eye-tracking interaction.

[0006] The present invention also provides a system for implementing screen interaction during a video call, used to execute the method for implementing screen interaction during a video call as described above, wherein the system for implementing screen interaction during a video call includes: The light spot acquisition module is used to place near-infrared LED point light sources with a wavelength of 850 nanometers at the four corners of the display, acquire images of the user's eyes through the camera, obtain images of four reflected light spots formed on the anterior surface of the cornea, and extract the pixel coordinate position of each reflected light spot. The light spot mapping modeling module is used to record the four light spot reference positions when the user looks directly at the center of the screen during the system initialization phase, and to establish a two-way mapping relationship between the light spot displacement and the horizontal and vertical rotation angles of the eyeball. The light spot displacement calculation module is used to calculate the displacement of each light spot in the current frame relative to the previous frame during real-time interaction, and extract the corresponding eye rotation angle based on the bidirectional mapping relationship. The pupil position estimation module is used to rotate the three-dimensional position vector of the pupil center relative to the eye rotation center by the eye rotation angle, and add it to the preset corneal curvature radius to obtain the real-time position of the pupil center in the camera coordinate system. The gaze point interaction module is used to map the real-time position to the display screen coordinate system to obtain the screen coordinates of the user's current gaze point; the screen coordinates are used to make the cursor move with the user's gaze point, and when the gaze point stays in the interactive area for more than a preset time threshold, a selection or confirmation operation is triggered to realize eye-tracking interaction.

[0007] The present invention also provides a terminal for implementing screen interaction during video calls, comprising: The display has near-infrared LED point light sources arranged at its four corners; A camera used to capture images of the user's eyes; processor; A memory storing a computer program that, when executed by the processor, implements a method for screen interaction during a video call as described in any of the preceding claims.

[0008] This invention can accurately acquire the rotation angle of a user's eyeballs and the three-dimensional position of the pupil center during video calls, thereby significantly improving the accuracy and reliability of eye-tracking interaction. During the calibration phase, by establishing an eye coordinate system with the eyeball rotation center as the origin and determining the initial pupil position, a stable spatial reference is provided for subsequent real-time tracking, ensuring consistency across different users and viewing angles.

[0009] By utilizing horizontal and vertical rotation angles to perform three-dimensional rotation calculations on the pupil position, the system can accurately capture minute eye movements and achieve precise positioning of the gaze direction. In the process of transforming the rotated pupil position to the camera coordinate system and combining it with corneal curvature correction, the influence of the corneal spherical shape on the optical reflection position is effectively compensated. This ensures that the gaze point mapped onto the screen is closer to the user's actual gaze area, thereby reducing accidental operations or drift. Simultaneously, the step design dynamically adapts to the user's fast or slow eye movements, making the cursor response smooth and natural, enhancing the fluidity and user experience of human-computer interaction.

[0010] Overall, this method combines spatial modeling, real-time angle calculation, and geometric correction, enabling eye-tracking systems to have both high-precision positioning capabilities and stable operation under various environmental conditions. It improves the accuracy and efficiency of gaze-based operations in video calls, provides a reliable technical foundation for contactless control, assisted operation, or intelligent interaction, and reduces the risk of misjudgment caused by eye movement deviations or optical distortions, making user operation more intuitive, convenient, and comfortable. Attached Figure Description

[0011] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating the steps of the method for implementing screen interaction during a video call according to the present invention. Figure 2 This is a schematic diagram illustrating the LED light source arrangement and light spot formation principle according to an embodiment of the present invention; Figure 3 This is a flowchart of a real-time spot tracking algorithm according to an embodiment of the present invention. Detailed Implementation

[0012] The technical method of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0013] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor methods and / or microcontroller methods.

[0014] It should be understood that although the terms "first," "second," etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are used merely to distinguish one unit from another. For example, without departing from the scope of the exemplary embodiments, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0015] To achieve the above objectives, please refer to Figures 1 to 3 This invention provides a method for implementing screen interaction during a video call, the method comprising the following steps: Step S1: Arrange near-infrared LED point light sources at the edge of the display, acquire images of the user's eyes through a camera, obtain images of four reflected light spots formed on the anterior surface of the cornea, and extract the pixel coordinate positions of each reflected light spot; Step S2: During the system initialization phase, record the four reference positions of the light spot when the user looks directly at the center of the screen, and establish a two-way mapping relationship between the light spot displacement and the horizontal and vertical rotation angles of the eyeball; Step S3: During real-time interaction, calculate the displacement of each light spot in the current frame relative to the previous frame, and extract the corresponding eye rotation angle based on the bidirectional mapping relationship. Step S4: Rotate the three-dimensional position vector of the pupil center relative to the eye rotation center by the eye rotation angle, and add it to the preset corneal curvature radius to obtain the real-time position of the pupil center in the camera coordinate system; Step S5: Map the real-time position to the display screen coordinate system to obtain the screen coordinates of the user's current gaze point; use the screen coordinates to make the cursor move with the user's gaze point, and trigger the selection or confirmation operation when the gaze point stays in the interactive area for more than a preset time threshold to realize eye-tracking interaction.

[0016] See Figure 2Four 850nm near-infrared LED light sources (LED1-LED4) are precisely positioned at the four corners of the 16:9 display. The user's eyeball is located 600mm above the display, with a corneal curvature radius of 7.8mm. The LED light sources illuminate the corneal surface of the eyeball, forming four reflected light spots A1-A4, which converge to the camera at the top of the display. The diagram clearly labels the symbols for each component: LED light source, reflected light spots, display outline, camera, eyeball cross-section, etc., providing a complete technical explanation of the system hardware layout and optical principles.

[0017] Optionally, determining the spot displacement in step S2 includes the following specific steps: During the system initialization phase, the display screen sequentially shows multiple calibration points distributed at preset grid positions, with each calibration point being displayed for 1-3 seconds. In one embodiment, during system initialization, multiple calibration points are first displayed sequentially on the screen according to a preset grid. These calibration points are typically designed as small dots or cross-shaped symbols, with colors and brightness that contrast sharply with the background, facilitating quick user positioning. Each calibration point is displayed individually for 1-3 seconds. For example, in a 5×5 grid, the system will first display the first point in the center of the screen, pause for 2 seconds, and then switch to the next point, until all calibration points are displayed sequentially. During the display process, the screen remains stable, and the near-infrared light source behind it continues to operate, ensuring that the reflected light spots from the eyes can be clearly captured by the camera.

[0018] For example, on a monitor with a resolution of 1920×1080, 25 calibration points can be displayed sequentially, with each point displayed for 2 seconds. The system automatically records the user's eye state during the viewing period.

[0019] As the user gazes at each calibration point, multiple frames of eye images are continuously acquired, and the displacement vector data of the four light spots relative to the central reference position in each frame are calculated. In one embodiment, when a user gazes at a calibration point, the camera continuously captures multiple frames of eye images, at a rate of dozens or even hundreds of frames per second. In each frame, the system identifies four light spots formed on the corneal surface and records their precise pixel positions. To determine the relative positions of the light spots, the system compares them with the position of a reference light spot captured when the user is looking directly at the center of the screen, thus obtaining the light spot offset for each frame. The entire calibration point display process lasts several seconds, during which the system obtains a series of continuous light spot displacement data.

[0020] For example, if the camera captures 120 frames of images within a 2-second display period at a certain calibration point, the system will obtain 120 sets of spot displacement data, which will be used for subsequent stability screening.

[0021] By removing blink and micro-motion interference from the displacement vector data, the stable spot displacement corresponding to each calibration point is obtained.

[0022] In one embodiment, after obtaining multiple frames of spot displacement data, the system needs to remove abnormal frames caused by interference such as blinking or slight head movements. For example, if the number of spots in a frame is less than four, or if the position of a spot suddenly deviates significantly from the majority of spots, that frame will be discarded. For the remaining valid frames, the system will further analyze the overall trend of spot changes; if the change in a particular frame is significantly higher than the normal gaze range, it will also be discarded. Finally, the system will extract the average or median from the remaining stable frames as the stable spot displacement result corresponding to that calibration point.

[0023] For example, a total of 120 frames of data were collected at a certain calibration point. Of these, 18 frames were discarded due to blinking or abnormal movement. In the remaining 102 frames, the system obtained a set of stable spot displacement values ​​by taking the median value and saved them for subsequent mapping modeling.

[0024] It should be noted that in practical applications, the number of grid points, the display duration of each point, and the criteria for removing interfering data can all be flexibly adjusted according to user habits and device performance. For example, for users using the system for the first time, the number of calibration points can be increased or the display time extended to improve the accuracy of modeling.

[0025] Optionally, the calculation of the horizontal and vertical rotation angles of the eyeball in step S2 is as follows: The horizontal and vertical rotation angles are calculated based on the screen position of the calibration point and the pre-acquired eye-screen distance. The horizontal rotation angle is the arctangent of the horizontal displacement of the light spot and the eye-screen distance, and the vertical rotation angle is the arctangent of the vertical displacement of the light spot and the eye-screen distance.

[0026] In one embodiment, during the calibration phase, the position of the current calibration point in the screen coordinate system and the pre-measured eye-to-screen distance are first obtained. The horizontal displacement is obtained by subtracting the horizontal position of the calibration point on the screen from the horizontal reference position at the center of the screen; similarly, the vertical displacement is obtained by subtracting the vertical position from the vertical reference position at the center of the screen. If the displacement is in pixels, the pixel displacement needs to be converted into a physical length unit (e.g., millimeters) that is the same as the eye-to-screen distance, based on the display pixel pitch or the calibration relationship between the camera and the screen. Then, the horizontal displacement is removed by the eye-to-screen distance and the arctangent of the resulting ratio is taken to obtain the corresponding horizontal rotation angle of the eyeball; the vertical displacement is removed by the eye-to-screen distance and the arctangent is taken to obtain the corresponding vertical rotation angle of the eyeball. This angle calculation is usually saved as the angle label of each calibration point after completion, for subsequent use in establishing displacement-angle mapping.

[0027] For example, on a monitor, the center of the screen is used as a reference point. A calibration point is horizontally offset 60 pixels to the right of the center, but vertically unchanged. Assuming each pixel on the screen corresponds to 0.264 millimeters, and the pre-measured eye-to-screen distance is 600 millimeters, the horizontal displacement is approximately 60 × 0.264 = 15.84 millimeters. The horizontal rotation angle is the arctangent of the ratio 15.84 divided by 600, approximately 1.5°. If another calibration point is vertically offset 80 pixels (approximately 21.12 millimeters) downwards, the vertical rotation angle can be calculated to be approximately 2.0°. These angle values, along with the corresponding spot displacement, are stored in a database for subsequent forward and reverse lookup operations.

[0028] Optionally, the construction of the bidirectional mapping relationship in step S2 includes: The spot displacement, horizontal rotation angle and vertical rotation angle of each calibration point are used to form an initial mapping data pair. Each data pair contains four two-dimensional displacement vectors of the spot and the corresponding two-dimensional rotation angle. In one embodiment, after calibration, the data collected at each calibration point is organized into an initial mapping record. The record includes the two-dimensional displacements of the four corneal reflective spots corresponding to the calibration point (the horizontal and vertical displacements of each spot), as well as the horizontal and vertical rotation angles calculated from the screen position and the distance between the eye and screen. All calibration points are sequentially saved into a table or database to form a complete initial mapping dataset, which facilitates subsequent filtering and modeling.

[0029] For example, for a 5×5 grid with 25 calibration points, the system will generate 25 records. Each record contains the displacement values ​​of four light spots and the corresponding two angle values, all of which are stored in the database for subsequent processing.

[0030] Select initial mapping data to establish temporary mapping relationships for some calibration points, and use the data of the remaining calibration points to verify the accuracy of the temporary mapping relationships. Use each calibration point as a verification point for iterative verification, and remove abnormal data pairs whose verification error angle exceeds the preset error threshold to obtain verification mapping pairs. In one embodiment, a subset of points is selected from the initial mapping set (e.g., through leave-one-out or proportional random sampling) to establish a temporary mapping relationship. The temporary mapping can employ easily implemented methods such as simple lookup table interpolation or local regression to map the spot displacement to a rotation angle. Subsequently, the remaining calibration points are used sequentially as verification points for iterative verification: for each verification point, the temporary mapping is used to predict its expected angle, and the predicted angle is compared with the actual measured angle at that point. If the error between the two exceeds a preset angle threshold (e.g., an empirical threshold, configurable), the verification point is marked as abnormal and removed. By iterating through each calibration point as a verification point, data pairs that exhibit inconsistent mapping relationships can be identified and eliminated, leaving only the "verification mapping pairs."

[0031] For example, when using the leave-one-out method for iterative verification among 25 points, if two points consistently show large errors in multiple verifications, the system will remove these two records from the mapping set, and the remaining 23 records will be used as verification mapping pairs.

[0032] In the horizontal and vertical directions of the verification mapping pair, virtual data points are inserted between adjacent calibration points to expand the initial points into a higher-density point set as a densified mapping grid.

[0033] In one embodiment, for verified mapping pairs, virtual data points are inserted between adjacent points in the horizontal direction (between adjacent points in the same row) and the vertical direction (between adjacent points in the same column) to increase grid density. The data for the virtual points can be generated by linear or bidirectional interpolation of adjacent real points, i.e., simultaneously interpolating the spot displacement and the corresponding angle to obtain new displacement-angle pairs, which are then written into the denser grid. This expands the sparse calibration point set into a higher resolution mapping table, facilitating subsequent real-time table lookups and fast interpolation.

[0034] For example, the original 5×5 verification point grid is modified by inserting a virtual point between each adjacent real point to obtain an approximately 9×9 denser grid. The system also saves these virtual points as lookup table entries.

[0035] It should be noted that the verification error threshold and interpolation strategy should be set according to the camera resolution, eye-screen distance, and actual noise level. Too strict a threshold will discard too much data, while too lenient a threshold will leave unreliable points. When inserting virtual points, if adjacent real points differ significantly (e.g., nonlinear behavior in edge regions), local smoothing or piecewise interpolation should be performed first to avoid generating unreasonable mapping values. It is recommended that outlier identification be based on a combination of multiple indicators (angle error, displacement consistency, spot intensity, etc.) rather than relying solely on a single factor to improve robustness.

[0036] Optionally, the construction of the bidirectional mapping relationship in step S2 further includes: Using a dense mapping mesh, the horizontal and vertical components of the spot displacement are used as two-dimensional indices to store the corresponding rotation angle values, and a forward lookup table is constructed. The resolution of the forward lookup table is set within a preset pixel precision range. In one embodiment, after completing the mesh densification mapping, a mapping table from displacement to angle can be established using the spot displacement as an index. Specifically, the horizontal and vertical components of the spot displacement at each grid point are selected and treated as two-dimensional coordinate index positions, where the previously calculated horizontal and vertical rotation angles are stored. This way, when a new spot displacement is subsequently acquired, the corresponding rotation angle can be quickly obtained by looking up the table.

[0037] For example, when a grid point has a horizontal displacement of 10 pixels and a vertical displacement of 5 pixels, the calculated rotation angle is 3 degrees horizontally and 2 degrees vertically. The angle (3,2) is stored at the (10,5) index in the lookup table.

[0038] It should be noted that the resolution of the forward lookup table can be set according to the precision of the display screen and camera, for example, by creating indexes in 0.5-pixel intervals to ensure sufficient precision.

[0039] Using a dense mapping mesh with the horizontal and vertical components of the rotation angle as two-dimensional indices, the corresponding spot displacement is stored to construct a reverse lookup table, where the resolution of the reverse lookup table is set within a preset angle accuracy range. In one embodiment, based on the forward lookup table, a reverse lookup table from angle to displacement can also be established. This is done by using the rotation angle of each point in the dense mapping mesh as a two-dimensional index, storing the corresponding spot displacement at that location. Thus, if the system needs to deduce the spot displacement from the rotation angle, it can directly look up the result in the table.

[0040] For example, at a certain grid point, the rotation angle is 2.5 degrees horizontally and 1.8 degrees vertically, and the corresponding spot displacement is (8,4) pixels. Then (8,4) is stored at the index (2.5,1.8) in the reverse lookup table.

[0041] It should be noted that the resolution here is usually set in terms of angular range, such as creating an index every 0.1 degrees to ensure accuracy during reverse lookup.

[0042] The angle obtained by forward lookup is converted back to displacement by reverse lookup using the forward lookup table and the reverse lookup table. If the conversion error exceeds 1 pixel, the mapping data of the area is locally smoothed to obtain a bidirectional mapping relationship.

[0043] In one embodiment, to ensure the consistency of the two lookup tables, bidirectional verification can be performed. Specifically, the forward lookup table is used to find the rotation angle based on the spot displacement, and then the angle is converted back to the displacement using the reverse lookup table. If the returned displacement differs significantly from the original input displacement, for example, by more than one pixel, it indicates that the mapping data in that area is not smooth or has accumulated errors. To address this, a smoothing interpolation method can be used within the local grid of that area to make the mapping relationship more continuous and stable.

[0044] For example, assuming the input displacement is (10, 6), the forward lookup yields the angle (3.2, 2.1), and the reverse lookup yields (11, 7), the error reaches [missing value]. If the number of pixels exceeds the threshold of 1 pixel, the mapping points of that area need to be corrected, for example, by using the average value of the surrounding points for local smoothing.

[0045] It should be noted that this smoothing process does not significantly alter the mapping relationship; it only makes minor adjustments to local abnormal areas to ensure the stability and accuracy of the table lookup process.

[0046] Optionally, step S3 includes the following steps: Step S31: Establish a search window in the current frame with the position of the light spot in the previous frame as a reference, and determine the precise position of the light spot through brightness gradient convergence analysis; In one embodiment, during each frame processing, a small pixel search window is established for each of the four light spots centered on the precise positions of the previous frame. Then, the precise center of the light spot is determined within the window using a brightness gradient convergence method. Specifically, the brightness gradient (or radial brightness change) of each pixel within the window is taken, and the location where the gradient converges most strongly is identified as the center of the light spot, thus achieving sub-pixel-level positioning. To balance speed and robustness, the search window size can be dynamically adjusted according to the eye's rotation speed: a small window (e.g., 11×11 pixels) is used when the rotation speed is low, and a larger window (e.g., 21×21 pixels) is used when the speed is high, to avoid losing the light spot during rapid movement.

[0047] For example, if a spot in the previous frame was located at (512, 400), and it is currently determined to be moving at a low speed, then gradient convergence is performed within an 11×11 window centered on that point, and finally the position is located at (512.4, 400.7) as the precise position in this frame.

[0048] Step S32: Calculate the displacement vector of each spot's precise position in the current frame relative to the spot's precise position in the previous frame, and calculate the consistency index of the four spot displacement vectors. When the deviation of any spot's displacement vector from the average of the other three spot displacement vectors exceeds 3 pixels, mark the spot as an abnormal spot. In one embodiment, the precise position of each spot in the current frame is subtracted from the precise position of the corresponding spot in the previous frame to obtain four displacement vectors (i.e., the pixel change of each spot in the horizontal and vertical directions). The average of these four displacement vectors is then calculated as the overall displacement reference, and the difference between the displacement of each spot and this average is calculated. When the difference of a spot exceeds a preset threshold (3 pixels in this scheme), the spot is marked as abnormal (e.g., momentarily occluded or affected by noise); otherwise, it is considered a normal spot and continues to be used.

[0049] For example, if the four displacements are (+7, +3), (+6, +2), (+7, +3), and (+12, +1), with an average of approximately (+8, +2.25), and the fourth spot deviates significantly from the average (more than 3 pixels), then it is marked as an anomaly.

[0050] Step S33: For normal light spots, input their displacement into a forward lookup table of bidirectional mapping relationship, and obtain the initial value of the corresponding eye rotation angle by looking up the table; for abnormal light spots, calculate the average value of the initial angle obtained from the other normal light spots in the current frame for compensation. In one embodiment, for light spots judged to be normal, their displacement values ​​are directly input into a pre-constructed forward lookup table to quickly find the corresponding horizontal and vertical rotation angles of the eyeball as the initial angle values ​​of the light spot; for light spots marked as abnormal, their displacement in this frame is not used directly, but the average value of the initial angle values ​​obtained from the lookup table of other normal light spots in the current frame is used for compensation, thereby generating an alternative initial angle value for the abnormal light spot, ensuring the continuity and robustness of subsequent angle calculations.

[0051] For example, if the initial angle values ​​of the three normal light spots obtained from the table are (2.9 degrees, 1.1 degrees), (2.8 degrees, 1.0 degrees), and (2.9 degrees, 1.2 degrees), then the average value of these three sets of angles (approximately 2.87 degrees, 1.1 degrees) is used to compensate for the initial angle value of the fourth abnormal light spot.

[0052] Step S34: The weighted average of the current initial angle value and the initial eye rotation angle values ​​of the previous two frames is set to 0.5, 0.3 and 0.2 respectively to obtain the eye rotation angle.

[0053] In one embodiment, to smooth out transient noise while maintaining response speed, the initial angle value obtained in the current frame is weighted and averaged with the initial angle values ​​of the previous two frames at fixed ratios of 0.5, 0.3, and 0.2, respectively, to obtain the final eye rotation angle output for the current frame. This allows for rapid response to changes in the current gaze point while suppressing short-term jitter.

[0054] For example, if the current initial value is 2.87 degrees, the previous frame was 2.90 degrees, and the frame before that was 2.85 degrees, then the weighted value is approximately 2.88 degrees (0.5×2.87+0.3×2.90+0.2×2.85).

[0055] It should be noted that if the number of anomalous light spots in the same frame reaches two or more, simply "compensating by averaging the remaining light spots" may not be reliable enough. In this case, it is necessary to revert to time-based smoothing estimation or use the angle history values ​​from the most recent multiple frames for predictive compensation, and trigger short-term relocalization or rapid recalibration if necessary. If an exact index value cannot be found when looking up the table, the initial angle value can be estimated using the nearest neighbor interpolation method; the table lookup accuracy and interpolation strategy should be consistent with the mesh densification strategy in the calibration stage. The weighting coefficients and anomaly detection thresholds can be optimized and adjusted according to the device resolution, camera frame rate, and user group characteristics; it is recommended to keep these parameters as configurable items in the implementation specification.

[0056] Of particular importance, step S31 includes: In the current frame image, a search window is established with the positions of each light spot in the previous frame as the center. The size of the search window is dynamically adjusted according to the eye rotation speed in the previous frame. When the rotation speed is less than 30 degrees per second, the window is set to 11×11 pixels, and when the rotation speed is greater than 30 degrees per second, the window is expanded to 21×21 pixels. In one embodiment, a search window is created in the current frame centered on the positions of each light spot from the previous frame, and its size is dynamically adjusted. Each light spot has a small window centered on its pixel position from the previous frame. The size of the window is automatically selected based on the estimated eye rotation speed from the previous frame: a smaller window (e.g., 11×11 pixels) is used when the rotation speed is less than 30 degrees / second to reduce computation and improve positioning accuracy; when the rotation speed is greater than 30 degrees / second, the window is enlarged (e.g., 21×21 pixels) to cover a larger displacement range and prevent the light spot from falling out of the search area due to rapid movement. The window is cropped and offset when it approaches the image boundary, and the window size and shape can be adjusted separately for the left and right eyes to adapt to lens distortion and viewing angle differences.

[0057] For example, if a spot in the previous frame is located at screen pixel (512,400) and the rotation speed estimated from the last two frames is 20 degrees / second, then the search is performed within an 11×11 pixel window at that location; if the speed is 45 degrees / second, then the search is expanded to 21×21 pixels to ensure that no spot is missed.

[0058] Within the search window, the center of the light spot is located by brightness gradient analysis. The radial gradient value of each pixel within the window is calculated, and the gradient convergence point is selected as the precise location of the light spot.

[0059] In one embodiment, the center of the light spot is located within the search window through brightness gradient convergence analysis. Specifically, the direction and intensity of brightness changes in each pixel within the window are analyzed, with a focus on calculating the convergence of radial brightness gradients from surrounding pixels towards a candidate point. At the corneal reflection point, the surrounding brightness gradients exhibit a clear centripetal or converging trend, and the intersection of gradient vectors is usually the true center of the light spot. The system finds the location with the strongest gradient convergence within the window as the precise location of the light spot and performs sub-pixel-level fine-tuning on this location (e.g., fine-tuning the center position based on local intensity centroids or small-range fitting) to improve positioning accuracy. When multiple gradient convergence points appear within the window, the point closest to the window center and with the largest gradient energy is selected first; if no obvious convergence point is found, the light spot in this frame is determined to be lost or occluded, and the system enters the anomaly handling process.

[0060] For example, if a significant radial gradient convergence is found at a certain pixel within an 11×11 window, the system fine-tunes that point to (512.6, 400.3) and uses it as the precise spot position for this frame.

[0061] It should be noted that the rotation speed can be estimated by dividing the angle change of the most recent few frames by the time difference. In the actual implementation, this speed should be smoothed to avoid frequent switching of window size caused by jitter. Expanding the window can improve robustness but also increases the amount of computation. For devices with high frame rates or limited computing power, pyramid / multi-scale search or coarse-to-fine stepwise search can be used to balance speed and accuracy. The determination of gradient convergence points should combine intensity threshold and gradient energy to avoid mistakenly taking noise or eyelid edges as convergence points. If a reliable convergence point cannot be located for several consecutive frames, a short-term relocation should be triggered or the center reference should be returned for recalibration.

[0062] See Figure 3 The algorithm begins by "reading the current frame" and establishes a dynamic search window of 11×11 or 21×21 pixels based on the eye rotation speed. Gradient convergence analysis is used to determine the precise position of the light spot, and the displacement vectors of the four light spots are calculated. When the light spot displacement deviation exceeds 3 pixels, it is considered an anomaly and is handled normally or with mean compensation. Finally, the current frame angle value is time-weighted and averaged with the previous two frames using a weighting of 0.5:0.3:0.2 to output a smooth eye rotation angle.

[0063] Optionally, step S4 includes the following steps: Step S41: Establish an eye coordinate system with the eye rotation center as the origin and determine the initial three-dimensional position vector of the pupil center, where the eye rotation center is located 13.5 mm behind the corneal apex and the initial position of the pupil center is located 9.9 mm in front of the origin of the eye coordinate system; In one embodiment, an eye coordinate system is established during the calibration phase, with the eye's rotation center as the origin. Conventionally, the direction facing the camera is defined as the Z-axis (forward), the X-axis (to the right), and the Y-axis (upward). The eye's rotation center is located approximately 13.5 mm behind the corneal apex (i.e., the corneal apex is approximately 13.5 mm directly in front of the coordinate system). Under this coordinate system, the initial three-dimensional position of the pupil center is denoted as "9.9 mm in front of the origin," meaning the initial pupil position is approximately 9.9 mm from the rotation center, located in the positive Z-axis direction, and serves as the reference point for subsequent rotation calculations.

[0064] For example, the initial position of the pupil in the eye coordinate system can be written as (0, 0, 9.9) mm, which means it is 9.9 mm in front of the center of rotation.

[0065] Step S42: Calculate the position of the pupil center after rotation using the eyeball rotation angle and the initial three-dimensional position vector, and record it as the pupil position after rotation; In one embodiment, after obtaining the horizontal and vertical rotation angles of the eyeball in real time, the initial three-dimensional position of the pupil is rotated to a new position in the order of first horizontal rotation and then vertical rotation. In practice, the rotation order of "first applying a horizontal angle around the vertical axis, and then applying a vertical angle around the horizontal direction" can be written into code or implemented by calling an existing three-dimensional rotation function. The rotation result maintains the same length as the initial vector but changes its direction, thus obtaining the "pupil position after rotation" for the current frame.

[0066] For example, if the current horizontal angle is 1.5° and the vertical angle is 2.0°, after rotating the initial position (0, 0, 9.9) mm horizontally and vertically, the pupil position after rotation is approximately (0.259, 0.345, 9.891) mm (the values ​​are approximate examples used to illustrate the calculation process).

[0067] Step S43: Convert the rotated pupil position from the eye coordinate system to the camera coordinate system, and add it to the preset corneal curvature radius to obtain the real-time position of the pupil center in the camera coordinate system.

[0068] In one embodiment, the rotated pupil position is transformed into the camera coordinate system using a rigid transformation (including translation and possible small-angle directional correction) from the eye coordinate system to the camera coordinate system measured during initialization, thereby mapping the "intraocular coordinates" to the "camera coordinates". To account for the geometric influence of the spherical shape of the cornea on the reflection point and the actual pupil position, a fine adjustment based on a preset corneal curvature radius is made at this position along the corneal normal direction (i.e., extrapolating the corneal radius length along a unit vector pointing from the rotation center to the pupil direction as a correction offset), finally obtaining the real-time position of the pupil center in the camera coordinate system for subsequent fixation point mapping.

[0069] For example, using the rotated position (0.259, 0.345, 9.891) mm from the previous example, if the corneal curvature radius is set to 7.8 mm, extrapolating along the same direction will yield a corrected position of approximately (0.463, 0.618, 17.683) mm (these are exemplary values; in actual applications, coordinate system calibration must be performed before calculation).

[0070] It should be noted that the definitions of coordinate axes and rotation order must be fixed during the calibration stage and strictly followed in real-time processing. Changing the rotation order will lead to different results. The camera coordinate transformation depends on the translation and rotation relationship obtained from the initial calibration. It is recommended that this calibration be completed and stored when the device leaves the factory or is installed for the first time. The corneal curvature radius can be the average value, or individualized calibration can be performed when needed to improve accuracy.

[0071] Optionally, step S42 includes: Calculate the trigonometric function values ​​of the eyeball rotation angle on the horizontal axis and the vertical axis respectively, and record them as the basic rotation data set; In one embodiment, trigonometric function values, including sine and cosine, are calculated for the real-time acquired horizontal and vertical eye rotation angles, and these values ​​are recorded as a basic rotation data set. These trigonometric function values ​​will serve as calculation parameters for subsequent three-dimensional spatial rotations. For example, assuming the current horizontal rotation angle is 2° and the vertical rotation angle is 1.5°, the horizontal angle cosine is calculated to be approximately 0.999 and the sine to be approximately 0.035; the vertical angle cosine is approximately 0.9997 and the sine to be approximately 0.026, forming the basic rotation data set.

[0072] The calculation parameters for three-dimensional spatial rotation are constructed using the basic rotation data set. The transformation coefficients of the X and Z axes are determined by the cosine and sine values ​​of the horizontal rotation angle, and the transformation coefficients of the Y and Z axes are determined by the cosine and sine values ​​of the vertical rotation angle, thus forming the spatial transformation coefficients. In one embodiment, transformation coefficients for three-dimensional spatial rotation are constructed using a base rotation data set. For example, before rotation, the X-axis coordinate is 0, the Y-axis coordinate is 0, and the Z-axis coordinate is 9.9 mm. The spatial transformation coefficients can be directly used for coordinate update calculations.

[0073] The X, Y, and Z coordinates of the initial 3D position vector are multiplied by the corresponding spatial transformation coefficients, and the theoretical position coordinates after rotation are calculated in the order of horizontal rotation first and then vertical rotation. In one embodiment, the coordinates of the initial three-dimensional position vector are multiplied by their corresponding spatial transformation coefficients, and the coordinates are updated sequentially in the order of horizontal rotation followed by vertical rotation to obtain the theoretically rotated pupil position coordinates. For example, the initial position (0, 0, 9.9 mm) is rotated horizontally by 2° and then vertically by 1.5°, and the theoretically rotated position is approximately (0.309, 0.259, 9.889) mm (the values ​​are for illustrative purposes only).

[0074] The distance from the theoretically rotated position coordinates to the eyeball rotation center is calculated. When the distance value is not at the position of the pupil after rotation, the position of the pupil after rotation is obtained.

[0075] In one embodiment, the actual distance from the theoretically rotated position to the center of eye rotation is calculated. If the distance deviates from a preset pupil radius range, correction is performed to obtain the final rotated pupil position. For example, if the calculated distance is 9.88 mm, while the actual distance from the pupil to the rotation center is 9.9 mm, a fine adjustment of 0.02 mm is made along the direction to maintain accurate positioning.

[0076] It should be noted that the rotation order in this step must strictly follow the principle of horizontal first and then vertical, otherwise the result will be inaccurate; it is recommended to use high-precision floating-point numbers to calculate trigonometric function values ​​to reduce accumulated errors; the correction of the deviation between the theoretical position and the actual pupil radius is an important step in ensuring the accuracy of three-dimensional spatial positioning.

[0077] Of particular importance, step S43 includes: Represent the position of the pupil after rotation as a three-dimensional vector in the eye coordinate system; obtain the position coordinates of the eye rotation center in the camera coordinate system; translate the origin of the eye coordinate system from the eye rotation center to the origin of the camera coordinate system, and the translation vector is equal to the camera coordinate of the eye rotation center; add the corresponding translation vector component to each coordinate component of the pupil position after rotation to complete the coordinate system transformation. Add a corneal curvature radius of 7.8 mm to the transformed Z-axis coordinates to compensate for the depth shift caused by corneal protrusion; verify the rationality of the final coordinates, the X and Y coordinates should be within ±50 mm, and the Z coordinate should be within 500 to 700 mm; if the coordinates exceed the reasonable range, use the coordinate values ​​of the previous valid frame to replace the current calculation results to ensure that the output real-time position is continuous and stable.

[0078] The present invention also provides a system for implementing screen interaction during a video call, used to execute the method for implementing screen interaction during a video call as described above, wherein the system for implementing screen interaction during a video call includes: The light spot acquisition module is used to place near-infrared LED point light sources with a wavelength of 850 nanometers at the four corners of the display, acquire images of the user's eyes through the camera, obtain images of four reflected light spots formed on the anterior surface of the cornea, and extract the pixel coordinate position of each reflected light spot. The light spot mapping modeling module is used to record the four light spot reference positions when the user looks directly at the center of the screen during the system initialization phase, and to establish a two-way mapping relationship between the light spot displacement and the horizontal and vertical rotation angles of the eyeball. The light spot displacement calculation module is used to calculate the displacement of each light spot in the current frame relative to the previous frame during real-time interaction, and extract the corresponding eye rotation angle based on the bidirectional mapping relationship. The pupil position estimation module is used to rotate the three-dimensional position vector of the pupil center relative to the eye rotation center by the eye rotation angle, and add it to the preset corneal curvature radius to obtain the real-time position of the pupil center in the camera coordinate system. The gaze point interaction module is used to map the real-time position to the display screen coordinate system to obtain the screen coordinates of the user's current gaze point; the screen coordinates are used to make the cursor move with the user's gaze point, and when the gaze point stays in the interactive area for more than a preset time threshold, a selection or confirmation operation is triggered to realize eye-tracking interaction.

[0079] The present invention also provides a terminal for implementing screen interaction during video calls, comprising: The display has near-infrared LED point light sources arranged at its four corners; A camera used to capture images of the user's eyes; processor; A memory storing a computer program that, when executed by the processor, implements a method for screen interaction during a video call as described in any of the preceding claims.

[0080] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of the equivalents of the application are intended to be included within the invention.

[0081] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features of the invention herein.

Claims

1. A method for implementing screen interaction during a video call, characterized in that, Includes the following steps: Step S1: Arrange near-infrared LED point light sources at the edge of the display, acquire images of the user's eyes through a camera, obtain images of four reflected light spots formed on the anterior surface of the cornea, and extract the pixel coordinate positions of each reflected light spot; Step S2: During the system initialization phase, record the four reference positions of the light spot when the user looks directly at the center of the screen, and establish a two-way mapping relationship between the light spot displacement and the horizontal and vertical rotation angles of the eyeball; Step S3: During real-time interaction, calculate the displacement of each light spot in the current frame relative to the previous frame, and extract the corresponding eye rotation angle based on the bidirectional mapping relationship. Step S4: Rotate the three-dimensional position vector of the pupil center relative to the eye rotation center by the eye rotation angle, and add it to the preset corneal curvature radius to obtain the real-time position of the pupil center in the camera coordinate system; Step S5: Map the real-time position to the display screen coordinate system to obtain the screen coordinates of the user's current gaze point; use the screen coordinates to make the cursor move with the user's gaze point, and trigger the selection or confirmation operation when the gaze point stays in the interactive area for more than a preset time threshold to realize eye-tracking interaction.

2. The method for implementing screen interaction during a video call according to claim 1, characterized in that, Determining the spot displacement in step S2 includes the following steps: During the system initialization phase, the display screen sequentially shows multiple calibration points distributed at preset grid positions, with each calibration point being displayed for 1-3 seconds. As the user gazes at each calibration point, multiple frames of eye images are continuously acquired, and the displacement vector data of the four light spots relative to the central reference position in each frame are calculated. By removing blink and micro-motion interference from the displacement vector data, the stable spot displacement corresponding to each calibration point is obtained.

3. The method for implementing screen interaction during a video call according to claim 2, characterized in that, The calculation of the horizontal and vertical rotation angles of the eyeball in step S2 is as follows: The horizontal and vertical rotation angles are calculated based on the screen position of the calibration point and the pre-acquired eye-screen distance. The horizontal rotation angle is the arctangent of the horizontal displacement of the light spot and the eye-screen distance, and the vertical rotation angle is the arctangent of the vertical displacement of the light spot and the eye-screen distance.

4. The method for implementing screen interaction during a video call according to claim 3, characterized in that, The construction of the bidirectional mapping relationship in step S2 includes: The spot displacement, horizontal rotation angle and vertical rotation angle of each calibration point are used to form an initial mapping data pair. Each data pair contains four two-dimensional displacement vectors of the spot and the corresponding two-dimensional rotation angle. Select initial mapping data to establish temporary mapping relationships for some calibration points, and use the data of the remaining calibration points to verify the accuracy of the temporary mapping relationships. Use each calibration point as a verification point for iterative verification, and remove abnormal data pairs whose verification error angle exceeds the preset error threshold to obtain verification mapping pairs. In the horizontal and vertical directions of the verification mapping pair, virtual data points are inserted between adjacent calibration points to expand the initial points into a higher-density point set as a densified mapping grid.

5. The method for implementing screen interaction during a video call according to claim 4, characterized in that, The construction of the bidirectional mapping relationship in step S2 also includes: Using a dense mapping mesh, the horizontal and vertical components of the spot displacement are used as two-dimensional indices to store the corresponding rotation angle values, and a forward lookup table is constructed. The resolution of the forward lookup table is set within a preset pixel precision range. Using a dense mapping mesh with the horizontal and vertical components of the rotation angle as two-dimensional indices, the corresponding spot displacement is stored to construct a reverse lookup table, where the resolution of the reverse lookup table is set within a preset angle accuracy range. The angle obtained by forward lookup is converted back to displacement by reverse lookup using the forward lookup table and the reverse lookup table. If the conversion error exceeds 1 pixel, the mapping data of the area is locally smoothed to obtain a bidirectional mapping relationship.

6. The method for implementing screen interaction during a video call according to claim 5, characterized in that, Step S3 includes the following steps: Step S31: Establish a search window in the current frame with the position of the light spot in the previous frame as a reference, and determine the precise position of the light spot through brightness gradient convergence analysis; Step S32: Calculate the displacement vector of each spot's precise position in the current frame relative to the spot's precise position in the previous frame, and calculate the consistency index of the four spot displacement vectors. When the deviation of any spot's displacement vector from the average of the other three spot displacement vectors exceeds 3 pixels, mark the spot as an abnormal spot. Step S33: For normal light spots, input their displacement into a forward lookup table of bidirectional mapping relationship, and obtain the initial value of the corresponding eye rotation angle by looking up the table; for abnormal light spots, calculate the average value of the initial angle obtained from the other normal light spots in the current frame for compensation. Step S34: The weighted average of the current initial angle value and the initial eye rotation angle values ​​of the previous two frames is set to 0.5, 0.3 and 0.2 respectively to obtain the eye rotation angle.

7. The method for implementing screen interaction during a video call according to claim 6, characterized in that, Step S4 includes the following steps: Step S41: Establish an eye coordinate system with the eye rotation center as the origin and determine the initial three-dimensional position vector of the pupil center, where the eye rotation center is located 13.5 mm behind the corneal apex and the initial position of the pupil center is located 9.9 mm in front of the origin of the eye coordinate system; Step S42: Calculate the position of the pupil center after rotation using the eyeball rotation angle and the initial three-dimensional position vector, and record it as the pupil position after rotation; Step S43: Convert the rotated pupil position from the eye coordinate system to the camera coordinate system, and add it to the preset corneal curvature radius to obtain the real-time position of the pupil center in the camera coordinate system.

8. The method for implementing screen interaction during a video call according to claim 7, characterized in that, Step S42 includes: Calculate the trigonometric function values ​​of the eyeball rotation angle on the horizontal axis and the vertical axis respectively, and record them as the basic rotation data set; The calculation parameters for three-dimensional spatial rotation are constructed using the basic rotation data set. The transformation coefficients of the X and Z axes are determined by the cosine and sine values ​​of the horizontal rotation angle, and the transformation coefficients of the Y and Z axes are determined by the cosine and sine values ​​of the vertical rotation angle, thus forming the spatial transformation coefficients. The X, Y, and Z coordinates of the initial 3D position vector are multiplied by the corresponding spatial transformation coefficients, and the theoretical position coordinates after rotation are calculated in the order of horizontal rotation first and then vertical rotation. The distance from the theoretically rotated position coordinates to the eyeball rotation center is calculated. When the distance value is not at the position of the pupil after rotation, the position of the pupil after rotation is obtained.

9. A system for enabling screen interaction during a video call, characterized in that, The system for performing the method for implementing screen interaction during a video call as described in claim 1, wherein the system for implementing screen interaction during a video call comprises: The light spot acquisition module is used to place near-infrared LED point light sources with a wavelength of 850 nanometers at the four corners of the display, acquire images of the user's eyes through the camera, obtain images of four reflected light spots formed on the anterior surface of the cornea, and extract the pixel coordinate position of each reflected light spot. The light spot mapping modeling module is used to record the four light spot reference positions when the user looks directly at the center of the screen during the system initialization phase, and to establish a two-way mapping relationship between the light spot displacement and the horizontal and vertical rotation angles of the eyeball. The light spot displacement calculation module is used to calculate the displacement of each light spot in the current frame relative to the previous frame during real-time interaction, and extract the corresponding eye rotation angle based on the bidirectional mapping relationship. The pupil position estimation module is used to rotate the three-dimensional position vector of the pupil center relative to the eye rotation center by the eye rotation angle, and add it to the preset corneal curvature radius to obtain the real-time position of the pupil center in the camera coordinate system. The gaze point interaction module is used to map the real-time position to the display screen coordinate system to obtain the screen coordinates of the user's current gaze point; the screen coordinates are used to make the cursor move with the user's gaze point, and when the gaze point stays in the interactive area for more than a preset time threshold, a selection or confirmation operation is triggered to realize eye-tracking interaction.

10. A terminal for enabling screen interaction during a video call, characterized in that, include: The display has near-infrared LED point light sources arranged at its four corners; A camera used to capture images of the user's eyes; processor; A memory storing a computer program that, when executed by the processor, implements a method for screen interaction during a video call as described in any one of claims 1 to 8.

Images