Television near field interaction control processing method, electronic device and storage medium

Abstract

The application provides a television near-field interaction control processing method, an electronic device and a storage medium. The method comprises: acquiring a plurality of human eye images in a continuous time period collected by a first acquisition device for a user's eye; determining a pupil track and an eyelid closure feature according to the plurality of human eye images, wherein the pupil track comprises pupil center coordinates of the user's pupil center at each time frame in the continuous time period; determining a target action instruction of the eye according to the pupil track, the eyelid closure feature and a preset control instruction set; converting each pupil center coordinate in the pupil track into a television coordinate in a television coordinate system according to a calibrated first coordinate conversion relationship; and sending the target action instruction and each television coordinate to a television end, so that the television end simulates the user's eye movement based on the action instruction and each television coordinate. End-to-end mapping from natural eye movement of the user to precise control of the television end and control level directional operation are realized, and non-perception interaction is realized.

Description

Technical Field

[0001] This application relates to the field of smart glasses technology, and more specifically, to a television near-field interactive control processing method, electronic device, and storage medium. Background Technology

[0002] In recent years, with the development of wearable devices and human-computer interaction technologies, smart glasses, as a new type of near-field interaction terminal, have shown application potential in scenarios such as home entertainment, remote collaboration, and barrier-free interaction. In the field of television remote control, some technologies have already attempted to use smart glasses as input terminals for remote control commands.

[0003] Current smart glasses TV remote control technology uses a monocular camera to capture images of the human eye and identifies the pupil position based on traditional image processing algorithms to achieve simple gaze positioning or blink-triggered operations. However, existing technologies suffer from problems such as disconnected interaction between smart glasses and the TV screen, insufficient recognition accuracy, coarse command mapping, and low naturalness of interaction. Summary of the Invention

[0004] The purpose of this application is to address the shortcomings of the prior art by providing a television near-field interactive control processing method, electronic device, and storage medium to improve the interactivity between smart glasses and the television screen.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a television near-field interactive control processing method, the method comprising: Acquire multiple frames of human eye images within a continuous time period captured by the first acquisition device targeting the user's eyes; The pupil trajectory and eyelid closure features within the continuous time period are determined based on the multi-frame human eye images. The pupil trajectory includes the pupil center coordinates of the user in each time frame of the continuous time period. The pupil center coordinates are two-dimensional coordinates in the coordinate system of the lens display interface. Based on the pupil trajectory, eyelid closure characteristics, and preset control command set, the target action command for the eye is determined; According to the calibrated first coordinate transformation relationship, the coordinates of each pupil center in the pupil trajectory are converted into television coordinates in the television coordinate system. The first coordinate transformation relationship is the coordinate transformation relationship between the lens interface coordinate system and the television screen coordinate system. The target action command and each of the TV coordinates are sent to the TV terminal so that the TV terminal can simulate the user's eye movements based on the action command and each of the TV coordinates.

[0006] Optionally, the calibration process of the first coordinate transformation relationship includes: Acquire a first television image captured by the second acquisition device and a second television image that is projected in real time onto the lens display interface via a transmissive semi-transparent lens. Control contours are extracted from the first television image and the second television image respectively to obtain a first set of feature points corresponding to the first television image and a second set of feature points corresponding to the second television image. The first coordinate transformation relationship is determined using the least squares method based on the first set of feature points, the second set of feature points, the spatial parameters between the lens display interface and the television.

[0007] Optionally, determining the pupil trajectory and eyelid closure features within the continuous time period based on the multiple frames of human eye images includes: Face detection, eye localization and feature extraction are performed sequentially on each human eye image in the multi-frame human eye images to obtain the pupil center coordinates and eye contour features corresponding to each human eye image. The coordinate system of the human eye image is the same as the coordinate system of the lens display interface. The eyelid closure features are determined based on the eye contour features corresponding to each of the aforementioned human eye images; Based on the pupil center coordinates corresponding to each of the aforementioned human eye images, a smoothing process is performed to obtain the pupil trajectory.

[0008] Optionally, the step of sequentially performing face detection, eye localization, and feature extraction on each of the multiple frames of human eye images to obtain the pupil center coordinates and eye contour features corresponding to each human eye image includes: Perform face detection on the human eye image to determine the user's face region; The face region is used to locate the eyes, the user's eye region is determined, and the eye region is input into a lightweight neural network model. The lightweight neural network model extracts eye features based on the eye region and determines the pupil center coordinates and eye contour features based on the eye features.

[0009] Optionally, determining the eyelid closure features based on the eye contour features corresponding to each person's eye image includes: The eyelid opening degree corresponding to each of the human eye images is determined based on the eye contour features corresponding to each of the human eye images; Based on the eyelid opening and closing degree corresponding to each of the aforementioned human eye images and a preset opening and closing degree threshold, the eyelid closing characteristics within a continuous time period are determined.

[0010] Optionally, determining the target eye movement command based on the pupil trajectory, eyelid closure characteristics, and a preset control command set includes: The pupil trajectory is identified and analyzed to determine the user's eye movements; The preset control instruction set selects an action instruction corresponding to a target action that matches the user's eye movement as the target action instruction, and selects an action instruction that matches the eyelid closure feature as the target action instruction.

[0011] Optionally, it also includes: Acquire the visible light brightness value of the current environment collected by the ambient light sensor; The current ambient light level is determined based on the visible light brightness value and multiple preset brightness thresholds. Adjust the photosensitivity gain of the first acquisition device according to the light level.

[0012] Optionally, adjusting the photosensitivity gain of the first acquisition device according to the illumination level includes: If the light level is higher, the photosensitivity gain of the first acquisition device is reduced; The lower the light level, the higher the photosensitivity gain of the first acquisition device.

[0013] Secondly, embodiments of this application also provide a television near-field interactive control processing device, the device comprising: The acquisition module is used to acquire multiple frames of human eye images collected by the first acquisition device within a continuous time period. The determination module is used to determine the pupil trajectory and eyelid closure features within the continuous time period based on the multi-frame human eye images. The pupil trajectory includes the pupil center coordinates of the user's pupil center in each time frame of the continuous time period, and the pupil center coordinates are two-dimensional coordinates in the coordinate system of the lens display interface. The determination module is used to determine the target action command of the eye based on the pupil trajectory, eyelid closure characteristics and preset control command set; The determination module is used to convert the coordinates of each pupil center in the pupil trajectory into television coordinates in the television coordinate system according to the calibrated first coordinate transformation relationship. The first coordinate transformation relationship is the coordinate transformation relationship between the lens interface coordinate system and the television screen coordinate system. The transmission module is used to send the target action command and each of the TV coordinates to the TV terminal, so that the TV terminal can simulate the user's eye movements based on the action command and each of the TV coordinates.

[0014] Optionally, the calibration module is used for: Acquire a first television image captured by the second acquisition device and a second television image that is projected in real time onto the lens display interface via a transmissive semi-transparent lens. Control contours are extracted from the first television image and the second television image respectively to obtain a first set of feature points corresponding to the first television image and a second set of feature points corresponding to the second television image. The first coordinate transformation relationship is determined using the least squares method based on the first set of feature points, the second set of feature points, the spatial parameters between the lens display interface and the television.

[0015] Optionally, the determining module is specifically used for: Face detection, eye localization and feature extraction are performed sequentially on each human eye image in the multi-frame human eye images to obtain the pupil center coordinates and eye contour features corresponding to each human eye image. The coordinate system of the human eye image is the same as the coordinate system of the lens display interface. The eyelid closure features are determined based on the eye contour features corresponding to each of the aforementioned human eye images; Based on the pupil center coordinates corresponding to each of the aforementioned human eye images, a smoothing process is performed to obtain the pupil trajectory.

[0016] Optionally, the determining module is specifically used for: Perform face detection on the human eye image to determine the user's face region; The face region is used to locate the eyes, the user's eye region is determined, and the eye region is input into a lightweight neural network model. The lightweight neural network model extracts eye features based on the eye region and determines the pupil center coordinates and eye contour features based on the eye features.

[0017] Optionally, the determining module is specifically used for: The eyelid opening degree corresponding to each of the human eye images is determined based on the eye contour features corresponding to each of the human eye images; Based on the eyelid opening and closing degree corresponding to each of the aforementioned human eye images and a preset opening and closing degree threshold, the eyelid closing characteristics within a continuous time period are determined.

[0018] Optionally, the determining module is specifically used for: The pupil trajectory is identified and analyzed to determine the user's eye movements; The preset control instruction set selects an action instruction corresponding to a target action that matches the user's eye movement as the target action instruction, and selects an action instruction that matches the eyelid closure feature as the target action instruction.

[0019] Optionally, the adjustment module is used for: Acquire the visible light brightness value of the current environment collected by the ambient light sensor; The current ambient light level is determined based on the visible light brightness value and multiple preset brightness thresholds. Adjust the photosensitivity gain of the first acquisition device according to the light level.

[0020] Optionally, the adjustment module is specifically used for: If the light level is higher, the photosensitivity gain of the first acquisition device is reduced; The lower the light level, the higher the photosensitivity gain of the first acquisition device.

[0021] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a memory, and a bus. The memory stores program instructions executable by the processor. When the application runs, the processor communicates with the memory via the bus, and the processor executes the program instructions to perform the steps of the television near-field interactive control processing method described in the first aspect.

[0022] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which is read and executes the steps of the television near-field interactive control processing method described in the first aspect.

[0023] The beneficial effects of this application are: This application provides a television near-field interactive control processing method, electronic device, and storage medium. By determining the pupil trajectory and eyelid closure features based on multiple frames of human eye images, the pupil trajectory includes the pupil center coordinates of the user in each time frame of a continuous time period. Based on the pupil trajectory, eyelid closure features, and a preset control command set, the target eye action command is determined, realizing the transformation from raw visual signals to high-level semantic commands. This allows users to operate the television device through natural eye movements, without relying on manual remote controls, voice input, or other additional interactive methods, significantly improving the convenience and immersion of human-computer interaction. According to a calibrated first coordinate transformation relationship, the pupil center coordinates in the pupil trajectory are converted into television coordinates in the television coordinate system. This achieves high-precision, low-latency mapping from user eye dynamics to television screen coordinates, providing a reliable positional basis for subsequent television-side simulation of user operations, improving the overall accuracy and naturalness of the interaction. The target action command and each television coordinate are sent to the television terminal, enabling the television terminal to simulate the user's eye movements based on the action command and television coordinates. It achieves end-to-end mapping from the user's natural eye movements to precise control on the TV, as well as control-level directional operation. The entire process requires no physical buttons, voice wake-up, or gesture operation, achieving seamless interaction and solving the problem of fragmented interaction in existing technologies. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is an exemplary scenario diagram provided for an embodiment of this application; Figure 2 A flowchart illustrating a television near-field interactive control processing method provided in an embodiment of this application; Figure 3 A flowchart illustrating the second television near-field interactive control processing method provided in this application embodiment; Figure 4 A flowchart illustrating the third television near-field interactive control processing method provided in this application embodiment; Figure 5 A schematic diagram of an apparatus for a television near-field interactive control processing method provided in an embodiment of this application; Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0027] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0028] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0029] Optionally, the television near-field interactive control processing method provided in this application embodiment is applied to an electronic device, which may be a server set in the cloud, or it may be set in a glasses device.

[0030] Figure 1 An exemplary scenario diagram provided for an embodiment of this application, such as... Figure 1 As shown, the scene includes remote-controlled glasses and a television device. The remote-controlled glasses also include a binocular camera module, a translucent semi-reflective lens, an ultra-wideband (UWB) near-field communication module, a microprocessor, and an ambient light sensor.

[0031] The binocular camera module includes a first acquisition device and a second acquisition device. The first acquisition device faces the user's eyes and is used to acquire dynamic images of the human eye. The second acquisition device faces the TV screen and is used to acquire the TV screen synchronously with the first acquisition device.

[0032] Among them, the transmissive semi-reflective translucent lens is used to project the television image captured by the second acquisition device onto the lens display area in real time, so that users can directly watch the synchronized image of the television screen through the lens without switching the visual focus, realizing the visual linkage between the line of sight, the lens and the television.

[0033] The UWB near-field communication module communicates with the UWB near-field module on the TV to establish a millimeter-level positioning connection, enabling spatial position calibration between the remote control glasses and the TV, and providing a basis for coordinate mapping.

[0034] An ambient light sensor is used to collect ambient light intensity in real time and send the collected ambient light intensity to a microprocessor. This allows the microprocessor to dynamically adjust the camera exposure parameters and lens display brightness based on the received ambient light intensity, thus avoiding image acquisition distortion caused by light interference.

[0035] Figure 2 This is a flowchart illustrating a near-field interactive control processing method for television provided in an embodiment of this application. The execution subject of this method is the aforementioned electronic device. Figure 2 As shown, the method includes: S101. Acquire multiple frames of human eye images within a continuous time period collected by the first acquisition device for the user's eyes.

[0036] The multi-frame human eye images are consecutive. This consecutive time period is an adaptive sliding time window, the duration of which can be dynamically adjusted based on user interaction intent. For example, this consecutive time period can be 2 seconds, 3 seconds, or 4 seconds. This consecutive time period can be represented by [t1, t2, t3, ..., tn], and the multi-frame human eye images can be represented by [F1, F2, F3, ..., Fn], where F1 is the human eye image captured at time frame t1, F2 is the human eye image captured at time frame t2, F3 is the human eye image captured at time frame t3, and so on.

[0037] Optionally, the first acquisition device is an infrared camera located inside the smart remote control glasses and facing the user's eyes. The first acquisition device may include an infrared supplementary lighting module. After the user wears the glasses and activates the interactive function, the first acquisition device begins to continuously acquire image sequences of the user's eye area at a preset frame rate, obtaining multiple frames of human eye images within a continuous time period.

[0038] S102. Determine the pupil trajectory and eyelid closure characteristics within a continuous time period based on multiple frames of human eye images.

[0039] The pupil trajectory includes the pupil center coordinates of the user at each time frame within a continuous time period. These pupil center coordinates are two-dimensional coordinates in the coordinate system of the lens display interface. One eye image is captured at each time frame, as mentioned above, the F1 eye image is captured at time frame t1. Multiple eye images constitute a continuous time-series image data, reflecting the dynamic changes in the user's eyes during this continuous time period. By performing frame-by-frame processing and cross-frame correlation analysis on these multiple eye images, semantically meaningful action features can be extracted.

[0040] Optionally, pupil trajectory can reflect the dynamic changes in the user's gaze within consecutive time frames, while eyelid closure features characterize the state of eyelid opening and closing and its temporal evolution, i.e., the transient physiological event characteristics of the user's intention trigger or state confirmation within that consecutive time period, serving as the basis for subsequent command mapping and coordinate transformation. Jointly analyzing the spatial motion path of the pupil and the temporal behavior features of the eyelids from multiple frames of human eye images not only improves the accuracy of individual feature recognition but also lays the foundation for subsequently converting natural eye movements into precise television control commands.

[0041] S103. Determine the target action command for the eye based on the pupil center, eyelid closure characteristics, and preset control command set.

[0042] The preset control instruction set includes a variety of different eye movements and corresponding action instructions for each eye movement, and may also include different eyelid closing features and corresponding action instructions for each eyelid closing feature.

[0043] Optionally, pupil trajectories can be used to identify eye movement-based interactive behaviors such as "swiping up", "swiping down", "translating left / right" or "fixing gaze"; eyelid closure features can be used to identify blinking types, such as single blink, rapid double blink, prolonged eye closure, etc.

[0044] For example, if the eye movement is looking up, the corresponding action instruction for looking up is to slide down.

[0045] S104. Based on the calibrated first coordinate transformation relationship, convert the coordinates of each pupil center in the pupil trajectory into television coordinates in the television coordinate system.

[0046] The first coordinate transformation relationship is the coordinate transformation between the lens interface coordinate system and the television screen coordinate system. This first coordinate transformation relationship is pre-calibrated and takes into account factors such as the relative spatial position, posture angle, optical imaging distortion, and display ratio between the remote control glasses and the television device, ensuring that the pupil trajectory collected from the glasses can accurately correspond to the specific UI control position on the television screen. This first coordinate transformation relationship can be represented, for example, by M, where M is a homogeneous transformation matrix that fuses UWB spatial parameters.

[0047] Optionally, the pupil center coordinates in the pupil trajectory obtained in step S102 are two-dimensional coordinates in the lens display interface coordinate system. Then, the formula [xp; yp; 1] = M × [xv; yv; 1] is used to transform the pupil center coordinates in the pupil trajectory to coordinates in the television screen coordinate system, where (xv, yv) are the pupil center coordinates in the pupil trajectory, i.e., coordinates in the lens display interface coordinate system; and (xp, yp) are the television coordinates obtained after transforming (xv, yv), i.e., coordinates in the television screen coordinate system.

[0048] S105. Send the target motion command and each TV coordinate to the TV terminal so that the TV terminal can simulate the user's eye movements based on the motion command and each TV coordinate.

[0049] Optionally, after identifying and mapping the target action commands, the target action commands and various TV coordinates can be encapsulated to generate a structured control command package. This command package can include action commands, TV coordinate sequences, and other information. The encapsulated command package can be sent to the TV via the UWB near-field communication module in the remote control glasses. Upon receiving the command package, the TV parses the action commands and TV coordinate sequences, invokes its internal user interface control system, and simulates the actual operation of the remote control glasses.

[0050] For example, if a confirmation command is received and the corresponding TV coordinate sequence corresponds to the "volume+" control, a click event on the "volume+" control is triggered to adjust the volume; if an upward swipe command and a continuous TV coordinate sequence are received, the page is flipped up or the channel is switched based on the TV coordinate sequence.

[0051] In this embodiment, pupil trajectory and eyelid closure features are determined based on multiple frames of human eye images. The pupil trajectory includes the pupil center coordinates of the user in each time frame of a continuous time period. Based on the pupil trajectory, eyelid closure features, and a preset control command set, the target eye action command is determined, realizing the transformation from raw visual signals to high-level semantic commands. This allows the user to operate the television device through natural eye movements, without relying on a manual remote control, voice input, or other additional interaction methods, significantly improving the convenience and immersion of human-computer interaction. The pupil center coordinates in the pupil trajectory are converted into television coordinates in the television coordinate system according to the calibrated first coordinate transformation relationship. This achieves high-precision, low-latency mapping from user eye dynamics to television screen coordinates, providing a reliable positional basis for subsequent television-side simulation of user operations, improving the accuracy and naturalness of the overall interaction. The target action command and each television coordinate are sent to the television terminal, enabling the television terminal to simulate the user's eye movements based on the action command and television coordinates. It achieves end-to-end mapping from the user's natural eye movements to precise control on the TV, as well as control-level directional operation. The entire process requires no physical buttons, voice wake-up, or gesture operation, achieving seamless interaction and solving the problem of fragmented interaction in existing technologies.

[0052] Figure 3 A flowchart illustrating the second television near-field interactive control processing method provided in this application embodiment is shown below. Figure 3 As shown, the calibration process for the first coordinate transformation relationship described above may include: S201. Acquire a first television image captured by the second acquisition device for the television screen and a second television image that is projected in real time onto the lens display interface via a transmissive semi-transparent lens.

[0053] Optionally, the first television image reflects the actual content and physical layout of the television screen in real space, and includes multiple controls. These may include various user interface elements such as channel list buttons, volume control sliders, menu borders, and icon borders.

[0054] Simultaneously, because the remote-controlled glasses are equipped with a translucent semi-reflective mirror, the television image, after being reflected and transmitted through this optical element, forms a virtual image on the lens display interface that is coplanar with the user's line of sight, achieving superimposed display of the television image. At this time, the same television image exists both on a distant television screen and simultaneously presented on the lens of the glasses in front of the user. In order to obtain the actual presentation form of this virtual image on the lens display interface, the second acquisition device performs secondary imaging on the image projected in the lens area, thereby obtaining a second television image, that is, the lens-side television image image after optical path transformation.

[0055] S202. Extract control contours from the first television image and the second television image respectively to obtain the first feature point set corresponding to the first television image and the second feature point set corresponding to the second television image.

[0056] The first television image is used to extract the layout features of controls on the television screen, such as menu buttons, icon borders, text boxes, and other prominent visual elements; while the second television image is used to extract the actual projection position of the corresponding controls in the lens display interface.

[0057] Optionally, an Oriented Fast and Rotated BRIEF (ORB) feature detection algorithm can be used to quickly identify and describe key structures in the image. This ORB algorithm can stably extract corner points, edge turning points, and high-contrast regions in the image under different scales and rotation conditions, and generate binary descriptors with directional information. By extracting control contours from the first television image, a first feature point set {Pn(xn, yn)} is obtained, where n is the total number of feature points, and (xn, yn) are the two-dimensional pixel coordinates of each feature point in the television screen coordinate system, i.e., the coordinates in the television physical coordinate system. By performing the same feature extraction on the second television image, a second feature point set {P'n(x'n, y'n)} corresponding to the second television image is identified, where (x'n, y'n) are the two-dimensional pixel coordinates of each feature point in the lens display interface coordinate system. The second feature point set can accurately reflect the screen layout in the user's visual perception.

[0058] Optionally, image preprocessing techniques, such as grayscale conversion, Gaussian filtering for noise reduction, and histogram equalization, can be combined during feature extraction to enhance image contrast, suppress ambient light interference, and improve the accuracy and robustness of feature point detection. Furthermore, high-quality feature points can be screened by setting a minimum feature response threshold and non-maximum suppression strategies, eliminating unstable points located in blurred areas at image edges or in low-texture regions, ensuring that the constructed feature point set has good matching consistency.

[0059] S203. Based on the first set of feature points, the second set of feature points, and the spatial parameters between the lens display interface and the television, determine the first coordinate transformation relationship using the least squares method.

[0060] Optionally, similarity matching can be performed on the first set of feature points and the second set of feature points based on feature descriptors to establish a one-to-one matching point pair set.

[0061] Specifically, UWB measures the spatial parameters between the lens display interface and the TV, including three-dimensional spatial parameters such as vertical distance, horizontal offset angle, and pitch angle. These spatial parameters can be used to construct a basic spatial transformation model and determine the global scaling, rotation, and offset references between the TV screen and the eyeglass lens image.

[0062] After determining the transformation model dimensions based on spatial parameters, the matching point pair set is substituted into the matrix, and the optimal 3×3 homogeneous transformation matrix M is iteratively fitted using the least squares method. M represents the first coordinate transformation relationship. M integrates all parameters, including UWB spatial geometric relationships, lens optical imaging distortion, and image scaling offset, ensuring that the coordinate mapping error is ≤1mm.

[0063] In this embodiment, the television screen is projected synchronously through a transmissive lens, so that the display screen on the lens display interface is consistent with the visual focus, avoiding visual fragmentation and improving operational continuity.

[0064] Figure 4 A flowchart illustrating the third television near-field interactive control processing method provided in this application embodiment is shown below. Figure 4 As shown, S102 above, determining the pupil trajectory and eyelid closure features within a continuous time period based on multiple frames of human eye images, may include: S301. Perform face detection, eye localization and feature extraction on each eye image in multiple frames of human eye images to obtain the pupil center coordinates and eye contour features corresponding to each eye image.

[0065] The coordinate system of the human eye image is the same as the coordinate system of the lens display interface.

[0066] Optionally, each frame of human eye image corresponds to a pupil center coordinate and an eye contour feature, which includes, but is not limited to, the pixel distribution of the upper and lower eyelid edges; the eye fissure height, i.e., the vertical distance between the upper and lower eyelids; the coordinates of key points at the corners of the eyes, such as the coordinates of the left corner of the eye, the right corner of the eye, the center coordinates of the upper eyelid, the center coordinates of the lower eyelid, the coordinates of the inner point of the upper eyelid, and the coordinates of the outer point of the upper eyelid; and the geometric shape information of the periorbital region.

[0067] By analyzing individual eye images, the transformation from human eye images to structured eye feature data was achieved, providing a reliable data foundation for subsequent pupil trajectory, eyelid state analysis, and control command recognition.

[0068] S302. Determine eyelid closure features based on the eye contour features corresponding to each person's eye image.

[0069] Among them, eye contour features refer to the geometric features extracted from human eye images that reflect the shape, opening and closing state, and dynamic changes of the eyes. They are the basis for subsequent judgments of pupil position, eyelid closure degree, blinking behavior, and other actions. By analyzing the eye contour features corresponding to each person's eye images, the eyelid closure features of the user within that continuous time period can be obtained.

[0070] Among these, eyelid closure characteristics can include rapid double blinking, prolonged eye closure, single blinking, and prolonged eye opening.

[0071] S303. Smooth the pupil center coordinates corresponding to each person's eye image to obtain the pupil trajectory.

[0072] Because the pupil center coordinates in individual eye images can be affected by various factors, such as camera sampling noise, minor eye tremors, instantaneous lighting fluctuations, or blink transition frames, the pupil coordinate sequence may exhibit some degree of jitter or abrupt changes. Directly using the unprocessed raw coordinates for subsequent action recognition can easily lead to misjudgments, such as misidentifying slight tremors as swipe commands or momentary occlusion as confirmation operations. Therefore, it is necessary to smooth the pupil center coordinates corresponding to each individual's eye images to extract stable and continuous pupil dynamic trajectories, thereby improving the accuracy and robustness of action recognition.

[0073] The pupil center coordinates corresponding to each person's eye image form a pupil center coordinate sequence. Smoothing refers to filtering or fitting multiple pupil center coordinates arranged in chronological order to remove random noise and outliers, retaining the core trajectory information reflecting the user's true gaze movement trend. For example, the Kalman filter algorithm can be used to smooth the pupil center coordinate sequence temporally. Kalman filtering is a recursive state estimation algorithm suitable for noise suppression and trajectory prediction in dynamic systems. Digital signal processing techniques such as moving average filtering, median filtering, or low-pass filtering can also be used to smooth the pupil coordinate sequence.

[0074] Optionally, the smoothed pupil center coordinate sequence is used as the vertical axis, with the corresponding time frame for each pupil center as the horizontal axis, to obtain the pupil trajectory. Here, the time frame corresponding to each pupil center refers to the time frame of the human eye image used to calculate the pupil center coordinates.

[0075] For example, the pupil trajectory could be: t0 (320,580); t1 (322,500); t2 (321,410); t3 (320,330). Here, (320,580) represents the pupil center coordinates at time frame t0.

[0076] Optionally, in step S301 above, performing face detection, eye localization, and feature extraction on each eye image in multiple frames of human eye images to obtain the pupil center coordinates and eye contour features corresponding to each eye image may include: Optionally, face detection is performed on the human eye image to determine the user's face region; eye localization is performed on the face region to determine the user's eye region, and the eye region is input into a lightweight neural network model, which extracts eye features based on the eye region and determines the pupil center coordinates and eye contour features based on the eye features.

[0077] A face detection algorithm is used to analyze human eye images, identifying and locating the user's face region within the image. This face detection algorithm can be, but is not limited to, a facial landmark detector, a face detection method based on a Haar cascade classifier, or a face detection network based on deep learning. Within the identified face region, further eye localization is performed to extract the local eye regions of the left and / or right eyes. The eye regions can be extracted using a preset facial landmark coordinate mapping, for example, by cropping region-of-interest (ROI) image blocks for both eyes based on detected periorbital landmarks. This eye region image focuses on the eyeball and surrounding tissues. The eye region image is then input into a neural network model, which can employ networks with low computational complexity and high inference speed, such as MobileNet, ShuffleNet, or Tiny-YOLO. This neural network model has the ability to simultaneously extract multiple deep visual features of the eye, including but not limited to features such as the gray-level gradient distribution at the pupil edge, iris texture structure, eyelid contour shape, and corneal reflective point location.

[0078] Optionally, determining the eyelid closure features based on the eye contour features corresponding to each person's eye image in S302 above may include: Optionally, the eyelid opening and closing degree corresponding to each person's eye image can be determined based on the eye contour features corresponding to each person's eye image. Then, based on the eyelid opening and closing degree corresponding to each person's eye image and a preset opening and closing degree threshold, the eyelid closing characteristics within a continuous time period can be determined.

[0079] First, based on the coordinates of key points at the corners of the eyes extracted from the eye contour features of each person's eye image, specifically the coordinates of the left and right corners of the eye, the center coordinates of the upper and lower eyelids, the coordinates of the inner and outer points of the upper eyelid, the eyelid opening degree corresponding to each person's eye image is calculated. The eyelid opening degree is the ratio of the vertical distance between the eyelids to the horizontal length of the palpebral fissure.

[0080] The eyelid opening and closing degree corresponding to each person's eye image is then compared with a preset opening and closing degree threshold to determine the eye state in each image, such as closed or open eyes. For example, if the eyelid opening and closing degree corresponding to two or three consecutive frames of human eye images is lower than the preset opening and closing degree threshold, it is considered that effective eye closure has occurred within that time frame. By performing temporal analysis on the eyelid opening and closing degree of all frames of human eye images within a continuous time period, different types of eyelid closure features can be identified, such as instantaneous blinking, prolonged eye closure, and rapid double blinking.

[0081] Eyelid closure features include not only the opening and closing state of a single frame, but also dynamic evolution characteristics over time, such as closure duration, closure frequency, closure speed, recovery rate, and the presence of periodic movements. Further analysis of the frequency, duration, and time interval between adjacent blink events identifies eyelid closure features. For example, if two blink events meeting the complete closure condition are detected consecutively within a short period, and the interval between the two events is within a preset time window, while the gaze position remains relatively stable during the two blinks, then the eyelid closure feature is determined to be a rapid double blink.

[0082] For example, if two brief eye-closing events are detected, with each closure lasting 100–300 ms and an interval of 300–600 ms between them, it can be identified as a rapid double blink eyelid closure feature. If the eyelid closure lasts for more than 1 second, it is identified as a prolonged eyelid closure feature. If the eyelid is slightly closed but does not fully reach the closure threshold, it can be marked as a half-closed or attentive eyelid closure feature. By setting a preset opening and closing degree threshold, it is determined whether the current frame is in a closed state. If the eyelid opening and closing degree of several consecutive frames is lower than the preset opening and closing degree threshold, it is considered that a blink has occurred.

[0083] In this embodiment, by converting the eye contour features of each frame into quantifiable eyelid opening and closing parameters, and combining them with preset thresholds and temporal logic analysis, the user's eyelid closing features can be accurately identified within a continuous time period, providing a reliable basis for subsequent action command recognition. This improves the ability to understand natural eye movements and enhances the robustness and user experience of the interactive system.

[0084] Optionally, the determination of the target eye movement command in S103 based on the pupil center, eyelid closure characteristics, and a preset control command set may include: Optionally, pupil trajectory can be identified and analyzed to determine the user's eye movements.

[0085] For example, if the pupil trajectory moves from the lower edge to the upper edge of the lens display area, the user's eye movement can be determined as "looking up". If the pupil trajectory moves from the upper edge of the lens display area downwards, the user's eye movement can be determined as "looking down". If the pupil trajectory moves from the left edge of the lens display area to the right, the user's eye movement can be determined as "looking from left to right". If the pupil center in the pupil trajectory continuously falls within the coordinate range of the TV control s within the lens display area for a period of more than a preset duration, the user's eye movement can be determined as "staring at the button s".

[0086] Optionally, the action instruction corresponding to the target action that matches the user's eye movement can be selected from the preset control instruction set as the target action instruction, and the action instruction that matches the eyelid closure feature can be selected from the preset control instruction set as the target action instruction.

[0087] For example, as mentioned above, the target action instruction corresponding to the eye movement "looking down" is "swipe down"; the target action instruction corresponding to the eye movement "looking down" is "swipe up"; the target action instruction corresponding to the eye movement "looking from left to right" is "swipe right"; the target action instruction corresponding to the eye movement "looking from right to left" is "swipe left"; and the target action instruction corresponding to the eye movement "staring at the button 's'" is "confirm". The target action instruction corresponding to the eyelid closure feature "rapid double blinking" is "return".

[0088] Optionally, the method may further include: Optionally, the visible light brightness value of the current environment is acquired by the ambient light sensor; the light level of the current environment is determined based on the visible light brightness value and multiple preset brightness thresholds; and the photosensitive gain of the first acquisition device is adjusted according to the light level.

[0089] Specifically, the microprocessor can compare the received visible light brightness value with multiple preset brightness thresholds to determine the current ambient light level. The light level can include, for example, strong light level, medium light level, weak light level, and dark level.

[0090] For example, if the visible light intensity value is less than a first preset threshold, the illumination level is dark; if the visible light intensity value is greater than or equal to the first preset threshold and less than a second preset threshold, the illumination level is determined to be low light; if the visible light intensity value is greater than or equal to the second preset threshold and less than a third preset threshold, the illumination level is determined to be medium light; and if the visible light intensity value is greater than the third preset threshold, the illumination level is determined to be high light. Wherein, the first preset threshold is less than the second preset threshold, and the second preset threshold is less than the third preset threshold.

[0091] Optionally, adjusting the photosensitivity gain of the first acquisition device according to the illumination level may include: Specifically, if the light level is higher, the light sensitivity gain of the first acquisition device is reduced; if the light level is lower, the light sensitivity gain of the first acquisition device is increased.

[0092] For example, in low light or dark conditions, the photosensitive gain of the first acquisition device is increased to enhance the image signal strength and prevent loss of detail in the human eye area due to underexposure. In medium or bright light conditions, the photosensitive gain of the first acquisition device is reduced to prevent overexposure of the image and avoid whitening of the pupil area or loss of iris texture information, thereby maintaining image contrast and feature clarity.

[0093] Figure 5 A schematic diagram of a device for a television near-field interactive control processing method provided in an embodiment of this application is shown below. Figure 5 As shown, the device includes: The acquisition module 401 is used to acquire multiple frames of human eye images within a continuous time period collected by the first acquisition device for the user's eyes; The determination module 402 is used to determine the pupil trajectory and eyelid closure features in the continuous time period based on the multi-frame human eye images. The pupil trajectory includes the pupil center coordinates of the user's pupil center in each time frame of the continuous time period. The pupil center coordinates are two-dimensional coordinates in the coordinate system of the lens display interface. The determining module 402 is used to determine the target action command of the eye based on the pupil trajectory, eyelid closure characteristics and preset control command set; The determining module 402 is used to convert the coordinates of each pupil center in the pupil trajectory into television coordinates in the television coordinate system according to the calibrated first coordinate transformation relationship. The first coordinate transformation relationship is the coordinate transformation relationship between the lens interface coordinate system and the television screen coordinate system. The transmission module 403 is used to send the target action command and each of the TV coordinates to the TV terminal, so that the TV terminal can simulate the user's eye movements based on the action command and each of the TV coordinates.

[0094] Optionally, the calibration module 404 is used for: Acquire a first television image captured by the second acquisition device and a second television image that is projected in real time onto the lens display interface via a transmissive semi-transparent lens. Control contours are extracted from the first television image and the second television image respectively to obtain a first set of feature points corresponding to the first television image and a second set of feature points corresponding to the second television image. The first coordinate transformation relationship is determined using the least squares method based on the first set of feature points, the second set of feature points, the spatial parameters between the lens display interface and the television.

[0095] Optionally, the determining module 402 is specifically used for: Face detection, eye localization and feature extraction are performed sequentially on each human eye image in the multi-frame human eye images to obtain the pupil center coordinates and eye contour features corresponding to each human eye image. The coordinate system of the human eye image is the same as the coordinate system of the lens display interface. The eyelid closure features are determined based on the eye contour features corresponding to each of the aforementioned human eye images; Based on the pupil center coordinates corresponding to each of the aforementioned human eye images, a smoothing process is performed to obtain the pupil trajectory.

[0096] Optionally, the determining module 402 is specifically used for: Perform face detection on the human eye image to determine the user's face region; The face region is used to locate the eyes, the user's eye region is determined, and the eye region is input into a lightweight neural network model. The lightweight neural network model extracts eye features based on the eye region and determines the pupil center coordinates and eye contour features based on the eye features.

[0097] Optionally, the determining module 402 is specifically used for: The eyelid opening degree corresponding to each of the human eye images is determined based on the eye contour features corresponding to each of the human eye images; Based on the eyelid opening and closing degree corresponding to each of the aforementioned human eye images and a preset opening and closing degree threshold, the eyelid closing characteristics within a continuous time period are determined.

[0098] Optionally, the determining module 402 is specifically used for: The pupil trajectory is identified and analyzed to determine the user's eye movements; The preset control instruction set selects an action instruction corresponding to a target action that matches the user's eye movement as the target action instruction, and selects an action instruction that matches the eyelid closure feature as the target action instruction.

[0099] Optionally, the adjustment module 405 is used for: Acquire the visible light brightness value of the current environment collected by the ambient light sensor; The current ambient light level is determined based on the visible light brightness value and multiple preset brightness thresholds. Adjust the photosensitivity gain of the first acquisition device according to the light level.

[0100] Optionally, the adjustment module 405 is specifically used for: If the light level is higher, the photosensitivity gain of the first acquisition device is reduced; The lower the light level, the higher the photosensitivity gain of the first acquisition device. Figure 6 This is a structural block diagram of an electronic device 500 provided in an embodiment of this application. For example... Figure 6 As shown, the electronic device may include: processor 501 and memory 502.

[0101] Optionally, a bus 503 may also be included, wherein the memory 502 is used to store machine-readable instructions executable by the processor 501. When the electronic device 500 is running, the processor 501 and the memory 502 communicate via the bus 503, and the processor 501 executes the machine-readable instructions to perform the method steps in the above method embodiments.

[0102] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the method steps described in the above-described television near-field interactive control processing method embodiment.

[0103] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0104] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0105] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for near-field interactive control processing in television, characterized in that, The method includes: Acquire multiple frames of human eye images within a continuous time period captured by the first acquisition device targeting the user's eyes; The pupil trajectory and eyelid closure features within the continuous time period are determined based on the multi-frame human eye images. The pupil trajectory includes the pupil center coordinates of the user in each time frame of the continuous time period. The pupil center coordinates are two-dimensional coordinates in the coordinate system of the lens display interface. Based on the pupil trajectory, eyelid closure characteristics, and preset control command set, the target action command for the eye is determined; According to the calibrated first coordinate transformation relationship, the coordinates of each pupil center in the pupil trajectory are converted into television coordinates in the television coordinate system. The first coordinate transformation relationship is the coordinate transformation relationship between the lens interface coordinate system and the television screen coordinate system. The target action command and each of the TV coordinates are sent to the TV terminal so that the TV terminal can simulate the user's eye movements based on the action command and each of the TV coordinates.

2. The television near-field interactive control processing method according to claim 1, characterized in that, The calibration process for the first coordinate transformation relationship includes: Acquire a first television image captured by the second acquisition device and a second television image that is projected in real time onto the lens display interface via a transmissive semi-transparent lens. Control contours are extracted from the first television image and the second television image respectively to obtain a first set of feature points corresponding to the first television image and a second set of feature points corresponding to the second television image. The first coordinate transformation relationship is determined using the least squares method based on the first set of feature points, the second set of feature points, the spatial parameters between the lens display interface and the television.

3. The television near-field interactive control processing method according to claim 1, characterized in that, The step of determining the pupil trajectory and eyelid closure features within the continuous time period based on the multiple frames of human eye images includes: Face detection, eye localization and feature extraction are performed sequentially on each human eye image in the multi-frame human eye images to obtain the pupil center coordinates and eye contour features corresponding to each human eye image. The coordinate system of the human eye image is the same as the coordinate system of the lens display interface. The eyelid closure features are determined based on the eye contour features corresponding to each of the aforementioned human eye images; Based on the pupil center coordinates corresponding to each of the aforementioned human eye images, a smoothing process is performed to obtain the pupil trajectory.

4. The television near-field interactive control processing method according to claim 3, characterized in that, The step of sequentially performing face detection, eye localization, and feature extraction on each of the multiple frames of human eye images to obtain the pupil center coordinates and eye contour features corresponding to each human eye image includes: Perform face detection on the human eye image to determine the user's face region; The face region is used to locate the eyes, the user's eye region is determined, and the eye region is input into a lightweight neural network model. The lightweight neural network model extracts eye features based on the eye region and determines the pupil center coordinates and eye contour features based on the eye features.

5. The method according to claim 3, characterized in that, The step of determining the eyelid closure features based on the eye contour features corresponding to each person's eye image includes: The eyelid opening degree corresponding to each of the human eye images is determined based on the eye contour features corresponding to each of the human eye images; Based on the eyelid opening and closing degree corresponding to each of the aforementioned human eye images and a preset opening and closing degree threshold, the eyelid closing characteristics within a continuous time period are determined.

6. The television near-field interactive control processing method according to claim 1, characterized in that, The step of determining the target movement command for the eye based on the pupil trajectory, eyelid closure characteristics, and a preset control command set includes: The pupil trajectory is identified and analyzed to determine the user's eye movements; The preset control instruction set selects an action instruction corresponding to a target action that matches the user's eye movement as the target action instruction, and selects an action instruction that matches the eyelid closure feature as the target action instruction.

7. The television near-field interactive control processing method according to claim 1, characterized in that, Also includes: Acquire the visible light brightness value of the current environment collected by the ambient light sensor; The current ambient light level is determined based on the visible light brightness value and multiple preset brightness thresholds. Adjust the photosensitivity gain of the first acquisition device according to the light level.

8. The television near-field interactive control processing method according to claim 7, characterized in that, The step of adjusting the photosensitivity gain of the first acquisition device according to the illumination level includes: If the light level is higher, the photosensitivity gain of the first acquisition device is reduced; The lower the light level, the higher the photosensitivity gain of the first acquisition device.

9. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program executable by the processor, and the processor executes the computer program to implement the steps of the method described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the television near-field interactive control processing method as described in any one of claims 1-8.

Images