Zoom control method based on eye signal, zoom control device and mobile terminal

Abstract

The application discloses a zoom control method and device based on eye signals and a mobile terminal. The application solves the problems of large signal noise, large individual difference of users, picture shaking, visual fatigue, and unbalanced sensitivity of bidirectional zooming when signal is saturated on one side in the prior art. The application realizes stable anti-shaking, balanced zooming in and zooming out, and continuous zooming by smoothing and reducing noise in the time domain of original eye signals, normalizing and scaling the progress by logarithmic transformation to fit human eye perception, dynamically adjusting the time step, and limiting the zooming range.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and human-computer interaction, and in particular to a zoom control method, zoom control device and mobile terminal based on eye signals. Background Technology

[0002] Gesture-free camera / magnification control is needed in outdoor observation (birdwatching, watching sports), barrier-free magnification for people with low vision and motor impairments, and in repair / medical scenarios where hands are occupied. Existing related technologies mainly include: Front camera face size → Rear camera zoom (absolute position mapping): For example, US20140184854A1, the front camera detects the face / interpupillary distance size and directly maps it proportionally to the rear camera zoom magnification, zooming in when close and zooming out when tilting back.

[0003] Squinting / Eyelid State → Magnification (proportional): For example, US5839000 judges squinting by the aspect ratio of the eyes / degree of openness, and the magnification is proportional to the degree of squinting.

[0004] Scaling based on gaze point / distance: such as Microsoft's "scaling changes according to the distance from the human eye to the screen", which maps the gaze point to the region of interest and focuses / scales it.

[0005] Rate scrolling based on gaze position: Some schemes adjust the scrolling rate according to the degree of deviation of the gaze point from the anchor point.

[0006] The existing technology has the following shortcomings: (a) Shaking directly enters the picture: The original eye signal has large frame-by-frame noise, and the absolute position mapping directly turns the shaking into the picture being sometimes large and sometimes small. There is a lack of systematic anti-shaking and dead zone processing, and "random zooming" is easy to occur near the neutral zone.

[0007] (b) Fatigue: Absolute / positional control requires the user to maintain a certain posture (continuously close, continuous squinting) in order to maintain a certain magnification, which leads to fatigue after prolonged use.

[0008] (c) Individual differences have not been eliminated: There are large differences in pupil distance and habitual eye opening degree among different users, making it difficult to apply a fixed threshold universally.

[0009] (d) Unprocessed nonlinear scaling perception: The human eye perceives magnification logarithmically (the "doubling" sensation is the same for 1×→2× and 5×→10×), while linear / proportional mapping is oversensitive at low magnification and insensitive at high magnification.

[0010] (e) Directional saturation of eye-opening signal: When "eye-opening degree / eye-opening probability" is used as the signal, normal eye-opening is already close to the upper limit (e.g., probability ≈ 0.99), and there is almost no usable dynamic range for "opening wider", resulting in severe asymmetry in bidirectional control: squinting and enlarging are sensitive, while eye-opening and shrinking are almost ineffective. Existing technology has not identified and solved this problem.

[0011] Therefore, existing technologies have shortcomings and need to be improved. Summary of the Invention

[0012] The technical problem to be solved by the present invention is to provide a zoom control method, zoom control device and mobile terminal based on eye signals, which achieves stable (anti-shake), fatigue-free zoom control that conforms to the logarithmic perception of the human eye and has balanced bidirectional (magnification / reduction) sensitivity under the conditions of high noise in the original signal, individual differences, and saturation at one end.

[0013] The technical solution of the present invention is as follows: A zoom control method based on eye signals is provided, comprising: 1) Calibration: After startup, collect a segment (e.g., 2-3 seconds) of eye signal in a relaxed state of the user, calculate the mean of its smooth value as a personalized baseline to eliminate individual differences; 2) Smoothing: Perform exponential moving average (EMA) de-jittering on each frame of the original signal s to obtain the smoothed value e; 3) Normalization deviation: Calculate δ=(e baseline) / max(|baseline|,ε), eliminates dimensional and individual differences; 4) Dead zone: When |δ| is less than the dead zone threshold, the magnification remains unchanged, suppressing micro-jitter near the neutral zone; 5) Speed ​​control + centering and focus lock: The zoom rate is determined by the portion of δ that exceeds the dead zone; when δ returns to the dead zone (centering), the rate is reduced to zero and the current magnification is locked. 6) Logarithmic perception exponential mapping: t=ln(zoom / min) / ln(max / min); t←clamp(t+rate·dt,0,1); zoom=min·(max / min)^t, making the doubling perception consistent throughout the process; 7)(Core) Eye-opening saturation asymmetry compensation: For signals of eye-opening degree that saturate near full opening, a smaller dead zone and a larger gain are used in the shrinking direction (eye-opening direction) than in the amplifying direction to compensate for its limited dynamic range and balance the bidirectional sensitivity. 8) Apply the calculated target magnification to the camera (optical / digital zoom).

[0014] The signal direction is uniformly agreed to be larger if the value is larger: the squinting scheme takes s= Eye opening degree; distance scheme is s = pupil distance or face frame width.

[0015] Specifically, the following steps are included: S1: Images are captured by a camera facing the user, and eye signals s representing the user's eye state are extracted from them; the eye signals s are normalized so that the larger the value, the more the user intends to increase the zoom ratio. S2: Determine the zoom rate rate based on the deviation δ between the eye signal s and the personalized baseline; where the sign of δ distinguishes two zoom directions: the first direction is the direction that increases the zoom magnification, and the second direction is the direction that decreases the zoom magnification. S3: When |δ| is not greater than the dead zone threshold corresponding to this direction, keep the current zoom ratio unchanged; when |δ| is greater than the dead zone threshold corresponding to this direction, adjust the zoom ratio according to rate=(|δ|). The dead zone threshold in that direction is multiplied by the rate gain in that direction, and the corresponding direction sign is assigned to determine the rate. S4: The dead zone threshold used in the second direction is less than the dead zone threshold used in the first direction, and the rate gain used in the second direction is greater than the rate gain used in the first direction. S5: Integrate the rate over time to obtain the target magnification, and apply the target magnification to the camera to perform zoom.

[0016] Furthermore, the eye signal s is selected from any of the following: (a) A quantity negatively correlated with the degree to which the user's eyes are open, which causes s to increase when the user squints; (b) A quantity positively correlated with the distance between the user and the screen, including interpupillary distance or face frame size, which causes s to increase when the user gets closer.

[0017] Further, in step S5, the method of obtaining the target zoom level by integrating rate over time is as follows: Let the control quantity t = ln(currentZoom / minZoom) / ln(maxZoom / minZoom), integrate rate on a logarithmic scale by t←clip(t+rate·dt,0,1), and calculate the target zoom level by zoom = minZoom·(maxZoom / minZoom)^t; where currentZoom is the current zoom level, dt is the inter-frame time interval, and minZoom and maxZoom are the minimum and maximum zoom levels, respectively.

[0018] Furthermore, the deviation δ is obtained in the following way: S2.1: The ocular signal s is smoothed in the time domain to obtain a smoothed value e, wherein the time domain smoothing is an exponential moving average e←α·s+(1 α)·e, where 0<α<1; S2.2:δ=(e The function is baseline) / max(|baseline|,ε), where ε is a small positive value to prevent division by zero.

[0019] Further, the personalized baseline is calibrated in the following manner: within a predetermined time period after starting or triggering calibration, collecting an eye signal of a user in a relaxed state, and taking the mean value of the smoothed value e within the time period as the baseline; and the zoom magnification is kept unchanged during the calibration period.

[0020] Further, denoted that the dead zone threshold in the first direction is deadZone, the rate gain is G, the dead zone threshold in the second direction is deadZone×k, and the rate gain is G×β, wherein 0<k<1 and β>1.

[0021] Further, the zoom control method based on an eye signal further comprises a position control mode: no integration is performed on rate, and the target magnification is directly obtained from the deviation δ through t=clip((|δ| dead zone threshold)·positionGain,0,1) and exponential mapping zoom=minZoom·(maxZoom / minZoom)^t, wherein positionGain is position sensitivity, and the dead zone threshold and the rate gain are selected according to the aforementioned direction asymmetric manner.

[0022] Further, the present invention further provides a zoom control device based on an eye signal, comprising: a signal acquisition module, configured to extract the eye signal s from a user-facing camera; a smoothing module, configured to perform time-domain smoothing on the eye signal s; a calibration module, configured to calibrate the personalized baseline; a control law module, configured to execute the step of determining a target magnification in the aforementioned zoom control method based on an eye signal; a zoom execution module, configured to apply the target magnification to a camera.

[0023] Further, the present invention further provides a mobile terminal, comprising a user-facing first camera, a user-backing second camera, a display screen and a processor; wherein the processor is configured to: enable the first camera and the second camera simultaneously, determine a target magnification according to an image collected by the first camera by the aforementioned zoom control method based on an eye signal, control the second camera to zoom according to the target magnification, and display a picture of the second camera on the display screen.

[0024] Further, the first camera and the second camera output images simultaneously through a concurrent camera session; and the second camera is a multi-camera virtual camera, which switches between lenses of different focal lengths to provide optical zoom during zooming.

[0025] Furthermore, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned zoom control method based on eye signals.

[0026] By adopting the above solution, the present invention provides a zoom control method, a zoom control device, and a mobile terminal based on eye signals, which have the following technical effects: Anti-shake and stable zoom: Calibration + EMA + Normalized dead zone eliminates individual differences and suppresses micro-shake near the neutral zone; No fatigue: Speed ​​control + center lock focus, once the target magnification is reached, it returns to a relaxed state and locks the focus, without the need to maintain the posture continuously; Conforms to human visual perception: Logarithmic perception index mapping, low magnification for fine detail, high magnification for fast speed, and linear and consistent feel throughout the entire process; Bidirectional equalization: Asymmetric saturation compensation solves the problem of sensitive amplification and blunt reduction caused by the saturation of the open-eye signal, making the open-eye reduction signal truly usable. This is a problem that the prior art has not solved, and this technical problem is the most critical technical problem that the present invention can solve.

[0027] The method of the present invention is independent of a specific camera / platform and can be implemented on a single device or dual devices. Attached Figure Description

[0028] Figure 1 This is a system block diagram of the method of the present invention; camera → feature extraction → control law → camera zoom → display, forming a closed loop.

[0029] Figure 2 The flowchart shows the control law of the method of this invention; calibration / smoothing / normalization / dead zone / direction / logarithmic mapping.

[0030] Figure 3 A graph showing the magnification mapping; linear vs. logarithmic perception (logarithmic curves are more precise at low magnification and faster at high magnification).

[0031] Figure 4 For the rate-deviation curve; (asymmetric dead zone + asymmetric gain): shrinking direction, smaller dead zone + larger gain.

[0032] Figure 5 The timing diagram is for calibration and centering lock-in; the magnification remains at a level during centering (no fatigue).

[0033] Figure 6 The diagram shows the hardware topology for three implementation methods: single-device multi-camera, single-device concurrency, and dual-device distributed. Detailed Implementation

[0034] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0035] Please see Figures 1-6 This invention provides a zoom control method based on eye signals, comprising the following steps: S1: Images are captured by a camera facing the user, and eye signals s representing the user's eye state are extracted from them; the eye signals s are normalized so that the larger the value, the more the user intends to increase the zoom ratio. S2: Determine the zoom rate rate based on the deviation δ between the eye signal s and the personalized baseline; where the sign of δ distinguishes two zoom directions: the first direction is the direction that increases the zoom magnification, and the second direction is the direction that decreases the zoom magnification. S3: When |δ| is not greater than the dead zone threshold corresponding to this direction, keep the current zoom ratio unchanged; when |δ| is greater than the dead zone threshold corresponding to this direction, adjust the zoom ratio according to rate=(|δ|). The dead zone threshold in that direction is multiplied by the rate gain in that direction, and the corresponding direction sign is assigned to determine the rate. S4: The dead zone threshold used in the second direction is less than the dead zone threshold used in the first direction, and the rate gain used in the second direction is greater than the rate gain used in the first direction. S5: Integrate the rate over time to obtain the target magnification, and apply the target magnification to the camera to perform zoom.

[0036] In this embodiment, the eye signal s is selected from any one of the following: (a) A quantity negatively correlated with the degree to which the user's eyes are open, which causes s to increase when the user squints; (b) A quantity positively correlated with the distance between the user and the screen, including interpupillary distance or face frame size, which causes s to increase when the user gets closer.

[0037] In this embodiment, in step S5, the method of obtaining the target zoom level by integrating rate over time is as follows: Let the control quantity t = ln(currentZoom / minZoom) / ln(maxZoom / minZoom), integrate rate on a logarithmic scale by t←clip(t+rate·dt,0,1), and calculate the target zoom level by zoom = minZoom·(maxZoom / minZoom)^t; where currentZoom is the current zoom level, dt is the inter-frame time interval, and minZoom and maxZoom are the minimum and maximum zoom levels, respectively.

[0038] In this embodiment, the deviation δ is obtained in the following way: S2.1: The ocular signal s is smoothed in the time domain to obtain a smoothed value e, wherein the time domain smoothing is an exponential moving average e←α·s+(1 α)·e, where 0 < α < 1; S2.2: δ=(e baseline) / max(|baseline|,ε), wherein ε is a small positive quantity for preventing division by zero.

[0039] In this embodiment, the personalized baseline is calibrated through the following method: within a predetermined time period after calibration is started or triggered, collect the eye signal of the user in a relaxed state, and take the mean value of the smoothed value e within this time period as the baseline; and keep the zoom magnification unchanged during the calibration time period.

[0040] In this embodiment, the dead zone threshold in the first direction is denoted as deadZone and the rate gain is denoted as G, the dead zone threshold in the second direction is denoted as deadZone×k and the rate gain is denoted as G×β, where 0 < k < 1 and β > 1.

[0041] In this embodiment, the zoom control method based on eye signals further includes a position control mode: rate is not integrated, and the target magnification is directly obtained from the deviation δ through t=clip((|δ| dead zone threshold)·positionGain, 0, 1) and exponential mapping zoom=minZoom·(maxZoom / minZoom)^t, wherein positionGain is the position sensitivity, and the dead zone threshold and the rate gain are selected according to the aforementioned directionally asymmetric manner.

[0042] The present invention also provides a zoom control device based on eye signals, comprising: a signal acquisition module, configured to extract the eye signal s by a user-facing camera; a smoothing module, configured to perform time-domain smoothing on the eye signal s; a calibration module, configured to calibrate the personalized baseline; a control law module, configured to execute the step of determining the target magnification in the aforementioned zoom control method based on eye signals; a zoom execution module, configured to apply the target magnification to a camera.

[0043] Further, the present invention also provides a mobile terminal, comprising: a first camera facing the user, a second camera facing away from the user, a display screen and a processor; the processor is configured to: enable the first camera and the second camera simultaneously, determine the target magnification according to the images collected by the first camera by the aforementioned zoom control method based on eye signals, control the second camera to zoom according to the target magnification, and display the image captured by the second camera on the display screen.

[0044] In this embodiment, the first camera and the second camera simultaneously output images through a concurrent camera session; and the second camera is a multi-camera virtual camera that switches between lenses of different focal lengths to provide optical zoom during zooming.

[0045] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned zoom control method based on eye signals.

[0046] The following is a specific embodiment of the present invention. Signal acquisition and face / eye feature extraction Images are captured by a user-facing camera and obtained through face detection: eye-opening degree o∈[0,1] (obtained from the eye aspect ratio EAR or the eye-opening probability output by the face model, with the average value of both eyes); pupil distance ipd (pixel distance between the centers of both eyes in the image); face bounding box width fw (backup distance proxy).

[0047] Signal direction normalization: Squinting scheme B takes s= o; Distance scheme A takes s=ipd (back to fw if missing).

[0048] Control Law (Pseudocode) Parameters: minZoom, maxZoom, deadZone, alpha (EMA), rateGain (amplification gain) outGain (reduces extra gain, >1), outDeadFactor (reduces dead zone coefficient, <1), calibSeconds Status: ema, baseline, currentZoom=minZoom, calibrating update(s, dt) per frame: ema = hasEMA ? alpha*s + (1-alpha)*ema : s# smoothing if calibrating:# calibrating Cumulative EMA is averaged and then used as a baseline; calibration ends when the specified duration is reached. return currentZoom# Calibration Lock Multiplier delta = (ema - baseline) / max(|baseline|, 1e-6) # Normalization deviation span = ln(maxZoom / minZoom);t = ln(currentZoom / minZoom) / span if delta>= 0:# Zoom in (squinting / approaching) if delta<deadZone: return currentZoom# Dead zone -> lock focus rate = (delta - deadZone) * rateGain else:# Zoom out (eyes open) - asymmetric compensation outDead = deadZone * outDeadFactor# Smaller dead zone if (-delta)<outDead: return currentZoom rate = -((-delta) - outDead) * rateGain * outGain # Larger gain t = clamp(t + rate*dt, 0, 1)# Logarithmic integration currentZoom = minZoom * (maxZoom / minZoom)^t # Exponential mapping back to magnification return currentZoom Typical parameters: deadZone=0.12, alpha=0.18, rateGain=3.0, outGain=3.0, outDeadFactor=0.3, calibSeconds=2.5, minZoom=1, maxZoom=10. Position control can also be alternatively set: t=clamp((δ deadZone)·positionGain,0,1).

[0049] Example 1: Multiple cameras on single device (iOS) The same mobile terminal uses a multi-camera session (AVCaptureMultiCamSession) to enable the front / rear cameras simultaneously: the front camera calculates eye openness / pupillary distance through facial key points → control law → the rear camera (a multi-camera virtual device, which automatically switches between ultra-wide-angle / wide-angle / telephoto for optical zoom during zooming) applies the target magnification with a smooth ramp; the screen displays the image captured by the rear camera; horizontal and vertical screen orientations are automatically adapted by the rotation coordinator.

[0050] Example 2: Concurrent cameras on single device (Android) Using concurrent cameras (CameraX ConcurrentCamera) to simultaneously operate front and rear cameras: Front camera image analysis is performed on a face detection model to obtain the probability of open eyes and key points of both eyes → control law → rear camera uses setZoomRatio to apply magnification. The device must support concurrent camera features.

[0051] Example 3: Dual-device distributed system The front-facing camera of the recognition unit (such as a computer) identifies eye signals and runs a control law. It then sends the target magnification command to the zoom unit (such as a mobile phone) via a network (P2P or LAN). The zoom unit then zooms in using its rear camera and sends the image back to the recognition unit for display. Note that this method is not limited to standalone operation. In summary, this invention provides a zoom control method, zoom control device, and mobile terminal based on eye signals, which have the following technical advantages: Anti-shake and stable zoom: Calibration + EMA + Normalized dead zone eliminates individual differences and suppresses micro-shake near the neutral zone; No fatigue: Speed ​​control + center lock focus, once the target magnification is reached, it returns to a relaxed state and locks the focus, without the need to maintain the posture continuously; Conforms to human visual perception: Logarithmic perception index mapping, low magnification for fine detail, high magnification for fast speed, and linear and consistent feel throughout the entire process; Bidirectional equalization: Asymmetric saturation compensation solves the problem of sensitive amplification and blunt reduction caused by the saturation of the open-eye signal, making the open-eye reduction signal truly usable. This is a problem that the prior art has not solved, and this technical problem is the most critical technical problem that the present invention can solve.

[0052] The method of the present invention is independent of a specific camera / platform and can be implemented on a single device or dual devices.

[0053] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A zoom control method based on eye signals, characterized in that, comprising the following steps: S1: collecting an image by a user-facing camera, and extracting an eye signal s representing the eye state of the user therefrom; the eye signal s is normalized such that a larger value represents that the user has a stronger intention to increase a zoom magnification; S2: determining a zoom rate rate according to a deviation δ between the eye signal s and a personalized baseline; wherein the sign of δ distinguishes two zoom directions: a first direction being a direction for increasing the zoom magnification, and a second direction being a direction for decreasing the zoom magnification; S3: When |δ| is not greater than the dead zone threshold corresponding to this direction, keep the current zoom ratio unchanged; when |δ| is greater than the dead zone threshold corresponding to this direction, adjust the zoom ratio according to rate=(|δ|). The dead zone threshold in that direction is multiplied by the rate gain in that direction, and the corresponding direction sign is assigned to determine the rate. S4: a dead zone threshold adopted in the second direction is smaller than a dead zone threshold adopted in the first direction, and a rate gain adopted in the second direction is larger than a rate gain adopted in the first direction; S5: integrating the rate in a time dimension to obtain a target magnification, and applying the target magnification to a camera to perform zooming.

2. The zoom control method based on eye signals according to claim 1, characterized in that, the eye signal s is selected from any one of the following: (a) a quantity negatively correlated with the degree of a user's eye opening, such that s increases when the user squints; (b) a quantity positively correlated with the distance from the user to a screen, comprising interpupillary distance of both eyes or a size of a face frame, such that s increases when the user moves closer.

3. The zoom control method based on eye signals according to claim 1, characterized in that, in the step S5, the way of integrating the rate in the time dimension to obtain the target magnification is: letting a control variable t=ln(currentZoom / minZoom) / ln(maxZoom / minZoom), integrating the rate on a logarithmic scale according to t←clip(t+rate·dt,0,1), and calculating the target magnification according to zoom=minZoom·(maxZoom / minZoom)^t; wherein currentZoom is a current magnification, dt is an inter-frame time interval, and minZoom and maxZoom are a minimum zoom magnification and a maximum zoom magnification respectively.

4. The zoom control method based on eye signals according to claim 1, characterized in that, the deviation δ is obtained by the following way: S2.1: The ocular signal s is smoothed in the time domain to obtain a smoothed value e, wherein the time domain smoothing is an exponential moving average e←α·s+(1 α)·e, where 0<α<1; S2.2:δ=(e The function is baseline) / max(|baseline|,ε), where ε is a small positive value to prevent division by zero.

5. The zoom control method based on eye signals according to claim 4, characterized in that, the personalized baseline is calibrated by the following way: within a predetermined time period after starting or triggering calibration, collecting eye signals of a user in a relaxed state, and taking a mean value of smoothed values e in the time period as the baseline; and keeping the zoom magnification unchanged during the calibration time period.

6. The zoom control method based on eye signals according to claim 1, characterized in that, denoting the dead zone threshold in the first direction as deadZone and the rate gain as G, the dead zone threshold in the second direction is deadZone×k and the rate gain is G×β, wherein 0<k<1 and β>1.

7. The zoom control method based on eye signals according to claim 1, characterized in that, It also includes a position control mode: instead of integrating the rate, the deviation δ is determined by t=clip((|δ|). The target magnification is directly obtained by using the dead zone threshold (positionGain, 0, 1) and the exponential mapping zoom = minZoom (maxZoom / minZoom)^t, where positionGain is the position sensitivity, and the dead zone threshold and rate gain are selected in the directional asymmetric manner as described in claim 1.

8. A zoom control device based on eye signals, characterized in that, comprising: a signal collection module, configured to extract the eye signal s by a user-facing camera; a smoothing module, configured to perform time-domain smoothing on the eye signal s; a calibration module, configured to calibrate the personalized baseline; a control law module, configured to execute the step of determining a target magnification in the eye signal-based zoom control method according to any one of claims 1 to 7; a zoom execution module, configured to apply the target magnification to a camera.

9. A mobile terminal, characterized in that, comprising: a first user-facing camera, a second camera facing away from the user, a display screen and a processor; The processor is configured to: simultaneously enable the first camera and the second camera, determine a target magnification based on the image captured by the first camera according to the zoom control method based on eye signals according to any one of claims 1 to 7, control the second camera to zoom according to the target magnification, and display the image of the second camera on the display screen.

10. The mobile terminal according to claim 9, characterized in that, The first camera and the second camera output images simultaneously through a concurrent camera session; and the second camera is a multi-camera virtual camera that switches between different focal length lenses to provide optical zoom during zooming.

11. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the zoom control method based on eye signals as described in any one of claims 1 to 7.

Images