Autofocus method, apparatus, computer device and computer readable storage medium
By using multi-target tracking detection and depth map to determine depth of field coverage, the problem of unstable focus in live broadcast footage has been solved, achieving stability and accuracy of autofocus and ensuring the quality of live broadcast footage.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- MALANSHAN AUDIO & VIDEO LABORATORY
- Filing Date
- 2026-04-23
- Publication Date
- 2026-07-10
AI Technical Summary
In live streaming scenarios, the host moves indoors and outdoors, sometimes close and sometimes far away, and multiple viewers connect or take videos together. When the camera zooms out, the pixels of the human eye are very small. In low light or backlight, the key points of the human eye fluctuate greatly, resulting in unstable focus and low quality of the live stream.
By performing multi-target tracking and detection on the preview image frame, the set of face bounding boxes and eye bounding boxes is obtained. The scale measurement is unified, the usable threshold of face and eye is calculated, and the depth map is combined to determine the depth coverage. The focus strategy is determined and the focus is switched to achieve automatic focus.
Ensure stable live stream footage and accurate focus, avoid blind focusing, and achieve a smooth transition between stable footage and accurate focus.
Smart Images
Figure CN122093663B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of focusing technology, and in particular to an automatic focusing method, apparatus, computer device, and computer-readable storage medium. Background Technology
[0002] In live streaming scenarios, the host moves indoors and outdoors, sometimes at a distance and sometimes close. Multiple viewers may connect or collaborate. When the camera zooms out, the pixels of the human eye are very small. In low light or backlight, the key points of the human eye fluctuate more, resulting in unstable focus and low quality of the live stream. Summary of the Invention
[0003] In view of this, the purpose of the present invention is to overcome the shortcomings of the prior art and provide an autofocus method, apparatus, computer device and computer-readable storage medium for autofocusing based on human eye size and depth of field prediction.
[0004] This invention provides the following technical solution:
[0005] In a first aspect, the present invention provides an autofocus method, the method comprising:
[0006] Multi-object tracking and detection are performed on the preview image frame to obtain the face bounding box set, the eye bounding box set, and the multi-object tracking parameters;
[0007] By unifying the scale measurement caliber of the face bounding box set and the eye bounding box set, the target face bounding box width and the target eye bounding box width are obtained.
[0008] The usable face threshold and usable eye threshold are calculated based on the reference face width threshold, the reference eye width threshold, the preset resolution, and the equivalent focal length parameters.
[0009] The multi-target tracking parameters are smoothed and normalized to obtain the smoothed and normalized multi-target tracking parameters;
[0010] The focus target is acquired, and the depth coverage determination result of the focus target is determined based on the depth map corresponding to the preview image frame;
[0011] Based on the target face bounding box width, target eye bounding box width, smoothed normalized multi-target tracking parameters, depth coverage determination results, and the face availability threshold and eye availability threshold corresponding to the focus target, a focus strategy is determined, and the focus target is switched according to the focus strategy.
[0012] In an optional implementation, the step of standardizing the scale measurement of the face bounding box set and the eye bounding box set to obtain the target face bounding box width and the target eye bounding box width includes:
[0013] For each face detection box in the face box set, the width of the face detection box is used as the initial face box width of the face detection box;
[0014] Divide the initial face frame width of each face detection box by the short side pixels of the current preview resolution to obtain the target face frame width of each face detection box;
[0015] For each pair of human eye detection boxes in the set of human eye boxes, the average or maximum value of the left and right eye widths of the human eye detection box pair is taken as the initial human eye box width of the human eye detection box pair.
[0016] Divide the initial eye frame width of each of the aforementioned eye detection box pairs by the short side pixels of the current preview resolution to obtain the target face frame width of each of the aforementioned eye detection box pairs.
[0017] In an optional implementation, the preset resolution includes the short-side pixels of the current preview resolution and the short-side pixels of the reference preview resolution; the equivalent focal length parameter includes the current equivalent focal length and the reference equivalent focal length; and the calculation of the usable face threshold and usable eye threshold based on the reference face width threshold, the reference eye width threshold, the preset resolution, and the equivalent focal length parameter includes:
[0018] Calculate the ratio of the short-side pixels of the current preview resolution to the short-side pixels of the reference preview resolution, and use this ratio as the resolution ratio;
[0019] Calculate the ratio of the current equivalent focal length to the reference equivalent focal length, and use it as the focal length ratio;
[0020] The product of the resolution ratio, the focal length ratio, and the reciprocal of the preset digital zoom ratio is used as the scaling factor.
[0021] The reference face width threshold and the reference eye width threshold are weighted according to the scaling factor to obtain the usable face threshold and the usable eye threshold.
[0022] In an optional implementation, the step of acquiring the focus target and determining the depth-of-field coverage result of the focus target based on the depth map corresponding to the preview image frame includes:
[0023] Obtain the current shooting parameters corresponding to the preview image frame, and determine the depth of field range based on the current shooting parameters;
[0024] Based on the depth map, the focus target is detected and located using human eye detection to obtain the eye region;
[0025] The distance to the eye is determined based on the depth value corresponding to the eye region.
[0026] By comparing the corresponding distance to the eye with the depth of field range, the depth of field coverage determination result is obtained.
[0027] In an optional implementation, the step of comparing the corresponding distance to the eye and the depth of field range to obtain the depth of field coverage determination result includes:
[0028] If the distance corresponding to the eye is within the depth of field range, then the depth of field coverage determination result is depth of field coverage;
[0029] If the distance corresponding to the eye is not within the depth of field range, the depth of field coverage determination result is no depth of field coverage.
[0030] In an optional implementation, the multi-target tracking parameters include face detection confidence, eye detection confidence, and eye tracking stability. The step of determining the focusing strategy based on the target face bounding box width, target eye bounding box width, smoothed and normalized multi-target tracking parameters, depth coverage determination results, and the face availability threshold and the eye availability threshold includes:
[0031] Determine whether the width of the target eye frame corresponding to the focus target is greater than or equal to the human eye usable threshold, whether the smoothed and normalized human eye detection confidence level corresponding to the focus target is greater than or equal to the human eye confidence threshold, and whether the human eye tracking stability corresponding to the focus target is stable.
[0032] If so, the focusing strategy prioritizes human eye focusing;
[0033] If not, then determine whether the width of the target face bounding box corresponding to the focus target is greater than or equal to the face availability threshold, whether the smoothed normalized face detection confidence corresponding to the focus target is greater than or equal to the face confidence threshold, and whether the depth coverage determination result is depth coverage.
[0034] If so, the focusing strategy is face-priority focusing;
[0035] If not, then the focusing strategy is to revert to the background.
[0036] In an optional implementation, the step of switching the focus on the focus target according to the focusing strategy includes:
[0037] If there are N consecutive preview image frames with the same focusing strategy, then the focus of the target is switched according to the focusing strategy.
[0038] In a second aspect, the present invention provides an autofocus device, the device comprising:
[0039] The detection and tracking module is used to perform multi-target tracking and detection on the preview image frame to obtain the face bounding box set, the eye bounding box set, and multi-target tracking parameters.
[0040] The scale unification module is used to unify the scale measurement caliber of the face frame set and the eye frame set to obtain the target face frame width and the target eye frame width.
[0041] The threshold calculation module is used to calculate the usable face threshold and usable eye threshold based on the reference face width threshold, the reference eye width threshold, the preset resolution and the equivalent focal length parameters.
[0042] The smoothing normalization module is used to smooth and normalize the multi-target tracking parameters to obtain smoothed and normalized multi-target tracking parameters;
[0043] The depth-of-field determination module is used to acquire the focus target and determine the depth-of-field coverage determination result of the focus target based on the depth map corresponding to the preview image frame.
[0044] The focus control module is used to determine a focus strategy based on the target face frame width, target eye frame width, smoothed normalized multi-target tracking parameters and depth coverage determination results corresponding to the focus target, as well as the face availability threshold and the eye availability threshold, and to switch the focus on the focus target according to the focus strategy.
[0045] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements the autofocus method as described in any of the foregoing embodiments.
[0046] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the autofocus method as described in any of the foregoing embodiments.
[0047] The automatic focusing method, apparatus, computer device, and computer-readable storage medium disclosed in this invention perform multi-target tracking and detection on a preview image frame to obtain a set of face bounding boxes, a set of eye bounding boxes, and multi-target tracking parameters; unify the scale measurement caliber of the face bounding box set and the eye bounding box set to obtain the target face bounding box width and the target eye bounding box width; calculate the face usability threshold and the eye usability threshold based on a reference face width threshold, a reference eye width threshold, a preset resolution, and equivalent focal length parameters; smooth and normalize the multi-target tracking parameters to obtain smoothed and normalized multi-target tracking parameters; acquire the focus target, determine the depth coverage determination result of the focus target based on the depth map corresponding to the preview image frame; determine a focusing strategy based on the target face bounding box width, the target eye bounding box width, the smoothed and normalized multi-target tracking parameters, the depth coverage determination result, the face usability threshold, and the eye usability threshold, and switch the focus on the focus target according to the focusing strategy. In this way, multi-target tracking ensures target continuity, scale normalization eliminates the impact of resolution and sensor differences, dynamic threshold calibration makes the semantics usable by the human eye consistent across devices, and depth-of-field coverage determination predicts in advance whether the eyes are still clear after focusing on the face, avoiding blind focusing, and then performing focus grading to ensure stable exported images, accurate focus, and smooth switching. Attached Figure Description
[0048] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope of protection of the present invention. In the various drawings, similar components are numbered similarly.
[0049] Figure 1 A flowchart of the autofocus method proposed in this embodiment is shown;
[0050] Figure 2 Another flowchart of the autofocus method proposed in this embodiment is shown;
[0051] Figure 3 A schematic diagram of the autofocus device proposed in this embodiment is shown.
[0052] Explanation of reference numerals in the attached diagram:
[0053] 300 - Autofocus device; 301 - Detection and tracking module; 302 - Scale unification module; 303 - Threshold calculation module; 304 - Smoothing and normalization module; 305 - Depth of field determination module; 306 - Focus control module. Detailed Implementation
[0054] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0055] The components of the embodiments of the invention described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0056] In the following, the terms “comprising,” “having,” and their cognates, which may be used in various embodiments of the invention, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as excluding, firstly, the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations thereof.
[0057] Furthermore, the terms "first," "second," and "third" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.
[0058] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the invention pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of the invention.
[0059] Example 1
[0060] This disclosure provides an autofocus method for autofocusing based on human eye size and depth-of-field prediction.
[0061] Please see Figure 1 The autofocus method includes steps S101 to S106, and each step is described in detail below.
[0062] Step S101: Perform multi-target tracking detection on the preview image frame to obtain the face bounding box set, the eye bounding box set, and the multi-target tracking parameters.
[0063] In this embodiment, face detection and eye key point extraction are performed on the preview image frame to achieve multi-target tracking and detection, resulting in a set of face bounding boxes. Human eye socket collection And multi-target tracking parameters.
[0064] For example, if a face is detected in the preview image frame, the eye bounding boxes (left and right eyes) are extracted, and the face target is assigned an ID for tracking. The face bounding boxes, eye bounding boxes and their corresponding tracking parameters of all targets in the preview image frame are integrated to obtain the face bounding box set, the eye bounding box set and the multi-target tracking parameters.
[0065] Understandably, by simultaneously outputting sets of face bounding boxes, sets of eye bounding boxes, and multi-target tracking parameters (including confidence and stability), a fine-grained, temporally coherent visual semantic perception foundation for live streaming scenarios is constructed. Compared to traditional focusing methods that rely solely on single face detection, this step achieves explicit localization and quality assessment of key regions at the human eye level, providing a data prerequisite for subsequent human eye-first strategies. Simultaneously, the multi-target tracking parameters can quantify the target's motion state, significantly improving the continuous trackability and detection reliability of targets in dynamic scenarios such as anchor movement, shaking, and occlusion, fundamentally alleviating the problem of misfocus jumps caused by key point drift.
[0066] Step S102: Standardize the scale measurement of the face frame set and the eye frame set to obtain the target face frame width and the target eye frame width.
[0067] In this embodiment, the scale measurement of the face bounding box set and the eye bounding box set is unified based on the pixel width to obtain the target face bounding box width and the target eye bounding box width. This ensures that the threshold semantics are consistent under different resolutions and eliminates the absolute size incomparability caused by different resolutions and different sensor sizes.
[0068] As an example, for each face detection box in the face box set, the width of the face detection box is used as the initial face box width of that face detection box.
[0069] Furthermore, the initial face frame width of each face detection box is divided by the short side pixels of the current preview resolution to obtain the target face frame width of each face detection box.
[0070] For each pair of human eye detection boxes in the human eye box set, the average or maximum value of the left and right eye widths of the human eye detection box pair is used as the initial human eye box width of that pair.
[0071] Furthermore, the initial eye frame width of each eye detection box pair is divided by the short side pixels of the current preview resolution to obtain the target face frame width of each eye detection box pair.
[0072] It should be noted that the focal length parameter can be used to correct the width of the target face frame and the width of the target eye frame, making the threshold comparable across different preview paths.
[0073] Step S103: Calculate the usable face threshold and usable eye threshold based on the reference face width threshold, the reference eye width threshold, the preset resolution, and the equivalent focal length parameters.
[0074] In this embodiment, by using preset resolution and equivalent focal length parameters, the reference face width threshold and reference eye width threshold can be calibrated in real time according to the current shooting link. This avoids premature triggering of eye focusing (misjudgment of small faces) or excessive delay (missed judgment of large faces) caused by applying fixed thresholds across wide-angle / telephoto and high-definition / standard-definition previews. This ensures the universality and generalization of the focusing strategy under diverse terminals and complex shooting parameters.
[0075] In one specific embodiment, the preset resolution includes the short-side pixels of the current preview resolution and the short-side pixels of the reference preview resolution. Step S103 includes: calculating the ratio of the short-side pixels of the current preview resolution to the short-side pixels of the reference preview resolution as the resolution ratio; calculating the ratio of the current equivalent focal length to the reference equivalent focal length as the focal length ratio; multiplying the resolution ratio, the focal length ratio, and the reciprocal of the preset digital zoom ratio as the scaling factor; and weighting the reference face width threshold and the reference eye width threshold according to the scaling factor to obtain the usable face threshold and the usable eye threshold.
[0076] In this embodiment, scaling factor The calculation formula is: In the formula, The shorter side pixels of the current preview resolution. For reference, the short side resolution is shown in pixels. For the current equivalent focal length, For reference equivalent focal length, Set the preset digital zoom magnification.
[0077] Among them, the short side pixels of the current preview resolution change with the application output resolution and the clipping link, while the short side pixels of the reference preview resolution are the reference short side selected by calibration or design. Both are used to construct the scaling factor for resolution normalization, so that the threshold of the same near / small semantics is consistent under different preview resolutions.
[0078] The current equivalent focal length is the equivalent focal length (or stop mapping value) under the current zoom / lens state, which changes with zoom; the reference equivalent focal length is the equivalent focal length under the reference focal length. The ratio of the two is used to adjust the scale and threshold with focal length (changes in viewing angle cause different target pixel scales at the same object distance), ensuring that the threshold and depth prediction are semantically consistent under different zoom levels.
[0079] Face availability threshold The calculation formula is: In the formula, For reference, the face width threshold; the threshold usable by the human eye. The calculation formula is: In the formula, This is a reference threshold for the width of the human eye.
[0080] Step S104: The multi-target tracking parameters are smoothed and normalized to obtain the smoothed and normalized multi-target tracking parameters.
[0081] In this embodiment, the multi-target tracking parameters include the face detection confidence of each target, the eye detection confidence of each target, the face tracking stability of each target, and the eye tracking stability of each target. The multi-target tracking parameters are then smoothed and normalized to obtain smoothed and normalized multi-target tracking parameters, which effectively suppresses noise abrupt changes and transient jitter interference in the temporal dimension.
[0082] Step S105: Obtain the focus target and determine the depth coverage determination result of the focus target based on the depth map corresponding to the preview image frame.
[0083] In this embodiment, based on the depth map corresponding to the preview image frame and combined with the current shooting parameters corresponding to the preview image frame, it is predicted whether the eyes of the focus target will still fall within an acceptable clear range when focusing on the face of the focus target, so as to obtain the depth of field coverage determination result. This solves the problem that the traditional solution blindly focuses on the face in shallow depth of field, resulting in blurred eyes and loss of focus for the viewer.
[0084] It should be noted that the focus target can be selected by the user or determined by combining factors such as face size, face tracking stability, and face position.
[0085] Please see Figure 2 In one specific embodiment, step S105 includes steps S1051 to S1054, and each step is described in detail below.
[0086] Step S1051: Obtain the current shooting parameters corresponding to the preview image frame, and determine the depth of field range based on the current shooting parameters.
[0087] In this embodiment, the current shooting parameters include the current focal length, current aperture, current focus distance, current target distance, and preset blur tolerance parameters. The depth-of-field range under face-focusing conditions is calculated based on these current shooting parameters.
[0088] It should be noted that the current focal length and current aperture are used to estimate the imaging ratio and relative aperture information required for depth of field, respectively; the current focusing distance is obtained from PDAF phase calculation, motor steps, or feedback from the relationship between lens position and object distance; the current target distance is obtained by combining the pixel scale of the focused target face with calibration; the preset blur tolerance parameters are the allowable circle of confusion diameter, acceptable blur threshold, or distance window that is still acceptable to the eye when the face is clear, etc., and are calibrated offline and written into the parameter set.
[0089] Step S1052: Perform human eye detection and positioning on the focus target based on the depth map to obtain the eye region.
[0090] In this embodiment, the depth map is used to detect and locate the human eye of the target in focus, thereby obtaining the eye region. Exemplarily, the depth map is used to detect and locate the human eye of the target in focus, obtaining the coordinates of the left eye keypoint and the right eye keypoint; a rectangular region is then extended from the center of the left and right eyes as the eye region.
[0091] Step S1053: Determine the corresponding distance of the eye based on the depth value corresponding to the eye region.
[0092] In this embodiment, all valid depth values within the eye region are obtained, and the median of the depth values is taken as the corresponding distance to the eye, which is used to represent the object distance of the eye of the focus target, approximating the object distance from the eye of the focus target to the entrance pupil of the camera along the optical axis of the camera.
[0093] Step S1054: Compare the corresponding distance to the eye with the depth range to obtain the depth coverage determination result.
[0094] In this embodiment, the depth-of-field coverage determination result of the focused target is obtained by comparing the positional relationship between the corresponding distance of the eye and the depth-of-field interval.
[0095] Specifically, if the distance to the eye is within the depth of field range, the depth of field coverage determination result is depth of field coverage; if the distance to the eye is not within the depth of field range, the depth of field coverage determination result is no depth of field coverage.
[0096] Step S106: Based on the target face bounding box width, target eye bounding box width, smoothed normalized multi-target tracking parameters and depth coverage determination results corresponding to the focus target, as well as the face availability threshold and the eye availability threshold, determine the focus strategy, and switch the focus on the focus target according to the focus strategy.
[0097] In this embodiment, a hierarchical focus decision engine is constructed based on the target face bounding box width, target eye bounding box width, smoothed and normalized multi-target tracking parameters, depth-of-field coverage determination results, and face and eye availability thresholds corresponding to the focus target. This significantly improves the continuity of user experience and system reliability in extreme scenarios (such as sudden darkness, rapid retraction, and overlapping of multiple people). Furthermore, the focus is switched on the focus target using a hierarchically determined focus strategy.
[0098] In one specific embodiment, step S106 includes: determining whether the width of the target eye frame corresponding to the focus target is greater than or equal to the available eye threshold, whether the smoothed normalized eye detection confidence level corresponding to the focus target is greater than or equal to the eye confidence threshold, and whether the eye tracking stability corresponding to the focus target is stable; if yes, the focusing strategy is eye-first focusing; if no, determining whether the width of the target face frame corresponding to the focus target is greater than or equal to the available face threshold, whether the smoothed normalized face detection confidence level corresponding to the focus target is greater than or equal to the face confidence threshold, and whether the depth-of-field coverage determination result is depth-of-field coverage; if yes, the focusing strategy is face-first focusing; if no, the focusing strategy is backtracking to distant scenes.
[0099] In this embodiment, the entry conditions for human eye-priority focusing are: the width of the target human eye frame corresponding to the focus target is greater than or equal to the human eye's usable threshold; the smoothed and normalized human eye detection confidence score corresponding to the focus target is greater than or equal to the human eye confidence threshold; and the human eye tracking stability corresponding to the focus target is stable. In other words, a sufficiently large and stable human eye is used as the threshold for entering human eye focusing.
[0100] If the entry conditions for human eye-first focusing are not met, but the width of the target face bounding box corresponding to the focus target is greater than or equal to the face availability threshold, the smoothed and normalized face detection confidence of the focus target is greater than or equal to the face confidence threshold, and the depth-of-field coverage determination result is depth-of-field coverage, then the focusing strategy is face-first focusing. That is, when human eye is unreliable, eye tracking is not continued, but depth-of-field coverage prediction is used to determine whether the eyes can still be guaranteed to be clear, thus reverting to face focusing.
[0101] If the width of the bounding box of the target face is smaller than the available face threshold, or if no face is detected in the preview image frame, the focusing strategy is to retreat to the distant view. That is, when the face is too small / extremely far away, directly enter the preset focus of the distant view / infinity, reducing meaningless searches.
[0102] It should be noted that the focusing strategy for human eye priority focus is: prioritize PDAF for coarse positioning, and CDAF for fine adjustment (fine adjustment of small ROI); the focusing strategy for face priority focus is: use PDAF for fast focusing, and CDAF for fine adjustment when necessary; the focusing strategy for retracting to a distant view is: push the motor to a preset position near infinity (or a distant view preset) and limit the search range to avoid focus hunting.
[0103] Understandably, human eye priority is only enabled when the width of the human eye frame is greater than or equal to the human eye's usable threshold, the human eye confidence level is met, and human eye tracking is stable, in order to avoid false human eye detection causing misfocusing in low light. When the human eye conditions are not met, face priority is further enabled when the face width meets the standard, the face confidence level meets the standard, and the depth of field coverage is met, to ensure that the main target is clear. If the width of the target face frame corresponding to the focus target is less than the face's usable threshold, or if no face is detected in the preview image frame, the system will actively fall back to the distant view strategy (such as focusing at infinity or the average distance of the scene) to prevent the system from getting stuck in an invalid focus loop or a black screen due to focus failure.
[0104] In one specific embodiment, step S106 includes: if there are N consecutive preview image frames with the same focusing strategy, then the focus target is switched according to the focusing strategy.
[0105] In this embodiment, to avoid frequent focus jumps around the threshold, if the focusing strategy is the same for N consecutive preview image frames, the focus target is switched according to the focusing strategy. For example, eye-priority focusing requires N consecutive frames before switching. Alternatively, no second focus switch is performed within a preset time window, or the focus target is not easily changed as long as the same focus target is still in the frame and has high stability, thereby reducing image shake and frequent subject switching.
[0106] The autofocus method proposed in this embodiment performs multi-target tracking and detection on the preview image frame to obtain a set of face bounding boxes, a set of eye bounding boxes, and multi-target tracking parameters. The scale measurement caliber of the face bounding box set and the eye bounding box set is unified to obtain the target face bounding box width and the target eye bounding box width. A face availability threshold and a eye availability threshold are calculated based on a reference face width threshold, a reference eye width threshold, a preset resolution, and equivalent focal length parameters. The multi-target tracking parameters are smoothed and normalized to obtain smoothed and normalized multi-target tracking parameters. The focus target is acquired, and the depth-of-field coverage determination result of the focus target is determined based on the depth map corresponding to the preview image frame. A focusing strategy is determined based on the target face bounding box width, the target eye bounding box width, the smoothed and normalized multi-target tracking parameters, the depth-of-field coverage determination result, the face availability threshold, and the eye availability threshold, and the focus is switched on the focus target according to the focusing strategy. In this way, multi-target tracking ensures target continuity, scale normalization eliminates the impact of resolution and sensor differences, dynamic threshold calibration makes the semantics usable by the human eye consistent across devices, and depth-of-field coverage determination predicts in advance whether the eyes are still clear after focusing on the face, avoiding blind focusing, and then performing focus grading to ensure stable exported images, accurate focus, and smooth switching.
[0107] Example 2
[0108] Furthermore, this disclosure provides an autofocus device 300, please refer to [link to relevant documentation]. Figure 3 The device includes:
[0109] The detection and tracking module 301 is used to perform multi-target tracking and detection on the preview image frame to obtain a set of face bounding boxes, a set of eye bounding boxes, and multi-target tracking parameters.
[0110] The scale unification module 302 is used to unify the scale measurement caliber of the face frame set and the eye frame set to obtain the target face frame width and the target eye frame width.
[0111] The threshold calculation module 303 is used to calculate the usable face threshold and the usable eye threshold based on the reference face width threshold, the reference eye width threshold, the preset resolution and the equivalent focal length parameters.
[0112] The smoothing normalization module 304 is used to smooth and normalize the multi-target tracking parameters to obtain smoothed and normalized multi-target tracking parameters;
[0113] The depth-of-field determination module 305 is used to acquire the focus target and determine the depth-of-field coverage determination result of the focus target based on the depth map corresponding to the preview image frame.
[0114] The focus control module 306 is used to determine a focus strategy based on the target face frame width, target eye frame width, smoothed normalized multi-target tracking parameters and depth coverage determination results corresponding to the focus target, as well as the face availability threshold and the eye availability threshold, and to switch the focus on the focus target according to the focus strategy.
[0115] In an optional implementation, the scale unification module 302 is used to, for each face detection box in the face box set, take the width of the face detection box as the initial face box width of the face detection box; divide the initial face box width of each face detection box by the short side pixels of the current preview resolution to obtain the target face box width of each face detection box; for each pair of eye detection boxes in the eye box set, take the average or maximum value of the left and right eye widths of the pair of eye detection boxes as the initial eye box width of the pair of eye detection boxes; divide the initial eye box width of each pair of eye detection boxes by the short side pixels of the current preview resolution to obtain the target face box width of each pair of eye detection boxes.
[0116] In an optional implementation, the preset resolution includes the short-side pixels of the current preview resolution and the short-side pixels of the reference preview resolution. The threshold calculation module 303 is used to calculate the ratio of the short-side pixels of the current preview resolution to the short-side pixels of the reference preview resolution as the resolution ratio; calculate the ratio of the current equivalent focal length to the reference equivalent focal length as the focal length ratio; multiply the resolution ratio, the focal length ratio, and the reciprocal of the preset digital zoom ratio as the scaling factor; and weight the reference face width threshold and the reference eye width threshold according to the scaling factor to obtain the usable face threshold and the usable eye threshold.
[0117] In an optional implementation, the depth-of-field determination module 305 is used to obtain the current shooting parameters corresponding to the preview image frame, determine the depth-of-field range based on the current shooting parameters, perform human eye detection and positioning on the focus target based on the depth map to obtain the eye region, determine the corresponding distance of the eye based on the depth value corresponding to the eye region, and compare the corresponding distance of the eye with the depth-of-field range to obtain the depth-of-field coverage determination result.
[0118] In an optional implementation, the depth-of-field determination module 305 is configured to determine the depth-of-field coverage result as depth-of-field coverage if the distance corresponding to the eye is within the depth-of-field range, and as non-depth-of-field coverage if the distance corresponding to the eye is not within the depth-of-field range.
[0119] In an optional implementation, the multi-target tracking parameters include face detection confidence, eye detection confidence, and eye tracking stability. The focus control module 306 is used to determine whether the width of the target eye frame corresponding to the focus target is greater than or equal to the available eye threshold, whether the smoothed and normalized eye detection confidence corresponding to the focus target is greater than or equal to the eye confidence threshold, and whether the eye tracking stability corresponding to the focus target is stable. If yes, the focus strategy is eye-priority focusing; if no, it determines whether the width of the target face frame corresponding to the focus target is greater than or equal to the available face threshold, whether the smoothed and normalized face detection confidence corresponding to the focus target is greater than or equal to the face confidence threshold, and whether the depth-of-field coverage determination result is depth-of-field coverage. If yes, the focus strategy is face-priority focusing; if no, the focus strategy is backtracking to distant scenes.
[0120] In an optional implementation, the focus control module 306 is used to switch the focus on the focus target according to the focus strategy if there are N consecutive preview image frames with the same focus strategy.
[0121] The apparatus provided in this embodiment can perform the steps of the autofocus method provided in Embodiment 1. To avoid repetition, the steps will not be described again.
[0122] The autofocus device proposed in this embodiment performs multi-target tracking and detection on the preview image frame to obtain a set of face bounding boxes, a set of eye bounding boxes, and multi-target tracking parameters. It then unifies the scale measurement caliber of the face bounding box set and the eye bounding box set to obtain the target face bounding box width and the target eye bounding box width. Based on a reference face width threshold, a reference eye width threshold, a preset resolution, and equivalent focal length parameters, it calculates a usable face threshold and a usable eye threshold. The multi-target tracking parameters are then smoothed and normalized to obtain smoothed and normalized multi-target tracking parameters. The device acquires the focus target and determines the depth-of-field coverage determination result of the focus target based on the depth map corresponding to the preview image frame. Based on the target face bounding box width, the target eye bounding box width, the smoothed and normalized multi-target tracking parameters, the depth-of-field coverage determination result, the usable face threshold, and the usable eye threshold, it determines a focusing strategy and switches the focus on the focus target according to the focusing strategy. In this way, multi-target tracking ensures target continuity, scale normalization eliminates the impact of resolution and sensor differences, dynamic threshold calibration makes the semantics usable by the human eye consistent across devices, and depth-of-field coverage determination predicts in advance whether the eyes are still clear after focusing on the face, avoiding blind focusing, and then performing focus grading to ensure stable exported images, accurate focus, and smooth switching.
[0123] Example 3
[0124] Furthermore, this disclosure provides a computer device including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the autofocus method described in Embodiment 1.
[0125] The device provided in this embodiment can perform the steps of the autofocus method provided in Embodiment 1. To avoid repetition, the steps will not be repeated.
[0126] Example 4
[0127] This disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the autofocus method described in Embodiment 1.
[0128] In this embodiment, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.
[0129] The computer-readable storage medium provided in this embodiment can implement the autofocus method provided in Embodiment 1. To avoid repetition, it will not be described again here.
[0130] In all examples shown and described herein, any specific values should be interpreted as merely exemplary and not as limitations; therefore, other examples of exemplary embodiments may have different values.
[0131] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0132] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.
Claims
1. An autofocus method, characterized in that, The method includes: Multi-object tracking and detection are performed on the preview image frame to obtain the face bounding box set, the eye bounding box set, and the multi-object tracking parameters; By unifying the scale measurement caliber of the face bounding box set and the eye bounding box set, the target face bounding box width and the target eye bounding box width are obtained. The usable face threshold and usable eye threshold are calculated based on the reference face width threshold, the reference eye width threshold, the preset resolution, and the equivalent focal length parameters. The multi-target tracking parameters are smoothed and normalized to obtain the smoothed and normalized multi-target tracking parameters; The focus target is acquired, and the depth coverage determination result of the focus target is determined based on the depth map corresponding to the preview image frame; Based on the target face bounding box width, target eye bounding box width, smoothed normalized multi-target tracking parameters, depth coverage determination results, and the face availability threshold and eye availability threshold corresponding to the focus target, a focus strategy is determined, and the focus target is switched according to the focus strategy.
2. The autofocus method according to claim 1, characterized in that, The process of standardizing the scale measurement of the face bounding box set and the eye bounding box set to obtain the target face bounding box width and the target eye bounding box width includes: For each face detection box in the face box set, the width of the face detection box is used as the initial face box width of the face detection box; Divide the initial face frame width of each face detection box by the short side pixels of the current preview resolution to obtain the target face frame width of each face detection box; For each pair of human eye detection boxes in the set of human eye boxes, the average or maximum value of the left and right eye widths of the human eye detection box pair is taken as the initial human eye box width of the human eye detection box pair. Divide the initial eye frame width of each of the aforementioned eye detection box pairs by the short side pixels of the current preview resolution to obtain the target face frame width of each of the aforementioned eye detection box pairs.
3. The autofocus method according to claim 1, characterized in that, The preset resolution includes the short side pixels of the current preview resolution and the short side pixels of the reference preview resolution. The equivalent focal length parameter includes the current equivalent focal length and the reference equivalent focal length. The calculation of the usable face threshold and usable eye threshold based on the reference face width threshold, the reference eye width threshold, the preset resolution, and the equivalent focal length parameter includes: Calculate the ratio of the short-side pixels of the current preview resolution to the short-side pixels of the reference preview resolution, and use this ratio as the resolution ratio; Calculate the ratio of the current equivalent focal length to the reference equivalent focal length, and use it as the focal length ratio; The product of the resolution ratio, the focal length ratio, and the reciprocal of the preset digital zoom ratio is used as the scaling factor. The reference face width threshold and the reference eye width threshold are weighted according to the scaling factor to obtain the usable face threshold and the usable eye threshold.
4. The autofocus method according to claim 1, characterized in that, The step of acquiring the focus target and determining the depth-of-field coverage result of the focus target based on the depth map corresponding to the preview image frame includes: Obtain the current shooting parameters corresponding to the preview image frame, and determine the depth of field range based on the current shooting parameters; Based on the depth map, the focus target is detected and located using human eye detection to obtain the eye region; The distance to the eye is determined based on the depth value corresponding to the eye region. By comparing the corresponding distance to the eye with the depth of field range, the depth of field coverage determination result is obtained.
5. The autofocus method according to claim 4, characterized in that, The comparison of the corresponding distance to the eye and the depth of field range to obtain the depth of field coverage determination result includes: If the distance corresponding to the eye is within the depth of field range, then the depth of field coverage determination result is depth of field coverage; If the distance corresponding to the eye is not within the depth of field range, the depth of field coverage determination result is no depth of field coverage.
6. The autofocus method according to claim 1, characterized in that, The multi-target tracking parameters include face detection confidence, eye detection confidence, and eye tracking stability. The step of determining the focusing strategy based on the target face bounding box width, target eye bounding box width, smoothed and normalized multi-target tracking parameters, depth coverage determination results, and the available face threshold and the available eye threshold includes: Determine whether the width of the target eye frame corresponding to the focus target is greater than or equal to the human eye usable threshold, whether the smoothed and normalized human eye detection confidence level corresponding to the focus target is greater than or equal to the human eye confidence threshold, and whether the human eye tracking stability corresponding to the focus target is stable. If so, the focusing strategy prioritizes human eye focusing; If not, then determine whether the width of the target face bounding box corresponding to the focus target is greater than or equal to the face availability threshold, whether the smoothed normalized face detection confidence corresponding to the focus target is greater than or equal to the face confidence threshold, and whether the depth coverage determination result is depth coverage. If so, the focusing strategy is face-priority focusing; If not, then the focusing strategy is to revert to the background.
7. The autofocus method according to claim 1, characterized in that, The step of switching the focus on the focus target according to the focusing strategy includes: If there are N consecutive preview image frames with the same focusing strategy, then the focus of the target is switched according to the focusing strategy.
8. An automatic focusing device, characterized in that, The device includes: The detection and tracking module is used to perform multi-target tracking and detection on the preview image frame to obtain the face bounding box set, the eye bounding box set, and multi-target tracking parameters. The scale unification module is used to unify the scale measurement caliber of the face frame set and the eye frame set to obtain the target face frame width and the target eye frame width. The threshold calculation module is used to calculate the usable face threshold and usable eye threshold based on the reference face width threshold, the reference eye width threshold, the preset resolution and the equivalent focal length parameters. The smoothing normalization module is used to smooth and normalize the multi-target tracking parameters to obtain smoothed and normalized multi-target tracking parameters; The depth-of-field determination module is used to acquire the focus target and determine the depth-of-field coverage determination result of the focus target based on the depth map corresponding to the preview image frame. The focus control module is used to determine a focus strategy based on the target face frame width, target eye frame width, smoothed normalized multi-target tracking parameters and depth coverage determination results corresponding to the focus target, as well as the face availability threshold and the eye availability threshold, and to switch the focus on the focus target according to the focus strategy.
9. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the autofocus method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the autofocus method as described in any one of claims 1 to 7.