Shooting method and electronic device

Abstract

A shooting method and electronic device are applied to the field of image processing technology. The method includes: displaying a first interface of a camera application, the first interface including a first control; in response to a user input operation on the first control, obtaining N registered facial features corresponding one-to-one to N faces in a gallery application; displaying a second interface of the camera application, the second interface including a first image frame captured by a camera, the first image frame including a first object; when the facial features of the first object match one of the N registered facial features, displaying a first focus frame on the first object and starting automatic focus on the first object; when the facial features of the first object do not match any of the N registered facial features, displaying a first selection frame on the first object, and in response to a user input operation on the first selection frame, starting focus on the first object. This method is used to prioritize automatic focus on the object that the user is most interested in, thereby meeting the user's shooting needs.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular, to a shooting method and electronic equipment. Background Art

[0002] With the increasing development of smart devices, the camera function has become an essential feature of smart terminals. Users can use electronic devices with camera functions to take photos or videos of the subject. During the photo or video recording process, the electronic device can use the focus tracking function to track the focus. For example, when shooting a portrait, the electronic device can use the focus tracking function to track the face. Focus tracking, also known as focus tracking, means that after the electronic device focuses on the subject, such as a face, it controls the focus to remain on the subject, thereby improving the quality of the shot.

[0003] However, when there are multiple shooting subjects, the shooting function of the existing technology cannot directly select the subject that needs to be focused, and the user needs to manually select the focus object, which brings inconvenience to the use and reduces the user experience. Summary of the Invention

[0004] In view of this, the present application provides a shooting method and electronic device that can achieve intelligent focus tracking, which helps to improve the user's shooting experience.

[0005] In a first aspect, an image processing method is provided, which is applied to an electronic device and includes: displaying a first interface of a camera application (a setting interface for a "main character tracking focus" function) in a camera application, wherein the first interface includes a first control (a control for turning on the "main character tracking focus" function); in response to a user's operation on the first control, obtaining N registered facial features corresponding one-to-one to N faces in a gallery application, where N is an integer greater than 0; after operating the first control, displaying a second interface of the camera application (a photo preview interface), wherein the second interface includes a first image frame captured by the camera, and when a first object is photographed, the first image frame includes the first object; thereafter, determining whether the facial features of the first object match one of the N registered facial features; when the facial features of the first object match one of the N registered facial features, displaying a first tracking frame (which may be a highlighted double frame) on the first object, and starting to automatically track focus on the first object. Examples of effects using the first tracking frame include changing the size, shape, color, contrast, and / or other visual features of the associated name.

[0006] When the facial features of a first subject do not match any of the N registered facial features, a first selection box (which can be a single white box) is displayed on the first subject, and the first tracking focus box is different from the first selection box. If the user wishes to manually set the first subject as the tracking focus target when the first subject does not match any of the N registered facial features, they can click the first selection box. In response to a second operation entered by the user on the first selection box, the first subject is tracked, and a second tracking focus box (which can be a single white box) is displayed on the first subject. The second tracking focus box is different from the first tracking focus box. This achieves automatic priority tracking and distinguishes the priority focus subject.

[0007] In one possible implementation, before obtaining N registered facial features corresponding to N faces in a gallery application, it is necessary to determine whether there are clustered faces in the gallery application. Clustered faces are generated by extracting facial features and clustering facial feature data using a clustering algorithm, and this facial feature data is stored in the gallery application. After determining that clustered faces exist in the gallery application, the first N clustered facial features in the gallery application are obtained as the N registered facial features. This allows the user to prioritize the focus on faces that they frequently photograph.

[0008] In one possible implementation, to determine whether clustered faces exist in the gallery app, the camera app must first perform a first query operation, which queries the storage status of clustered faces. If the gallery app stores clustered faces, it writes tag 1 to the database; if the gallery app does not have facial feature data, it writes tag 2 to the database. Therefore, the camera app can determine whether clustered faces exist in the gallery app based on the tags obtained from querying the database. When the camera app obtains the first tag, it determines that clustered faces are stored. This prevents the camera from frequently searching for faces in the gallery app, reducing power consumption.

[0009] In one possible implementation, in order to obtain N registered facial features corresponding one-to-one to N faces in the gallery application, the gallery application is also required to query the on status of the first shooting mode ("protagonist focus tracking" shooting mode). There is a mark in the database, and the "protagonist focus tracking" mode status initially stored in the database is marked as OFF, which means that the "protagonist focus tracking" mode is off, and ON means that the "protagonist focus tracking" mode is on. Therefore, when the gallery application obtains the second mark (ON mark), it is determined that the first shooting mode is on. Therefore, the protagonist focus tracking will only be performed when the "protagonist focus tracking" shooting mode is turned on, thereby avoiding an increase in power consumption.

[0010] In one possible implementation, before determining whether the facial features of the first subject match one of the N registered facial features, the first image (obtained via a callback from the photo preview interface) needs to be sent to the target recognition algorithm to identify the facial features of the subject therein. The algorithm then identifies the facial features of different subjects in the image separately, such as the first facial features of the first subject. This allows automatic acquisition of subject information in the photo preview image.

[0011] In one possible implementation, when a first facial feature matches one of N registered facial features, the autofocus algorithm first sets the first object as the focal point of the first image and focuses on it. Then, the next image frame (the second image) is acquired, and the first object is set as the focal point of the second image and focuses on it. Not only is the autofocus algorithm used to focus on the first object, but a first focus frame is also drawn on the first object based on the first facial feature. This allows for automatic priority focus and allows the prioritized subject to be displayed distinctly from other subjects.

[0012] In one possible implementation, when the first image frame also includes a second object that matches the registered facial features, and when the facial features of the second object match one of the N registered facial features, the areas occupied by the first and second objects in the second interface are compared. When the area occupied by the first object in the second interface is larger than the area occupied by the second object in the second interface, the second object is considered to be a registered face but not the highest priority for tracking focus, and a second tracking focus frame is displayed on the second object, and the second tracking focus frame is displayed separately from the first tracking focus frame. This allows automatic priority focusing on the registered face with the highest priority while distinguishing other registered faces for user selection.

[0013] In one possible implementation, when the first image frame includes a third subject, and if the third subject's facial features do not match any of the N registered faces, a fourth focus frame is displayed on the third subject. This allows automatic focus to prioritize the highest-priority registered face while also highlighting other unregistered faces or subjects for user selection.

[0014] In one possible implementation, when the first object moves so that the second image frame does not include the first object, the tracking focus on the first object is stopped, thereby enabling automatic priority focusing on the registered face with the highest priority and then stopping the focusing.

[0015] In a second aspect, an electronic device is provided, comprising a unit for executing any one of the methods in the first aspect. The electronic device may be a terminal or a chip within the terminal. The electronic device includes an input unit, a display unit, and a processing unit.

[0016] When the electronic device is a terminal, the processing unit may be a processor, the input unit may be a communication interface, and the display unit may be a graphics processing module and a screen; the terminal may also include a memory for storing computer program code, and when the processor executes the computer program code stored in the memory, the terminal executes any one of the methods in the first aspect.

[0017] When the electronic device is a chip in a terminal, the processing unit may be a logic processing unit inside the chip, the input unit may be an input interface, pin or circuit, etc., and the display unit may be a graphics processing unit inside the chip; the chip may also include a memory, which may be a memory inside the chip (for example, a register, a cache, etc.) or a memory located outside the chip (for example, a read-only memory, a random access memory, etc.); the memory is used to store computer program code, and when the processor executes the computer program code stored in the memory, the chip executes any one of the methods of the first aspect.

[0018] According to a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer program code. When the computer program code is executed by an electronic device, the electronic device executes any one of the methods according to the first aspect.

[0019] In a fourth aspect, a computer program product is provided, comprising: a computer program code, wherein when the computer program code is executed by an electronic device, the electronic device executes any one of the methods in the first aspect.

[0020] In a fifth aspect, a chip is provided, the chip including a processor. When the processor executes instructions, the processor executes any one of the methods in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of an interface provided in an embodiment of the present application;

[0022] Figure 2 A hardware structure diagram of an electronic device provided in an embodiment of the present application;

[0023] Figure 3 A software architecture diagram of an electronic device provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of the interface provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of module interaction provided in an embodiment of the present application;

[0026] Figure 6A schematic diagram of an interface provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of an interface provided in an embodiment of the present application;

[0028] Figure 8 A schematic diagram of module interaction provided in an embodiment of the present application;

[0029] Figure 9 A schematic diagram of an interface provided in an embodiment of the present application;

[0030] Figure 10 A schematic diagram of an interface provided in an embodiment of the present application;

[0031] Figure 11 A schematic diagram of module interaction provided in an embodiment of the present application;

[0032] Figure 12 A schematic diagram of an interface provided in an embodiment of the present application;

[0033] Figure 13 A schematic diagram of an interface provided in an embodiment of the present application;

[0034] Figure 14 A schematic diagram of an interface provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0036] It should be understood that the terms "first," "second," and the like in the specification, claims, and drawings of this application are used to distinguish between different objects, rather than to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0037] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0038] See also Figure 1 , Figure 1 is a schematic diagram of a shooting scene. Figure 1 As shown in Figure A, the user interface 100 displays a page with application icons, which may include multiple application icons (for example, a weather application icon, a calendar application icon, an album application icon, a note application icon, an email application icon, an application store application icon, a settings application icon, etc.). A page indicator may also be displayed below the above-mentioned multiple application icons to indicate the positional relationship between the currently displayed page and other pages. There are multiple application icons (for example, a camera application icon 11) below the page indicator. These application icons remain displayed when the page is switched. It can be understood that the camera application icon 11 is an icon for a camera application (i.e., a camera application). The camera application icon 11 can be used to trigger the launch of the camera application.

[0039] The electronic device may detect a user operation on the camera application icon 11, and in response to the operation, the electronic device may display a Figure 1 Preview interface 101 is shown in Figure B. Preview interface 101 is the default shooting mode interface of the camera application, where users can preview images and take photos. When the subject includes multiple subjects, such as people and pets, the camera's preview interface 101 displays the face of the person or pet and its focus frame. The user can select the focus subject, and the electronic device focuses on the selected focus subject. When there are multiple subjects, the user needs to manually select the focus subject, which is inconvenient and reduces the user experience of focusing and shooting.

[0040] The present application provides a shooting method. The shooting method can be applied to an electronic device equipped with a camera. According to the shooting method, in a shooting scene where multiple shooting objects are in the same frame, the electronic device can automatically focus on one of the shooting objects with the highest priority, and always display the "focusing frame" of the priority-focused object. In some embodiments of the present application, the electronic device can automatically focus on the shooting object with the highest priority and multiple objects with lower priorities and objects that are not in the priority list at the same time, and display the "focusing frame" of the focused object according to the priority.

[0041] As understood, "focusing" in this application refers to the process of achieving a clear image of the focused object by changing the distance and position of the camera's focus mechanism. "Focus tracking" involves continuously focusing after setting a focus point, maintaining that focus point even if the focused object moves. This refers to continuous focusing.

[0042] The following combination Figure 2 Describe the hardware architecture used in the embodiments of the present application.

[0043] refer to Figure 2 , Figure 2 A hardware system of an electronic device suitable for the present application is shown.

[0044] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0045] Among them, the sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0046] It should be noted that Figure 2 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include Figure 2 More or fewer components than those shown, or the electronic device 100 may include Figure 2 Combinations of some of the components shown, or alternatively, the electronic device 100 may include Figure 2 Subassemblies of some of the components shown. Figure 2 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0047] The processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). The different processing units may be independent devices or integrated devices.

[0048] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0049] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0050] In some embodiments, the processor 110 may include one or more interfaces. For example, the processor 110 may include at least one of the following interfaces: an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and a USB interface. Figure 2 The connection relationship between the modules shown is only for illustrative purposes and does not limit the connection relationship between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of the multiple connection modes in the above embodiments.

[0051] Electronic device 100 can implement display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0052] Display screen 194 can be used to display images or videos. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini light-emitting diode (Mini LED), a micro light-emitting diode (Micro LED), a micro OLED, or a quantum dot light-emitting diode (QLED).

[0053] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0054] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can perform algorithmic optimization on image noise, brightness, and color. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0055] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard red, green, blue (RGB), YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0056] The processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). The different processing units may be independent devices or integrated devices.

[0057] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals.

[0058] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0059] The NPU is a processor that draws inspiration from biological neural network architecture, such as the transmission patterns between neurons in the human brain, to rapidly process input information and continuously learn. The NPU enables intelligent cognitive functions in electronic device 100, such as image recognition, face recognition, voice recognition, and text comprehension.

[0060] The external memory interface 120 may be used to connect an external memory card, such as a secure digital (SD) card, to expand the storage capacity of the electronic device 100 .

[0061] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, and applications required for at least one function (for example, a sound playback function and an image playback function). The data storage area may store data created during the use of the electronic device 100 (for example, audio data and a phone book). In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as: at least one disk storage device, a flash memory device, and a universal flash storage (UFS). The processor 110 executes various processing methods of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0062] The electronic device 100 can implement audio functions, such as music playback and recording, through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0063] The distance sensor 180F is used to measure distance. The electronic device 1000 can measure distance using infrared or laser. In some embodiments, for example, in a photography scenario, the electronic device 1000 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0064] Ambient light sensor 180L is used to sense ambient light brightness. Electronic device 1000 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light brightness. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether electronic device 1000 is in a pocket to prevent accidental touches.

[0065] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 1000 can use the collected fingerprint characteristics to implement functions such as unlocking, accessing application locks, taking photos, and answering calls.

[0066] The touch sensor 180K is also referred to as a touch-sensitive device. The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also referred to as a touch screen. The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor 180K can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100 and at a different location from the display screen 194.

[0067] The buttons 190 include a power button and a volume button. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive button input signals and implement functions related to the button input signals.

[0068] The technical solutions involved in the following embodiments can all be implemented in the electronic device 100 having the above hardware architecture.

[0069] For ease of understanding, the following examples of this application will be described with Figure 2 Taking the electronic device with the structure shown as an example, the shooting method provided in the embodiment of the present application is specifically explained in combination with the accompanying drawings and application scenarios.

[0070] The following combination Figure 3 The software structure of an electronic device provided in an embodiment of the present application is introduced.

[0071] based on Figure 3 The software structure of the electronic device shown introduces a "protagonist tracking focus" shooting method provided in an embodiment of the present application.

[0072] like Figure 3As shown, the storage module in the gallery application stores the captured raw image data. The gallery then transmits this raw image data via the image transfer module in the application framework layer to the facial feature extraction and clustering algorithms in the algorithm library for processing. The facial feature extraction and clustering algorithm performs feature extraction and clustering on images of identical faces and returns the clustered images to the gallery's storage module. Upon receiving the clustered image data, the gallery application's storage module clusters and stores the images of identical faces. The gallery application's verification module verifies whether the "Main Character Tracking" switch in the camera application is enabled. If it is, the gallery application's storage module transmits the stored clustered image data to the camera HAL via the transfer module in the application framework layer. Optionally, the storage module only transmits the feature data of N clustered faces after clustering identical faces. It is understood that the feature data of a clustered face can be a facial feature model calculated based on multiple images after recognizing the same face in multiple different images. Optionally, to generate a facial feature model, it is necessary to have at least X photos of a single face (such as person A) in the gallery, so that a facial feature model of person A is generated based on the X photos. This application does not limit the number of X. In this application, the feature data of N clustered faces is called "registered face" data. The embodiment of this application introduces "registered face" to describe the portrait data in the gallery. A registered face refers to a face that an electronic device automatically focuses on through a focus frame. The registered face can be predetermined. The embodiment of this application does not specifically limit the method of determining the registered face. It can be understood that the description here is based on "registered face" as an implementation method, and the embodiment of this application is not limited to this. For example, the registered face can also be other registered objects, such as cats, dogs, and other objects.

[0073] The image captured by the electronic device's camera is transmitted to the face detection algorithm through the camera HAL. The face detection algorithm identifies facial features in the image and transmits them back to the camera HAL. The camera HAL provides the facial characteristics determined by the face detection algorithm and the "registered face" feature data to the target tracking algorithm for target recognition. The target recognition algorithm transmits the identified subject information back to the camera HAL, which reports the tracked target to the camera application. The camera application's display module displays the "focus frame" of the tracked target in a way that distinguishes it from other subjects. The target recognition algorithm can then transmit the optimal "registered face" data to the autofocus algorithm, which then adjusts the camera focus based on the optimal identified target to track the target.

[0074] Users can manually select an area to track. If the target recognition algorithm identifies a subject in the user-selected area, the identified subject will be sent to the autofocus algorithm for tracking. If the target recognition algorithm does not identify a subject in the user-selected area, the autofocus algorithm will focus on the user-selected area.

[0075] In this application, the above-mentioned mode of adjusting the camera focus tracking according to the optimal recognition target by the autofocus algorithm is referred to as "protagonist focus tracking". The "protagonist focus tracking" used here is not limited to the focus tracking effect on the person described and shown. However, the term "protagonist" is intended to indicate any form of effect that will distinguish it from other photographed objects. Examples of such effects include changing the size, shape, color, contrast and / or other visual features of the relevant name, or other items. In other embodiments, "protagonist" refers to displaying one or more photographed objects in a specific area of the screen or window to highlight them as distinguished from other display methods. In other embodiments, highlighting is achieved using non-visual methods, and those skilled in the art should think of other optional embodiments.

[0076] The following combination Figure 4-Figure 6 The specific implementation of the above embodiment is described.

[0077] 1. Data Preparation for "Registered Face"

[0078] It should be noted that the availability of "Main Character Focus Tracking" depends on whether portrait data has been generated in the gallery app. If the gallery data has been generated, "Main Character Focus Tracking" is available; if not, "Main Character Focus Tracking" is unavailable.

[0079] Among them, the portrait data in the gallery has been generated, which can be understood as multiple registered faces have been determined.

[0080] The following describes the "face registration" scenario:

[0081] Figure 4 The following figure shows an interface diagram of the gallery discovery page showing "Register Face". Figure 4 As shown in (1), the screen display system of the electronic device displays the current output interface content, which displays multiple application APPs. The user can click the "Gallery" application icon 301 on the electronic device desktop to instruct the electronic device to open the gallery application, and the electronic device displays the following Figure 4 The interface shown in (2) is shown in the figure. Figure 4 As shown in (2), the gallery displays a photo page, which contains photos taken by the user (such as photos taken by the user today and photos taken by the user yesterday). The gallery interface also includes albums, moments, and a discovery control 302. When the user clicks the discovery control 302, the gallery's discovery page is displayed. Figure 4 As shown in (3). Figure 4 As shown in (3), the discovery page includes a portrait area 303. In the dotted box shown in the portrait area 303, portrait data in the gallery is displayed, for example, Figure 4(3) shows four portraits. Optionally, Figure 4 The four portraits shown in (3) are sorted in descending order according to the number of people who appear in the photo. The dotted box in the portrait area 303 also includes a "more" control 304. When the user clicks the control 304, the portrait page is entered, and the portrait page can display the people that the electronic device focuses on by default, such as Figure 4 The interface shown in (4).

[0082] like Figure 4 As shown in (4), the dotted box 306 is used to display the statistical data of multiple portraits. Optionally, the statistical data of the multiple portraits displayed in the dotted box 306 include the names of the portraits and the corresponding numbers. Optionally, the multiple portraits in the dotted box 306 can be displayed in order from large to small in number. For example, the dotted box 306 includes: a portrait named Dad (corresponding number is 58), a portrait named Mom (corresponding number is 53), a portrait named Sister (corresponding number is 46), a portrait named Brother (corresponding number is 45), and a portrait named Friends (corresponding numbers are 42 and 30 respectively). Among them, the name of each portrait is in the lower left corner of the portrait frame, and the number of each portrait appearing in the photo is in the lower right corner of the portrait frame. For example, taking the portrait 307 in the dotted box 306 as an example, the name of the portrait 307 is "Dad" and the number of portraits appearing in the photo is 58. It can be understood that Figure 4 The portrait statistics shown in (4) are illustrated by taking photos of the “personal” category as an example, and the present application embodiment is not limited thereto. For example, Figure 4 The portrait interface shown in (4) also includes a "group photo" category. When the user clicks on the "group photo" category, the portrait statistics under the group photo category can be displayed in the interface.

[0083] The following will be combined Figure 5 Steps S500-S505 in the figure introduce the data preparation process for "registering a face":

[0084] S500: The image library application sends the original image data to the facial feature extraction and clustering algorithm.

[0085] Exemplarily, the electronic device can be in a charging screen-off state, and the storage module sends the original image data, thereby avoiding the power consumption of image processing affecting the normal user experience. Specifically, the storage module can send the original image data to the face data transmission module of the framework layer, and then the face data transmission module sends the original image data to the face feature extraction and clustering algorithm. Optionally, the electronic device determines whether the device is currently in a charging screen-off state and whether the current power is greater than or equal to a preset power threshold. If the electronic device is currently in a charging screen-off state and the current power is greater than or equal to the preset power threshold, the gallery application sends the original image data to the face feature extraction and clustering module for processing. If the electronic device is not currently in a charging screen-off state, or the current power is less than the preset power threshold, the gallery application will not send the original image data for the time being.

[0086] S501: The facial feature recognition, extraction and clustering module processes the facial image and returns the facial feature data to the gallery application.

[0087] Exemplarily, when the original image includes a face, the original image can be called a face image. The facial feature extraction can be achieved by using technologies such as histogram of oriented grids (HOG) or convolutional neural network (CNN) to generate facial feature data. Optionally, high-frequency facial feature data can be generated preferentially. High-frequency faces refer to faces that appear multiple times in images stored by electronic devices. Alternatively, it can be understood as faces that are photographed multiple times by electronic devices. High-frequency faces can be users (N is an integer) that rank in the top N corresponding to the number of photos containing characteristic faces in the gallery application, sorted from large to small. For example Figure 4 As shown in (4), after the multiple portraits are arranged in descending order, the high-frequency faces are the portrait named Dad (corresponding to a number of 58), the portrait named Mom (corresponding to a number of 53), and the portrait named Sister (corresponding to a number of 46). The high-frequency faces can also be users preset by the user. The specific method for determining the high-frequency faces is not limited in this embodiment of the application.

[0088] Optionally, the facial feature extraction and clustering algorithm can compare and cluster the extracted features with any one of the multiple features of the high-frequency face. Alternatively, the facial feature extraction and clustering algorithm can calculate the similarity between the extracted features and the features of the high-frequency face and compare them. In an embodiment of the present application, the electronic device can cluster the original images of the same face based on the extracted features and the high-frequency face, and when the calculated cosine value (cos value) is less than a preset value, the extracted features are the features corresponding to the high-frequency user, and the comparison results are consistent.

[0089] Specifically, after the facial feature extraction and clustering algorithms complete clustering of high-frequency facial feature data, the high-frequency facial feature data is returned to the image library application. It is understood that the high-frequency facial feature data in the algorithm library is returned to the image library application via the facial data transmission module at the framework layer. Optionally, after processing the facial image, the facial feature recognition, extraction, and clustering module can return all facial feature data to the image library application, not just the high-frequency facial feature data.

[0090] After the facial feature extraction and clustering algorithm completes the clustering of the high-frequency facial feature data, one implementation method is to use the top N high-frequency faces (the top N corresponding users in the gallery application sorted from large to small in the number of photos containing feature faces) as the facial feature data that meets the "registered face" conditions referred to in this application.

[0091] Another implementation method is to use the facial feature data of high-frequency faces and more than X photos containing characteristic faces as the "registered face" data for "protagonist tracking" referred to in this application. Figure 4 In the interface shown in (4), assuming that the high-frequency faces are the top 5 faces, the top 5 high-frequency faces automatically determined according to the number of photos can be: a portrait named Dad (corresponding to 58), a portrait named Mom (corresponding to 53), a portrait named Sister (corresponding to 46), a portrait named Brother (corresponding to 45), and a portrait named Friend (corresponding to 42). These 5 high-frequency faces can be used as "registered faces" for automatic focus tracking by electronic devices. An optional implementation is to use the top 5 faces automatically determined by the number of photos and the number of photos greater than M as "registered faces": Optionally, if M is selected as 50, in the following example Figure 4 In (4), only the father’s portrait (the corresponding number is 58) meets the conditions, so the father’s portrait is selected as the “registered face”.

[0092] It is understood that step S501 is an embodiment of this application where the top N most frequently viewed faces are used as facial feature data that meets the "registered face" criteria. It is also understood that when the "registered face" can be another subject, such as a human body, a cat, or a dog, a feature recognition and extraction algorithm can be used to process the image and return the corresponding subject feature data to the image library application.

[0093] S502: The gallery application stores the facial feature status identifier.

[0094] It is understood that when the gallery application's storage module contains facial feature data, the gallery application writes tag 1 to the database; when the storage module does not contain facial feature data, the gallery application writes tag 2 to the database. Alternatively, when the storage module contains facial feature data that meets the requirements for a "registered face," the gallery application writes tag 1 to the database; when the storage module does not contain facial feature data that meets the requirements for a "registered face," the gallery application writes tag 2 to the database. The database can be a database encapsulated in the framework layer that provides data sharing for application layer apps and is used to store user preferences.

[0095] S503: The gallery application sends a query request for the camera's main character focus tracking mode status.

[0096] It's understood that each time a user turns the "Main Character Tracking" control on or off (see below for details on how to do this), the "Main Character Tracking" mode status flag is saved to the database. The "ON" flag indicates that "Main Character Tracking" mode is currently on, and the "OFF" flag indicates that "Main Character Tracking" mode is currently off. The Gallery app can send a query to the database to request information about the "Main Character Tracking" mode status.

[0097] S504: In response to the query request of the gallery application for the state of the camera's main subject tracking focus mode, the database returns an identifier.

[0098] It is understandable that before the user makes any settings, since "Main Character Tracking" is initially in the default off state, the database initially stores the "Main Character Tracking" mode status flag as OFF. When the user turns on the "Main Character Tracking" switch for the first time, the camera application writes the flag ON to the database. If the user turns off the "Main Character Tracking" switch, the camera application writes the flag OFF to the database again. Upon receiving a request from the camera application to query the "Main Character Tracking" mode status, the database returns the latest flag representing the current "Main Character Tracking" on / off state to the gallery application based on the stored "Main Character Tracking" mode status flags.

[0099] Optionally, if the database returns an indication to the image library application indicating that the "main character tracking focus" mode is on, step S505 may be further included after step S504. If the database returns an indication to the image library application indicating that the "main character tracking focus" mode is off, step S506 will be executed.

[0100] S505: When the subject tracking mode is turned on and facial feature data exists, the gallery application sends the most recently acquired facial feature data to the camera HAL.

[0101] It is understandable that when the database returns the flag ON to the gallery application, it means that the "main character tracking focus" mode is turned on. As mentioned above, when the electronic device is in the charging and screen-off state, it will extract and cluster the facial feature data of the face image and store it in the gallery application. Optionally, before step S505, when the facial feature data stored by the gallery application last time is changed compared to the facial feature data stored last time and the "main character tracking focus" mode is turned on, the gallery application can send the facial feature data stored last time to the camera HAL. Optionally, the gallery application can send the "registered face" feature data stored last time to the camera HAL. Optionally, the gallery application can send the "registered face" feature information stored last time to the camera HAL when the "registered face" feature data stored by the gallery application last time is changed compared to the "registered face" feature data stored last time and the "main character tracking focus" mode is turned on.

[0102] Optionally, the gallery application sends facial feature data to the camera HAL through a transfer module of the framework layer.

[0103] It is understood that this application does not limit the order in which the electronic device executes step S502 and steps S503-S504. The electronic device may execute steps S503-S504 first, may execute step S502 first, or may execute steps S502 and steps S503-S504 simultaneously.

[0104] 2. "Protagonist Focus" mode is turned on

[0105] An optional implementation is to enable the "main character tracking focus" mode in the camera application. Figure 5 Steps S506-S516 in the figure introduce the process of the "main character tracking focus" mode:

[0106] S506: The user starts the camera application.

[0107] It is understandable that the user can trigger the electronic device to start the camera application. Accordingly, the camera application in the electronic device can receive the request to start the camera application.

[0108] For example, Figure 1 As shown in (1), the user can click the camera application icon 101, and accordingly, the electronic device can receive a request to start the camera application. In response to the request to start the camera application, the electronic device can start the camera application and display Figure 1 The user interface 101 shown in (2) is shown in FIG. It is understood that the user can also start the camera application by voice, gesture, etc., and this application does not limit the specific method in which the user triggers the electronic device to start the camera application.

[0109] It is understood that the camera can start after receiving a request to start the camera from the camera application. Furthermore, after starting, the camera can continue to collect raw image data. The camera can send the collected raw image data to the camera HAL. It is understood that when the camera application is in preview mode, the camera HAL can process the raw image data and send the processed image to the display screen for display. It is understood that the preview mode mentioned here can include a photo preview mode, a video preview mode, etc.

[0110] It should be understood that steps S500 to S505 are not necessary steps before step S506. The electronic device may first execute steps S500 to S505, and may first execute step S506.

[0111] S507: The camera application queries the facial feature data storage status

[0112] It is understandable that when the database stores an identifier representing whether the gallery application has facial feature data, the camera application can request the database to obtain the identifier. Optionally, if the database stores facial feature data representing whether the gallery application meets the requirements of a "registered face", the camera application can request the database to obtain the identifier, thereby obtaining the information on the "registered face" storage status of the gallery application without directly interacting with the gallery application. Optionally, when the database stores a "protagonist focus tracking" mode status identifier, the camera application can also request the database to obtain the identifier when querying the facial feature data storage status. This application does not impose any restrictions on the order in which the electronic device queries the facial feature data storage status and the "protagonist focus tracking" mode status. The camera application can first query the facial feature data storage status and the "protagonist focus tracking" mode status, or it can query them at the same time.

[0113] S508: The database returns the facial feature data identifier

[0114] It is understood that when receiving a request from the camera application to query the storage status of facial feature data, the database returns either ID 1 or ID 2 to the camera application based on the stored facial feature data status identifier. Specifically, returning ID 1 indicates that the facial feature data identifier exists in the gallery application (or facial feature data that meets the requirements of a "registered face"); returning ID 2 indicates that the facial feature data identifier does not exist in the gallery application (or facial feature data that meets the requirements of a "registered face").

[0115] Optionally, if in step S507 the camera application also sends a request to the database to query the status of the "main character tracking" mode, when receiving the request from the camera application to query the status of the main character tracking mode, the database returns an identifier representing the current "main character tracking" on / off status to the gallery application based on the stored "main character tracking" mode status identifier.

[0116] Optionally, as an embodiment, when the facial feature data storage status flag is present in the database, it indicates that the "main character tracking focus" mode is enabled. At this point, if the "main character tracking focus" mode switch is off, the electronic device may prompt the user to turn on the "main character tracking focus" mode switch. Specifically, after executing steps S506-S508, steps S509-S512 may also be included. It should be understood that steps S509-S512 are not required steps after steps S506-S508.

[0117] S509: When the main character focus tracking mode is turned off and facial feature data exists, the first interface is displayed.

[0118] For example, when the database facial feature data storage status flag is 1 and the "main character tracking focus" mode on / off flag is OFF, it means that there is "registered face" data in the application gallery and the main character tracking focus mode is off. After the electronic device starts the camera application, when the electronic device recognizes that the gallery has generated available face data, it can display the following information: Figure 6 The user interface shown in (1) is shown in the photo preview interface. Window 601 (i.e., an implementation of the first interface) is displayed, and window 601 is used to prompt the user whether to turn on the "main character tracking focus" function. For example, the content displayed in window 601 is: Do you want to turn on the "main character tracking focus" function? Identify the portraits that are often taken in the gallery and give priority to tracking focus. This prompts the user to turn on the "main character tracking focus" mode. It is understandable that the user can also trigger the electronic device to enter the tracking focus mode through voice, gestures, etc.

[0119] S510: In response to the user's operation of turning on the subject tracking focus mode, the camera application writes a subject tracking focus mode status identifier into the database.

[0120] For example, when the prompt box appears, when the user operates to turn on the "protagonist focus" mode, for example, Figure 6 The opening control 63 shown in (1) (i.e., an implementation of the first control) will turn on the "main character tracking focus" mode. Specifically, the camera application will write the flag ON to the database, indicating that the main character tracking focus mode has been turned on. It is understood that the original OFF flag will be replaced by the new flag written this time or the return priority when it is requested will be lowered. It is understood that after the "main character tracking focus" mode is turned on, at this time, Figure 6 In the setting interface shown in (2), option 61 of "Intelligent Focus Tracking" and option 62 of "Main Character Focus Tracking" are turned on.

[0121] S511: The camera application calls the gallery application interface and sends a request to refresh facial feature data to the gallery application.

[0122] It is understandable that the gallery application can refresh the stored facial feature data to the camera HAL, and the camera application can send a request to refresh the facial feature data to the gallery by calling the gallery application interface.

[0123] S512: Upon receiving a request to refresh facial feature data, the gallery application queries the focus tracking mode status of the protagonist.

[0124] It is understood that the database stores the "main character tracking focus" mode status flag. In this embodiment, in step S510, the camera application has written the flag ON into the database. The flag ON indicates that the "main character tracking focus" mode is currently on, so the gallery application can query the "main character tracking focus" mode status.

[0125] S513: In response to the query request of the gallery application for the state of the camera's subject tracking focus mode, the database returns an identifier.

[0126] In this embodiment, in step S510 , the camera application has written the flag ON into the database. The flag ON represents that the “main character tracking focus” mode is currently on, that is, the flag ON returned by the database.

[0127] If the "main character tracking focus" mode switch is already on, the electronic device needs to execute steps S514-S515 before shooting in the "main character tracking focus" mode, and send the facial feature data to the algorithm library so that the tracking algorithm can determine the face that needs to be tracked.

[0128] S514: When the subject tracking mode is turned on, the gallery application sends facial feature data to the camera HAL.

[0129] In this embodiment, optionally, the most recently saved facial feature data may be sent to the camera HAL with reference to step S505. Optionally, this step may only send the most recently saved facial feature data that meets the "registered face" condition.

[0130] S515: The camera HAL sends the facial feature data to the algorithm library.

[0131] As an embodiment, the camera HAL may save the most recently acquired facial feature data and send it to the target recognition algorithm in the algorithm library. Alternatively, the camera HAL may only send the most recently acquired facial feature data that meets the "registered face" criteria to the target recognition algorithm in the algorithm library.

[0132] In some embodiments of the present application, after step S515, the following steps may be further included:

[0133] S516: When the subject tracking mode is turned on, in response to the user's operation of turning on the subject tracking mode, the camera application sends an instruction to turn on the registered face tracking.

[0134] In this embodiment, it is understandable that the camera application can send a registered face focus tracking instruction to the algorithm library through the camera HAL, so that the algorithm can track the registered face. The specific tracking process will be introduced below.

[0135] An optional implementation method is to enable the "main character tracking focus" mode in addition to the camera application, and also enable the "main character tracking focus" mode in the settings.

[0136] Figure 7 This is an example interface diagram of camera settings (i.e., another implementation of the first interface). Figure 7 (1) shows the interface of the state where "Main Character Focus" is unavailable. When the registered face data in the gallery is effective, the interface will be displayed as follows Figure 7 As shown in (2), the option 62 of "Main Character Focus Tracking" in the setting interface is in an available state. The available state can also be understood as a non-grayed state, that is, the user can operate it.

[0137] Figure 7 The option 62 of "Main Character Focus Tracking" shown in (2) can be clicked by the user (i.e., another embodiment of the first control). When the user clicks the option 62 of "Main Character Focus Tracking", the "Main Character Focus Tracking" is turned on and the interface is displayed as follows: Figure 7 As shown in (3), option 62 of "Main Character Focus Tracking" is turned on. The "Main Character Focus Tracking" mode is turned on. Specifically, the camera application writes a flag "ON" to the database, indicating that the "Main Character Focus Tracking" mode has been turned on.

[0138] It can be understood that the implementation steps after turning on the "Main Character Focus Tracking" switch in the camera settings interface can refer to steps S510-S516 of turning on the "Main Character Focus Tracking" through the prompt box in the camera application, and will not be repeated here.

[0139] It is understood that whether "Main Character Tracking" mode is enabled in the camera settings interface or in the camera app, the activation of "Main Character Tracking" depends on whether the registered facial data in the gallery is valid. If the registered facial data in the gallery is determined to be valid, or if the gallery has generated usable portrait data, the "Main Character Tracking" option is enabled. If the registered facial data in the gallery is not valid, the "Main Character Tracking" option is disabled, for example, by graying it out to inform the user that "Main Character Tracking" is unavailable.

[0140] 3. Turn off the "Main Character Focus" mode

[0141] Furthermore, users can also click on the "Smart Focus Tracking" option to turn it off. Since "Smart Focus Tracking" and "Main Character Focus Tracking" are linked, and the premise for turning on "Main Character Focus Tracking" is that "Smart Focus Tracking" is already turned on, when the "Smart Focus Tracking" option is turned off, the "Main Character Focus Tracking" option is also automatically turned off. For example, if a user clicks Figure 7 In (3), click on the option 61 of “Smart Focus Tracking” to turn off “Smart Focus Tracking”. The interface will be displayed as follows: Figure 7 As shown in (4). Figure 7 As shown in (4), option 61 of "Intelligent Focus Tracking" is turned off, and option 62 of "Main Character Focus Tracking" is not displayed in the setting options.

[0142] In some embodiments of the present application, in order to disable the “main character tracking focus” mode, after step S516, the following steps may be further performed:

[0143] S517: In response to the user's operation of turning off the main character tracking mode, the camera application writes the main character tracking mode status identifier into the database.

[0144] Specifically, the camera app writes the flag OFF to the database, indicating that the subject's focus tracking mode is off. It is understood that any existing ON flag will be replaced by the newly written flag or have its priority lowered when requested.

[0145] S518: When the subject tracking mode is off, in response to the user's operation of turning on the subject tracking mode, the camera application sends an instruction to turn off the registered face tracking.

[0146] In this embodiment, optionally, it is understandable that the camera application can send a command to turn off the "registered face" focus tracking to the algorithm library through the camera HAL, thereby stopping the algorithm from focusing on the "registered face".

[0147] S519: When the subject tracking mode is turned off, the camera application sends an instruction to clear facial feature data.

[0148] Optionally, when the "main character tracking focus" mode is turned off, the gallery application sends an instruction to clear facial feature data to the camera HAL through the transfer module of the framework layer.

[0149] It is understood that the present application does not limit the order in which the electronic device executes steps S517-S519. The electronic device may also execute S517, S518, and S519 simultaneously.

[0150] 4. Automatic "Protagonist Focus"

[0151] like Figure 8As shown in the flowchart, when an electronic device enters the camera app and starts shooting after turning on the "Main Subject Focus Tracking" mode, the electronic device automatically identifies whether the subject is a registered face. If a registered face is recognized, the electronic device performs focus tracking on the "registered face."

[0152] After the camera application sends the registered face focus tracking start instruction to the algorithm library in step S516, the following steps may be performed:

[0153] S600: The terminal obtains a first preview image of the scene to be photographed and determines a target subject.

[0154] After the camera application is started, the camera can continuously capture images. When the terminal previews the captured image, it can send the image (i.e., the first image) to the camera HAL. Optionally, the camera application calls back the preview stream through a callback function and processes the image obtained by the callback. For example, taking the processing of image 1 as an example, the camera HAL sends image 1 to the target recognition algorithm. The target recognition algorithm performs image recognition on image 1 to obtain the identified subject information category label in the preview image. The subject information category label may include: face, human body, cat, dog, etc., which is not limited in this solution.

[0155] S601: When there is an identifiable subject, report the identified subject information

[0156] Optionally, taking the example of recognizing a face in image 1 input to a target recognition algorithm, key points of the image are detected and located to identify key point detection and location results, which include key points of the face in the image, and the face is located using the key points of the face. The face image is cropped from the image using the key points of the face, and an affine transformation is performed on the face image to correct the face. The feature extraction of the corrected face is then performed using a convolutional neural network (CNN) to obtain facial features.

[0157] For images of faces in the gallery app, the image matching module similarly detects and locates key points in the images. Using these key points, the face is cropped from the image and affine transformed to correct it. Feature extraction is then performed on the corrected face to obtain its characteristics. The module then determines whether the obtained facial features match those of the "registered face." It is understood that the object recognition algorithm can also identify subjects such as humans, cats, and dogs.

[0158] The target recognition algorithm transmits the identified subject information back to the camera HAL, which then reports the tracked target to the camera application. Optionally, along with the identified subject information, the tracking ID, tracking status, tracking area, tracking status, and registered face identification data can also be reported as "identification data." The tracking ID indicates the tracking subject ID, used to identify the tracking subject; the tracking type includes face, body, cat, and dog to distinguish the tracking subject; and the registered face identification identifies whether the facial features meet the "registered face" requirement.

[0159] S602: Autofocus based on the highest priority target

[0160] Since the image library has already sent the facial feature data that meets the "registered face" criteria to the target recognition algorithm in step S515, the target recognition algorithm can determine the priority of the recognized subjects based on the matching of all recognized subject information with the facial feature data of the "registered face" criteria. The target recognition algorithm then sends all recognized subject information and its priority information to the autofocus algorithm.

[0161] In one possible implementation, objects other than "registered faces" can also be other objects, such as cats, dogs, and other objects. Therefore, the priority can be determined by referring to the method of registered faces > unregistered faces > human bodies > cats > dogs. That is, when the recognized subject information (face) contains facial feature data that meets the "registered face" conditions, the subject with facial feature data that meets the "registered face" conditions has the highest priority. In one possible implementation, when the recognized subject information (face) contains multiple facial feature data that meet the "registered face" conditions, the face with the largest area among the multiple facial feature data that meet the "registered face" conditions has the highest priority, that is, the priority is: registered faces with large facial areas > registered faces with small facial areas > unregistered faces > human bodies > cats > dogs. In other implementations, feature conditions such as gender, age, race, makeup, and hairstyle can also be set to determine the face with the highest priority among the multiple faces with facial feature data that meet the "registered face" conditions.

[0162] Optionally, the target recognition algorithm sends all the recognized subject information and the subject information with the highest priority to the autofocus algorithm, and the autofocus algorithm gives priority to adjusting the camera to autofocus and track the target on some subjects. It can be understood that when the autofocus algorithm focuses on some subjects, it can focus on the subject with a high priority first. One implementation method is that when the priority is set according to whether the "registered face" condition is met, the autofocus algorithm gives priority to focusing on the subject (face) that meets the "registered face" condition in the subject information (face). When there are multiple faces that meet the "registered face" condition, the autofocus algorithm gives priority to focusing on the subject (registered face) with the highest priority.

[0163] S603: Display the focus frame of the tracking target

[0164] The camera determines the focus based on the recognized subject information, draws the focus frame, and displays the focus frame.

[0165] In single-person and multi-person scenarios, the camera application will use the target recognition algorithm to determine whether the subject information it identifies contains a registered face, and the display of the focus frame will be different. Please refer to the previous article for specific scenarios.

[0166] It is understandable that the present application does not limit the order in which the electronic device executes steps S602-S603, and the electronic device can also execute S602 and S603 at the same time.

[0167] When "Smart Focus Tracking" and "Main Subject Focus Tracking" are enabled, the device automatically identifies the subject of the shot as a registered face when the user is shooting. If a registered face is recognized, the device performs focus tracking on that face. The following details the implementation of automatic "Main Subject Focus Tracking" upon recognition of a registered face, in conjunction with various interfaces.

[0168] Figure 9 The following figure shows an example of the camera preview interface after turning on the "Intelligent Focus Tracking" and "Main Character Focus Tracking" functions. When there is only one person, the target recognition algorithm recognizes face A, and face A is a "non-registered face." The target recognition algorithm sends the information of face A to the autofocus algorithm, and the autofocus algorithm focuses on face A. The target recognition algorithm reports the information of face A and "identification data" such as the registered face identification to the camera application for drawing the focus frame for face A. Figure 9 As shown in (1), the focus frame for the "unregistered face" A is the regular focus frame 901. The regular focus frame 901 will automatically disappear after a period of time (for example, 5 seconds). The optional regular focus frame 901 is usually displayed in white in the preview interface. Optionally, the regular focus frame is displayed normally and only includes one main frame.

[0169] As the lens moves, when face B appears in the photo preview interface, the target recognition algorithm recognizes face B as a registered face and reports face B's information and "identification data" such as the registered face identifier to the camera application for drawing a focus frame for face B. The focus frame for face B at this time is the main focus frame proposed in this application. Figure 9 (2) is described by taking a photo of a single registered face as an example. Figure 9 As shown in (2), the preview interface displays face B and the main character tracking focus frame 902. When face B moves, the autofocus algorithm will continue to focus on face B, and the main character tracking focus frame 902 will also move with face B. For example, when face B moves, Figure 9The picture frame shown in (2) and Figure 9 In the picture frame shown in (3), the main character tracking focus frame 902 moves along with the face B. Moreover, the size of the main character tracking focus frame also changes as the face B moves.

[0170] The main character tracking focus frame 902 is used to perform main character tracking (or continuous tracking) on face B. Optionally, the main character tracking focus frame is highlighted and includes a main frame and a sub-frame. It should be understood that Figure 9 Medium (2) or Figure 9 The main character tracking frame shown in (3) can be yellow or other bright colors in a specific implementation to distinguish it from the conventional tracking frame. It is understandable that those skilled in the art can make other designs on the color, display, size, shape, etc. of the conventional tracking frame and the main character tracking frame to distinguish them. Figure 9 Medium (2) or Figure 9 The main character tracking frame 902 shown in (3) is just an example.

[0171] above Figure 9 (2) or Figure 9 The interface examples shown in (3) are all for shooting scenes with a single registered face. For shooting scenes with multiple people, the embodiments of the present application are still applicable.

[0172] As an embodiment, a multi-person shooting scene refers to a situation where there may be multiple shooting subjects during shooting. For multiple shooting subjects, if the target recognition algorithm does not recognize a registered face among the multiple shooting subjects, that is, the multiple shooting subjects are all unregistered faces. The target recognition algorithm determines the information of face Z with the highest priority among the multiple unregistered faces (refer to the previous text for the determination method), and sends the information of face Z to the autofocus algorithm, and the autofocus algorithm focuses on face Z. The target recognition algorithm reports the information of the multiple unregistered faces identified and the "identification data" that identifies the registered faces to the camera application for drawing focus frames for the multiple unregistered face objects. The camera application uses a regular focus frame for each unregistered face. Referring to the above, the regular focus frame will automatically disappear after a period of time (for example, 5 seconds). The regular focus frame 901 is usually displayed in white in the preview interface. It can be understood that as an embodiment, when shooting multiple non-registered face objects, in the focus tracking function, the autofocus algorithm can only focus on the target with the highest priority, and the regular focus tracking frame will be displayed for all recognized subjects on the camera preview interface.

[0173] If the target recognition algorithm identifies multiple subjects as registered faces, i.e., multiple registered faces exist, the target recognition algorithm, as one embodiment, sorts the registered faces according to priority. The target recognition algorithm determines information about face Y, the highest priority face among the multiple non-registered faces (see above for the determination method), and transmits this information to the autofocus algorithm for focus tracking. The target recognition algorithm reports the information about the multiple registered faces it recognizes, along with "identification data" identifying the registered faces, to the camera application for drawing focus frames for the multiple subjects. The camera application draws a subject focus frame for the registered face with the highest priority, and draws non-subject focus frames for the registered faces excluding the highest priority face. Both the subject focus frame and the non-subject focus frames can be considered registered focus frames, i.e., focus frames for registered faces. The difference between the non-subject focus frames and the subject focus frame is their different colors. Compared to the non-subject focus frames, the subject focus frame is highlighted in the interface to emphasize the subject's image. For example, the lines of the main character's tracking frame are yellow, and the lines of the non-main character's tracking frame are white.

[0174] It is understandable that if the target recognition algorithm identifies multiple subjects as registered faces and non-registered faces among multiple subjects, as an embodiment, the target recognition algorithm focuses on the registered face with the highest priority, and the camera application draws a protagonist tracking focus frame for the registered face with the highest priority, draws non-protagonist tracking focus frames for the registered faces other than the registered face with the highest priority among the multiple registered faces, and draws a regular tracking focus frame for the non-registered faces. The regular tracking focus frame will automatically disappear after a period of time (for example, 5 seconds). In addition, the non-protagonist tracking focus frame has the same frame type as the protagonist tracking focus frame, that is, the non-protagonist tracking focus frame includes a main frame and a sub-frame. The regular tracking focus frame is a single frame. Therefore, from the perspective of frame type, the non-protagonist tracking focus frame is different from the regular tracking focus frame.

[0175] Figure 10 An example diagram of a multi-person shooting scene in photo mode is shown. Figure 10 As shown in (1), when face A, face B, face C, and face D appear in the photo preview interface on the tracking focus frame display, if the electronic device recognizes that face A is an unregistered face, and face B, face C, and face D are all registered faces, and face B is the face closest to the camera among multiple registered faces (such as face B, face C, and face D), then the conventional tracking focus frame 901 is used for face A, the tracking focus frame for face B is the protagonist tracking focus frame 902 proposed in this application, and the tracking focus frames for faces C and face D are the non-protagonist tracking focus frames 903 proposed in this application. For the relevant description of the conventional tracking focus frame 901, please refer to Figure 10The main character frame 902 and the non-main character tracking focus frame 903 are both composed of a main frame and a sub-frame. The difference is that the main character frame 902 is highlighted compared to the non-main character tracking focus frame 903 to highlight the main character portrait. After a period of time, the regular tracking focus frame 901 for face A automatically disappears. For example, after 5 seconds, the interface displays the following Figure 10 As shown in (2), the regular focus frame 901 for face A disappears, while the main focus frame 902 for face B and the non-main focus frames 903 for faces C and D do not disappear.

[0176] In the focus tracking function, an optional implementation is that the autofocus algorithm focuses on the registered face with the highest priority. If the face with the largest area among multiple facial feature data that meet the "registered face" conditions is the face with the highest priority, as the face or the lens moves, the order of the faces in the picture close to the lens changes, and the priority of the photographed subject will change. For example, Figure 10 As shown in (3), as the face or lens moves, face C becomes the largest face among multiple registered faces (such as face B, face C, and face D). At this time, the main focus frame is used for face C, and the focus frame for face B is switched to a non-main focus frame. When the focus frame is redrawn, the autofocus algorithm switches focus from face B to face C.

[0177] In other embodiments, the focus types including human bodies, cats, dogs, etc. may also belong to "unregistered faces". Optionally, when a human body appears in the photo preview interface, a focus frame may not be drawn for the human body. When a cat or a dog appears in the photo preview interface, a conventional focus frame may be drawn, and this embodiment does not limit this. It should be understood that different focus frames are used for display of "registered faces" and "unregistered faces". Those skilled in the art can adjust the color, shape, size, display mode, etc. of the focus frame, such as using a large frame for a "registered face" and a small frame for an "unregistered face". Or the focus frame for a "registered face" may be displayed in a flashing manner, and a still display for an "unregistered face".

[0178] It is understood that in other embodiments, the autofocus algorithm can also automatically or manually focus on one or more other objects, and this embodiment is not limited here. The following details the situation where the user manually intervenes in the focus tracking mode in "Main Character Focus Tracking".

[0179] 5. Manual Intervention After Automatic Focus Tracking

[0180] The above describes embodiments related to focus tracking when there is no human intervention during shooting. The following describes embodiments related to focus tracking when there is human intervention. Human intervention refers to the user actively clicking on an area or object in the interface to select the object to be focused. Interfaces include but are not limited to the photo preview interface, the video preview interface, or the video recording interface (i.e., the screen during video recording).

[0181] Before the interface is manually intervened, the electronic device automatically identifies the subject to be photographed and performs tracking focus. An optional implementation is, as described above, if the registered face is not recognized, a regular tracking focus frame is displayed for the subject to be photographed, and the object with the highest focus priority can be determined by referring to the method of faces with large face area > faces with small face area > human body > cat > dog. If a registered face is recognized, a registered tracking focus frame is displayed for the face in the facial feature data that meets the "registered face" conditions, and the face with the highest focus priority is tracked. However, when there is human intervention in the interface, for example, the user actively clicks on a face in the photo preview interface or the video preview interface, the electronic device will take the person or object selected by the user as the subject to be tracked, regardless of whether the facial feature data meets the "registered face" conditions.

[0182] Specifically, such as Figure 11 After displaying the tracking focus frame of the tracking target in S603, the following steps may also be performed:

[0183] S604: Receive the user's selection of a specific target area and identify the target area

[0184] In response to the user selecting a specific target area, the camera application provides the target area to the target recognition algorithm, which then identifies whether a target subject exists in the target area. Optionally, the camera application sends the target area to the camera HAL, which then sends the target area to the target recognition algorithm in the algorithm library.

[0185] S605: When there is an identifiable subject, report the identified subject information

[0186] When there is a recognizable subject, the target recognition algorithm will send the recognized subject information back to the camera HAL, and the camera HAL will report the tracking target to the camera application.

[0187] S606: Automatically tracking focus based on the identified subject information

[0188] The target recognition algorithm sends the identified subject information to the autofocus algorithm, which then adjusts the camera focus based on the identified subject information to track the target. Optionally, the autofocus algorithm can focus not only on the subject identified in the target area but also on subjects identified in non-target areas.

[0189] S607: Display focus frame

[0190] The camera determines the focus based on the recognized subject information, draws the focus frame, and displays the focus frame.

[0191] It is understandable that this application does not limit the order in which the electronic device executes steps S606-S607.

[0192] The focus frame display can be referred to as follows Figure 12 As shown in (1), in the preview interface of the single-person photo scene, the normal focus frame 901 is displayed for the non-registered face A and the focus is tracked. When the user clicks on the face A, the interface is displayed as follows Figure 12 (2). Figure 12 As shown in (2), in the photo preview interface, a highlight focus frame 904 and a small sun icon 905 indicating the exposure level are displayed for face A. The exposure level can be adjusted by operating the small sun icon 905. It is understood that the small sun icon is only an example and can actually be replaced with other icons indicating the exposure level.

[0193] Figure 13 The figure shows an example of a human intervention interface in a multi-person photo shooting scene when a registered face is recognized. Figure 13 As shown in (1), in the preview interface of a multi-person photo scene, a regular focus frame 901 is displayed for non-registered face A, a main focus frame 902 is displayed for registered face B and focus is performed, and a non-main focus frame 903 is displayed for registered face C. The user can actively click on face C to select face C as the target focus object.

[0194] For example, when the user clicks on face C, the autofocus algorithm focuses on face C and the camera interface displays the following: Figure 13 As shown in (2). Figure 13 As shown in (2), in the photo preview interface, a highlighted focus frame 904 and a small sun icon 905 are displayed for face C, and the regular focus frame 901 of face A and the protagonist focus frame 902 of face B both disappear.

[0195] The above describes the scenario where the "Main Character Focus Tracking" mode is enabled. A user manually selects a face for focus tracking. In one embodiment, the user can also click on a non-face area or an area where the algorithm cannot identify the object type. As shown in the figure, after S604, the following may also be included:

[0196] S608: When there is no identifiable subject in the target area, automatically focus on the target area and display the focus frame

[0197] When the target recognition algorithm does not recognize any subject information or the recognized subject information does not meet the conditions, the camera application will focus on the selected target area and display the focus frame. Figure 14 In (1), the normal focus frame 901 is displayed for the non-registered face A and the focus is tracked. When the user clicks on area 1, the automatic focus algorithm tracks the focus on area 1, and the camera interface displays as follows Figure 14 As shown in (2). In the photo preview interface, a highlighted focus frame 904 and a small sun icon 905 are displayed in area 1. It is understood that if there is a registered face, after automatically focusing on the registered face, if the user manually intervenes to select a certain area for focus, the autofocus algorithm will focus on the user-selected area and display the focus frame on the user-selected area instead of the registered face.

[0198] In one possible implementation, after the user selects a subject in a specific target area for tracking focus, the electronic device can use the subject as a "registered face" and set its tracking priority to the highest. This will not be described in conjunction with the accompanying drawings.

[0199] In one possible implementation, when the "registered face" is a pet or a building, the electronic device can also use the above-mentioned object tracking method to perform "protagonist tracking".

[0200] In one possible implementation, a "registered face" can be preset by the user in a gallery application on an electronic device. For example, a user can select face A as a "registered face" by selecting face A in the gallery. The face can be stored in the gallery application's storage space, and the gallery application's personal album can read the "registered face" image from the storage space to display all images of the "registered face" in the personal album. Furthermore, to distinguish between "registered face" images and non-"registered face" images, a distinguishing mark can be added to the "registered face" images.

[0201] In one possible implementation, the electronic device can use objects with the same label in the gallery application as "registered faces." For example, if a personal album in the gallery application contains images labeled "family" and "friends," all objects labeled "family" in the gallery application can be used as "registered faces."

[0202] In some other possible implementations, the "registered face" can be derived from facial data obtained through facial recognition by the device owner, such as facial data obtained during face unlocking. The "registered face" can also be derived from data stored on other electronic devices. This application does not limit the source of the "registered face" data.

[0203] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0204] The present application also provides a computer program product, which, when executed by a processor, implements the method described in any method embodiment of the present application.

[0205] The computer program product can be stored in a memory and finally converted into an executable target file that can be executed by a processor through preprocessing, compilation, assembly and linking.

[0206] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, implements the method described in any method embodiment of the present application. The computer program can be a high-level language program or an executable target program.

[0207] The computer-readable storage medium may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0208] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes and technical effects produced by the above-described devices and equipment can refer to the corresponding processes and technical effects in the aforementioned method embodiments, and will not be repeated here.

[0209] In the several embodiments provided in this application, the disclosed systems, devices and methods can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not executed. The device embodiments described above are merely schematic, and the division of units is only a logical function division. There may be other division methods in actual implementation, and multiple units or components may be combined or integrated into another system. In addition, the coupling between the units or the coupling between the components may be direct coupling or indirect coupling, and the above coupling includes electrical, mechanical or other forms of connection.

[0210] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0211] Additionally, the terms "system" and "network" are often used interchangeably. The term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the related objects.

[0212] The terms (or numbers) "first", "second", ... etc. that appear in the embodiments of the present application are only used for descriptive purposes, that is, they are only used to distinguish different objects, such as different "calibration data", etc., and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second", ... etc. may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, "at least one (item)" refers to one or more. "Multiple" means two or more. "At least one of the following (item)" or similar expressions refers to any combination of these items, including any combination of a single (item) or plural (items).

[0213] For example, the meaning of expressions similar to "the item includes at least one of the following: A, B, and C" in the embodiments of the present application, unless otherwise specified, generally means that the item can be any one of the following: A; B; C; A and B; A and C; B and C; A, B and C; A and A; A, A and A; A, A and B; A, A and C, A, B and B; A, C and C; B and B, B, B and B, B, B and C, C and C; C, C and C, and other combinations of A, B and C. The above examples use A, B, and C as an example to illustrate the optional items of the item. When the expression is "the item includes at least one of the following: A, B, ..., and X", that is, when the expression has more elements, the items that can be applied to the item can also be obtained according to the above rules.

[0214] In short, the above description is only a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included in the scope of protection of this application.

Claims

1. A shooting method, characterized in that: The method is applied to an electronic device, the electronic device including a gallery application, the gallery application storing image data collected by the electronic device, and the method comprising: In the charging screen-off state, determining N faces as N registered faces based on the image data, where N is an integer greater than 0; Display a first interface of a camera application, where the first interface includes a first control; In response to a first operation input by a user on the first control, obtaining N registered facial features corresponding one-to-one to the N registered faces; displaying a second interface of the camera application, the second interface including a first image frame captured by the camera, the first image frame including a first object, a second object, and a third object, the first object occupying a larger area in the first image frame than the second object in the first image frame; facial features of the first object matching one of the N registered facial features, facial features of the second object matching one of the N registered facial features, and facial features of the third object not matching the N registered facial features; Start displaying a first focus tracking frame on the first object, displaying a second focus tracking frame on the second object, displaying a third focus tracking frame on the third object, and start automatically focusing on the first object; After automatically focusing on the first object for a first period of time, canceling display of the third focusing frame on the third object and maintaining display of the second focusing frame on the second object; When a second operation input by the user to the second focusing frame is received, focusing on the second object begins.

2. The method according to claim 1, characterized in that Determining N faces as N registered faces according to the image data includes: Performing face clustering on the image data to obtain a plurality of clustered faces; sorting the clustered faces according to the number of photos; The N faces in the clustered multiple groups of faces whose number of photos is greater than a first threshold and are ranked in the top N are used as the N registered faces.

3. The method according to claim 2, characterized in that Before obtaining N registered facial features corresponding one-to-one to the N registered faces in response to a first operation input by a user on the first control, the method includes: The camera application performs a first query operation, where the first query operation is used to query the storage status of the clustered multiple faces; The camera application obtains a first tag, wherein the first tag is a tag generated by the gallery application after storing the clustered multiple faces.

4. The method according to claim 1, wherein Before obtaining N registered facial features corresponding one-to-one to the N registered faces in response to a first operation input by the user on the first control, the method includes: The gallery application performs a second query operation, where the second query operation is used to query the activation status of the first shooting mode; The first shooting mode is used to automatically focus on an object that matches the N registered facial features; When the gallery application obtains the second mark, determining that the first shooting mode is turned on; The second mark is a mark generated in response to a first operation input by the user on the first control.

5. The method according to claim 1, wherein Also includes: Before starting to display a first focus tracking frame on the first object, acquiring a first image, where the first image is the same as the first image frame; sending the first image to an object recognition algorithm; The target recognition algorithm identifies a first facial feature of the first object; Determine whether the first facial feature matches the registered facial feature.

6. The method according to claim 1, characterized in that The starting of automatic focusing on the first object includes: An autofocus algorithm sets the first object as the focus of the first image frame and performs focusing; Acquire a second image frame, where the second image frame is a next image frame of the first image frame; The auto-focus algorithm sets the first object as the focus of the second image frame and performs focusing.

7. The method according to claim 5, characterized in that The starting to display a first focus frame on the first object includes: Obtaining the first facial feature; Drawing a first focus tracking frame on the first object according to the first facial feature; The first object and the first focus frame are displayed.

8. The method according to claim 1, characterized in that include: The first tracking focus frame is different from the second tracking focus frame, the second tracking focus frame is different from the third tracking focus frame, and the third tracking focus frame is different from the first tracking focus frame.

9. The method according to any one of claims 1 to 8, characterized in that include: Start displaying a first focusing frame on the first object, displaying a second focusing frame on the second object, and displaying a third focusing frame on the third object. After starting to automatically focus on the first object, when the area occupied by the second object in the second interface is larger than the area occupied by the first object in the second interface, start displaying the first focusing frame on the second object, displaying the second focusing frame on the first object, and start automatically focusing on the second object.

10. The method according to any one of claims 1 to 8, characterized in that include: When the second interface does not include the first object; Stop automatically tracking focus on the first object.

11. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the processor and the memory are coupled, and the memory is used to store a computer program. When the computer program is executed by the processor, the electronic device executes the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 10.

13. A chip, characterized in that: The method comprises a processor, wherein when the processor executes instructions, the processor performs the method according to any one of claims 1 to 10.

Images