Focus tracking method, electronic device and storage medium

Abstract

The present application provides a focus tracking method, electronic device and storage medium. The method includes: matching the multi-level information of the candidate object in the current preview interface with the multi-level information corresponding to N historical focus tracking objects, and determining whether there is a historical focus tracking object matching the candidate object among the N historical focus tracking objects; if so, taking the candidate object as the current focus tracking object, and focusing on the focus tracking part of the current focus tracking object. The focus tracking method can autonomously identify the focus tracking object in the current preview interface, and automatically maintain the focus on the focus tracking part of the focus tracking object, and can lock the focus on the focus tracking part of the focus tracking object, effectively improving the focus tracking effect.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a focus tracking method, an electronic device and a storage medium. Background Art

[0002] Currently, users can capture a variety of images through electronic devices. If the subject (such as human body, animal, etc.) is in motion, the captured image may be out of focus, or the focus may shift from the subject to other objects. To avoid the above situation, users can use electronic devices to track the focus of the subject. However, existing tracking solutions cannot automatically maintain the focus. Summary of the invention

[0003] The present application provides a focusing method, an electronic device and a storage medium, which can autonomously identify a focusing object and automatically maintain focusing on a focusing part of the focusing object, and can lock the focus on the focusing part of the focusing object, thereby effectively improving the focusing effect.

[0004] In the first aspect, the embodiment of the present application provides a focusing method, which includes: matching the multi-level information of the candidate object in the current preview interface with the multi-level information corresponding to N historical focusing objects, and determining whether there is a historical focusing object matching the candidate object among the N historical focusing objects; the candidate object includes at least one first part, and a first part includes at least one second part, and the multi-level information of the candidate object includes at least two of the object information of the candidate object, the information of each first part, or the information of each second part; N is a positive integer; if it exists, the candidate object is used as the current focusing object, and the focusing part of the current focusing object is focused. Among them, the N historical focusing objects are the stored and recorded focusing objects.

[0005] In the method described in this aspect, a candidate object that matches the historical focus object can be autonomously identified in the current preview interface, and used as the focus object, and the focus portion of the focus object can be automatically kept in focus. For example, when the focus object leaves the preview interface and re-enters the preview interface, the focus object can be autonomously identified, and the focus portion of the focus object can be automatically kept in focus. For another example, when switching the shooting mode, the focus object can be autonomously identified, and the focus portion of the focus object can be automatically kept in focus. Moreover, the method focuses on the focus portion of the focus object, and locking the focus on a specific focus portion can effectively improve the focus effect.

[0006] In addition, the multi-level information of the method includes at least two items of object information, first part information, or second part information. The multi-level information has richer information types, which is helpful to improve the matching accuracy when matching is performed based on the multi-level information.

[0007] It is understandable that the focus object refers to the object that is locked in focus. The focus objects involved in the present application include but are not limited to all human bodies that can be photographed by the electronic device, but all photographed objects. For example, the focus object can also be animals, plants, etc. in the shooting environment. It should be noted that the focus objects of the embodiments of the present application may include the current focus object and the historical focus object. The current focus object refers to the object that is being locked in focus in the current preview interface, and the historical focus object refers to the object that has been locked in focus in the historical preview interface. The multi-level information of the historical focus object can be recorded and stored, that is, the historical focus object is the recorded and stored focus object. The focus moment of the current focus object is later than the focus moment of the historical focus object, that is, the focus moment of the historical focus object is earlier than the current moment. It is understandable that the historical focus object can maintain focus within a focus time period, and each focus moment within the focus time period of the historical focus object is earlier than the current moment.

[0008] Optionally, a photographing object (including a focus tracking object or a candidate object) of the present application may include at least one first part. For example, taking the photographing object as a human body, the first part of the human body may be a face, a torso, and limbs.

[0009] Optionally, a first part of the present application may include at least one second part. For example, taking the first part being a face as an example, the second parts of the human body may be eyes, nose, lips, ears, etc.

[0010] In combination with the first aspect, in an optional implementation, the focus portion is a portion with the highest priority of the current focus object in the current preview interface.

[0011] For example, if the current focus object is a person who is playing football, and the priority of the face of the person who is playing football is higher than that of the torso, and the priority of the torso is higher than that of the limbs, if the current preview interface includes the face, torso and limbs, the face can be used as the focus part of the person who is playing football; if the current preview interface includes the torso and limbs, the torso can be used as the focus part of the person who is playing football.

[0012] For example, if the current focus object is a person who is playing football, and the priority of the second part of the person who is playing football is higher than the priority of the trunk, and the priority of the trunk is higher than the limbs. If the current preview interface includes eyes, trunk and limbs, the eyes can be used as the focus part of the person who is playing football.

[0013] In combination with the first aspect, in an optional embodiment, the priority of the part of the current focus object is determined based on at least one of the following: the priority set by the system by default; the position information of the part in the current preview interface; the relative position information between the part and other parts in the current preview interface.

[0014] For example, it is simpler and more convenient to use the priority of the system default setting as the priority of the part. For another example, based on the position information of the part in the current preview interface, the priority of the system default setting is updated to obtain the updated priority, and the updated priority is used as the priority of the part. The priority can be adaptively adjusted based on the position information of the part, which is more flexible. For another example, based on the relative position information between the part and other parts in the current preview interface, the priority of the system default setting is updated to obtain the updated priority, and the updated priority is used as the priority of the part. The priority can be adaptively adjusted based on the relative position information between the parts, and the priority of each part can reflect the distribution of each part, which is more accurate.

[0015] In combination with the first aspect, in an optional implementation, the focus portion is the first portion of the current focus object, and after focusing on the focus portion of the current focus object, the method further includes: using the second portion included in the first portion of the current focus object as the updated focus portion. That is, when the current preview interface does not include the first portion, but includes the second portion included in the first portion, the second portion with the highest priority in the current preview interface can be used as the updated focus portion, and the second portion with the highest priority belongs to the original focus portion.

[0016] In combination with the first aspect, in an optional implementation, in a scenario where a candidate object enters the current preview interface, a step is performed of matching the multi-level information of the candidate object in the current preview interface with the multi-level information corresponding to N historical focus objects.

[0017] In combination with the first aspect, in an optional implementation, the method further includes: the candidate object leaves the current preview interface; and the time between the moment when the candidate object leaves the current preview interface and the moment when the candidate object enters the current preview interface is less than a preset time. It can be seen that when the focus object leaves the preview interface and re-enters the preview interface, the focus object can be autonomously identified and the focus portion of the focus object can be automatically maintained in focus.

[0018] In combination with the first aspect, in an optional implementation, the camera application is controlled to switch from the first shooting mode to the second shooting mode; when the camera application is in the second shooting mode, the step of matching the multi-level information of the candidate object in the current preview interface with the multi-level information corresponding to the N historical focus objects is performed. When switching the shooting mode, the focus object in the current preview interface can be autonomously identified, and the focus portion of the focus object can be automatically kept in focus.

[0019] In combination with the first aspect, in an optional implementation, before matching the multi-level information of the candidate object in the current preview interface with the multi-level information corresponding to the N historical focus objects, the method further includes:

[0020] A multi-level detection algorithm is called to detect the preview image in the current preview interface to obtain multi-level information of the candidate object in the current preview interface; the multi-level detection algorithm includes at least two of the object detection algorithm, the first part detection algorithm, or the second part detection algorithm.

[0021] In combination with the first aspect, in an optional implementation, after the candidate object is used as the current focus object, the method further includes: storing multi-level information of the current focus object through a tree structure;

[0022] The tree structure includes a root node, a first child node, and a second child node; the root node is associated with at least one first child node, and a first child node is associated with at least one second child node; the root node is used to store the object information of the current focus object, the first child node is used to store the information of each first part of the current focus object, and the second child node is used to store the information of each second part of the current focus object. Through the tree structure, the object information of the current focus object and the corresponding first part information and / or second part information can be effectively distinguished. At the same time, when matching information, there is no need to traverse the entire multi-level information, which is conducive to quickly locating the target information.

[0023] In combination with the first aspect, in an optional embodiment, the storage order of the information of the first parts of the current focus object corresponding to the first sub-node is related to the priority of the first parts of the current focus object; the storage order of the information of the second parts of the current focus object corresponding to the second sub-node is related to the priority of the second parts of the current focus object, and / or the first part to which the second part belongs.

[0024] In combination with the first aspect, in an optional implementation, there is a historical focus object that matches the candidate object among the N historical focus objects when at least two of the following conditions are met: the object information of the candidate object matches the object information of a historical focus object; the information of the first part of the candidate object matches the information of the first part of the historical focus object; the information of the second part of the candidate object matches the information of the second part of the historical focus object. It can be seen that in this implementation, when the candidate object matches at least two of the object information, the information of the first part, or the information of the second part of the historical focus object, it is determined that the candidate object matches the historical focus object. Compared with the scheme of only comparing object information, this implementation matches more information, thereby making the matching result more accurate.

[0025] In a second aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors, one or more memories; the one or more memories are coupled to the one or more processors, the one or more memories are used to store computer program code, the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method described in the first aspect.

[0026] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, including computer instructions, which, when executed on an electronic device, causes the electronic device to execute the method described in the first aspect.

[0027] In a fourth aspect, an embodiment of the present application provides a chip or a chip system, including a processing circuit and an interface circuit, the interface circuit is used to receive code instructions and transmit them to the processing circuit, and the processing circuit is used to run the code instructions to execute the method described in the first aspect.

[0028] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and when the computer program is run on a computer, it enables the computer to execute the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments of the present application will be described below.

[0030] Figure 1 This is a schematic diagram of a scenario provided by an embodiment of the present application;

[0031] Figure 2A-2Q is a schematic diagram of a user interface provided in an embodiment of the present application;

[0032] Figure 3 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0033] Figure 4 It is a schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0034] Figure 5 It is a flowchart of a focus tracking method provided in an embodiment of the present application;

[0035] Figure 6 It is a structural schematic diagram of a multi-level detection algorithm provided in an embodiment of the present application;

[0036] Figure 7 It is a structural diagram of multi-level information stored in a tree structure provided by an embodiment of the present application;

[0037] Figure 8It is a flow chart of another focus tracking method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0039] It should be understood that the terms "first", "second", etc. in the specification, claims and drawings of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0040] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0041] The present application provides a focusing method. In the focusing method, the electronic device can match the multi-level information of the candidate object in the current preview interface with the multi-level information corresponding to N historical focusing objects, respectively, to determine whether there is a historical focusing object that matches the candidate object among the N historical focusing objects; if so, the candidate object is used as the current focusing object, and the focusing part of the current focusing object is focused. The focusing method can autonomously identify the focusing object in the current preview interface, and automatically maintain the focusing on the focusing part of the focusing object. For example, when the focusing object leaves the preview interface and re-enters the preview interface, it can autonomously identify the focusing object and automatically maintain the focusing on the focusing part of the focusing object. For another example, when switching the shooting mode, it can autonomously identify the focusing object and automatically maintain the focusing on the focusing part of the focusing object. The present application solution can realize real-time focusing when the focusing object moves and the preview interface changes greatly, so as to achieve a better focusing effect.

[0042] See also Figure 1 , Figure 1 A schematic diagram of a scenario provided for an embodiment of the present application.

[0043] like Figure 1 As shown, there are multiple people in motion in the shooting environment (for example, people who are playing football and people who are walking). The user can use an electronic device to shoot. After the user determines the object to be photographed in the shooting environment, if the object to be photographed is in motion, the captured image may be out of focus. Alternatively, the object to be photographed is easily disturbed by other objects in the shooting environment, resulting in a focus shift problem, that is, the focus is transferred from the object to other objects. In order to avoid the above situation, the user can turn on the focus tracking function and focus on the focus tracking area of ​​the focus tracking object. Even if the focus tracking object is in motion, there will be no out of focus problem, and the focus tracking object is not easily disturbed by other objects, thereby avoiding the focus shift problem.

[0044] It can be understood that the focus tracking object refers to the object that is locked in focus. For example, the photographed object can be Figure 1 The person who is playing football or walking in the video. The focus tracking objects involved in this application include but are not limited to all human bodies that can be photographed by the electronic device, but all shooting objects. For example, in other scenes, the focus tracking objects can also be animals, plants, etc. in the shooting environment. It should be noted that the focus tracking objects in the embodiments of the present application may include the current focus tracking object and the historical focus tracking object. The current focus tracking object refers to the object that is being focused on in the current preview interface, and the historical focus tracking object refers to the object that has been focused on in the historical preview interface. The multi-level information of the historical focus tracking object can be recorded and stored, that is, the historical focus tracking object is the recorded and stored focus tracking object. The focus tracking moment of the current focus tracking object is later than the focus tracking moment of the historical focus tracking object, that is, the focus tracking moment of the historical focus tracking object is earlier than the current moment. It is understandable that the historical focus tracking object can maintain focus within a focus tracking time period, and each focus tracking moment within the focus tracking time period of the historical focus tracking object is earlier than the current moment.

[0045] Optionally, a photographing object (including a focus tracking object or a candidate object) in an embodiment of the present application may include at least one first part. For example, taking the photographing object as a human body, the first part of the human body may include a face, a torso, and limbs.

[0046] Optionally, a first part of an embodiment of the present application may include at least one second part. For example, taking the first part being a face as an example, the second parts of the human body may be eyes, nose, lips, ears, etc.

[0047] It should be noted that, similarly, in other embodiments, a second part may also include at least one third-level part, a third-level part may also include at least one fourth-level part, and so on, and the present application does not impose any limitation on this.

[0048] Optionally, the focus portion of the focus object refers to a specific locked portion of the focus object. Figure 1 For example, if the person who is playing football in the video is playing football, the focus part can be the face of the person who is playing football. Figure 1 Taking the person who is playing football in the video as an example, the focus-tracking part may also be the eyes of the person who is playing football. That is, the focus-tracking part of the focus-tracking object may include at least one of the first part of the focus-tracking object or the second part of the focus-tracking object.

[0049] The following is a detailed introduction to the shooting scenes provided by this application in conjunction with a set of user interface schematic diagrams.

[0050] It is understood that the terms "interface" and "user interface" in the specification, claims and drawings of this application are media interfaces for interaction and information exchange between applications or operating systems and users, which realize the conversion between the internal form of information and the form acceptable to users. The commonly used form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an interface element such as an icon, window, control, etc. displayed on the display screen of an electronic device, where the control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visual interface elements.

[0051] 1. Enable the focus tracking function ( Figure 2A to Figure 2E )

[0052] like Figure 2AAs shown, the user interface 100 displays a page with application icons, which may include multiple application icons (for example, a clock application icon, a calendar application icon, a weather application icon, a memo application icon, a calculator application icon, a recorder application icon, a music application icon, and a settings application icon, etc.). A page indicator may also be displayed below the above-mentioned multiple application icons to indicate the positional relationship between the currently displayed page and other pages. There are multiple application icons (for example, a dial application icon, a contact application icon, a message application icon, and a camera application icon 101) below the page indicator, and the application icon remains displayed when the page is switched. It should be noted that the camera application icon 101 is an icon of a camera application. The camera application icon 101 can be used to trigger the launch of a camera application. The camera application is an image capture application on electronic devices such as smartphones and tablets. This application does not limit the name of the application.

[0053] The electronic device may detect a user operation on the camera application icon 101, and in response to the operation, the electronic device may display a Figure 2B The user interface 200-1 or Figure 2C User interface 200-2 is shown. User interface 200-1 or user interface 200-2 may be a preview interface of a camera application, and a user may preview an image on the preview interface and complete taking a photo. That is, a user may open user interface 200-1 or user interface 200-2 of a camera application by clicking on camera application icon 101.

[0054] It is understandable that the user operations mentioned in this application may include but are not limited to touch (for example, click, slide, etc.), voice control, gesture and other operations, and this application does not impose any restrictions on this.

[0055] like Figure 2BAs shown, the user interface 200-1 may include a preview area 201, a shooting mode list 202, an album shortcut control 203, a shutter control 204, a camera flip control 205, and a status list 206. Among them, the preview area 201 can be used to display a preview image, which is an image collected by the electronic device in real time through a camera. One or more shooting mode options may be displayed in the shooting mode list 202. The one or more shooting mode options may include: a portrait mode option 2021, a video mode option, a photo mode option 2022, and more options. It should be noted that the portrait mode option 2021 is a control for the portrait mode. The portrait mode option 2021 can be used to trigger the portrait mode of the camera application. The photo mode option 2022 is a control for the photo mode. The photo mode option 2022 can be used to trigger the photo mode of the camera application. The portrait mode and the photo mode are different shooting modes in the camera application, and at least one of the shooting parameters, camera functions, or image optimization effects corresponding to the two shooting modes may be different. It can be understood that the shooting mode list 202 may also include more or fewer shooting mode options. The album shortcut control 203 can be used to open the album application. After the user triggers the electronic device to open the album application through the album shortcut control 203, the captured images and videos can be viewed. The album application is an image management application on the electronic device, which can also be called "album". The name of this application is not limited in this embodiment. In addition, the electronic device can also display thumbnails of saved images or videos in the album shortcut control 203. The shutter control 204 can be used to monitor the user operation of triggering the photo. The camera flip control 205 can be used to monitor the user operation of triggering the flip camera. One or more status options can be displayed in the status list 206. These one or more status options may include: focus tracking option 2061, AI scene enhancement option, flash option and setting option 2062. It is understandable that the status list 206 may also include more or fewer options. The focus tracking option 2061 is used to turn on or off the focus tracking function of the camera application. The setting option 2062 is used to manage more status options, including but not limited to watermark options, reference line options, or shutter sound options, etc. ( Figure 2B not shown).

[0056] Optionally, in other embodiments, the state list 206 may not include a focus tracking option, and the focus tracking option is managed by the setting option 2062. Figure 2CFIG. 2 shows a schematic diagram of a user interface 200-2. The controls included in the user interface 200-2 are substantially the same as those included in the user interface 200-1, except that the state list 206 of the user interface 200-2 does not include the focus tracking option 2061. In this case, the user can trigger the electronic device to display the user interface 300 by setting the option 2062. The user interface 300 may include the focus tracking option 301. Figure 2D The user interface 300 is a setting interface of the camera application, and the focus tracking option 301 is used to turn on or off the focus tracking function of the camera application. Figure 2B The user interface 200 - 1 shown is taken as an example for illustrative description.

[0057] The electronic device may detect a user operation on the focus tracking option 2061. In response to the user operation, the electronic device may display a Figure 2E User interface 400 is shown. User interface 400 is a preview interface of the camera application when the focus tracking function is turned on. The controls included in user interface 400 are substantially the same as those included in user interface 200-1, except that user interface 200-1 is a preview interface of the camera application when the focus tracking function is turned off, and user interface 400 is a preview interface of the camera application when the focus tracking function is turned on.

[0058] like Figure 2E As shown, the preview area 201 of the user interface 400 shows two people in motion. The preview area 201 may include an object frame 401 and an object frame 402. The object frame 401 is used to frame the person who is playing football, and the object frame 402 is used to frame the person who is walking.

[0059] It can be understood that the above embodiment shows a specific implementation method of enabling the focus tracking function based on user operation. In other embodiments, the focus tracking function of the camera application can also be enabled by default, that is, the user can open the user interface 400 of the camera application by clicking the camera application icon 101, without limitation.

[0060] 2. Select the focus object during preview and lock the focus on the focus part of the focus object ( Figure 2F to Figure 2G )

[0061] For example, after the electronic device detects the user operation on the object frame 401, it can be determined that the focus object is a person who is playing football, and the electronic device can display Figure 2F The user interface 500-1 shown or Figure 2G User interface 500-2 is shown. User interface 500-1 and user interface 500-2 are preview interfaces when the focus tracking function is turned on and the focus tracking is successful.

[0062] like Figure 2F As shown, the user interface 500-1 may include a focus frame 501, and the focus frame 501 is used to frame and select a focus portion of the focus object, and the focus portion is a first portion of a human face. Figure 2G As shown, the user interface 500-2 may include a focus frame 502, and the focus frame 502 is used to frame the focus part of the focus object, and the focus part is the second part of the eye. Figure 2F The user interface 500-1 is described.

[0063] It is understandable that the embodiments of the present application do not limit the shape, color, and size of the object frame and the tracking focus frame, as long as the user can distinguish between the object frame and the tracking focus frame. Figure 2E The combination of the virtual frame and the symbol "+" shown, the focus tracking frame can be Figure 2F and Figure 2G As shown in the virtual frame, in other embodiments, the object frame may also be a combination of a virtual frame and a circle, and the focus tracking frame may be a circle.

[0064] 3. The focus area leaves the preview interface ( Figure 2H to Figure 2I )

[0065] In some embodiments of the present application, when the focus tracking function is turned on in the preview state, due to the movement of the focus tracking object or the adjustment of the framing range of the electronic device, the electronic device may detect that the focus tracking part of the focus tracking object has left the preview interface, and the electronic device may update the focus tracking part of the focus tracking object based on the priority of the part. It can be understood that the framing range of the electronic device is related to the field of view angle and the focal length.

[0066] In an optional implementation, when the electronic device detects that the original focus part of the focus object has partially left the preview interface, the electronic device may use the first part included in the original focus part as the updated focus part. Taking the original focus part as the first part as an example, when the electronic device detects that the preview interface does not include the first part, but includes the second part included in the first part, the electronic device may use the second part with the highest priority in the preview interface as the updated focus part, and the second part with the highest priority belongs to the original focus part.

[0067] like Figure 2HAs shown, the electronic device can display a user interface 600, and the controls included in the user interface 600 are substantially the same as those included in the user interface 500-1. The difference is that the user interface 500-1 includes the first part of the face, and the user interface 600 does not include the first part of the face, but includes the second part of the eyes, mouth and ears. The electronic device can use the second part of the eyes with the highest priority in the preview interface as the updated focus part, that is, the user interface 600 can include a focus frame 601, and the focus frame 601 is used to frame the second part of the eyes of the person who is playing football. In this embodiment, when the focus part partially leaves the preview interface, the first part included in the original focus part can be automatically updated to the focus part based on the priority, and the updated focus part can be kept in focus, so that a better focus effect can be achieved when the focus object moves.

[0068] It is understandable that, as the focus position changes, the focus frame 601 may have corresponding changes compared to the focus frame 501. For example, Figure 2H The focus frame 601 shown is Figure 2F For example, in other embodiments, at least one of the shape, color, or size of tracking focus frame 601 and tracking focus frame 501 may be different, such as the size of tracking focus frame 601 is smaller than the size of tracking focus frame 501.

[0069] In another optional implementation, when the electronic device detects that all the original focus parts of the focus object have left the preview interface, the electronic device may use the part of the focus object with the highest priority in the preview interface as the updated focus part of the focus object based on the priority of each part. Fig.2I As shown, the electronic device may display a user interface 700, which may include a focus frame 701, which is used to frame the torso of the person who is kicking the ball. In this implementation, when the focus part leaves the preview interface, the focus part may be automatically updated based on the priority, and the updated focus part may be kept in focus, so that a better focus effect can be achieved when the focus object moves.

[0070] Similarly, it can be understood that as the focus position changes, the focus frame 701 may have corresponding changes compared to the focus frame 501, which will not be elaborated.

[0071] 4. The focus object leaves the preview interface and re-enters the preview interface ( Figure 2J to Figure 2K )

[0072] In some embodiments of the present application, in a preview state with the focus tracking function turned on, the electronic device may detect that the focus tracking object has left the preview interface due to movement of the focus tracking object or adjustment of the framing range of the electronic device.

[0073] like Figure 2JAs shown, the electronic device may display a user interface 800. In the image displayed in the preview area 201 of the user interface 800, the person who is playing football has completely exceeded the framing range of the electronic device, and the electronic device detects that the focus object has left the preview interface.

[0074] In some embodiments of the present application, after the electronic device detects that the focus object has left the preview interface, it may be detected that the focus object re-enters the preview interface due to movement of the focus object or adjustment of the framing range of the electronic device.

[0075] like Figure 2K As shown, the electronic device can display a user interface 900. In the image displayed in the preview area 201 of the user interface 900, the person who is playing football re-enters the framing range of the electronic device, and the electronic device detects that the focus object has entered the preview interface. At this time, the preview area 201 may include a focus frame 901, which is used to frame the second part of the eye of the person who is playing football. In this implementation, when the focus object leaves the preview interface and re-enters the preview interface, the electronic device can automatically identify the focus object in the preview interface. Compared with the method of manually re-determining the focus object after the focus object leaves the preview interface, this solution can automatically maintain the focus on the focus part of the focus object when the object moves, thereby achieving a better focus effect.

[0076] Optionally, when the electronic device detects that the time length after the focus object leaves the preview interface is greater than or equal to the preset time length, the electronic device may automatically exit the focus lock on the original focus object. That is to say, when the electronic device detects that the time length after the focus object leaves the preview interface is less than the preset time length, the electronic device may continue to maintain the focus lock on the original focus object, and when the electronic device detects that the time length after the focus object leaves the preview interface is greater than or equal to the preset time length, the electronic device may automatically exit the focus lock on the original focus object. The value of the preset time length may be 2 seconds (s), 3s, 6s, etc. The preset time length may be set based on actual needs, and this application does not limit this. In this implementation, when the time length after the focus object leaves the preview interface is greater than or equal to the preset time length, the focus lock on the focus object may be exited in a timely manner, which is conducive to timely deletion of historical data related to the focus object (such as multi-level information) and saving storage space.

[0077] like Figure 2L As shown, the electronic device may display user interface 1000, which may include object frame 1001 and object frame 1002. Object frame 1001 is used to frame a person who is playing football, and object frame 1002 is used to frame a person who is walking, so that the user can select the focus object again.

[0078] It can be understood that the duration of the focus object leaving the preview interface refers to the duration between the moment the focus object leaves the preview interface and the moment the focus object re-enters the preview interface. In other words, the focus object can be continuously focused in the preview interface. When the focus object leaves the preview interface, the focus object is stopped, and when the focus object re-enters the preview interface, the focus object is continued.

[0079] 5. Image capture and saving when the focus tracking function is turned on ( Figure 2M to Figure 2N )

[0080] Electronic devices can detect the effects of Figure 2F The user operation on the shutter control 204 included in the user interface 500-1 shown in FIG. In response to the user operation, the electronic device can capture an image and display the image as shown in FIG. Figure 2M The user interface 1100 shown. The controls included in the user interface 1100 are substantially the same as those included in the user interface 500-1, except that the album quick control 203 included in the user interface 1100 can display thumbnails of images captured in the preview area 201 stored in the electronic device.

[0081] In some embodiments, the electronic device may detect the photo album shortcut control 203 included in the user interface 1100. In response to the user operation, the electronic device may display the following Figure 2N The user interface 1200 shown in FIG. 1 may include a display area 1201. The display area 1201 may be used to display a captured image.

[0082] It is understandable that when the focus area of ​​the focus object is different, the captured image may be different. For example, if the face of a person who is playing football is used as the focus area, the face area shown in the captured image is clearer, and other areas (such as the torso area) are blurred; for another example, if the torso area of ​​a person who is playing football is used as the focus area, the torso area shown in the captured image is clearer, and other areas (such as the face area) are blurred.

[0083] 6. Switch shooting mode ( Fig.2O )

[0084] In some embodiments of the present application, in the preview state where the focus tracking function is turned on, the electronic device can detect the effect on Figure 2F In response to the user operation, the camera application of the electronic device can switch from the photo mode to the portrait mode and display the following image: Fig.2OUser interface 1300 is shown. The controls included in user interface 1300 are substantially the same as those included in user interface 500-1, except that user interface 500-1 is a user interface when the camera application is in a photo mode, and user interface 1300 is a user interface when the camera application is in a portrait mode.

[0085] It is understandable that this implementation is explained by taking the switching of the camera mode to the portrait mode as an example, and the present application does not limit the first shooting mode before the switching and the second shooting mode after the switching.

[0086] In this implementation, when switching the shooting mode, the electronic device can automatically identify the focus object in the preview interface. Compared with the method of manually re-determining the focus object after switching the shooting mode, this solution can automatically maintain the focus on the focus part of the focus object, thereby achieving a better focus effect.

[0087] It is understandable that the electronic device can also perform focus tracking preview in the horizontal screen state. Figure 2P As shown, the electronic device can display user interface 1400. The controls included in user interface 1400 are substantially the same as the controls included in user interface 400. The electronic device can detect a user operation on object frame 403 included in user interface 1400. In response to the user operation, the electronic device can display Figure 2Q The user interface 1500 is shown. The user interface 1500 may include a focus frame 1501, which is used to frame the focus portion of the focus object. That is, after the electronic device detects the user operation on the object frame 403, it can determine that the focus object is a person who is playing football, and lock the focus on the face of the person who is playing football.

[0088] It should be noted that when an electronic device is in landscape mode and performing focus preview, the focus area may leave the preview interface, the focus object may leave the preview interface, the focus object may re-enter the preview interface, an image may be captured, or the shooting mode may be switched. For details, please refer to the interface diagram in portrait mode (such as FIG. 2A to FIG. 2O ), no more details.

[0089] It should be noted that the above user interfaces are only some examples provided by this application and should not be regarded as limitations of this application. In other words, the above user interfaces can display more or less content, and this application does not limit this.

[0090] It is understandable that the electronic device can specifically be a mobile phone, a tablet computer, a desktop computer, a wearable device (such as a laptop computer, etc.), a handheld computer, a notebook computer, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA) or a dedicated camera (such as a SLR camera, a card camera) and other electronic devices. The embodiments of the present application do not impose any restrictions on the specific type of the electronic device.

[0091] The electronic equipment will be described in detail below.

[0092] See also Figure 3 , shows a schematic diagram of the hardware structure of an electronic device. The electronic device may include: a processor, an external memory interface, an internal memory, a Universal Serial Bus (USB) interface, a charging management module, a power management module, a battery, an antenna 1, an antenna 2, a mobile communication module, a wireless communication module, a sensor module, a button, a motor, an indicator, a camera, a display screen, and a SIM card slot, etc. The audio module may include a speaker, a receiver, a microphone, an earphone interface, etc. The sensor module may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0093] It is to be understood that the structures illustrated in the embodiments of the present application do not constitute specific limitations on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0094] The processor may include one or more processing units, for example, the processor may include an application processor (AP), a modem processor (Modem), a central processing unit (CPU), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors. The processor is the nerve center and command center of electronic devices.

[0095] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module, wireless communication module, and Modem, etc. In some embodiments, the antenna 1 of the electronic device is coupled with the mobile communication module, and the antenna 2 is coupled with the wireless communication module, so that the electronic device can communicate with the network side device and other electronic devices through wireless communication technology.

[0096] Among them, the touch sensor is also called a "touch control device". The touch sensor can be set on the display screen, and the touch sensor and the display screen form a touch screen, also called a "touch control screen". The touch sensor is used to detect touch operations acting on or near it. The touch sensor can transmit the detected touch operation to the AP to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen. In other embodiments, the touch sensor can also be set on the surface of the electronic device, which is different from the position of the display screen. In an embodiment of the present application, the electronic device can detect whether there is a user operation acting on the display screen of the electronic device through the touch sensor. After the touch sensor detects the user operation acting on the camera application, the electronic device can take an image.

[0097] In addition, operating systems are running on the above components, such as the iOS operating system developed by Apple, the Android open source operating system developed by Google, and the Windows operating system developed by Microsoft.

[0098] The operating system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device. It should be noted that although the embodiment of the present application is illustrated by taking the Android system as an example, its basic principle is also applicable to electronic devices based on operating systems such as iOS or Windows.

[0099] See also Figure 4 , which is a software structure block diagram of an electronic device provided in an embodiment of the present application. The software structure adopts a layered architecture, which divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. Taking the Android system as an example, in some embodiments, the Android system is divided from top to bottom into an application layer, an application framework layer (Framework), a hardware abstraction layer (HAL), a kernel layer (Kernel) and a hardware layer.

[0100] The application layer may include a series of application packages, including camera, gallery, calendar, call, map, WLAN, Bluetooth, music, video, short message and other APPs.

[0101] Among them, the application framework layer provides an application programming interface (Application Programming Interface, API) and a programming framework for the application of the application layer. The application framework layer includes some predefined functions. For example, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, etc. The embodiment of the present application does not impose any restrictions on this. Exemplarily, the above-mentioned window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The above-mentioned content provider is used to store and obtain data, and make these data accessible to the application. The data may include video, images, audio, dialed and received calls, browsing history and bookmarks, phone books, etc. The above-mentioned view system can be used to build the display interface of the application. Each display interface can be composed of one or more controls. Generally speaking, controls can include interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. The above-mentioned resource manager provides various resources for the application, such as localized strings, icons, images, layout files, video files, etc. The above notification manager uses an application to display notification information in the status bar, which can be used to convey notification-type messages and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the completion of downloads, message reminders, etc. The notification manager can also be a notification that appears in the top status bar of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is prompted in the status bar, a prompt sound is emitted, a vibration is made, an indicator light flashes, etc. In some embodiments, the application framework layer may also include a camera service. The camera service can interact with the hardware abstraction layer.

[0102] Among them, the hardware abstraction layer is an interface layer located between the kernel layer and the hardware, which can be used to abstract the hardware. In some embodiments, the hardware abstraction layer includes a hardware abstraction layer interface definition language (HIDL) interface. Among them, the hardware abstraction layer may include: camera HAL, display HAL, sensor HAL, etc. Optionally, in some embodiments, the camera HAL may include but is not limited to an image processing module, a multi-level detection module, a storage module, and an automatic focus (Automatic Focus, AF) module. Among them, the multi-level detection module can be used to call a multi-level detection algorithm to detect the image to obtain multi-level information of the object (including candidate objects and focus objects); the storage module can be used to store multi-level information of the focus object; the AF module is used to focus on the focus part of the focus object; the image processing module is used to determine the focus object and the focus part of the focus object.

[0103] The kernel layer is the foundation of the Android operating system, responsible for hardware drivers, network, power, system security, memory management and other functions. The kernel layer is an intermediate layer between hardware and software, and its function is to pass application requests to the hardware. The kernel layer contains at least camera drivers, sensor drivers, display drivers and audio drivers.

[0104] The hardware layer may include one or more hardware components. For example, the hardware layer may include a camera.

[0105] Combine the following Figure 4 The software structure diagram of FIG. 1 details the interaction between the various modules involved in the electronic device when executing the focus tracking method. Figure 5 As shown, the electronic device may include but is not limited to a camera, an image processing module, a multi-level detection module, a storage module and an AF module.

[0106] S101: A camera shoots a current shooting environment to obtain a preview image.

[0107] In order to better explain step S101, Figure 4 The software structure diagram of the embodiment details the software interaction diagram of the camera collecting and previewing images.

[0108] s11. The camera application receives a request message for starting the camera application.

[0109] It is understandable that the user can trigger the electronic device to start the camera application. Correspondingly, the camera application in the electronic device can receive the request information for starting the camera application.

[0110] For example, Figure 2A As shown, the user can click the camera application icon 101. Accordingly, the electronic device can receive a request message for starting the camera application. In response to the request message for starting the camera application, the electronic device can start the camera application and display the Figure 2B User interface 200-1 is shown. It is understandable that the user can also start the camera application by voice, gesture, etc., and the present application does not limit the specific method in which the user triggers the electronic device to start the camera application.

[0111] s12. The camera application sends a request message to the camera service to start the camera service.

[0112] After receiving the request information for starting the camera application, the camera application can send the request information for starting the camera service to trigger the camera service to start. Correspondingly, the camera service can receive the request information for starting the camera service sent by the camera application and start the camera service.

[0113] s13. The camera service sends a request message to the camera HAL to start the camera HAL.

[0114] After receiving the request information for starting the camera service sent by the camera application, the camera service may send a request information for starting the camera HAL to trigger the camera HAL to start.

[0115] s14. Camera HAL sends a request message to the camera to start the camera.

[0116] After the camera HAL receives the request information for starting the camera HAL sent by the camera service, it can send the request information for starting the camera to trigger the camera to start.

[0117] s15. The camera shoots the current shooting environment to obtain a preview image.

[0118] It can be understood that the activated camera can shoot the current shooting environment to obtain a preview image.

[0119] It is understandable that the specific process (such as step s11 to step s15) can be referred to the relevant technical documents and will not be described in detail here.

[0120] Optionally, after S101, the method may further include: the camera sends the preview image to the camera application through the camera HAL and the camera service, so that the camera application displays the preview image in the current preview interface. The specific process can be referred to in the relevant technical documents and will not be described in detail.

[0121] S102: The camera sends the preview image to an image processing module, and the image processing module sends the preview image to a multi-level detection module.

[0122] Correspondingly, the multi-level detection module receives the preview image.

[0123] Optionally, in other implementations, the camera may also directly send the preview image to the multi-level detection module.

[0124] S103: The multi-level detection module calls a multi-level detection algorithm to detect the preview image, and obtains multi-level information of the candidate object in the current preview interface.

[0125] The multi-level detection algorithm includes at least two of an object detection algorithm, a first part detection algorithm, or a second part detection algorithm. Figure 6 , shows a schematic diagram of the structure of a multi-level detection algorithm. Figure 6 As shown, the object detection algorithm may include but is not limited to a human body detection algorithm, an animal detection algorithm, a plant detection algorithm, and a car detection algorithm; the first part detection algorithm may include but is not limited to a face detection algorithm, a trunk detection algorithm, and a limb detection algorithm; the second part detection algorithm may include but is not limited to an eye detection algorithm, a nose detection algorithm, a lip detection algorithm, and an ear detection algorithm, as well as an arm detection algorithm and a leg detection algorithm. It is understood that in some examples, the object detection algorithm may be associated with at least one first part detection algorithm, for example, the human body detection algorithm may be associated with a face detection algorithm, a trunk detection algorithm, and a limb detection algorithm; a first part detection algorithm may be associated with at least one second part detection algorithm, for example, a face detection algorithm may be associated with an eye detection algorithm, a nose detection algorithm, a lip detection algorithm, and an ear detection algorithm. It is understood that the various detection algorithms involved in the present application may be provided by the electronic device itself or by a third-party application, without limitation. It can be seen that a variety of detection algorithms can be integrated in a multi-level detection module, and different levels of parts of different objects can be detected by scheduling the detection algorithms.

[0126] Optionally, different detection algorithms are used to detect different contents in the preview image. For example, an object detection algorithm is used to detect each object in the preview image; a first part detection algorithm is used to detect a first part in the preview image; and a second part detection algorithm is used to detect a second part in the preview image.

[0127] Optionally, the detection algorithm can be obtained by directly modeling and training based on the filter convolution image, or can be constructed based on the detection algorithm of the next level associated with the detection algorithm. For example, the human body detection algorithm can be constructed by combining the first part detection algorithms such as the face detection algorithm, the trunk detection algorithm, and the limb detection algorithm with the position information of each first part of the human body. For another example, the face detection algorithm can be constructed by combining the second part detection algorithms such as the eye detection algorithm, the nose detection algorithm, the lip detection algorithm, and the ear detection algorithm with the position information of each second part of the face (i.e., the facial features).

[0128] Among them, the candidate object includes at least one first part, a first part includes at least one second part, and the multi-level information of the candidate object includes at least two of the object information of the candidate object, the information of each first part, or the information of each second part. The object information of the candidate object includes but is not limited to at least one of the identification, feature vector or position information of the candidate object; the information of the first part includes but is not limited to at least one of the identification, feature vector, position information or priority of the first part; the information of the second part includes but is not limited to at least one of the identification, feature vector, position information or priority of the second part. It should be noted that the feature vector mentioned in this application can be used to describe at least one of color features, texture features, or spatial features. It should be noted that the position information mentioned in this application may include but is not limited to the position coordinates of the first coordinate system and / or the position coordinates of the second coordinate system.

[0129] Optionally, the priority of a part may be determined based on at least one of the following: a priority set by the system by default, position information of the part in the current preview interface, or relative position information of the part with respect to other parts in the current preview interface.

[0130] A. Priority of system default settings

[0131] For example, the system defaults to setting the priority of the face to 4, the priority of the trunk to 3, and the priority of the limbs to 2. The priority of the face is higher than the priority of the trunk, and the priority of the trunk is higher than the priority of the limbs. In this implementation, it is simpler and more convenient to set the priority by system default.

[0132] B. Determine the priority of the part based on the system default setting priority and the location information of the part in the current preview interface

[0133] In an optional implementation, the priority of the system default setting can be updated based on the position information of the part in the current preview interface to obtain an updated priority, and the updated priority is used as the priority of the part. For example, taking the system default setting of the priority of the limbs as 2 as an example, if the limbs are located at the center of the current preview interface, the priority of the limbs can be increased, such as adjusting the priority of the limbs from 2 to 3. In this implementation, the priority can be adaptively adjusted based on the position information of the part, which is more flexible.

[0134] C. Determine the priority of the part based on the system default setting priority and the relative position information between the part and other parts in the current preview interface

[0135] In an optional implementation, the priority of the system default setting can be updated based on the relative position information between the part and other parts in the current preview interface to obtain an updated priority, and the updated priority is used as the priority of the part. For example, taking the system default setting of the priority of the limbs as 2 as an example, if the relative distance between the limbs and the face in the current preview interface is less than a threshold, the priority of the limbs can be increased, such as adjusting the priority of the limbs from 2 to 3. In this implementation, the priority can be adaptively adjusted based on the relative position information between the parts, and the priority of each part can reflect the distribution of each part more accurately.

[0136] It should be noted that in other implementations, the priority of the system default setting can also be updated based on the distance between the part and the camera to obtain an updated priority, and the updated priority is used as the priority of the part. This is conducive to using the clearer part that the camera can capture as the tracking focus part to improve image quality.

[0137] Optionally, for the priority of the second part, the priority of the second part is lower than the priority of the first part to which the second part belongs. For example, if the second part is an eye and the first part to which the eye belongs is a face, the priority of the eye is lower than the priority of the face. For another example, if the second part is an arm and the first part to which the arm belongs is a limb, the priority of the arm is lower than the priority of the limbs.

[0138] Further, optionally, the priority order of the second part relative to other parts is related to the priority of the first part to which the second part belongs. For example, taking the second part as an eye, the first part to which the eye belongs as a face, and the other parts as limbs, if the priority of the first part to which the second part eye belongs as a face is higher than the priority of the limbs, then the priority of the eye is higher than the priority of the limbs. If the priority of the first part to which the second part eye belongs as a face is lower than the priority of the limbs, then the priority of the eye is lower than the priority of the limbs.

[0139] Optionally, the priority of an object is higher than the priority of all parts included in the object. For example, the priority of a human body is higher than the priority of a first part, a face; for another example, the priority of a human body is higher than the priority of a second part, an eye, and so on.

[0140] It should be noted that, different from the implementation described in step S103, in other implementations, the priority of the first part and the priority of the second part may also be determined by the image processing module without limitation.

[0141] S104. The multi-level detection module sends the multi-level information of the candidate object to the image processing module.

[0142] Correspondingly, the image processing module receives multi-level information of the candidate object.

[0143] S105. The image processing module obtains multi-level information corresponding to N historical focus objects from the storage module.

[0144] Wherein, N is a positive integer. The N historical focus objects are the focus objects that are stored and recorded.

[0145] It can be understood that, similar to the candidate object, a historical focus object includes at least one first part, a first part includes at least one second part, and the multi-level information of the historical focus object includes at least two items of the object information of the historical focus object, the information of each first part of the historical focus object, or the information of each second part.

[0146] In an optional implementation, the multi-level information of each historical focus object is stored in a tree structure. The tree structure includes a root node, a first child node, and a second child node. The root node is associated with at least one first child node, and a first child node is associated with at least one second child node. The root node of the tree structure is used to store the object information of the historical focus object, the first child node of the tree structure is used to store the information of each first part of the historical focus object, and the second child node of the tree structure is used to store the information of each second part of the historical focus object. Please refer to Figure 7 , shows a schematic diagram of a structure of multi-level information stored in a tree structure. Figure 7As shown, taking the case where the historical focus object is a human body, the first part includes the face, torso and limbs, and the second part includes the eyes, nose, lips, ears, arms and legs as an example, the root node can be used to store information about the human body, the first child node can be used to store information about the face, torso and limbs, and the second child node can be used to store information about the eyes, nose, lips, ears, arms and legs. It can be seen that in this embodiment, a specific tree structure can be constructed based on the basic data structure, and the multi-level information of the historical focus object can be stored through the tree structure. Through the tree structure, the object information of each historical focus object, as well as the information of the first part and / or the second part corresponding to each historical focus object can be effectively distinguished. At the same time, there is no need to traverse the entire multi-level information when matching information, which is conducive to quickly locating the target information.

[0147] Optionally, the multi-level information of the historical focus object can take into account the priority of each part when it is stored in a tree structure. For example, the higher the priority of a part, the higher the traversal order of the information of the part in the tree structure. Then, the image processing module can obtain the information of each part in sequence according to the priority of the part. When the information of each part is obtained in sequence based on the priority of the part, it can help avoid duplicate matching information and save computing resources.

[0148] It should be noted that, in other implementations, the embodiments of the present application may also use other storage methods (such as lists) to store multi-level information, which will not be elaborated herein.

[0149] S106. The image processing module matches the multi-level information of the candidate object in the current preview interface with the multi-level information corresponding to the N historical focus objects, and determines whether there is a historical focus object that matches the candidate object among the N historical focus objects.

[0150] For each historical focus object, step S106 may include: the image processing module matches the multi-level information of the historical focus object with the multi-level information of the candidate object, and if the multi-level information of the historical focus object matches the multi-level information of the candidate object, it is determined that the historical focus object matches the candidate object. If the multi-level information of the historical focus object does not match the multi-level information of the candidate object, it is determined that the historical focus object does not match the candidate object.

[0151] The multi-level information of the candidate object includes at least two of the object information of the candidate object, the information of each first part, or the information of each second part.

[0152] In an optional embodiment, exemplarily, taking the case that the multi-level information of the candidate object includes the object information of the candidate object and the information of each first part, if the object information of the candidate object matches the object information of the historical focus object, and the information of the first part of the candidate object matches the information of the first part of the historical focus object, then it is determined that the candidate object matches the historical focus object, otherwise, the candidate object does not match the historical focus object. Exemplarily, taking the case that the multi-level information of the candidate object includes the object information of the candidate object, the information of each first part and the information of each second part, if at least two of the following conditions are met: the object information of the candidate object matches the object information of the historical focus object, the information of the first part of the candidate object matches the information of the first part of the historical focus object, and the information of the second part of the candidate object matches the information of the second part of the historical focus object, then it is determined that the candidate object matches the historical focus object, otherwise, the candidate object does not match the historical focus object. It can be seen that in this embodiment, when at least two of the object information, the information of the first part, or the information of the second part of the candidate object match the object information of the historical focus object, then it is determined that the candidate object matches the historical focus object. Compared with the scheme of only comparing the object information, this embodiment matches more information, thereby making the matching result more accurate.

[0153] Furthermore, in another optional embodiment, it is also possible to determine whether the candidate object matches the historical focus object based on the matching rate between the candidate object's multi-level information and the multi-level information of the historical focus object. For example, if the matching rate between the candidate object's multi-level information and the multi-level information of the historical focus object is greater than the matching threshold, then the candidate object is determined to match the historical focus object; if the matching rate between the candidate object's multi-level information and the multi-level information of the historical focus object is less than or equal to the matching threshold, then the candidate object is determined to not match the historical focus object. It should be noted that the matching threshold can be set based on business needs or experience. For example, the matching threshold can be 0.75, 0.85, etc. Optionally, the matching rate between the candidate object's multi-level information and the multi-level information of the historical focus object can be determined based on a weighted sum algorithm. Exemplarily, taking the case where the multi-level information of the candidate object includes the object information of the candidate object, the information of the first part, and the information of the second part, and the weighting coefficient of the object information is 0.5, the weighting coefficient of the information of the first part is 0.3, and the weighting coefficient of the information of the second part is 0.2, if the object information of the candidate object matches the object information of the historical focus object, and the information of the first part of the candidate object matches the information of the first part of the historical focus object, then it can be determined that the matching rate between the multi-level information of the candidate object and the multi-level information of the historical focus object is 0.8; if the object information of the candidate object matches the object information of the historical focus object, and the information of the second part of the candidate object matches the information of the second part of the historical focus object, then it can be determined that the matching rate between the multi-level information of the candidate object and the multi-level information of the historical focus object is 0.7. It can be seen that in this embodiment, the matching rate between the multi-level information of the candidate object and the multi-level information of the historical focus object can be determined based on the weighted sum algorithm, and the importance of each information can be adaptively adjusted using the weighting coefficient, which can meet the diverse matching needs of users, thereby making the matching result more accurate.

[0154] Optionally, if there are multiple first parts of the candidate object, when there is a first part among the multiple first parts of the candidate object that matches the first part of the historical focus object, then the first part of the candidate object is determined to match the first part of the historical focus object, that is, the information of the first part of the candidate object matches the information of the first part of the historical focus object. Similarly. If there are multiple second parts of the candidate object, when there is a second part among the multiple second parts of the candidate object that matches the second part of the historical focus object, then the second part of the candidate object is determined to match the second part of the historical focus object, that is, the information of the second part of the candidate object matches the information of the second part of the historical focus object. It should be noted that it is also possible to determine whether the first part of the candidate object matches the first part of the historical focus object based on a weighted sum algorithm, and / or to determine whether the second part of the candidate object matches the second part of the historical focus object based on a weighted sum algorithm, which will not be elaborated on.

[0155] In an optional implementation, the image processing module may match each piece of information of the candidate object with each piece of information of the historical focus object in order of priority. For example, if the priority of the object information is higher than the priority of the information of the first part, the object information of the candidate object may be compared with the object information of the historical focus object first, and then the candidate object may be compared with the information of the first part of the historical focus object.

[0156] S107: If so, the image processing module uses the candidate object as the current focus object.

[0157] In an optional implementation, if there is a historical focus object matching the candidate object among the N historical focus objects, the candidate object is used as the current focus object, and steps S108 to S109 are performed. If not, the steps are terminated.

[0158] For example, the current preview interface is Figure 2K Taking the user interface 900 as an example, the candidate objects may include a person who is playing football and a person who is walking. The image processing module matches the multi-level information of the person who is playing football with the multi-level information corresponding to the N historical focus objects, and determines that there is a historical focus object matching the person who is playing football among the N historical focus objects, and then uses the person who is playing football as the current focus object. In other words, the storage module pre-stores the multi-level information of the person who is playing football (such as in Figure 2FIn the user interface 500-1 shown, multi-level information is stored when a person who is playing football is used as a focus object. When it is detected that the person who is playing football re-enters the preview interface, the person who is playing football can be automatically used as the current focus object. In other words, in the scenario where the candidate object enters the current preview interface, the multi-level information of the candidate object in the current preview interface is matched with the multi-level information corresponding to N historical focus objects. The present application solution can automatically identify the focus object in the current preview interface and automatically keep the focus on the focus part of the focus object.

[0159] In an optional implementation, before detecting that the person who is playing football has re-entered the preview interface, it is also possible to detect that the person who is playing football has left the current preview interface; the duration between the moment the candidate object leaves the current preview interface and the moment the candidate object enters the current preview interface is less than a preset duration. Only when the duration between the moment the candidate object leaves the current preview interface and the moment the candidate object enters the current preview interface is less than a preset duration, the storage module stores the multi-level information of the historical focus objects that match the candidate object. Otherwise, the storage module does not store the multi-level information of the historical focus objects that match the candidate object, and the historical focus objects that match the candidate object cannot be found from the storage module, and the focus cannot be automatically maintained subsequently, that is, the focus lock on the candidate object is automatically exited. Exemplarily, the current preview interface is Figure 2L Taking the user interface 1000 shown as an example, if the time the person playing football leaves the preview interface is greater than or equal to the preset time, the storage module has deleted the multi-level information of the person playing football, and the image processing module cannot identify the person playing football in the user interface 1000 as the current focus object.

[0160] In an optional implementation, the camera application is controlled to switch from a first shooting mode to a second shooting mode; when the camera application is in the second shooting mode, the steps of matching the multi-level information of the candidate object in the current preview interface with the multi-level information corresponding to N historical focus objects are performed.

[0161] Exemplarily, taking the case where the camera application switches from the photo mode shown in user interface 500-1 to the portrait mode of user interface 1300 as an example, the candidate objects in the current preview interface (i.e., user interface 1300) may include a person who is playing football and a person who is walking. The image processing module matches the multi-level information of the person who is playing football with the multi-level information corresponding to the N historical focus objects, and determines that there is a historical focus object that matches the person who is playing football among the N historical focus objects, and then uses the person who is playing football as the current focus object. In other words, when switching the shooting mode, the focus object in the current preview interface can be automatically identified. Compared with the method of manually re-determining the focus object after switching the shooting mode, this solution can automatically keep the focus on the focus part of the focus object, thereby achieving a better focus effect.

[0162] It should be noted that, in a scenario where there are multiple candidate objects in the current preview interface, the number of candidate objects that match the historical focus object may be multiple, and the current focus object may be selected from the multiple candidate objects based on the following methods: (1) selecting any candidate object that matches the historical focus object as the current focus object; (2) determining the current focus object based on the shooting time of the historical focus object, for example, selecting the candidate object that matches the historical focus object whose shooting time is closest to the current time as the current focus object.

[0163] S108. The image processing module determines a focus portion of the current focus object, and sends focus information of the focus portion to the AF module.

[0164] The focus part is the part with the highest priority of the current focus object in the current preview interface. That is to say, the higher the priority of the part, the easier it is to be used as the focus part. For example, if the current focus object is a person who is playing football, and the priority of the first part of the person who is playing football is higher than the priority of the torso, and the priority of the torso is higher than the priority of the limbs, in the current preview interface, Figure 2F When the current preview interface includes the face, torso and limbs, the face can be used as the focus tracking part of the person who is playing football; when the current preview interface is Fig.2I When the user interface 700 is displayed, the current preview interface includes the torso and limbs, and the torso can be used as the focus part of the person who is kicking the ball.

[0165] For example, if the current focus tracking object is a person who is playing football, and the priority of the second part of the person who is playing football is higher than the priority of the torso, and the priority of the torso is higher than the priority of the limbs. Figure 2H When the user interface 600 is shown, the current preview interface includes eyes, torso and limbs, and the eyes can be used as the focus part of the person who is kicking the ball.

[0166] Optionally, the focus information may include but is not limited to at least one of the position information of the focus part, the identification of the focus part, multi-level information of the current focus object, or the object identification of the current focus object.

[0167] S109. The AF module performs focusing on the focusing part based on the focusing information of the focusing part.

[0168] Optionally, when the focus tracking information is position information of the focus tracking part, the AF module performs focus tracking directly based on the position information.

[0169] Optionally, when the focus tracking information is an identifier of the focus tracking part, the AF module may obtain the position information of the focus tracking part based on the identifier, and perform focus tracking based on the position information of the focus tracking part.

[0170] Optionally, when the focus information is multi-level information of the current focus object, the AF module may extract position information of the focus part from the multi-level information of the current focus object, and perform focus tracking based on the position information of the focus part.

[0171] Optionally, when the focus information is the object identifier of the current focus object, the AF module can obtain multi-level information of the current focus object based on the object identifier, extract the position information of the focus part from the multi-level information of the current focus object, and perform focus based on the position information of the focus part.

[0172] In an optional implementation, step S108 is performed when the electronic device has a focus tracking function turned on. Specifically, in response to a user operation on the focus tracking option, the electronic device may turn on the focus tracking function through a camera application. FIG. 2A to FIG. 2E The relevant embodiments are not described in detail.

[0173] Optionally, in other implementations, the AF module can also focus on the current focus object, which is not limited in this application.

[0174] Furthermore, the embodiments of the present application can also store multi-level information of the current focus object through a storage module. That is, after the image processing module uses the candidate object as the current focus object, the method also includes: the image processing module sends the multi-level information of the current focus object to the storage module, and the storage module stores the multi-level information of the current focus object through a tree structure; the root node of the tree structure is used to store the object information of the current focus object, the first child node of the tree structure is used to store the information of each first part of the current focus object, and the second child node of the tree structure is used to store the information of each second part of the current focus object. For related elaboration, please refer to step S105 on the relevant embodiments of the multi-level information of the historical focus object, which will not be repeated. It can be seen that in this embodiment, the multi-level information of the current focus object can be stored. In the subsequent time period, the current focus object can be used as the historical focus object, and the multi-level information of the current focus object can be used as the multi-level information of the historical focus object, which is conducive to timely updating the multi-level information of the historical focus object.

[0175] It can be seen that in this embodiment, whether to keep focusing on the candidate object can be determined based on the matching result between the multi-level information of the historical focus object and the multi-level information of the candidate object. The focus object can be autonomously identified in the current preview interface, and the focus portion of the focus object can be automatically kept in focus, which is conducive to real-time and effective automatic focus in complex focus scenes. For example, when the focus object leaves the preview interface and re-enters the preview interface, the focus object can be autonomously identified, and the focus portion of the focus object can be automatically kept in focus. For another example, when switching the shooting mode, the focus object can be autonomously identified, and the focus portion of the focus object can be automatically kept in focus. In addition, the embodiment of the present application locks the focus on a specific focus portion, which can effectively improve the focus effect.

[0176] In addition, the multi-level detection module of the embodiment of the present application can not only detect the candidate objects in the preview image to obtain object information by calling the multi-level detection algorithm, but also detect the various parts of the candidate objects to obtain the information of the first part and / or the information of the second part. The multi-level information of the embodiment of the present application includes at least two of the object information, the information of the first part, or the information of the second part. The multi-level information has richer information types, which is beneficial to improving the accuracy of matching when matching is based on the multi-level information.

[0177] In some scenarios, the current focus object can also be manually determined. For example, when the multi-level information of historical focus objects is not stored in the storage module, the current focus object can be determined through user operation. Figure 8 , Figure 8 A flowchart of another focus tracking method provided in an embodiment of the present application. The focus tracking method may include but is not limited to the following steps:

[0178] S201: The camera shoots the current shooting environment to obtain a preview image.

[0179] S202: The camera sends the preview image to the image processing module, and the image processing module sends the preview image to the multi-level detection module.

[0180] S203: The multi-level detection module calls a multi-level detection algorithm to detect the preview image, and obtains multi-level information of each candidate object in the current preview interface.

[0181] S204: The multi-level detection module sends the multi-level information of each candidate object to the image processing module.

[0182] It should be noted that the specific implementation of steps S201 to S204 can refer to steps S101 to S104 and will not be described in detail.

[0183] S205: The image processing module determines the position information of each candidate object based on the multi-level information of each candidate object.

[0184] In an optional implementation, when the multi-level information of the candidate object includes object information of the candidate object, the image processing module directly obtains the position information of the candidate object from the object information of the candidate object.

[0185] In another optional embodiment, when the multi-level information of the candidate object does not include the object information of the candidate object, the image processing module may obtain the position information of each first part from the information of each first part, and / or obtain the position coordinates of each second part from the information of each second part, and determine the position information of the candidate object based on the position information of each first part and / or the position information of each second part.

[0186] S206: The image processing module sends the location information of each candidate object to the camera service, and the camera service sends the location information of each candidate object to the camera application.

[0187] In one implementation, the coordinate systems used by the hardware abstraction layer and the application layer may be the same or different. In the case where the coordinate systems used by the hardware abstraction layer and the application layer are different, the coordinate conversion step can also be performed by a coordinate conversion module. It should be noted that the coordinate conversion module can be located in any layer of the hardware abstraction layer, the application framework layer, or the application layer. The following is an exemplary description using the coordinate conversion module located in the hardware abstraction layer as an example.

[0188] Exemplarily, taking the hardware abstraction layer using the first coordinate system and the application layer using the second coordinate system as an example, the image processing module of the hardware abstraction layer can send the position coordinates of each candidate object in the first coordinate system to the coordinate conversion module of the hardware abstraction layer. After the coordinate conversion module converts the position coordinates in the first coordinate system into the position coordinates in the second coordinate system, it sends the position coordinates of each candidate object in the second coordinate system to the camera application of the application layer through the camera service of the application framework layer. It can be seen that through coordinate conversion, the image processing module of the hardware abstraction layer can send the accurate position information of each candidate object to the camera application of the application layer.

[0189] S207: The camera application displays object frames respectively associated with each candidate object in the preview interface based on the position information of each candidate object.

[0190] For example, Figure 2E As shown, the preview interface may display an object frame 401 and an object frame 402, wherein the object frame 401 is used to frame a person who is playing football, and the object frame 402 is used to frame a person who is walking.

[0191] Optionally, the position information of each object frame is determined based on the position information of each candidate object.

[0192] S208: The camera application detects a first operation and determines location information of the first operation.

[0193] For example, Figure 2E As shown, when the user clicks on the object frame 401, the camera application can detect the click operation and determine the position information of the click operation.

[0194] S209: The camera application sends the location information of the first operation to the camera service, and the camera service sends the location information of the first operation to the image processing module.

[0195] Similar to step S206, when the coordinate systems used by the hardware abstraction layer and the application layer are different, a coordinate conversion step also needs to be performed by the coordinate conversion module.

[0196] Exemplarily, taking the hardware abstraction layer using the first coordinate system and the application layer using the second coordinate system as an example, the camera application of the application layer sends the position coordinates of the first operation in the second coordinate system to the camera service of the application framework layer. After the camera service receives the position coordinates of the first operation in the second coordinate system, it can send the position coordinates of the first operation in the second coordinate system to the coordinate conversion module of the hardware abstraction layer. After the coordinate conversion module converts the position coordinates of the first operation in the second coordinate system into the position coordinates of the first coordinate system, it can send the position coordinates of the first operation in the first coordinate system to the image processing module. It can be seen that through coordinate conversion, the camera application of the application layer can send the accurate position information of the first operation to the image processing module of the hardware abstraction layer.

[0197] S210: The image processing module determines a current focus object based on the position information of the first operation and the position information of each candidate object.

[0198] In an optional implementation, the image processing module determines the distance between each candidate object and the first operation based on the position information of the first operation and the position information of each candidate object, and uses the candidate object closest to the first operation as the current focus object.

[0199] S211. The image processing module determines a focus portion of the current focus object, and sends focus information of the focus portion to the AF module.

[0200] S212. The AF module performs focusing on the focusing part based on the focusing information of the focusing part.

[0201] It should be noted that for the relevant description of step S211 to step S212, please refer to step S108 to step S109 and will not be repeated here.

[0202] Similarly, in this embodiment, the multi-level information of the current focus object can also be stored by the storage module. For related description, please refer to Figure 5 The relevant embodiments are not described in detail.

[0203] It can be seen that in this embodiment, when using the camera application, the user may have a need to track the focus of the candidate object in the current preview interface. At this time, the electronic device can detect the first operation and determine the current focus object based on the position information of the first operation. At the same time, the embodiment of the present application introduces a coordinate conversion module, based on which the position information can be accurately transmitted between the hardware abstraction layer and the application layer.

[0204] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0205] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0206] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A focus tracking method, characterized in that: The method comprises: In a scenario where a candidate object leaves a current preview interface and re-enters the current preview interface, the multi-level information of the candidate object in the current preview interface is matched with the multi-level information corresponding to N historical focus objects to determine whether there is a historical focus object matching the candidate object among the N historical focus objects; the candidate object includes at least one first part, a first part includes at least one second part, and the multi-level information of the candidate object includes at least two of the object information of the candidate object, the information of each first part, or the information of each second part; N is a positive integer; the N historical focus objects are stored and recorded focus objects; If so, the candidate object is used as the current focus object, and the focus portion of the current focus object is focused; Before matching the multi-level information of the candidate object in the current preview interface with the multi-level information corresponding to the N historical focus objects, the method further includes: A multi-level detection algorithm is called to detect the preview image in the current preview interface to obtain multi-level information of the candidate object in the current preview interface; the multi-level detection algorithm includes at least two of an object detection algorithm, a first part detection algorithm, or a second part detection algorithm.

2. The method according to claim 1, characterized in that The focus portion is the portion with the highest priority of the current focus object in the current preview interface.

3. The method according to claim 2, characterized in that The priority of the part of the current focus object is determined based on at least one of the following: The priority of the system default settings; The position information of the part in the current preview interface; Relative position information between the part and other parts in the current preview interface.

4. The method according to claim 1, characterized in that The focus tracking portion is a first portion of the current focus tracking object. After focusing on the focus tracking portion of the current focus tracking object, the method further includes: The second part included in the first part of the current focus object is used as the updated focus part of the current focus object.

5. The method according to any one of claims 1 to 4, characterized in that: After taking the candidate object as the current focus object, the method further includes: Storing multi-level information of the current focus object through a tree structure; Among them, the tree structure includes a root node, a first child node and a second child node; the root node is associated with at least one first child node, and a first child node is associated with at least one second child node; the root node is used to store the object information of the current focus object, the first child node is used to store the information of each first part of the current focus object, and the second child node is used to store the information of each second part of the current focus object.

6. The method according to claim 5, characterized in that The storage order of the information of the first parts of the current focus object in the first sub-node is related to the priority of the first parts of the current focus object; The storage order of the information of the second parts of the current focus object in the second sub-node is related to the priority of the second parts of the current focus object and / or the first part to which the second parts belong.

7. The method according to any one of claims 1 to 4, characterized in that: When at least two of the following conditions are met, there is a historical focus object matching the candidate object among the N historical focus objects: The object information of the candidate object matches the object information of a historical focus object; The information of the first part of the candidate object matches the information of the first part of the historical focus object; The information about the second part of the candidate object matches the information about the second part of the historical focus object.

8. An electronic device, characterized in that: include: One or more processors and one or more memories; the one or more memories are coupled to the one or more processors, the one or more memories are used to store computer program codes, the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 7.

10. A chip system, characterized in that: The method comprises a processing circuit and an interface circuit, wherein the interface circuit is used to receive code instructions and transmit the code instructions to the processing circuit, and the processing circuit is used to run the code instructions to execute the method according to any one of claims 1 to 7.

Images