Focus control method, electronic device and medium

Abstract

The present application discloses a focus control method, electronic device and medium, and relates to the field of computer technology. The method comprises: acquiring an image frame, generating multiple candidate focus areas according to the image frame, wherein at least two of the candidate focus areas have different proportions of the target object in their respective candidate focus areas; determining a target focus area according to the confidence of the multiple candidate focus areas; and focusing on the target object using the target focus area. In the method, multiple candidate focus areas are generated during the process of focusing on the target object. Compared with one focus area, multiple candidate focus areas can reduce the situation where a single focus area includes too much background to a certain extent. Selecting a target focus area with a larger confidence from the multiple candidate focus areas, and using the target focus area to focus on the target object can reduce the situation where the background is focused on.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a focus control method, electronic equipment and medium. Background Art

[0002] Currently, electronic devices (such as mobile phones, tablet computers, etc.) all support a shooting function. Based on this shooting function, a user can shoot a target object (such as a person, a landscape, a plant, etc.) to obtain an image of the target object.

[0003] When an electronic device is photographing a target object, the electronic device can determine the position of the target object based on a target detection algorithm, use the area at the position as a focus area, and then automatically focus on the target object based on an autofocus (AF) algorithm and the focus area, thereby making the target object in the captured image clearer.

[0004] However, when electronic devices use autofocus algorithms, the focus area not only covers the target object but also the background other than the target object. This may cause the electronic device to focus on the background, thereby reducing the quality of the captured image. Summary of the invention

[0005] The present application provides a focus control method, an electronic device and a medium, which can reduce the situation where the electronic device focuses on the background.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a focus control method, which can be performed by an electronic device, for example, an electronic device with a camera. The method includes: the electronic device acquires an image frame, and then generates multiple candidate focus areas based on the image frame, wherein at least two candidate focus areas have different proportions of target objects in their respective candidate focus areas, and then the electronic device can determine the target focus area based on the confidence of the multiple candidate focus areas, and finally use the target focus area to focus on the target object.

[0008] In this method, during the process of focusing on the target object, multiple candidate focus areas are generated. Compared with one candidate focus area, multiple candidate focus areas can reduce the situation where a single focus area includes too much background to a certain extent. A target focus area with a larger confidence is selected from the multiple candidate focus areas, and the target object is focused on using the target focus area, which can reduce the situation where the background is focused.

[0009] In some possible implementations, the electronic device generates multiple candidate focus areas according to the image frame by: firstly, the electronic device generates an initial focus area according to the image frame, and then generates at least one focus area by shrinking the initial focus area. For example, at least one focus area can be generated within the initial focus area, thereby obtaining multiple candidate focus areas.

[0010] In some other possible implementations, the electronic device may generate multiple candidate focus areas according to the image frame by: firstly generating an initial focus area according to the image frame, and then generating at least one focus area by expanding the initial focus area. For example, at least one focus area may be generated outside the initial focus area, thereby obtaining multiple candidate focus areas.

[0011] In some other possible implementations, the electronic device generates multiple candidate focus areas according to the image frame by: the electronic device first generates an initial focus area according to the image frame, and then generates at least one focus area by tiling the initial focus area. For example, at least one focus area is tiled in the initial focus area, or at least one focus area is tiled outside the initial focus area, or at least one focus area is tiled in the initial focus area and outside the initial focus area, respectively, to obtain multiple candidate focus areas.

[0012] In some possible implementations, the electronic device generates an initial focus area based on an image frame, which may be: the electronic device first determines whether the image frame includes a face, and if the image frame includes a face, generates an initial focus area in the face area and / or body area, that is, the position of the initial focus area is in the face area and / or body area in the image frame. If the image frame does not include a face, a saliency detection may be performed on the image frame to obtain a saliency area, and then an initial focus area is generated in the saliency area, that is, the position of the initial focus area is in the saliency area. In this method, the electronic device generates the initial focus area in different ways based on different situations, thereby improving the accuracy of the initial focus area, thereby reducing excessive background in subsequent candidate focus areas, and further improving the accuracy of obtaining the target focus area, so as to facilitate the subsequent focusing process, so that the proportion of the target object in the focus area is larger and the proportion of the background is smaller.

[0013] In some possible implementations, the electronic device may further screen the candidate focus areas. For example, the electronic device may screen the candidate focus areas by the depth information of the candidate focus areas. For example, if the electronic device can obtain the depth information of multiple candidate focus areas, the quasi-focus position corresponding to each candidate focus area is determined according to the depth information and the correspondence between the preset depth information and the quasi-focus position, and then the multiple candidate focus areas are screened according to the quasi-focus position corresponding to each candidate focus area and the difference between the phase difference detection value and the first threshold. On this basis, the electronic device may determine the target focus area based on the confidence of the screened candidate focus areas. The method further filters out the focus areas whose depth information does not match, thereby further increasing the accuracy of the candidate focus areas, thereby reducing the situation where the focus is on the background.

[0014] In some possible implementations, the electronic device may also determine the distant view grid and the near view grid in each candidate focus area according to the phase difference detection value of each grid in each candidate focus area and the second threshold, and then determine whether each candidate focus area is a multi-depth scene according to the first statistical value of the phase difference detection value of the distant view grid in each candidate focus area and the second statistical value of the phase difference detection value of the near view grid and the third threshold. Further, the electronic device may filter the candidate focus areas after the above screening, filter out the candidate focus areas of the multi-depth scene, and then determine the target focus area from the candidate focus areas that are not multi-depth scenes. For example, the target focus area is determined based on the confidence of the candidate focus areas that are not multi-depth scenes. In this method, the electronic device further filters out the candidate focus areas of the multi-depth scenes, thereby further reducing the number of focus areas that do not meet the requirements, and then determines the target focus area from the focus areas that are not multi-depth scenes, thereby reducing the situation of focusing on the background.

[0015] In some other possible implementations, the electronic device cannot obtain the depth information of the candidate focus areas, then the electronic device determines the distant view grids and the near view grids in the respective candidate focus areas based on the phase difference detection value of each grid in the respective candidate focus areas and the second threshold, then determines whether each candidate focus area is a multi-depth of field scene based on the first statistical value of the phase difference detection value of the distant view grid in the respective candidate focus area and the second statistical value of the phase difference detection value of the near view grid and the third threshold, then retains the candidate focus areas that are not multi-depth of field scenes from the multiple candidate focus areas, and determines the target focus area based on the confidence of the candidate focus areas that are not multi-depth of field scenes.

[0016] In some possible implementations, the target candidate focus area is the candidate focus area with the greatest confidence or any one of the first N candidate focus areas after sorting the confidences from large to small, where N is a positive integer.

[0017] In some possible implementations, the electronic device may determine the confidence of the candidate focus area based on the proportion of the target object in the candidate focus area. For example, the greater the proportion of the target object in the focus area, the greater the confidence of the focus area, and the smaller the proportion of the target object in the focus area, the smaller the confidence of the focus area. Furthermore, in a focus area with a greater confidence, the greater the proportion of the target object in the focus area, and focusing based on the target focus area can reduce the situation of focusing on the background.

[0018] In a second aspect, the present application provides an electronic device, including: a camera, a processor, and a memory;

[0019] The camera collects image frames;

[0020] One or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the electronic device executes any method in the first aspect.

[0021] In a third aspect, the present application provides a computer storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes any method as described in the first aspect.

[0022] In a fourth aspect, the present application provides a computer program product, comprising instructions; when the instructions are executed by an electronic device, the electronic device executes any one of the methods in the first aspect above.

[0023] The technical solution of this application has the following beneficial effects:

[0024] The present application provides a focus control method, which includes acquiring an image frame, generating multiple candidate focus areas based on the image frame, wherein at least two of the candidate focus areas have different proportions of the target object in their respective candidate focus areas; determining a target focus area based on the confidence of the multiple candidate focus areas; and focusing on the target object using the target focus area. In this method, multiple candidate focus areas are generated during the process of focusing on the target object. Compared with one focus area, multiple candidate focus areas can reduce the situation where a single focus area includes too much background to a certain extent. Selecting a target focus area with a larger confidence from the multiple candidate focus areas, and using the target focus area to focus on the target object can reduce the situation where the background is focused on.

[0025] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of features or beneficial effects means that specific technical features, technical solutions or beneficial effects are included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in the present embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be realized without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in a specific embodiment that does not embody all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0028] Figure 3 A flowchart of a focus control method provided in an embodiment of the present application;

[0029] Figure 4A A schematic diagram of a retracted candidate focus area provided in an embodiment of the present application;

[0030] Figure 4B A schematic diagram of another indented candidate focus area provided in an embodiment of the present application;

[0031] Figure 4C A schematic diagram of another indented candidate focus area provided in an embodiment of the present application;

[0032] Figure 4D A schematic diagram of an outward-expanding candidate focus area provided in an embodiment of the present application;

[0033] Figure 4E A schematic diagram of another outward-expanding candidate focus area provided in an embodiment of the present application;

[0034] Figure 4F A schematic diagram of another outward-expanding candidate focus area provided in an embodiment of the present application;

[0035] Figure 5 A schematic diagram of determining a motor position after adjustment provided in an embodiment of the present application;

[0036] Figure 6A flowchart of another focus control method provided in an embodiment of the present application;

[0037] Figure 7 A schematic diagram of an image frame including a face provided in an embodiment of the present application;

[0038] Figure 8 A schematic diagram of an image frame not including a face provided in an embodiment of the present application;

[0039] Fig. 9 A schematic diagram of a focusing effect provided in an embodiment of the present application;

[0040] Fig.10 A schematic diagram of another focusing effect provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The terms "first", "second", "third", etc. in the specification, claims and drawings of this application are used to distinguish different objects rather than to limit a specific order.

[0042] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0043] In order to make the description of the following embodiments clear and concise, a brief introduction to the related technology is first given:

[0044] The target detection algorithm is used to automatically detect and locate targets in an image or video. The target detection algorithm can identify different targets in the image, such as people, cars, animals, etc., and mark the location of the target in the image. In some examples, the target detection algorithm may include a face detection algorithm, which is used to automatically detect and locate faces in images or videos. For example, based on the face detection algorithm, it can be determined whether the image includes a face. In the face detection algorithm, some specific features are used to detect faces, such as the outline of the face, eyes, mouth, etc. These features can be learned and recognized through machine learning or deep learning algorithms. Exemplarily, the electronic device can determine whether the image includes a face based on the face detection algorithm, and if a face is included, mark the area of ​​the face.

[0045] The saliency detection algorithm is used to determine the most salient area in the image. The saliency detection algorithms include saliency detection algorithms based on global contrast, saliency detection algorithms based on frequency domain analysis, saliency detection algorithms based on region, saliency detection algorithms based on deep learning, and saliency detection algorithms based on visual attention.

[0046] An autofocus algorithm is an algorithm that automatically focuses on the content being photographed. The autofocus algorithm can automatically adjust the focal length of the lens by analyzing the image, thereby improving the clarity of the target object in the image. The focus area is the information required by the autofocus algorithm. For example, the electronic device can determine the position of the target object in the image based on the above-mentioned target detection algorithm, and determine the focus area based on the position of the target object. The electronic device can also determine the most prominent area in the image based on the saliency detection algorithm, and determine the focus area based on the most prominent area. Then the electronic device performs autofocus based on the autofocus algorithm and the focus area. Exemplarily, the electronic device can determine the position of a face in the image based on a face detection algorithm, and determine the focus area based on the position of the face to perform subsequent autofocus operations.

[0047] However, in traditional technical solutions, electronic devices often only determine one focus area, and the focus area often covers the background except the target object, which will cause the electronic device to focus on the background instead of the target object, and thus make the target object in the captured image blurred and unclear, reducing the quality of the captured image and affecting the shooting experience.

[0048] In view of this, an embodiment of the present application provides a focus control method, which can be applied to a photo-taking scenario of an electronic device. In this scenario, the method includes: the electronic device acquires an image frame and determines the position of a target object in the image frame, and generates multiple candidate focus areas according to the position of the target object, wherein at least two candidate focus areas have different proportions of the target object in their respective corresponding candidate focus areas, for example, the proportion of the target object in the first candidate focus area is 80%, and the proportion of the target object in the second candidate focus area is 90%, and then the confidence of each candidate focus area is determined respectively, for example, the confidence of the candidate focus area can be determined based on the proportion of the candidate focus area occupied by the target object and / or other information of the candidate focus area, and then the target focus area is determined based on the confidence of each candidate focus area, and the target focus area can be determined, which can be the candidate focus area with the highest confidence, or any one of the top N candidate focus areas after the confidence is ranked from large to small, and then, focusing is performed based on the target focus area.

[0049] In this method, the electronic device generates multiple candidate focus areas during the process of focusing on the target object. Compared with a single focus area, multiple candidate focus areas provide more options for the electronic device to perform automatic focusing, thereby reducing the situation where a single focus area includes too much background to a certain extent. The electronic device selects a target focus area with a larger confidence (for example, the highest confidence or the top N confidences in descending order) from the multiple candidate focus areas, and uses the target focus area to focus on the target object, which can reduce the situation where the electronic device focuses on the background.

[0050] In some embodiments, the electronic device may be a mobile phone, a camera (e.g., a digital camera), a tablet computer, a desktop, a laptop, a notebook computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), a wearable electronic device, a smart watch, etc. The present application does not impose any special restrictions on the specific form of the above electronic devices. In this embodiment, the structure of the electronic device may be as follows: Figure 1 As shown, Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0051] like Figure 1 As shown, the electronic device may include a processor 110 , an internal memory 121 , a motor 191 , a camera 193 , and a display screen 194 .

[0052] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0053] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices or integrated into one or more processors. For example, in the present application, the processor can obtain an image frame; based on the image frame, generate multiple candidate focus areas, at least two of the candidate focus areas have different proportions of the target object in each candidate focus area; determine the target focus area based on the confidence of the multiple candidate focus areas; and focus on the target object using the target focus area.

[0054] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0055] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0056] The electronic device implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0057] The display screen 194 is used to display images, videos, etc. In some examples, the display screen 194 can be used to display images captured by the camera 193 or images after autofocusing. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oled, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0058] A series of graphical user interfaces (GUIs) can be displayed on the display screen 194 of the electronic device, and these GUIs are the main screen of the electronic device. Generally speaking, the size of the display screen 194 of the electronic device is fixed, and only limited controls can be displayed on the display screen 194 of the electronic device. A control is a GUI element, which is a software component included in an application program, and controls all data processed by the application program and interactive operations on the data. Users can interact with the control through direct manipulation to read or edit relevant information of the application program. Generally speaking, a control can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc.

[0059] The electronic device can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0060] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0061] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device may include 1 or N cameras 193, where N is a positive integer greater than 1. When the electronic device uses the camera 193 to shoot, it can automatically focus on the content being shot based on an autofocus algorithm, thereby making the content being shot clearer.

[0062] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and continuously self-learn. NPU can realize applications such as intelligent cognition of electronic devices, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0063] The internal memory 121 may be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 121. For example, in this embodiment, the processor 110 may execute the following steps by executing the instructions stored in the internal memory 121:

[0064] Acquire an image frame; generate multiple candidate focus areas based on the image frame, wherein at least two of the candidate focus areas have different proportions of the target object in their respective candidate focus areas; determine a target focus area based on the confidence levels of the multiple candidate focus areas; and focus on the target object using the target focus area.

[0065] Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0066] Motor 191 can be an autofocus motor and can be used as one of the components of the lens module. The autofocus motor can be used to control the focus and zoom of the lens, thereby improving the clarity of the image or making the focus more accurate. Generally, after the user triggers the shooting operation (such as pressing the shutter or shooting control), the autofocus motor will automatically adjust the focus of the lens to ensure that the target object (also called the subject) in the image is clear.

[0067] In an embodiment of the present application, the electronic device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. Among them, the hardware layer may include hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also called main memory). The operating system of the operating system layer can be any one or more computer operating systems that implement business processing through processes, such as Linux operating system, Unix operating system, Android operating system, iOS operating system, or Windows operating system. The application layer may include applications such as browsers, address books, word processing software, and instant messaging software.

[0068] In order to make the technical solution of the present application clearer and easier to understand, the application scenarios of the focus control method provided by the present application are introduced below.

[0069] like Figure 2As shown, this figure is a schematic diagram of an application scenario provided by an embodiment of the present application. In this scenario, a photographer 211 can use an electronic device 212 to shoot a subject 213. The electronic device 212 can obtain an image frame 220 captured by a camera, and the image frame 220 includes a target object 221 and a background 222, wherein the target object 221 can be a user, and the background 222 can be the scenery behind the user. For example, the background 222 can be a tree, of course, it can also be other plants, or it can be a landscape, etc. Figure 2 In the description, only the background 222 is taken as an example of trees.

[0070] Then, the electronic device can generate multiple candidate focus areas based on the image frame 220. Taking the case where the number of candidate focus areas generated by the electronic device is 3 as an example, such as the first candidate focus area 231, the second candidate focus area 232 and the third candidate focus area 233, wherein the second candidate focus area 232 can be the initial focus area, and the first candidate focus area 231 can be obtained by expanding the second candidate focus area 232, for example, the first candidate focus area 231 is obtained after expanding by 20%, and the third candidate focus area 233 can be obtained by shrinking the second candidate focus area 232, for example, the third candidate focus area 233 is obtained after shrinking by 20%.

[0071] Next, the electronic device may sort the three candidate focus areas according to the confidence level, for example, the three candidate focus areas may be sorted in descending order based on the confidence level to obtain a sorting result. The confidence level of the candidate focus area may be calculated by an autofocus algorithm, and the confidence level may be affected by a variety of factors, such as the proportion of the target object in the focus area, the light, the clarity of the focus area, etc. In some examples, the confidence levels of the three candidate focus areas are sorted from large to small as follows: the third candidate focus area 233, the second candidate focus area 232, and the first candidate focus area 231.

[0072] Exemplarily, the electronic device may determine the candidate focus area with the highest confidence as the target focus area, and then perform autofocus based on the target focus area; or, the electronic device may select any one of the top N candidate focus areas in terms of confidence as the target focus area, and perform autofocus using the target focus area. Wherein, N is a positive integer, and N is less than the total number of candidate focus areas. In the above scenario, the value of N can be 2. The electronic device focuses based on the target focus area to obtain a focused image 240, in which the background 222 is partially blurred and the target object 221 is partially clear. It can be seen that when focusing on the target focus area, the background 222 is not focused on, but on the target object 221.

[0073] In this scenario, the electronic device generates multiple candidate focus areas, which provide more options for the electronic device to perform autofocus compared to one focus area, thereby reducing the situation where a single focus area includes too much background to a certain extent. For example, the third candidate focus area 231 mentioned above covers a smaller part of the background, and then performing autofocus based on the third candidate focus area 231 can reduce the possibility of the electronic device focusing on the background and reduce the situation where the electronic device focuses on the background.

[0074] In order to make the technical solution of the present application clearer and easier to understand, the focus control method provided in the embodiment of the present application is introduced from the perspective of an electronic device in combination with the accompanying drawings.

[0075] like Figure 3 As shown in FIG. 1 , the figure is a flow chart of a focus control method provided in an embodiment of the present application, the method comprising:

[0076] S301: The electronic device acquires an image frame.

[0077] An image frame refers to an image captured by an electronic device through a camera. The image frame may be a record of the captured content. The captured content may refer to the above Figure 2 In some examples, the electronic device can obtain an image frame through data collected by a camera.

[0078] S302: The electronic device generates a plurality of candidate focus areas according to the image frame.

[0079] After the electronic device acquires the image frame, it can generate multiple candidate focus areas. In some examples, the electronic device can first detect the image frame to obtain the position of the target object in the image frame, and then generate multiple candidate focus areas based on the position of the target object. The position of the target object can be represented by coordinates. Exemplarily, the electronic device can determine the position of the above-mentioned target object through a target detection algorithm. In other examples, the electronic device can perform saliency detection on the image frame to obtain the most salient area in the image frame, and then generate multiple candidate focus areas based on the most salient area.

[0080] Exemplarily, the electronic device generates an initial focus area based on the position of the target object, and the center point of the initial focus area is consistent with the position of the target object. The area size of the initial focus area is smaller than the area size of the image frame. For example, the initial focus area can be 55% of the image frame, and the initial focus area can be 45% of the image frame. Those skilled in the art can design a specific ratio based on actual needs, and the specific ratio is not limited in the implementation of this application.

[0081] Next, the electronic device may regenerate at least one focus area in a variety of ways, which are described below.

[0082] The first one:

[0083] Based on the initial focus area, the electronic device can generate at least one focus area in the initial focus area by shrinking, and then the electronic device generates multiple candidate focus areas. There are also multiple ways for the electronic device to generate at least one focus area in the initial focus area, which are described below in conjunction with the accompanying drawings.

[0084] like Figure 4A As shown in FIG. 4 , the figure is a schematic diagram of a retracted candidate focus area provided by an embodiment of the present application. After the electronic device generates the initial focus area 401 , it can generate a focus area, such as the candidate focus area 402 , within the initial focus area 401 .

[0085] It should be noted that the embodiment of the present application does not specifically limit the style of the focus area. The focus area can be a rectangular area or an area of ​​other styles. This embodiment only takes the focus area as a rectangle as an example, and the present application does not limit this.

[0086] like Figure 4B As shown, this figure is a schematic diagram of another indented candidate focus area provided in an embodiment of the present application. After the electronic device generates the initial focus area 401, multiple focus areas can be generated within the initial focus area 401. After the electronic device generates the initial focus area 401, a focus area, such as candidate focus area 403, is generated again by indentation within the initial focus area 401, and a focus area, such as candidate focus area 404, is generated again by indentation within the candidate focus area 403, and so on, until the number of candidate focus areas meets the requirements. Among them, the number of candidate focus areas can be pre-set or calculated in real time.

[0087] like Figure 4C As shown, this figure is a schematic diagram of a tiled candidate focus area provided in an embodiment of the present application. After the electronic device generates an initial focus area 401, multiple focus areas are generated in the initial focus area 401 by tiling, such as candidate focus area 405, candidate focus area 406, candidate focus area 407, and candidate focus area 408. Among them, the number and position of the above-mentioned candidate focus areas are only examples. In actual application, there may be intervals between different candidate focus areas, or there may be no intervals.

[0088] It should be noted that the embodiment of the present application does not specifically limit the method for generating the focus area. In other examples, the electronic device may also generate the focus area. Figure 4B and Figure 4C The method of generating candidate focus areas shown in the figure is combined. For example, the electronic device generates a secondary initial focus area in the initial focus area by shrinking, and then generates multiple focus areas in the secondary initial focus area by tiling.

[0089] Second type:

[0090] Based on the initial focus area, the electronic device can generate at least one focus area outside the initial focus area by expanding the initial focus area, thereby generating multiple candidate focus areas. There are also multiple ways for the electronic device to generate at least one focus area outside the initial focus area, which are described below in conjunction with the accompanying drawings.

[0091] like Figure 4D As shown, this figure is a schematic diagram of an outward-expanding candidate focus area provided in an embodiment of the present application. After the electronic device generates the initial focus area 401, it can generate a focus area outside the initial focus area 401, such as the candidate focus area 409, and the candidate focus area 409 can cover the initial focus area 401.

[0092] like Figure 4E As shown, this figure is a schematic diagram of another outward-expanding candidate focus area provided in an embodiment of the present application. After the electronic device generates the initial focus area 401, multiple focus areas can be generated outside the initial focus area 401. After the electronic device generates the initial focus area 401, it generates a focus area such as candidate focus area 410 by outward expansion outside the initial focus area 401, and generates another focus area such as candidate focus area 411 by outward expansion outside the candidate focus area 410, and so on, until the number of candidate focus areas meets the requirements.

[0093] like Figure 4F As shown, this figure is a schematic diagram of another tiled candidate focus area provided in an embodiment of the present application. After the electronic device generates the initial focus area 401, it generates at least one focus area outside the initial focus area 401 by tiling, such as candidate focus area 412, candidate focus area 413, candidate focus area 414, and candidate focus area 415. Among them, the number and position of the above-mentioned candidate focus areas are only examples. In actual application, there may or may not be gaps between different candidate focus areas. The focus area generated by tiling may not cover the initial focus area 401.

[0094] It should be noted that the embodiment of the present application does not specifically limit the method for generating the focus area. In other examples, the electronic device may also generate the focus area. Figure 4E and Figure 4F The method of generating candidate focus areas shown in the figure is combined. For example, the electronic device generates a secondary initial focus area outside the initial focus area 401 by expanding it, and then generates multiple focus areas outside the secondary initial focus area by tiling it.

[0095] It should be noted that in the embodiment of the present application, the first and second methods of generating multiple candidate focus areas can also be combined, that is, a shrinking method and an expanding method are used. Those skilled in the art can choose a method of generating candidate focus areas based on actual needs, and the present application does not limit this.

[0096] S303: The electronic device determines a target focus area from multiple candidate focus areas.

[0097] For example, the electronic device may determine the confidence of each candidate focus area based on the proportion of the target object in the focus area. Then, the electronic device may determine the target focus area from multiple candidate focus areas based on the confidence of each candidate focus area. Exemplarily, the electronic device may obtain the confidence of the candidate focus area based on the PD algorithm library, and the confidence may be used to characterize the credibility of the PD value of the candidate focus area.

[0098] In some examples, the electronic device may sort the candidate focus areas based on the confidence of each candidate focus area, for example, it may sort them from large to small based on the confidence. After obtaining the sorting result, the electronic device may use the candidate focus area with the highest confidence as the target focus area, or it may select any one of the top N candidate focus areas as the target focus area.

[0099] S304: The electronic device determines a phase difference detection value corresponding to the target focus area.

[0100] Phase Detection (PD) refers to the phase difference between adjacent pixels. The target focus area may include multiple grids, and the PD value of the target focus area may be obtained by weighted averaging the PD values ​​of each grid. The weighting coefficient of each grid may be preset or modified in real time. The PD value of each grid may be determined by the phase difference between two adjacent sub-pixels in the grid, and the two adjacent sub-pixels are usually adjacent in the horizontal direction or in the vertical direction. Exemplarily, the PD value may be calculated by the following formula: PD1 = (LR) / (L + R). PD1 is the phase difference detection value of the grid, L is the light intensity of the left pixel, and R is the light intensity of the right pixel. When a grid includes multiple pixels, the PD value of the grid may be obtained by calculating the average PD value of all pixels in the grid. In engineering implementation, the electronic device may automatically calculate the phase difference detection value corresponding to the target focus area.

[0101] S305: The electronic device adjusts the position of the motor according to the phase difference detection value to perform focusing.

[0102] The motor may be an autofocus motor, which is one of the components of the lens module and is used to control the focus and zoom of the lens.

[0103] like Figure 5 As shown in the figure, this figure is a schematic diagram of determining the motor position provided by an embodiment of the present application. Among them, PDrawl and PD rawr can be images in RAW format, and the image in RAW format refers to the original image data that has not been compressed and processed, and can be the original image data directly output by a camera or other digital imaging device. The image in RAW format includes information such as brightness and color. Among them, PD rawl and PD rawr can correspond to different areas in an image respectively. For example, PD rawl can be the left half of the RAW image, and PD rawr can be the right half of the RAW image.

[0104] PDbib is an image processing library used in computer vision, which can be used for image processing and analysis in cameras. For example, the PD rawl and PD rawr are processed based on PDlib to obtain the PD value of the target focus area.

[0105] DCC stands for Defocus Conversion Coefficient, a parameter used to calculate the depth information of a camera lens. When a camera lens is in focus, it captures a sharp image, and when it is out of focus, the image becomes blurry. DCC is generally between 0.1 and 0.5. The larger the DCC, the more accurate the depth information.

[0106] Defocus refers to the amount of defocus. Electronic devices can determine the amount of defocus based on the PD value and DCC of the target focus area.

[0107] After obtaining the defocus amount, the electronic device can determine the target position of the motor based on the defocus amount and the current position of the motor, for example, by adding the current position of the motor and the defocus amount to obtain the target position of the motor. Then, the electronic device adjusts the motor according to the target position, that is, adjusts the motor to the target position, thereby achieving focus.

[0108] Based on the above description, an embodiment of the present application provides a focus control method, in which an electronic device generates multiple candidate focus areas based on an image frame. Compared with generating only one candidate focus area, multiple candidate focus areas provide more choices for the electronic device during the automatic focusing process, thereby reducing the situation where a single focus area includes too much background to a certain extent. The electronic device selects a target focus area with a higher confidence level from the multiple candidate focus areas, and uses the target focus area to focus on the target object, which can reduce the situation where the electronic device focuses on the background.

[0109] The present application also provides a focus control method. Figure 3 Compared with the method shown in FIG. 1 , the method distinguishes the types of target objects, which will be described below in conjunction with the accompanying drawings. Figure 6 As shown in FIG. 1 , this figure is a flow chart of another focus control method provided in an embodiment of the present application, the method comprising:

[0110] S601: The electronic device acquires an image frame.

[0111] It should be noted that the manner in which the electronic device acquires the image frame can be referred to in the above S301 and will not be described in detail here.

[0112] S602, the electronic device determines whether the image frame includes a human face; if so, execute S603, if not, execute S604.

[0113] When the image frame includes a human face or does not include a human face, the electronic device may generate a candidate focus area in different ways.

[0114] In some examples, the electronic device can detect the image frame based on the face detection algorithm, and then determine whether the image frame includes a face. Exemplarily, the image frame can be preprocessed first, for example, the image frame can be scaled, grayed, etc., and then feature extraction is performed, and then the extracted features are sent to a pre-trained separator to obtain a result of whether the image frame includes a face.

[0115] If the image frame includes a human face, the electronic device executes S603 ; if the image frame does not include a human face, the electronic device executes S604 .

[0116] S603: The electronic device generates a plurality of candidate focus areas in the face area and / or body area.

[0117] like Figure 7 As shown, this figure is a schematic diagram of an image frame including a face provided by an embodiment of the present application. The image frame includes a target object 711 and a background 712, wherein the target object 711 may be a person, and the target object 711 includes a face area 713 and a body area 714.

[0118] In some examples, the electronic device may generate multiple candidate focus areas in the face area 713, such as candidate focus area 721 and candidate focus area 722, wherein candidate focus area 721 may overlap with the face area 713, and candidate focus area 722 may be a focus area obtained by shrinking the candidate focus area 721.

[0119] In other examples, the electronic device may also generate multiple candidate focus areas in the body area 714, such as candidate focus area 723 and candidate focus area 724, wherein candidate focus area 723 may be obtained by moving the face area 713 downward, and candidate focus area 724 may be a focus area obtained by shrinking the candidate focus area 723.

[0120] In other examples, the electronic device may generate at least one candidate focus area in the face area 713 and at least one candidate focus area in the body area 714. For example, candidate focus area 725 and candidate focus area 726 are generated in the face area 713, and candidate focus area 727 is generated in the body area 714.

[0121] It should be noted that the present application does not specifically limit the number of candidate focus areas generated, and the electronic device can generate multiple candidate focus areas.

[0122] S604: The electronic device generates a plurality of candidate focus areas in the salient area.

[0123] like Figure 8As shown, the figure is a schematic diagram of an image frame that does not include a face provided by an embodiment of the present application. The image frame includes a target object 811 and a background 812, and the target object can be grass 813. Among them, grass 813 can be a salient area. In some examples, the electronic device can use a saliency detection algorithm to perform saliency detection on the image frame, and then obtain a salient area in the image frame, that is, grass 813. After the electronic device determines the salient area, it can generate multiple candidate focus areas in the salient area.

[0124] In some examples, the electronic device may generate multiple candidate focus areas in the area of ​​grass 813, such as candidate focus area 814, candidate focus area 815, candidate focus area 816, candidate focus area 817, and candidate focus area 818. Specifically, the electronic device generates candidate focus area 814 in the area of ​​grass 813, and then generates candidate focus area 815, candidate focus area 816, candidate focus area 817, and candidate focus area 818 within the candidate focus area 814 by tiling.

[0125] It should be noted that the above-mentioned method of generating candidate focus areas is merely an exemplary introduction, and candidate focus areas may also be generated in other ways, such as a retracting method, an expanding method, and the like.

[0126] S605: The electronic device calculates the PD value of each candidate focus area.

[0127] The way in which the electronic device calculates the PD value of each candidate focus area can refer to the way in which the electronic device determines the PD value of the target focus area, such as S304 above, and will not be repeated here. In other embodiments, the electronic device may also perform calculations when the PD value of the candidate focus area is needed. Since in some examples, the electronic device will remove some candidate focus areas, but the PD value is not used at this time, there is no need to calculate the PD value, thereby reducing the amount of calculations and improving processing efficiency.

[0128] S606: The electronic device determines the confidence level of each candidate focus area.

[0129] The electronic device may determine the confidence of the candidate focus area. In some examples, the candidate focus area may be divided into multiple grids, each grid having various corresponding confidences and weights, and the electronic device may perform weighted average on the confidence of each grid to obtain the confidence of the candidate focus area. Based on a similar approach, the electronic device may determine the confidence of each candidate focus area.

[0130] S607: The electronic device screens candidate focus areas according to the confidence threshold.

[0131] After determining the confidence of each candidate focus area, the electronic device can screen each candidate focus area based on the confidence of each candidate focus area and the confidence threshold. Exemplarily, when the confidence of a candidate focus area is lower than the confidence threshold, the electronic device removes the candidate focus area and retains the remaining candidate focus areas, thereby removing the candidate focus areas with poor reliability.

[0132] S608. The electronic device determines whether the depth device is valid; if so, execute S609; if not, execute S610.

[0133] Exemplarily, the electronic device can determine whether the depth device is valid by judging whether the depth device is in place. When the depth device is in place, the depth device is considered valid, and when the depth device is not in place, the depth device is considered invalid. Among them, the depth device in place means that the electronic device includes a depth device, and the depth device is valid means that the depth device can output depth information. The depth device of the electronic device is used to obtain the depth information of each position in the image frame, and the depth interval can be achieved in a variety of different ways, such as structured light, time of flight, and binocular vision.

[0134] If the electronic device determines that the depth device is valid, S609 is further executed. If the electronic device determines that the depth device is invalid, S610 is executed.

[0135] In some embodiments, the above S605 may also be executed after S608.

[0136] S609: The electronic device screens the candidate focus areas again according to the PD value of each candidate focus area and the depth information output by the depth device.

[0137] When the electronic device determines that the depth device is valid, the electronic device can obtain the depth information of the candidate focus area based on the depth device, and then determine the quasi-focus position corresponding to the depth information based on the depth information and the predetermined correspondence between the depth information and the quasi-focus position, and then compare the quasi-focus position with the PD value of the candidate focus area. For example, the quasi-focus position and the PD value may be subjected to difference processing to obtain a difference. If the difference is greater than or equal to a preset threshold, the candidate focus area is removed. If the difference is less than the preset threshold, the candidate focus area is retained. Based on a similar principle, the electronic device can perform similar processing on each candidate focus area, thereby screening the candidate focus areas again. The preset threshold may be the first threshold.

[0138] This method further screens the candidate focus areas through the depth information output by the depth device, removes the candidate focus areas with poor reliability, and thus further reduces the possibility of focusing on the background.

[0139] S610: The electronic device removes candidate focus areas with multiple depths of field according to the PD values ​​of the screened candidate focus areas.

[0140] If the electronic device determines that the depth device is invalid, then S610 is directly executed. If the electronic device determines that the depth interval is valid, then S609 is first executed, and then S610 is executed. In this way, the selected candidate focus area can be the candidate focus area retained in S609 or the candidate focus area retained in S607.

[0141] After determining the selected candidate focus areas, the electronic device can remove the candidate focus areas with multiple depths of field based on the PD values ​​of the selected candidate focus areas. Exemplarily, the electronic device can determine whether the candidate focus area is a multiple depth of field scene based on the PD values ​​of the candidate focus areas.

[0142] In some examples, the candidate focus area may be composed of multiple grids, each grid has its own corresponding PD value, and then the electronic device may compare the PD value of the grid with the distant view threshold. When the PD value of the grid is greater than or equal to the distant view threshold, the grid is determined to be distant, and when the PD value of the grid is less than the distant view threshold, the grid is determined to be near. Next, the electronic device may average the PD values ​​of the distant grids to obtain an average PD value of the distant grids, average the PD values ​​of the near grids to obtain an average PD value of the near grids, and then perform a difference process on the average PD value of the distant grids and the average PD value of the near grids to obtain a difference. If the difference is greater than or equal to the average value threshold, it indicates that the candidate focus area is a multi-depth scene. If the difference is less than the average value threshold, it indicates that the candidate focus area is not a multi-depth scene. Among them, the distant view threshold is the second threshold. In this example, the first statistical value and the second statistical value are consistent, both of which are average values.

[0143] In other examples, the electronic device may also perform a difference process on the maximum and minimum PD values ​​of the grid in the candidate focus area to obtain a difference. If the difference is greater than or equal to the difference threshold, the candidate focus area is characterized as a multi-depth scene. If the difference is less than the difference threshold, the candidate focus area is not characterized as a multi-depth scene. Among them, the difference threshold and the average value threshold can both be the third threshold, and the above average value, maximum value, and minimum value are all statistical values. In this example, the first statistical value is the maximum value, and the second statistical value is the minimum value. It can be seen that the first statistical value and the second statistical value may be the same or different.

[0144] It should be noted that the present application does not specifically limit the implementation method of determining the scene of the candidate focus area. The above two methods are merely exemplary introductions. Those skilled in the art may choose other methods for determination based on actual needs.

[0145] After the electronic device determines the scene of the candidate focus area, it can remove the candidate focus areas of the multi-depth of field scene, including the candidate focus areas that are not multi-depth of field scenes. The candidate focus areas that are not multi-depth of field scenes include fewer background areas, which can further reduce the situation where the electronic device focuses on the background.

[0146] S611: The electronic device determines a target focus area according to the confidence levels of the remaining candidate focus areas.

[0147] After the electronic device completes the above screening of the candidate focus areas, the electronic device can determine the target focus area from the remaining candidate focus areas based on the confidence of the candidate focus areas. The specific process can be referred to in the above S303 and will not be repeated here.

[0148] S612: The electronic device performs automatic focusing according to the target focusing area.

[0149] The process of the electronic device performing automatic focusing according to the target focusing area can be referred to above S304-S305, which will not be described in detail here. Fig. 9 As shown in FIG. 1 , the figure is a schematic diagram of a focusing effect provided by an embodiment of the present application. It can be seen from the figure that when the image frame includes a face, the electronic device can successfully focus on the face area and body area of ​​the target object instead of focusing on the background area, thereby improving the clarity of the face area and body area of ​​the target object. Fig.10 As shown, the figure is another schematic diagram of the effect after focusing provided in an embodiment of the present application. It can be seen from the figure that when the image frame does not include a face, the electronic device can successfully focus on the salient area (such as grass) instead of focusing on the background area, thereby improving the clarity of the salient area.

[0150] Based on the above description, an embodiment of the present application provides a focus control method, which further distinguishes whether the image frame includes a face, and then selects different methods to generate multiple candidate focus areas, so that the generated candidate focus areas are more adapted to the target object in the image frame, thereby improving the reliability of the generated candidate focus areas. Then, the method screens the candidate focus areas in a variety of ways to further remove candidate focus areas with poor reliability. Finally, the electronic device determines the target focus area from the remaining candidate focus areas, and uses the target focus area for automatic focusing, which can further reduce the situation of focusing on the background, thereby improving the quality of the captured image.

[0151] It should be noted that the above is only an introduction to the focus control method provided in the embodiment of the present application from the perspective of an electronic device. Exemplarily, the operating system of the electronic device may include a hardware abstraction layer, and the focus control method may be executed by a focus control module in the hardware abstraction layer. The present application does not specifically limit the execution subject of the method, and the method may also be executed by other software modules.

[0152] This embodiment also provides a computer-readable storage medium, which includes instructions. When the instructions are executed on an electronic device, the electronic device executes Figure 3 and Figure 6 The relevant method steps in the embodiment are used to implement the method in the above embodiment.

[0153] This embodiment also provides a computer program product including instructions. When the computer program product is run on an electronic device, the electronic device executes the following Figure 3 and Figure 6 The relevant method steps in the embodiment are used to implement the method in the above embodiment.

[0154] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A focus control method, characterized in that: include: Get image frame; Generating an initial focus area by executing a target detection algorithm or a saliency detection algorithm on the image frame; On the basis of the initial focus area, a plurality of candidate focus areas are generated by shrinking or expanding, and at least two of the candidate focus areas have different proportions of the target object in the respective candidate focus areas; If the depth information of the plurality of candidate focus areas can be obtained, determining the in-focus position corresponding to each candidate focus area according to the depth information and a correspondence between the preset depth information and the in-focus position; Screening the plurality of candidate focus areas according to a difference between the in-focus position and the phase difference detection value corresponding to each candidate focus area and a first threshold; Determining a distant view grid and a near view grid in each candidate focus area according to the phase difference detection value of each grid in the respective candidate focus area and a second threshold; Determining whether each candidate focus area is a multi-depth of field scene according to a first statistic of phase difference detection values ​​of distant view grids and a second statistic of phase difference detection values ​​of near view grids in each candidate focus area and a third threshold; From the screened candidate focus areas, retain the candidate focus areas that are not multi-depth-of-field scenes; Determining a target focus area according to the confidence of the candidate focus area that is not a multi-depth of field scene; The target object is focused using the target focusing area.

2. The method according to claim 1, characterized in that The method further comprises: Based on the initial focus area, multiple candidate focus areas are generated by tiling.

3. The method according to claim 1, characterized in that: Generating an initial focus area according to the image frame includes: If the image frame includes a human face, an initial focus area is generated in the human face area and / or body area.

4. The method according to claim 1, characterized in that Generating an initial focus area according to the image frame includes: If the image frame does not include a human face, performing saliency detection on the image frame to obtain a salient area; In the saliency area, an initial focus area is generated.

5. The method according to claim 1, characterized in that The method further comprises: If the depth information of the plurality of candidate focus areas cannot be obtained, determining the distant view grid and the near view grid in the respective candidate focus areas according to the phase difference detection value of each grid in the respective candidate focus areas and the second threshold; Determining whether each candidate focus area is a multi-depth of field scene according to a first statistic of phase difference detection values ​​of distant view grids and a second statistic of phase difference detection values ​​of near view grids in each candidate focus area and a third threshold; From the plurality of candidate focus areas, retaining candidate focus areas that are not multi-depth-of-field scenes; A target focus area is determined according to the confidence of the candidate focus area that is not a multi-depth of field scene.

6. The method according to any one of claims 1 to 5, characterized in that: The target focus area is a candidate focus area with the highest confidence or any one of the first N candidate focus areas after sorting the confidence from large to small, where N is a positive integer.

7. The method according to any one of claims 1 to 6, characterized in that: The confidence of the candidate focus area is obtained by: The confidence level of the candidate focus area is determined based on the proportion of the target object in the candidate focus area to the candidate focus area.

8. An electronic device, characterized in that: include: Camera, processor and memory; The camera is used to collect image frames; One or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 7.

9. A computer storage medium, characterized in that: The method comprises computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 7.

Images