Method, device, electronic device and computer-readable storage medium for controlling focus

Abstract

The embodiments of the present application disclose a method, device, electronic device and computer-readable storage medium for controlling focus. The method includes: determining the motion state of the target object based on a historical frame image; analyzing the defocus amount of the current frame image based on the motion state, wherein the historical frame image is an image captured by a camera before the current frame image; if the defocus amount is determined to be a normal defocus amount based on the analysis result, then controlling the focus module to perform a focusing operation based on the defocus amount. The above-mentioned method, device, electronic device and computer-readable storage medium for controlling focus can improve the accuracy of autofocus, improve the imaging effect of the target object, and prevent the lens from being pushed or pulled.

Description

Technical Field

[0001] The present application relates to the field of imaging technology, and in particular to a method, device, electronic device and computer-readable storage medium for controlling focus. Background Art

[0002] When the camera is capturing images, it needs to focus first to ensure that the target object is clearly imaged in the image. For a target object in motion, the camera needs to track the target object in real time and determine the actual object distance of the target object, so as to move the lens to the focal plane where the target is clear and ensure that the moving target is clearly focused.

[0003] However, in actual use of the camera device, in order to ensure the imaging effect after the image of the target object is captured, the captured image usually needs to be fully processed first, including autofocus, auto exposure, white balance processing and noise reduction, etc. Therefore, the tracking process of the target object will be delayed, and the tracking result of the target object by the camera device will deviate from the actual position of the target object, resulting in inaccurate autofocus, which reduces the imaging effect of the target object. Summary of the invention

[0004] The embodiments of the present application disclose a method, device, electronic device and computer-readable storage medium for controlling focus, which can improve the accuracy of autofocus, improve the imaging effect of the target object, and prevent the lens from being pushed or pulled.

[0005] The embodiment of the present application discloses a method for controlling focus, including:

[0006] Determine the motion state of the target object according to the historical frame images;

[0007] Analyzing the defocus amount of the current frame image according to the motion state, the historical frame image is an image captured by a camera before the current frame image;

[0008] If it is determined according to the analysis result that the defocus amount is a normal defocus amount, the focusing module is controlled to perform a focusing operation according to the defocus amount.

[0009] The embodiment of the present application discloses a device for controlling focus, comprising:

[0010] A state determination module, used to determine the motion state of the target object according to the historical frame image;

[0011] An analysis module, used for analyzing the defocus amount of the current frame image according to the motion state, wherein the historical frame image is an image captured by a camera before the current frame image;

[0012] The control module is used for controlling the focusing module to perform a focusing operation according to the defocus amount if it is determined according to the analysis result that the defocus amount is a normal defocus amount.

[0013] An embodiment of the present application discloses an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements the method described above.

[0014] An embodiment of the present application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described above is implemented.

[0015] The method, device, electronic device and computer-readable storage medium for controlling focus disclosed in the embodiments of the present application determine the motion state of the target object based on the historical frame image, analyze the defocus amount of the current frame image based on the motion state, and if the defocus amount is determined to be a normal defocus amount based on the analysis result, control the focus module to perform a focusing operation based on the defocus amount, and can detect whether the defocus amount of the current frame image is normal based on the motion state. Focusing is performed only when the defocus amount of the current frame image is a normal defocus amount, thereby improving the accuracy of autofocus and the imaging effect of the target object, and preventing the lens from pushing or pulling due to the abnormal defocus amount of the current frame image, thereby improving the stability of focusing. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1A A block diagram of an electronic device implementing target tracking in an ideal state in an embodiment;

[0018] Figure 1B A block diagram of an electronic device implementing target tracking in actual use in an embodiment;

[0019] Figure 2 is a block diagram of an image processing circuit in one embodiment;

[0020] Figure 3 is a flow chart of a method for controlling focus in one embodiment;

[0021] Figure 4 is a flow chart of a method for controlling focus in another embodiment;

[0022] Figure 5Ais a schematic diagram of analyzing the defocus amount according to the motion state of a target object in one embodiment;

[0023] Figure 5B is a schematic diagram of analyzing the defocus amount according to the motion state of the target object in another embodiment;

[0024] Figure 6 is a flow chart of a method for controlling focus in another embodiment;

[0025] Figure 7 is a block diagram of a device for controlling focus in one embodiment;

[0026] Figure 8 FIG. 4 is a structural block diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present application and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0029] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first region of interest may be referred to as a second region of interest, and similarly, a second region of interest may be referred to as a first region of interest. Both the first region of interest and the second region of interest are regions of interest, but they are not regions of interest in the same frame image.

[0030] Figure 1A FIG. 1 is a block diagram of an electronic device implementing target tracking in an ideal state in an embodiment. Figure 1AAs shown, the electronic device may include an image sensor 110, a target determination module 120, a tracking module 130 and a focusing module 140. Focus tracking refers to tracking and focusing on a moving target. The target determination module 120 can be used to determine the target object to be tracked. During the movement of the target object, the image sensor 110 can capture image data. The tracking module 130 can determine the position of the target object in each frame of the image based on the image data captured by the image sensor 110, and track the target object. The focusing module 140 can focus based on the tracking result of the tracking module 130, and can use the position of the target object in the image output by the tracking module 130 as the focus area to determine the actual object distance of the target object, thereby moving the lens to a focal plane where the target is clear to ensure that the target object is clearly focused.

[0031] In an ideal focus tracking process, the focus module 140 is placed after the tracking module 130, so that the tracking result received by the focus module 140 each time is the position of the target object in the current frame image, thereby ensuring accurate focus. However, in the actual focus tracking process, in order to ensure the imaging effect, it is usually required that the image data captured by the image sensor 110 is fully processed, and the processing may include but is not limited to auto focus, auto exposure, white balance processing and noise reduction, etc. Therefore, the tracking module 130 is placed after the focus module 140.

[0032] Figure 1B FIG. 1 is a block diagram of an electronic device implementing target tracking in actual use in an embodiment. Figure 1B As shown, after the tracking module 130 is placed in the focusing module 140, the focusing module 140 is in the front position during the image acquisition process, which will cause the tracking result output by the tracking module 130 to be delayed and fed back to the focusing module 140, that is, the tracking result output by the tracking module 130 is the tracking result of the previous frames of images, rather than the tracking result of the target object in the image that the focusing module 140 currently needs to focus on. For a target object in a moving state, the delayed feedback of the tracking module 130 will cause a deviation between the tracking result of the target object and the actual position of the target object, resulting in an inaccurate focus area of ​​the focusing module 140, and the target object will not be clear in the captured image, thereby reducing the imaging effect of the target object. And because the focusing module 140 focuses on the current frame image based on the tracking results of the previous frames of images, when there is a large deviation between the focus area and the actual position of the target object, it will cause the lens to have a push-pull effect, and the focus tracking process will be unstable.

[0033] A method, device, electronic device and computer-readable storage medium for controlling focus are provided in the embodiments of the present application. Whether the defocus amount of the current frame image is normal can be detected according to the motion state. Focusing is performed only when the defocus amount of the current frame image is a normal defocus amount, thereby improving the accuracy of autofocus and the imaging effect of the target object, and preventing the lens from being pushed or pulled due to the abnormal defocus amount of the current frame image, thereby improving the stability of focus.

[0034] An embodiment of the present application provides an electronic device. The electronic device includes an image processing circuit, which can be implemented using hardware and / or software components and can include various processing units that define an ISP (Image Signal Processing) pipeline. Figure 2 FIG. 1 is a block diagram of an image processing circuit in one embodiment. For ease of explanation, Figure 2 Only various aspects of the image processing technology related to the embodiments of the present application are shown.

[0035] like Figure 2 As shown, the image processing circuit includes an ISP processor 240 and a control logic 250. The image data captured by the imaging device 210 is first processed by the ISP processor 240, and the ISP processor 240 analyzes the image data to capture image statistics that can be used to determine one or more control parameters of the imaging device 210. The imaging device 210 may include one or more lenses 212 and an image sensor 214. The image sensor 214 may include a color filter array (such as a Bayer filter), and the image sensor 214 may obtain light intensity and wavelength information captured by each imaging pixel, and provide a set of raw image data that can be processed by the ISP processor 240. The posture sensor 220 (such as a three-axis gyroscope, a Hall sensor, an accelerometer, etc.) can provide the collected image processing parameters (such as anti-shake parameters) to the ISP processor 240 based on the interface type of the posture sensor 220. The posture sensor 220 interface can use a SMIA (Standard Mobile Imaging Architecture) interface, other serial or parallel camera interfaces, or a combination of the above interfaces.

[0036] It should be noted that although Figure 2 Only one imaging device 210 is shown in the figure, but in the embodiment of the present application, at least two imaging devices 210 may be included, each imaging device 210 may correspond to one image sensor 214, or multiple imaging devices 210 may correspond to one image sensor 214, which is not limited here. The working process of each imaging device 210 may refer to the above description.

[0037] In addition, the image sensor 214 may also send raw image data to the gesture sensor 220 , and the gesture sensor 220 may provide the raw image data to the ISP processor 240 based on the interface type of the gesture sensor 220 , or the gesture sensor 220 may store the raw image data in the image memory 230 .

[0038] The ISP processor 240 processes the raw image data pixel by pixel in a variety of formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the ISP processor 240 may perform one or more image processing operations on the raw image data and collect statistical information about the image data. The image processing operations may be performed at the same or different bit depth precisions.

[0039] The ISP processor 240 may also receive image data from the image memory 230. For example, the gesture sensor 220 interface sends raw image data to the image memory 230, and the raw image data in the image memory 230 is then provided to the ISP processor 240 for processing. The image memory 230 may be a part of a memory device, a storage device, or an independent dedicated memory within an electronic device, and may include a DMA (Direct Memory Access) feature.

[0040] When receiving raw image data from the image sensor 214 interface or from the posture sensor 220 interface or from the image memory 230, the ISP processor 240 may perform one or more image processing operations, such as time domain filtering. The processed image data may be sent to the image memory 230 for further processing before being displayed. The ISP processor 240 receives the processed data from the image memory 230 and performs image data processing in the original domain and in the RGB and YCbCr color spaces on the processed data. The image data processed by the ISP processor 240 may be output to the display 260 for viewing by the user and / or further processed by a graphics engine or GPU (Graphics Processing Unit). In addition, the output of the ISP processor 240 may also be sent to the image memory 230, and the display 260 may read the image data from the image memory 230. In one embodiment, the image memory 230 may be configured to implement one or more frame buffers.

[0041] The statistical data determined by the ISP processor 240 may be sent to the control logic 250. For example, the statistical data may include image sensor 214 statistical information such as vibration frequency of a gyroscope, automatic exposure, automatic white balance, automatic focus, flicker detection, black level compensation, lens 212 shading correction, etc. The control logic 250 may include a processor and / or microcontroller that executes one or more routines (such as firmware), and the one or more routines may determine control parameters of the imaging device 210 and control parameters of the ISP processor 240 based on the received statistical data. For example, the control parameters of the imaging device 210 may include attitude sensor 220 control parameters (such as gain, integration time for exposure control, anti-shake parameters, etc.), camera flash control parameters, camera anti-shake displacement parameters, lens 212 control parameters (such as focal length for focus or zoom), or a combination of these parameters. The ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (for example, during RGB processing), and lens 212 shading correction parameters.

[0042] For example, in combination Figure 2 The image processing circuit of the present application is used to illustrate the method for controlling focus provided in the embodiment of the present application. The image sensor 214 in the image processing circuit can send the captured raw image data to the ISP processor 240, and the ISP processor 240 can process the raw image data sent by the image sensor 214 to obtain an image that can be displayed on the display 260. The ISP processor 240 can identify the target object area in each frame image obtained by processing. Optionally, after the ISP processor 240 identifies the target object area of ​​the current frame image, the motion state of the target object can be determined based on the difference between the target object area of ​​the current frame image and the target object area of ​​the previous frame image. At the same time, the focus area of ​​the next frame (or the next N frames) of the image can be determined based on the target object area of ​​the current frame image, and the focus area is the area of ​​interest of the next frame image.

[0043] When the ISP processor 240 receives the raw image data of the next frame image sent by the image sensor 214, the next frame image is used as the new current frame image. It can determine whether the region of interest of the current frame (i.e., the focus area determined by the previous frame image) is a normal region of interest based on the lens stabilization strategy and motion state. The normal region of interest can be understood as the target object being within the region of interest, that is, it is accurate to use the region of interest as the focus area. If the region of interest of the current frame is a normal region of interest, the ISP processor 240 can send the region information of the region of interest of the current frame to the control logic 250, and the control logic 250 determines the focus area based on the region information of the region of interest, and generates corresponding lens control parameters to control the movement of the lens, so that the imaging device 210 focuses on the focus area, thereby achieving accurate tracking of the target device.

[0044] If the ROI of the current frame is an abnormal ROI, it means that the target object is not in the ROI or only a part of it is in the ROI, that is, it is inaccurate to use the ROI as the focus area. The IPS processor 240 can control the lens to remain stationary through the control logic 250 and not perform a focus operation based on the ROI.

[0045] Further, the ISP processor 240 can analyze the defocus amount of the current frame image according to the motion state. If the defocus amount is determined to be a normal defocus amount according to the analysis result, it can be explained that the region of interest of the current frame is a normal region of interest, and the control logic 250 can generate corresponding lens control parameters according to the defocus amount to control the movement of the lens, thereby performing a focusing operation. If the defocus amount is determined to be an abnormal defocus amount according to the analysis result, the defocus amount can be filtered to keep the lens still.

[0046] like Figure 3 As shown, in one embodiment, a method for controlling focus is provided, which can be applied to the above-mentioned electronic devices, which may include but are not limited to mobile phones, tablet computers, smart wearable devices, vehicle-mounted terminals, monitoring systems, laptop computers and PCs (Personal Computers), etc. The electronic device may be provided with one or more cameras, and may also be connected to one or more external camera devices. The image processing method may include the following steps:

[0047] Step 310: determine the motion state of the target object according to the historical frame images.

[0048] The target object may refer to the object of focus of the electronic device, that is, the main shooting object of the camera. The target object may include but is not limited to various objects such as people, animals, inanimate objects, etc. The target object may be the entire object or only a part of the object. For example, the target object may be a whole person or a person's face. The target object may be an entire building or a door or floor of a building.

[0049] The image sensor captures the raw data of the image and sends the captured image to the processor. The image most recently captured by the image sensor can be used as the current frame image, and the historical frame image can refer to the image collected by the camera before the current frame image. The historical frame image can be the original image captured by the image sensor or the image obtained after processing by the processor.

[0050] In some embodiments, the electronic device may determine the motion state of the target object frame by frame. After acquiring each frame of image, the processor of the electronic device may identify the target object in each frame of image through the above-mentioned tracking module 130, and determine the target object area of ​​the target object in each frame of image. The current motion state of the target object may be determined based on the target object area of ​​the target object in multiple consecutive frames of image.

[0051] Exemplarily, the target object area in the acquired Nth frame image can be compared with the target object area in one or more frames of images before the Nth frame image (such as the N-1th frame image, the N-2th frame image, etc.), and the motion state of the target object can be determined based on the comparison result. The motion state of the target object can be used to verify whether the area of ​​interest of the next image that needs to be focused (such as the N+1th frame image) is accurate.

[0052] Step 320 , analyzing the defocus amount of the current frame image according to the motion state, and judging whether the defocus amount is a normal defocus amount according to the analysis result, if so, executing step 330 , if not, executing step 340 .

[0053] After identifying the target object in each frame of the image, the tracking module 130 can determine the target object area of ​​the target object in each frame of the image, and determine the region of interest of the next image that needs to be focused based on the determined target object area. Since the tracking module 130 is placed after the focusing module 140, there is a frame difference in the tracking result fed back by the tracking module 130 to the focusing module 140. In some embodiments, after the tracking module 130 identifies the target object area in the image, the target object area can be fed back to the focusing module 140, and the focusing module 140 can use the target object area as the region of interest of the image that currently needs to be focused, and the region of interest refers to the area to be focused. That is, the region of interest of the image that currently needs to be focused is determined based on the target object area in the previous historical frame image.

[0054] For example, after the tracking module 130 identifies the target object region of the Nth frame image, the target object region of the Nth frame image may be used as the region of interest of the N+1th frame image.

[0055] The defocus amount can be used to indicate the degree of deviation of the target object relative to the focusing plane of the camera head. The larger the defocus amount, the greater the degree of deviation. When the defocus amount is large, the camera may consider that the image is currently out of focus, and thus drive the lens to refocus. After the image sensor collects the original image data of the current frame image, the blur parameter corresponding to the current frame image can be calculated based on the original image data, and the defocus amount of the current frame image can be determined based on the blur parameter. The blur parameter can be used to characterize the blur degree of the current frame image. The larger the defocus amount, the greater the blur degree.

[0056] In some embodiments, one or more lens stabilization strategies may be pre-set. The lens stabilization strategy may be used to determine whether the lens needs to be moved, thereby ensuring that the lens does not experience unstable situations such as pushing and pulling. The lens stabilization strategy may be set according to actual needs. When the target object is in different motion states, different lens stabilization strategies may be used to correspond to each other.

[0057] Whether the defocus amount of the current frame image is a normal defocus amount can be judged according to the lens stabilization strategy corresponding to the motion state, so as to determine whether the first region of interest of the current frame image is a normal region of interest. Optionally, the normal region of interest refers to a region where there is no deviation or a small deviation between the first region of interest of the current frame image and the actual position of the target object. For example, when the target object is stationary or moves on the same plane, the depth information of the target object between adjacent frame images will not change, so the resulting defocus amount should be small. If the defocus amount of the current frame image is a large defocus amount at this time, it can be explained that the defocus amount is an abnormal defocus amount, that is, there is a large deviation between the first region of interest of the current frame image and the actual position of the target object, and the first region of interest is inaccurate and is an abnormal region of interest.

[0058] The electronic device can determine whether the defocus amount of the current frame image is a normal defocus amount based on a preset lens stabilization strategy and the motion state of the target object. If the defocus amount is a normal defocus amount, it means that focusing on the first area of ​​interest of the current frame image is accurate, and the focus module can be controlled to perform a focusing operation based on the first area of ​​interest.

[0059] If the defocus amount of the current frame image is an abnormal defocus amount, it means that there is a large deviation between the first area of ​​interest of the current frame image and the actual position of the target object. If the focus is performed on the first area of ​​interest, the focus will be on the non-target object area (such as the background area, etc.), and the focus will be re-focused back to the target object area in the next frame, thereby producing a lens push-pull effect, and the focus tracking process is unstable. Therefore, if it is determined that the defocus amount of the current frame image is an abnormal defocus amount, the focus module can be controlled not to perform the focus operation, and continue to perform steps 310 to 340. The tracking module identifies the target object in the current frame image, determines the target object area in the current frame image, determines the motion state of the target object based on the target object area of ​​the current frame image, and determines the area of ​​interest of the next frame image. The current frame image is the historical frame image of the next frame image, and the next frame image is used as the new current frame image. When the defocus amount of the new current frame image is a normal defocus amount, the focus operation is performed to continue to track the target object.

[0060] Step 330: Control the focus module to perform a focus operation based on the first region of interest.

[0061] In some embodiments, the focus module may perform the focus operation using a focus method such as phase focus or contrast focus. In the phase focus method, the lens position of the camera may be adjusted according to the phase difference information of each pixel point contained in the focus area in the original image data captured by the image sensor, so that the phase difference of each pixel point contained in the focus area is 0, which indicates that the focus area is a clear image area, thereby achieving the focus on the focus area. In the contrast focus method, the clarity of the focus area in the original image data captured by the image sensor may be calculated, and the lens position of the camera may be adjusted based on the clarity. A hill climbing algorithm may be used to find the best lens position, and the focus area corresponding to the best lens position has the highest clarity, thereby achieving the focus on the focus area. It should be noted that the focus module may also perform the focus operation in other ways, which are not limited here.

[0062] Step 340: Filter the defocus amount so that the focus module does not re-execute the focus operation.

[0063] If the defocus amount of the current frame image is an abnormal defocus amount, the defocus amount can be filtered. Optionally, filtering the defocus amount may include setting the defocus amount to 0, or setting the defocus amount to a fixed value less than a first defocus amount threshold, etc., wherein the first defocus amount threshold is the defocus amount threshold that triggers the movement of the lens. When the defocus amount of the image is greater than the first defocus amount threshold, the lens will be triggered to move and re-focus.

[0064] The electronics can filter out abnormal amounts of defocus, keeping the lens still and preventing the focus module from having to refocus, thus preventing the lens from pushing or pulling.

[0065] In an embodiment of the present application, the motion state of the target object is determined based on the historical frame image, and the defocus amount of the current frame image is analyzed based on the motion state. If the defocus amount is determined to be a normal defocus amount based on the analysis result, the focus module is controlled to perform a focusing operation based on the defocus amount. Whether the defocus amount of the current frame image is normal can be detected based on the motion state. Focusing is performed only when the defocus amount of the current frame image is a normal defocus amount, thereby improving the accuracy of autofocus and the imaging effect of the target object, and preventing the lens from pushing and pulling due to the abnormal defocus amount of the current frame image, thereby improving the focusing stability.

[0066] like Figure 4 As shown, in one embodiment, another method for controlling focus is provided, which can be applied to the above electronic device, and the method may include the following steps:

[0067] Step 402: determine the motion state of the target object according to the historical frame images.

[0068] In some embodiments, the electronic device determines the motion state of the target object based on the historical frame images, which may include: comparing the target object area in at least two consecutive historical frame images before the current frame image to obtain position change and size change information of the target object area, and determining the motion state of the target object based on the position change and size change information.

[0069] When determining the motion state of the target object frame by frame, the electronic device may determine the motion state of the target object based on the change of the target object area in at least two adjacent frames of images. For example, the first position and size information of the target object area in the Nth frame image may be obtained, and the second position and size information of the target object area in the N-1th frame image may be obtained. The first position and size information may be used to characterize the image position and size of the target object in the Nth frame image, and the second position and size information may be used to characterize the image position and size of the target object in the N-1th frame image. Among them, the image position may be represented by pixel coordinates or image coordinates, and the size may be represented by the pixel area occupied by the target object area, etc., but is not limited thereto. The first position and size information may be compared with the second position and size information to obtain the position change and size change information of the target object in the N-1th frame image and the Nth frame image.

[0070] Optionally, the motion state of the target object may include no movement, movement on the same plane (i.e., translation), movement on different planes, etc., wherein movement on different planes may include forward and backward movement and diagonal movement. When the target object moves on the same plane, the depth information of the target object will not change, i.e., the object distance remains unchanged, so the position of the target object area changes, but the size does not change. When the target object moves on different planes, the depth information of the target object will change, i.e., the object distance will change, so the size of the target object area will change, but the position may or may not change.

[0071] As a specific implementation, the determination conditions for determining the motion state of the target object based on the position change and size change information of the target object region may be as shown in Table 1.

[0072] Table 1

[0073]

[0074]

[0075] Based on the position change and size change information of the target object area in the continuous multi-frame images, the motion state of the target object can be accurately identified, so that the target object can be tracked more accurately, thereby improving the accuracy of tracking.

[0076] Step 404 , when the motion state is no movement or movement on the same plane, determine whether the defocus amount of the current frame image is greater than a first defocus amount threshold, if so, execute step 408 , if not, execute step 406 .

[0077] In some embodiments, when the motion state of the target object is not moving or moving on the same plane, it can be determined whether the defocus amount of the current frame image is greater than the first defocus amount threshold. If the defocus amount of the current frame image is greater than the first defocus amount threshold, it can be determined that the defocus amount is a normal defocus amount. If the defocus amount is greater than the first defocus amount threshold, it can be determined that the defocus amount is an abnormal defocus amount.

[0078] When the target object is stationary or moves on the same plane, the image plane of the target object in the camera will not be greatly offset, and the distance from the best focus plane will not change greatly. Therefore, the defocus amount generated should be small. Whether the defocus amount of the current frame image is greater than a first defocus amount threshold, the first defocus amount threshold may refer to a defocus amount threshold for triggering the movement of the lens. When the defocus amount is greater than the first defocus amount threshold, the lens will be triggered to move.

[0079] When the target object is stationary or moves on the same plane, the lens does not need to move. If the defocus amount of the current frame image is greater than the first defocus amount threshold, the defocus amount can be determined to be an abnormal defocus amount, indicating that the first region of interest of the current frame image is an abnormal region of interest. The defocus amount of the current frame image can be filtered, and the defocus amount of the current frame image can be set to 0, or to a fixed value less than the first defocus amount threshold, so as not to trigger the focus module to perform the focus operation, so that the lens remains stationary.

[0080] In some embodiments, the first defocus threshold is determined according to the current position of the lens, and the corresponding first defocus threshold may be different when the lens is in different positions. The current position of the lens may be obtained, and the first defocus threshold corresponding to the current position may be calculated based on at least two groups of reference values, wherein each group of reference values ​​may include a reference lens position and a reference defocus threshold corresponding to the reference lens position. For example, the first group of reference values ​​includes a first reference lens position LensPos1 and a first reference defocus threshold DefThresh1, and the second group of reference values ​​includes a second reference lens position LensPos2 and a second reference defocus threshold DefThresh2.

[0081] Specifically, a first position difference between the current position LensPos of the lens and the first reference lens position LensPos1, and a second position difference between the second reference lens position LensPos2 and the first reference lens LenPos1 may be calculated, and a ratio of the first position difference to the second position difference may be calculated, and a first defocus threshold DefThresh corresponding to the current position LensPos of the lens may be determined based on the ratio. Alternatively, the first defocus threshold DefThresh corresponding to the current position LensPos of the lens may be calculated using the following formula (1):

[0082]

[0083] It should be noted that the above-mentioned first reference lens position LensPos1, first reference defocus threshold DefThresh1, second reference lens position LensPos2, and second reference defocus threshold DefThresh2 can be obtained through multiple experiments, and the electronic device can also calculate the first defocus threshold corresponding to the current position of the lens based on three groups, four groups, or other groups of reference values, and is not limited to the above-mentioned two groups. In other embodiments, the electronic device can also pre-calculate the first defocus threshold corresponding to each position of the lens, and store the correspondence between the lens position and the defocus threshold. After obtaining the defocus of the current frame image, the first defocus threshold corresponding to the current position of the lens can be searched according to the stored correspondence, and it can be determined whether the defocus of the current frame image is greater than the first defocus threshold.

[0084] Step 406 , determining that the defocus amount is a normal defocus amount, and controlling the focus module to perform a focus operation according to the defocus amount.

[0085] If the defocus amount of the current frame image is a normal defocus amount, it can be said that the first area of ​​interest of the current frame image is a normal area of ​​interest. The focus module can be controlled to perform a focusing operation based on the defocus amount so that the focus falls within the first area of ​​interest, and the first area of ​​interest is used as the focus area for focusing.

[0086] Step 408: determine that the defocus amount is an abnormal defocus amount, and filter the defocus amount so that the focus module does not re-execute the focus operation.

[0087] If the defocus amount of the current frame image is an abnormal defocus amount, the abnormal defocus amount can be filtered out, and the focus module is not triggered to perform a focus operation, so that the lens remains stationary and continues to track the target object.

[0088] For example, Figure 5A FIG. 1 is a schematic diagram of analyzing the defocus amount according to the motion state of the target object in one embodiment. Figure 5AAs shown, image 502 is the N-1th frame image, and image 504 is the Nth frame image. Image recognition can be performed on image 502 to obtain a target image area 512, and a first region of interest 514 of image 504 can be determined based on the target image area 512. It can be calculated that the defocus amount of image 504 is small. Since the target object (person) is in translational motion, the image 502 and the image 504 change very little, and the defocus amount is a normal defocus amount, indicating that the first region of interest 514 is a normal region of interest, and the focusing operation can be performed normally.

[0089] For example, Figure 5B Schematic diagram of analyzing the defocus amount according to the motion state of the target object in another embodiment. Image 506 is the N-1th frame image, and image 508 is the Nth frame image. Image recognition can be performed on image 506 to obtain a target image area 516, and a first region of interest 518 of image 508 is determined based on the target image area 516. It can be calculated that the defocus amount of image 504 is large. Since the target object (person) is in translational motion, the image 506 and the image 508 change very little, so the defocus amount is an abnormal defocus amount, indicating that the first region of interest 518 is an abnormal region of interest, and no focusing operation is performed, which can prevent the abnormal first region of interest 518 from being focused on in the current frame, and the normal region of interest (the upper body region of the person) from being focused on in the next frame, thereby causing the lens to push and pull.

[0090] In the embodiment of the present application, the defocus amount of the current frame image can be analyzed according to the motion state to determine whether the defocus amount of the current frame image is normal, and when the defocus amount of the current frame image is abnormal, the defocus amount is filtered to keep the lens still, thereby avoiding the situation where the lens is pushed or pulled due to abnormal defocus, and improving the stability of tracking the target object. In addition, by analyzing whether the defocus amount is normal in combination with the motion state, the accuracy of the analysis can be further improved, thereby improving the accuracy of the autofocus.

[0091] like Figure 6 As shown, in one embodiment, another method for controlling focus is provided, which can be applied to the above electronic device, and the method may include the following steps:

[0092] Step 602: determine the motion state of the target object according to the historical frame image.

[0093] Step 604: Acquire the first region of interest of the current frame image.

[0094] After the electronic device captures the current frame image through the image sensor, the first region of interest determined according to the target object region in the historical frame image can be obtained. As an optional implementation, the target object region of the target object in the target historical frame image can be determined, and the target object region in the target historical frame image can be used as the first region of interest of the current frame image. The target historical frame image can be the previous frame image of the current frame image, or the previous two frames of the current frame image. For example, if the current frame image is the 5th frame image, the target historical frame image can be the 4th frame image, or the 3rd frame image, etc., but is not limited thereto.

[0095] Optionally, the frame difference between the target historical frame image and the current frame image is positively correlated with the duration of determining the target object area in the image. The longer the duration of the target object area in the image identified by the tracking module 130, the greater the frame difference between the current frame image and the target historical frame image.

[0096] Step 606: Determine the similarity between the first region of interest and the second region of interest of the reference frame image.

[0097] Since the motion state of the target object is detected frame by frame, the position and size information of the target object area of ​​the target object in two adjacent frames may not change. If the motion state of the target object is oblique movement, since the position and size information of the target object area of ​​the target object in two adjacent frames may not change, the oblique movement may be identified as translation or forward and backward movement. In order to further distinguish the actual motion state of the target object, in an embodiment of the present application, the similarity between the second region of interest of the current frame image and the reference frame image may be combined to assist in the judgment.

[0098] The reference frame image belongs to a historical frame image captured before the current frame image. The reference frame image may be a historical frame image of the previous frame of the current frame image. Optionally, the reference frame image may be a historical frame image of the previous frame of the current frame image whose region of interest is a normal region of interest. In some embodiments, the current frame image and the reference frame image for similarity calculation may be phase difference images, original images captured by an image sensor (such as RAW data in MIPI format), or other images in the ISP processor that are not auto-focused, which are not limited here.

[0099] The electronic device may extract a first image feature of the first region of interest, and extract a second image feature of the second region of interest of the reference frame image, and calculate the similarity between the first region of interest and the second region of interest based on the first image feature and the second image feature. Optionally, the first image feature and the second image feature may include a grayscale histogram feature, a color histogram feature, a texture feature, etc., but are not limited to these, wherein the grayscale histogram feature may be used to describe the grayscale distribution of each pixel in the region of interest, and the color histogram feature may be used to describe the color distribution of each pixel in the region of interest.

[0100] In some embodiments, the similarity between the first region of interest and the second region of interest may be calculated based on the grayscale histogram feature. The first pixel distribution number of the first region of interest in each grayscale interval may be counted respectively, and the first grayscale distribution probability of the first region of interest may be determined based on the first pixel distribution number. The second pixel distribution number of the second region of interest in each grayscale interval may be counted respectively, and the second grayscale distribution probability of the second region of interest may be determined based on the second pixel distribution number.

[0101] The grayscale of the image can be divided into multiple grayscale intervals. For example, if the grayscale of the image is 256, the grayscale can be divided into 4 grayscale intervals, that is, [0,64), [64,128), [128,192) and [192,255]. The number of first pixel distributions of the first region of interest in each grayscale interval can be counted respectively, and the number of second pixel distributions of the second region of interest in each grayscale interval can be counted respectively. For example, the number of pixels of the first region of interest in the four grayscale intervals [0,64), [64,128), [128,192) and [192,255] can be counted respectively, and the number of pixels of the second region of interest in the four grayscale intervals [0,64), [64,128), [128,192) and [192,255] can be counted respectively.

[0102] According to the first pixel distribution number of the first region of interest in each gray level interval, the proportion of the number of first pixels in each gray level interval to the first region of interest can be calculated to obtain the first gray level distribution probability. For example, the first region of interest contains 1000 pixels, and the number of pixels in the four gray level intervals [0,64), [64,128), [128,192) and [192,255] can be counted as 320, 120, 220 and 340 respectively. Then, the proportion of the number of pixels in each gray level interval to the first region of interest can be calculated, and the first gray level distribution probability can be obtained as [0.32, 0.12, 0.22, 0.34].

[0103] Similarly, the proportion of the number of second pixels in each gray level interval to the second region of interest can be calculated according to the number of second pixels distributed in each gray level interval in the second region of interest to obtain the second gray level distribution probability.

[0104] The similarity between the first region of interest and the second region of interest may be calculated based on the first grayscale distribution probability and the second grayscale distribution probability. The algorithm for calculating the similarity may include but is not limited to the JSD (Jensen-Shannon Divergence) distance algorithm, the KL divergence (Kullback-Leibler Divergence) algorithm, the JS divergence algorithm, etc. As a specific embodiment, the distance between the first grayscale distribution probability and the second grayscale distribution probability may be calculated, and the similarity between the first region of interest and the second region of interest may be determined based on the distance. The smaller the distance between the first grayscale distribution probability and the second grayscale distribution probability, the greater the similarity between the first region of interest and the second region of interest.

[0105] Taking the JDS distance algorithm as an example, the similarity between the first region of interest and the second region of interest can be calculated by the following formulas (2) to (3):

[0106]

[0107] Herein, P may represent the first grayscale distribution probability, and Q may represent the second grayscale distribution probability.

[0108] Step 608 , analyzing the defocus amount of the current frame image according to the motion state and the similarity, and judging whether the defocus amount is a normal defocus amount according to the analysis result, if so, executing step 610 , if not, executing step 612 .

[0109] When the target object moves on different planes, the target object may move forward and backward or move diagonally. Since the defocus amounts corresponding to the forward and backward movement and diagonal movement of the target object are different, the similarity between the first region of interest and the second region of interest can be used to distinguish them, and different lens stabilization strategies can be set respectively.

[0110] In some embodiments, when the motion state is moving on different planes, it can be determined whether the similarity is greater than the similarity threshold. If the similarity is greater than the similarity threshold, it can be determined that the defocus amount of the current frame image is a normal defocus amount. When the target object moves forward and backward, the focus frame will always remain on the target object, and the difference between the regions of interest of two adjacent frames of images is very small. Therefore, when the motion state is moving on different planes, if the similarity between the first region of interest and the second region of interest is greater than the similarity threshold, it can be considered that the target object is currently moving forward and backward.

[0111] When the target object is moving forward and backward, the lens needs to move to focus on the target object. When the object moves forward and backward very quickly, a large defocus amount will be generated. At this time, the defocus amount cannot be filtered, otherwise out of focus will occur. Therefore, the defocus amount of the current frame image can be determined as a normal defocus amount, and the focus module can be controlled to push the lens to the optimal position according to the defocus amount to focus on the target object.

[0112] In some embodiments, when the motion state is moving on different planes, if the similarity between the first region of interest and the second region of interest is not greater than the similarity threshold, it can be further determined whether the defocus amount of the current frame image is greater than the second defocus amount threshold. If the defocus amount of the current frame image is greater than the second defocus amount threshold, it can be determined that the defocus amount is an abnormal defocus amount. If the defocus amount of the current frame image is not greater than the second defocus amount threshold, it can be determined that the defocus amount is a normal defocus amount.

[0113] When the target object moves obliquely, the difference between the regions of interest of two adjacent frames of images will be greater than the difference when the target object moves forward and backward. When the motion state is moving on different planes, if the similarity between the first region of interest and the second region of interest is not greater than the similarity threshold, it can be considered that the target object is currently moving obliquely. When the target object moves obliquely, the lens also needs to move to focus on the target object, but when the defocus is too large, it can be considered that the focus is on the background area, that is, the first region of interest is an abnormal region of interest, and the defocus needs to be filtered. A second defocus threshold can be set, and the second defocus threshold can be a fixed value set after multiple tests, and when the defocus of the current frame image is greater than the second defocus threshold, the defocus of the current frame image is filtered to keep the lens motionless.

[0114] As a specific implementation, the similarity between the first region of interest and the second region of interest can be represented by a JSD distance, and it can be determined whether the JSD distance between the first region of interest and the second region of interest is less than a set JSD distance threshold. If it is less than the set JSD distance threshold, it can be considered that the first region of interest is similar to the second region of interest, and it can be determined that the target object moves forward and backward. If it is not less than the set JSD distance, it can be considered that the difference between the first region of interest and the second region of interest is large, and it can be determined that the target object moves obliquely.

[0115] In some embodiments, if the similarity between the first region of interest and the second region of interest is greater than a similarity threshold, the current frame image may be updated as a new reference frame image for similarity calculation of subsequently acquired images.

[0116] It should be noted that if the target object is not moving or moving on the same plane, the method for determining whether the defocus amount of the current frame image is a normal defocus amount can refer to the description in the above embodiment, which will not be repeated here.

[0117] Step 610: Control the focus module to perform a focus operation according to the defocus amount.

[0118] Step 612: Filter the defocus amount so that the focus module does not re-execute the focus operation.

[0119] In an embodiment of the present application, the motion state of the target object can be further divided based on the similarity between the first region of interest in the current frame image and the second region of interest in the reference frame image, and different lens stabilization modules can be set for the motion states in different situations, so that the control of the lens is more in line with the actual situation when focusing on the moving target, the accuracy of focusing is improved, and the situation where the lens is pushed and pulled due to abnormal defocus can be avoided, thereby improving the stability of focusing on the target object.

[0120] In some embodiments, the electronic device can also obtain the first region of interest of the current frame image by prediction. The step of obtaining the first region of interest of the current frame image may include: obtaining the target object region of the target object in multiple frames of historical frame images, analyzing the target object region of the target object in multiple frames of historical frame images through a filter, predicting the predicted region of the target object in the current frame image, and using the predicted region as the first region of interest of the current frame image.

[0121] Taking the current frame image as the i+Nth frame image as an example, the region of interest of the i+Nth frame image can be predicted based on the i-th frame image, the iN-th frame image and other historical frame images, wherein N can be a normal number and can be set according to actual needs. Optionally, N can be used to characterize the delayed frame number, which can refer to the delayed frame number of feedback from the tracking module 130 to the focusing module 140, that is, the difference between the frame number corresponding to the tracking result fed back by the tracking module 130 and the frame number focused by the focusing module 140 based on the tracking result. For example, when the tracking module 130 identifies the second frame image and obtains the tracking result of the target object, and then feeds back to the focusing module 140, the focusing module 140 focuses on the fifth frame image based on the tracking result, and there are 3 frames of delayed frames between the tracking module 130 and the focusing module 140.

[0122] In some embodiments, the electronic device may calculate the change speed information of the target object in the i-th frame image based on the target object area in the iN-th frame image and the target object area in the i-th frame image, and the change speed information may include information such as the position change speed and the size change speed. The first predicted area information of the target object in the i+N-th frame image may be predicted based on the change speed information.

[0123] Optionally, the filter may include a Kalman filter, which is a data filtering model that uses the state of a linear system and can estimate the state of a dynamic system from a series of data with measurement noise based on known conditions. The Kalman filter can obtain a predicted value of the Kalman filter based on the target object region of the target object in the i-th frame image, and the first predicted region information obtained above can be used as the observed value of the Kalman filter, and the predicted value of the Kalman filter can be corrected using the predicted value to obtain the second predicted region information, so that the second predicted region information can be used as the first region of interest of the i+N-th frame image.

[0124] After predicting the first region of interest of the current frame image using multiple frames of historical frame images, since the prediction results may be inaccurate, the defocus amount of the current frame image can be detected by the methods described in the above embodiments, so as to determine whether the predicted first region of interest of the current frame image is accurate, which can further improve the accuracy of the prediction results and ensure the accuracy of focusing.

[0125] In an embodiment of the present application, filters and multiple frames of historical frame images can be used to predict the first region of interest of the current frame image, which can reduce the deviation between the tracking result and the actual position of the target object, improve the accuracy of the first region of interest, and thus improve the accuracy of autofocus.

[0126] like Figure 7 As shown, in one embodiment, a device 700 for controlling focus is provided, which can be applied to the above electronic device. The device 700 for controlling focus can include: a state determination module 710 , an analysis module 720 and a control module 730 .

[0127] The state determination module 710 is used to determine the motion state of the target object according to the historical frame images.

[0128] The analysis module 720 is used to analyze the defocus amount of the current frame image according to the motion state, and the historical frame image is an image collected by the camera before the current frame image.

[0129] The control module 730 is configured to control the focus module to perform a focus operation according to the defocus amount if it is determined according to the analysis result that the defocus amount is a normal defocus amount.

[0130] In one embodiment, the control module 730 is further configured to filter the defocus amount so that the focusing module does not re-execute the focusing operation if the defocus amount is determined to be an abnormal defocus amount according to the analysis result.

[0131] In one embodiment, the control module 730 is also used to set the defocus amount to 0, or set the defocus amount to a fixed value less than a first defocus amount threshold if the defocus amount is determined to be an abnormal defocus amount according to the analysis results, and the first defocus amount threshold is the defocus amount threshold that triggers lens movement.

[0132] In an embodiment of the present application, the motion state of the target object is determined based on the historical frame image, and the defocus amount of the current frame image is analyzed based on the motion state. If the defocus amount is determined to be a normal defocus amount based on the analysis result, the focus module is controlled to perform a focusing operation based on the defocus amount. Whether the defocus amount of the current frame image is normal can be detected based on the motion state. Focusing is performed only when the defocus amount of the current frame image is a normal defocus amount, thereby improving the accuracy of autofocus and the imaging effect of the target object, and preventing the lens from pushing and pulling due to the abnormal defocus amount of the current frame image, thereby improving the focusing stability.

[0133] In one embodiment, the state determination module 710 is also used to compare the target object area in at least two consecutive historical frame images before the current frame image to obtain the position change and size change information of the target object area, and determine the motion state of the target object based on the position change and size change information.

[0134] In one embodiment, the analysis module 720 is also used to determine whether the defocus amount of the current frame image is greater than a first defocus amount threshold when the motion state is not moving or moving on the same plane. If the defocus amount is not greater than the first defocus amount threshold, the defocus amount is determined to be a normal defocus amount; if the defocus amount is greater than the first defocus amount threshold, the defocus amount is determined to be an abnormal defocus amount.

[0135] Optionally, the first defocus threshold is determined according to a current position of the lens.

[0136] In the embodiment of the present application, the defocus amount of the current frame image can be analyzed according to the lens stabilization strategy corresponding to the motion state to determine whether the defocus amount of the current frame image is normal, and when the defocus amount of the current frame image is abnormal, the defocus amount is filtered to keep the lens still, thereby avoiding the situation where the lens is pushed or pulled due to abnormal defocus, and improving the stability of tracking the target object. In addition, by analyzing whether the defocus amount is normal in combination with the motion state, the accuracy of the analysis can be further improved, thereby improving the accuracy of the autofocus.

[0137] In one embodiment, the above-mentioned apparatus 700 for controlling focus, in addition to the state determination module 710 , the analysis module 720 and the control module 730 , further includes a region acquisition module and a similarity determination module.

[0138] The region acquisition module is used to acquire a first region of interest of a current frame image, where the first region of interest is determined based on a target object region in a historical frame image.

[0139] In one embodiment, the region acquisition module is also used to determine the target object region of the target object in the target historical frame image, and use the target object region in the target historical frame image as the first region of interest of the current frame image, and the frame difference between the target historical frame image and the current frame image is positively correlated with the duration of determining the target object region in the image.

[0140] The similarity determination module is used to determine the similarity between the first region of interest and the second region of interest of a reference frame image, where the reference frame image is a historical frame image collected before the current frame image.

[0141] In one embodiment, the similarity determination module includes a feature extraction unit and a calculation unit.

[0142] The feature extraction unit is used to extract a first image feature of a first region of interest and to extract a second image feature of a second region of interest of a reference frame image.

[0143] The calculation unit is used to calculate the similarity between the first region of interest and the second region of interest according to the first image feature and the second image feature.

[0144] In one embodiment, the feature extraction unit is also used to count the number of first pixel distributions of the first region of interest in each gray level interval, and determine the first gray level distribution probability of the first region of interest based on the first pixel distribution number; and to count the number of second pixel distributions of the second region of interest in each gray level interval, and determine the second gray level distribution probability of the second region of interest based on the second pixel distribution number.

[0145] The calculation unit is further used to calculate the similarity between the first region of interest and the second region of interest according to the first gray level distribution probability and the second gray level distribution probability.

[0146] The analysis module 720 is further configured to analyze the defocus amount of the current frame image according to the motion state and the similarity.

[0147] In one embodiment, the analysis module 720 is also used to, when the motion state is moving on different planes, if the similarity is greater than the similarity threshold, determine that the defocus amount of the current frame image is a normal defocus amount; if the similarity is not greater than the similarity threshold, and the defocus amount is not greater than the second defocus threshold, determine that the defocus amount is a normal defocus amount; if the similarity is not greater than the similarity threshold, and the defocus amount is greater than the second defocus threshold, determine that the defocus amount is an abnormal defocus amount.

[0148] In an embodiment of the present application, the motion state of the target object can be further divided based on the similarity between the first region of interest in the current frame image and the second region of interest in the reference frame image, and different lens stabilization modules can be set for the motion states in different situations, so that the control of the lens is more in line with the actual situation when focusing on the moving target, the accuracy of focusing is improved, and the situation where the lens is pushed and pulled due to abnormal defocus can be avoided, thereby improving the stability of focusing on the target object.

[0149] In one embodiment, the region acquisition module is also used to obtain the target object region of the target object in multiple frames of historical frame images, analyze the target object region of the target object in multiple frames of historical frame images through a filter, predict the predicted region of the target object in the current frame image, and use the predicted region as the first region of interest of the current frame image.

[0150] In an embodiment of the present application, filters and multiple frames of historical frame images can be used to predict the first region of interest of the current frame image, which can reduce the deviation between the tracking result and the actual position of the target object, improve the accuracy of the first region of interest, and thus improve the accuracy of autofocus.

[0151] Figure 8 FIG. 1 is a structural block diagram of an electronic device in an embodiment. Figure 8 As shown, the electronic device 800 may include one or more of the following components: a processor 810, and a memory 820 coupled to the processor 810, wherein the memory 820 may store one or more computer programs, and the one or more computer programs may be configured to implement the methods described in the above embodiments when executed by one or more processors 810.

[0152] The processor 810 may include one or more processing cores. The processor 810 uses various interfaces and lines to connect various parts of the entire electronic device 800, and executes various functions and processes data of the electronic device 800 by running or executing instructions, programs, code sets or instruction sets stored in the memory 820, and calling data stored in the memory 820. Optionally, the processor 810 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 810 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 810, but may be implemented separately through a communication chip.

[0153] The memory 820 may include a random access memory (RAM) or a read-only memory (ROM). The memory 820 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 820 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also store data created by the electronic device 800 during use, etc.

[0154] It is understandable that the electronic device 800 may include more or fewer structural elements than those in the above structural block diagram, for example, a power module, physical buttons, a WiFi (Wireless Fidelity) module, a speaker, a Bluetooth module, a sensor, etc., and is not limited here.

[0155] An embodiment of the present application discloses a computer-readable storage medium storing a computer program, wherein the computer program implements the method described in the above embodiment when executed by a processor.

[0156] An embodiment of the present application discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program can be executed by a processor to implement the methods described in the above embodiments.

[0157] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a ROM, etc.

[0158] As used herein, any reference to memory, storage, database, or other medium may include nonvolatile and / or volatile memory. Suitable nonvolatile memory may include ROM, programmable ROM (Programmable ROM, PROM), erasable PROM (Erasable PROM, EPROM), electrically erasable PROM (Electrically Erasable PROM, EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache memory. By way of illustration and not limitation, RAM may be in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus direct RAM (RDRAM) and direct memory bus dynamic RAM (DRDRAM).

[0159] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. Those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required for the present application.

[0160] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the above-mentioned processes does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0161] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0163] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several requests for a computer device (which can be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to perform some or all of the steps of the above-mentioned methods of various embodiments of the present application.

[0164] The above is a detailed introduction to a method, device, electronic device and computer-readable storage medium for controlling focus disclosed in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those of ordinary skill in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A method for controlling focus, characterized in that: include: Determine the motion state of the target object according to the historical frame images; Analyzing the defocus amount of the current frame image according to the motion state, the historical frame image is an image captured by a camera before the current frame image; If it is determined according to the analysis result that the defocus amount is a normal defocus amount, controlling the focus module to perform a focus operation according to the defocus amount; If the defocus amount is determined to be an abnormal defocus amount according to the analysis result, the defocus amount is filtered so that the focus module does not re-execute the focus operation.

2. The method according to claim 1, characterized in that The filtering of the defocus amount includes: setting the defocus amount to 0, or setting the defocus amount to a fixed value less than a first defocus amount threshold, where the first defocus amount threshold is a defocus amount threshold that triggers lens movement.

3. The method according to any one of claims 1-2, characterized in that: The analyzing the defocus amount of the current frame image according to the motion state includes: When the motion state is no movement or movement on the same plane, determining whether the defocus amount of the current frame image is greater than a first defocus amount threshold; If the defocus amount is not greater than the first defocus amount threshold, determining that the defocus amount is a normal defocus amount; If the defocus amount is greater than the first defocus amount threshold, the defocus amount is determined to be an abnormal defocus amount.

4. The method according to claim 3, characterized in that The first defocus threshold is determined according to the current position of the lens.

5. The method according to any one of claims 1-2, characterized in that: Before analyzing the defocus amount of the current frame image according to the motion state, the method further includes: Acquire a first region of interest of a current frame image, where the first region of interest is determined according to a target object region in the historical frame image; Determine a similarity between the first region of interest and a second region of interest of a reference frame image, the reference frame image being a historical frame image acquired before the current frame image; The analyzing the defocus amount of the current frame image according to the motion state includes: The defocus amount of the current frame image is analyzed according to the motion state and the similarity.

6. The method according to claim 5, characterized in that The determining the similarity between the first region of interest and the second region of interest of the reference frame image includes: Extracting a first image feature of the first region of interest, and extracting a second image feature of a second region of interest of the reference frame image; The similarity between the first region of interest and the second region of interest is calculated according to the first image feature and the second image feature.

7. The method according to claim 6, characterized in that The extracting of the first image feature of the first region of interest and the extracting of the second image feature of the second region of interest of the reference frame image include: Counting the number of first pixel distributions in each gray level interval of the first region of interest respectively, and determining a first gray level distribution probability of the first region of interest according to the number of first pixel distributions; Counting the number of second pixel distributions of the second region of interest in each of the gray level intervals respectively, and determining a second gray level distribution probability of the second region of interest according to the number of second pixel distributions; The calculating the similarity between the first region of interest and the second region of interest according to the first image feature and the second image feature includes: The similarity between the first region of interest and the second region of interest is calculated according to the first gray level distribution probability and the second gray level distribution probability.

8. The method according to claim 5, characterized in that The analyzing the defocus amount of the current frame image according to the motion state and the similarity includes: When the motion state is moving on different planes, if the similarity is greater than a similarity threshold, determining that the defocus amount of the current frame image is a normal defocus amount; If the similarity is not greater than the similarity threshold, and the defocus amount is not greater than a second defocus amount threshold, determining that the defocus amount is a normal defocus amount; If the similarity is not greater than the similarity threshold, and the defocus amount is greater than the second defocus amount threshold, the defocus amount is determined to be an abnormal defocus amount.

9. The method according to claim 5, characterized in that The step of acquiring the first region of interest of the current frame image includes: A target object area of ​​a target object in a target historical frame image is determined, and the target object area in the target historical frame image is used as the first region of interest of a current frame image, wherein a frame difference between the target historical frame image and the current frame image is positively correlated with a duration of determining the target object area in the image.

10. The method according to claim 5, characterized in that The step of acquiring the first region of interest of the current frame image includes: Obtaining a target object region of a target object in multiple historical frame images; The target object region of the target object in multiple frames of historical frame images is analyzed by a filter, a predicted region of the target object in the current frame image is predicted, and the predicted region is used as the first region of interest of the current frame image.

11. The method according to any one of claims 1-2, 6-10, characterized in that: Determining the motion state of the target object according to the historical frame image includes: Compare the target object area in at least two consecutive historical frame images before the current frame image to obtain position change and size change information of the target object area; The motion state of the target object is determined according to the position change and size change information.

12. A device for controlling focus, characterized in that: include: A state determination module, used to determine the motion state of the target object according to the historical frame image; An analysis module, used for analyzing the defocus amount of the current frame image according to the motion state, wherein the historical frame image is an image captured by a camera before the current frame image; A control module, configured to control a focusing module to perform a focusing operation according to the defocus amount if it is determined according to the analysis result that the defocus amount is a normal defocus amount; The control module is further configured to filter the defocus amount so that the focusing module does not re-execute the focusing operation if the defocus amount is determined to be an abnormal defocus amount according to the analysis result.

13. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images