Shooting method and device, electronic equipment and storage medium

Abstract

The invention relates to a shooting method and device, electronic equipment and a storage medium. The method comprises the following steps: in response to a detected shooting instruction, collecting a sequence image comprising a target object; wherein the sequence image comprises a previous frame image and a current frame image which are adjacent to each other; determining a first position of a target object in the preorder frame image; determining the displacement of the target object in the current frame image relative to the target object in the preorder frame image based on the first position of the target object in the preorder frame image; and carrying out image stabilization processing on the current frame image based on the displacement of the target object in the preorder frame image and the current frame image, and outputting a target image. Through the method, the image stabilization effect can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of photographing technology, and in particular to a photographing method and device, electronic equipment, and storage medium. Background Art

[0002] At present, with the development of media technology, the requirements for shooting are getting higher and higher. For example, in video shooting, the video shooting needs to be stable and there is no jitter between image frames; when previewing selfies, a face frame is usually displayed in the preview screen, and the position and size of the face frame in the preview screen also need not fluctuate dramatically between image frames. How to improve shooting quality has always been a focus of attention. Summary of the invention

[0003] The present invention provides a photographing method and device, an electronic device, and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided a shooting method, including:

[0005] In response to detecting a shooting instruction, acquiring a sequence of images including a target object; wherein the sequence of images includes adjacent previous frame images and a current frame image;

[0006] Determine a first position of the target object in the preceding frame image;

[0007] Based on a first position of the target object in the previous frame image, determining a displacement of the target object in the current frame image relative to the target object in the previous frame image;

[0008] Based on the displacement of the target object in the previous frame image and the current frame image, image stabilization is performed on the current frame image, and a target image is output.

[0009] In some embodiments, determining the displacement of the target object in the current frame image relative to the target object in the previous frame image based on the first position of the target object in the previous frame image includes:

[0010] Sampling pixel points in a first position area of ​​the target object in the previous frame image to obtain a first pixel point set;

[0011] Using a forward optical flow method, estimate that each pixel in the first pixel set corresponds to a second pixel set in the current frame image;

[0012] Using the inverse optical flow method, estimate that each pixel in the second pixel set corresponds to a third pixel set in the previous frame image;

[0013] Based on the first pixel point set, the second pixel point set and the third pixel point set, a displacement of the target object in the current frame image relative to the target object in the previous frame image is estimated.

[0014] In some embodiments, estimating the displacement of the target object in the current frame image relative to the target object in the previous frame image based on the first pixel point set, the second pixel point set, and the third pixel point set includes:

[0015] Based on the difference between the first pixel point set and the third pixel point set, selecting a first key point whose difference is less than a preset difference threshold from the first pixel point set;

[0016] Filtering out a second key point corresponding to the first key point from the second pixel point set;

[0017] Preliminarily estimating a second position of the target object in the current frame image based on a first position of the target object in the previous frame image, a first key point of the previous frame image, and a second key point of the current frame image;

[0018] Based on a first position of the target object in the previous frame image and a preliminarily estimated second position of the target object in the current frame image, a displacement of the target object in the current frame image relative to the target object in the previous frame image is determined.

[0019] In some embodiments, based on the difference between the first pixel point set and the third pixel point set, selecting a first key point having a difference less than a preset difference threshold from the first pixel point set includes:

[0020] Determine a median value of the coordinate difference based on the coordinate difference between each pixel point in the first pixel point set and the corresponding pixel point in the third pixel point set;

[0021] A pixel point in the first pixel point set whose coordinate difference corresponding to the pixel point is less than the median value of the coordinate difference is determined as the first key point.

[0022] In some embodiments, the method further comprises:

[0023] Based on the median of the coordinate differences, a first confidence level of the displacement of the target object is determined.

[0024] In some embodiments, preliminarily estimating the second position of the target object in the current frame image based on the first position of the target object in the previous frame image, the first key point of the previous frame image, and the second key point of the current frame image includes:

[0025] Based on the coordinates of the corresponding pixel points in the first key point and the second key point, determine the median value of the coordinate translation in the horizontal direction and the median value of the coordinate translation in the vertical direction;

[0026] Preliminarily determining the coordinates of the predetermined position of the target object in the current frame image based on the coordinates of the predetermined position of the target object in the previous frame image, and the median value of the coordinate translation in the horizontal direction and the median value of the coordinate translation in the vertical direction;

[0027] Calculating the distance between any pixels in the first key points and determining a first distance median, and calculating the distance between any pixels in the second key points and determining a second distance median;

[0028] Based on the width and height of the target object in the previous frame image and the ratio of the second distance median to the first distance median, the width and height of the target object in the current frame image are preliminarily determined.

[0029] In some embodiments, performing image stabilization processing on the current frame image based on the displacement of the target object in the previous frame image and the current frame image, and outputting the target image, includes:

[0030] Filtering the displacement of the target object in the previous frame image and the current frame image to obtain a filtered displacement;

[0031] The current frame image is subjected to image stabilization processing based on the filtered displacement, and the target image is output.

[0032] In some embodiments, filtering the displacement of the target object in the previous frame image and the current frame image to obtain the filtered displacement includes:

[0033] Determine a Kalman gain according to the first key point and the second key point, a first position of the target object in the previous frame image and a preliminarily estimated second position of the target object in the current frame image;

[0034] Determining, according to the first key point and the second key point, a displacement observation value of the target object in the current frame image and the previous frame image;

[0035] The displacement estimated based on the optical flow method is subjected to Kalman filtering based on the Kalman gain and the displacement observation value to obtain the filtered displacement.

[0036] In some embodiments, determining the Kalman gain according to the first key point and the second key point, a first position of the target object in the previous frame image and a preliminarily estimated second position of the target object in the current frame image includes:

[0037] Calculating a first center point of the first key point and a second center point of the second key point;

[0038] Calculating a first coordinate difference between the center position of the target object in the previous frame image and the first center point, and a second coordinate difference between the center position of the target object in the current frame image and the second center point that is preliminarily estimated;

[0039] Determining a covariance of the displacement observation value according to the first coordinate difference and the second coordinate difference;

[0040] The Kalman gain is determined according to the covariance of the displacement observations.

[0041] In some embodiments, determining the covariance of the displacement observation value according to the first coordinate difference and the second coordinate difference includes:

[0042] Determining a mean value of the differences according to the first coordinate difference and the second coordinate difference;

[0043] determining a standard deviation of the differences based on the first coordinate difference, the second coordinate difference, and a mean of the differences;

[0044] The covariance of the displacement observations is determined based on a ratio of a standard deviation of the differences to a mean of the differences.

[0045] In some embodiments, determining the displacement observation value of the target object in the current frame image and the previous frame image according to the first key point and the second key point includes:

[0046] The difference between the second center point and the first center point is determined as the displacement observation value.

[0047] In some embodiments, the method further comprises:

[0048] Correcting a first position of the target object in the previous frame image based on the filtered displacement to obtain a third position of the target object in the current frame image;

[0049] Acquire a fourth position of the target object in the current frame image detected by a target object detection method other than the optical flow method;

[0050] Based on the difference between the third position and the fourth position, a second confidence level of the displacement of the target object is determined.

[0051] In some embodiments, the method further comprises:

[0052] In response to the existence of the first confidence level and the second confidence level, a target confidence level of the displacement amount of the target object is determined based on the first confidence level and the second confidence level.

[0053] In some embodiments, the method further comprises:

[0054] Acquiring a displacement of an electronic device for collecting the sequence of images determined based on motion sensor data;

[0055] Obtaining the confidence level of the displacement of the target object;

[0056] Based on the confidence level, determining a first weight of the displacement determined based on the motion sensor data and a second weight of the displacement of the target object;

[0057] The performing image stabilization processing on the current frame image and outputting a target image comprises:

[0058] Based on the displacement determined by the motion sensor data and the corresponding first weight, based on the displacement of the target object and the corresponding second weight, the current frame image is stabilized, and the target image is output.

[0059] In some embodiments, determining, based on the confidence level, a first weight of the displacement determined based on the motion sensor data and a second weight of the displacement of the target object comprises:

[0060] Obtaining a confidence difference between the confidence and a preset confidence threshold;

[0061] Based on the confidence difference, the confidence is filtered using a predetermined filter function to obtain a filtered confidence;

[0062] The first weight and the second weight are determined based on the post-filtering confidence; wherein the second weight is positively correlated with the post-filtering confidence, and the first weight is negatively correlated with the post-filtering confidence.

[0063] According to a second aspect of an embodiment of the present disclosure, there is provided a photographing device, including:

[0064] A collection module, configured to collect a sequence of images including a target object in response to detecting a shooting instruction; wherein the sequence of images includes adjacent previous frame images and a current frame image;

[0065] A first determination module is configured to determine a first position of a target object in the preceding frame image;

[0066] A second determination module is configured to determine a displacement of the target object in the current frame image relative to the target object in the previous frame image based on a first position of the target object in the previous frame image;

[0067] The image stabilization module is configured to perform image stabilization processing on the current frame image based on the displacement of the target object in the previous frame image and the current frame image, and output the target image.

[0068] In some embodiments, the second determination module is further configured to sample pixel points within the first position area of ​​the target object in the previous frame image to obtain a first pixel point set; use a forward optical flow method to estimate that each pixel point in the first pixel point set corresponds to a second pixel point set in the current frame image; use a reverse optical flow method to estimate that each pixel point in the second pixel point set corresponds to a third pixel point set in the previous frame image; based on the first pixel point set, the second pixel point set and the third pixel point set, estimate the displacement of the target object in the current frame image relative to the target object in the previous frame image.

[0069] In some embodiments, the second determination module is further configured to screen out a first key point whose difference is less than a preset difference threshold from the first pixel point set based on the difference between the first pixel point set and the third pixel point set; screen out a second key point corresponding to the first key point from the second pixel point set; preliminarily estimate the second position of the target object in the current frame image based on the first position of the target object in the previous frame image, the first key point of the previous frame image, and the second key point of the current frame image; determine the displacement of the target object in the current frame image relative to the target object in the previous frame image based on the first position of the target object in the previous frame image and the preliminarily estimated second position of the target object in the current frame image.

[0070] In some embodiments, the second determination module is further configured to determine the median of the coordinate difference based on the coordinate difference between each pixel point in the first pixel point set and the corresponding pixel point in the third pixel point set; and determine the pixel point whose coordinate difference corresponding to the pixel point in the first pixel point set is less than the median of the coordinate difference as the first key point.

[0071] In some embodiments, the apparatus further comprises:

[0072] The third determination module is configured to determine a first confidence level of the displacement of the target object based on a median value of the coordinate difference.

[0073] In some embodiments, the second determination module is further configured to determine the median value of the coordinate translation in the horizontal direction and the median value of the coordinate translation in the vertical direction based on the coordinates of each corresponding pixel point in the first key point and the second key point; preliminarily determine the coordinates of the predetermined position of the target object in the current frame image based on the coordinates of the predetermined position of the target object in the previous frame image, and the median value of the coordinate translation in the horizontal direction and the median value of the coordinate translation in the vertical direction; calculate the distance between any pixel points in the first key point and determine the median value of the first distance, and calculate the distance between any pixel points in the second key point and determine the median value of the second distance; preliminarily determine the width and height of the target object in the current frame image based on the width and height of the target object in the previous frame image, and the ratio of the median value of the second distance to the median value of the first distance.

[0074] In some embodiments, the image stabilization module is further configured to filter the displacement of the target object in the previous frame image and the current frame image to obtain a filtered displacement; perform image stabilization processing on the current frame image based on the filtered displacement, and output the target image.

[0075] In some embodiments, the image stabilization module is further configured to determine the Kalman gain based on the first key point and the second key point, the first position of the target object in the previous frame image and the preliminarily estimated second position of the target object in the current frame image; determine the displacement observation value of the target object in the current frame image and the previous frame image based on the first key point and the second key point; and perform Kalman filtering on the displacement estimated based on the optical flow method based on the Kalman gain and the displacement observation value to obtain the filtered displacement.

[0076] In some embodiments, the image stabilization module is further configured to calculate a first center point of the first key point and a second center point of the second key point; calculate a first coordinate difference between the center position of the target object in the previous frame image and the first center point, and a second coordinate difference between the preliminarily estimated center position of the target object in the current frame image and the second center point; determine a covariance of the displacement observation value based on the first coordinate difference and the second coordinate difference; and determine the Kalman gain based on the covariance of the displacement observation value.

[0077] In some embodiments, the image stabilization module is further configured to determine a mean of the difference based on the first coordinate difference and the second coordinate difference; determine a standard deviation of the difference based on the first coordinate difference, the second coordinate difference and the mean of the difference; and determine a covariance of the displacement observation value based on a ratio of the standard deviation of the difference to the mean of the difference.

[0078] In some embodiments, the image stabilization module is further configured to determine the difference between the second center point and the first center point as the displacement observation value.

[0079] In some embodiments, the apparatus further comprises:

[0080] an obtaining module configured to correct the first position of the target object in the previous frame image based on the filtered displacement to obtain a third position of the target object in the current frame image;

[0081] A first acquisition module is configured to acquire a fourth position of the target object in the current frame image detected by a target object detection method other than an optical flow method;

[0082] The fourth determination module is configured to determine a second confidence level of the displacement of the target object based on a difference between the third position and the fourth position.

[0083] In some embodiments, the apparatus further comprises:

[0084] The fifth determination module is configured to determine a target confidence level of the displacement of the target object based on the first confidence level and the second confidence level in response to the existence of the first confidence level and the second confidence level.

[0085] In some embodiments, the apparatus further comprises:

[0086] A second acquisition module is configured to acquire a displacement of an electronic device for collecting the sequence of images determined based on the motion sensor data;

[0087] A third acquisition module is configured to acquire the confidence of the displacement of the target object;

[0088] a weight determination module configured to determine, based on the confidence level, a first weight of the displacement determined based on the motion sensor data and a second weight of the displacement of the target object;

[0089] The image stabilization module is further configured to perform image stabilization processing on the current frame image based on the displacement determined by the motion sensor data and the corresponding first weight, based on the displacement of the target object and the corresponding second weight, and output the target image.

[0090] In some embodiments, the weight determination module is further configured to obtain a confidence difference between the confidence and a preset confidence threshold; filter the confidence using a predetermined filtering function based on the confidence difference to obtain a filtered confidence; determine the first weight and the second weight based on the filtered confidence; wherein the second weight is positively correlated with the filtered confidence, and the first weight is negatively correlated with the filtered confidence.

[0091] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0092] processor;

[0093] a memory for storing processor-executable instructions;

[0094] Wherein, the processor is configured to execute the shooting method as described in the first aspect above.

[0095] According to a fourth aspect of an embodiment of the present disclosure, there is provided a storage medium, including:

[0096] When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the shooting method as described in the first aspect above.

[0097] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0098] In an embodiment of the present disclosure, based on the first position of the target object in the previous frame image, the displacement of the target object in the previous frame image and the current frame image is determined for image stabilization processing, which can improve the effect of image stabilization processing on the current frame image.

[0099] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0101] Figure 1 It is a flow chart of a shooting method shown in an embodiment of the present disclosure.

[0102] Figure 2 This is an example diagram of confidence filtering disclosed in an embodiment of the present disclosure.

[0103] Figure 3 The present invention is a flowchart showing an example of a shooting method according to an embodiment of the present invention.

[0104] Figure 4is a diagram showing a shooting device according to an exemplary embodiment.

[0105] Figure 5 It is a block diagram of an electronic device shown in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0106] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0107] The present disclosure provides a shooting method. Figure 1 is a flow chart of a shooting method shown in an embodiment of the present disclosure, such as Figure 1 As shown, the shooting method includes the following steps:

[0108] S11, in response to detecting a shooting instruction, collecting a sequence of images including a target object; wherein the sequence of images includes adjacent previous frame images and a current frame image;

[0109] S12, determining a first position of the target object in the preceding frame image;

[0110] S13, determining a displacement of the target object in the current frame image relative to the target object in the previous frame image based on the first position of the target object in the previous frame image;

[0111] S14. Based on the displacement of the target object in the previous frame image and the current frame image, perform image stabilization processing on the current frame image, and output a target image.

[0112] The shooting method provided in the embodiments of the present disclosure may be implemented by electronic devices such as smart phones, tablet computers or cameras, and the embodiments of the present disclosure are not limited thereto.

[0113] In step S11, after the electronic device detects the shooting instruction, it collects a sequence of images including the target object. In some embodiments, the shooting instruction may be an instruction to start a camera application, and the electronic device collects a sequence of images of the preview screen according to the instruction; in other embodiments, the shooting instruction may be a video recording instruction, and the electronic device collects a sequence of images according to the instruction. In the embodiment of the present disclosure, the sequence of images may be images within a time window, and each frame image in the sequence of images is adjacent in time, for example, the preceding frame image is I1, and the adjacent current frame image is I2; the target object may be a face, a human body, or other animals, vehicles, and other objects.

[0114] In step S12, the electronic device determines the first position of the target object in the previous frame image. The first position of the target object can be the position of the target object in the previous frame image after image stabilization processing, or can be the position of the target object in the previous frame image before image stabilization processing.

[0115] Taking the target object as a face as an example, face information mostly relies on face detection of the FaceDection (FD) platform. The FD platform is implemented based on deep learning algorithms and uses efficient algorithms and hardware acceleration technologies. It can complete face detection tasks in a short time and can perform face detection in various scenes such as different lighting, angles, and expressions. Therefore, in the embodiment of the present disclosure, the first position of the target object in the preceding frame image before or after image stabilization processing can be obtained based on the FD platform detection. The first position can be identified by a face frame. Exemplarily, in the embodiment of the present disclosure, the first position of I1's face is called the first face frame, which can be expressed as Including the four boundary values ​​in the {top, left, bottom, right} image coordinate system, the four vertices of the face frame can be expressed as the following formula (1):

[0116] p tl ={top,left},p tr ={top,right},p bl ={bottom,left},p br ={bottom,right}(1)

[0117] It should be noted that in the embodiment of the present disclosure, the electronic device can process the sequence images in real time, and for the first frame image in the sequence images, the face detection result of the FD platform can be directly used as the first position of the target object in the first frame image to assist the next frame in performing image stabilization based on the optical flow method of the embodiment of the present disclosure. In addition, if any preceding frame image other than the first frame in the sequence images cannot detect a face after image stabilization, then the embodiment of the present disclosure can also use the face detection result of the FD platform of the preceding frame image before stabilization as the first position of the face in the preceding frame image to perform image stabilization on the current frame image.

[0118] In step S13, the electronic device determines the displacement of the target object in the current frame image relative to the target object in the previous frame image based on the first position of the target object in the previous frame image. For example, the displacement of the target object can be determined based on the optical flow method, or after detecting the position in the current frame image based on a detection method of the target object other than the optical flow method, the displacement is determined based on the difference between the position and the first position. Among them, the optical flow method is a method for calculating the movement direction and speed of a pixel point in an image in time, which calculates motion information by comparing the brightness changes of the pixel points between adjacent frame images. The optical flow method assumes that the brightness of the same target in two adjacent frames in the time domain usually does not change significantly, and the movement distance will be very small. Therefore, the change of pixel brightness in the image frame in the time domain and the correlation of the pixel points in the spatial domain can be used to calculate the pixel motion information between the video frame images. Generally, optical flow refers to the instantaneous rate, which is also equivalent to the displacement of the target point when the time interval is very small (such as between two consecutive frames of the video).

[0119] Exemplarily, when estimating the displacement based on the optical flow method, the unidirectional optical flow method can be used to determine the displacement of the target object in the current frame image relative to the target object in the previous frame image through a single estimation; the bidirectional optical flow method can also be used for a secondary estimation to determine the displacement of the target object in the current frame image relative to the target object in the previous frame image. Among them, the unidirectional method can be the Lucas-Kanade algorithm, the Horn-Schunck algorithm, etc., and the bidirectional optical flow can include one or more of the aforementioned optical flow algorithms, which is not limited in the embodiments of the present disclosure.

[0120] In the embodiment of the present disclosure, the displacement of the target object in the current frame image relative to the target object in the previous frame image can be the relative displacement of the center of the target object in the current frame image and the previous frame image. In the embodiment of the present disclosure, diff box Indicates that it includes the displacement in the horizontal direction and the displacement in the vertical direction.

[0121] In step S14, the electronic device performs image stabilization processing on the current frame image based on the displacement of the target object in the previous frame image and the current frame image, and outputs the target image.

[0122] In some embodiments, the electronic device may determine a motion transformation matrix between the current frame image and the previous frame image according to the displacement corresponding to the target object, and then obtain the current frame image after image stabilization processing based on the previous frame image and the motion transformation matrix.

[0123] In other embodiments, the electronic device may also perform image stabilization processing on the current frame image based on the displacement amount and the displacement amount detected by a motion sensor in the electronic device.

[0124] It should be noted that the jitter between the current frame image after image stabilization and the previous frame image is small, and the user can experience a smoother video viewing experience. Taking face video recording or selfie preview as an example, a face frame will be displayed on the screen. The electronic device can perform image stabilization on each current frame image based on the method of the disclosed embodiment, and then perform face detection on the current frame image after image stabilization and display the face frame. Since the jitter between the current frame image after image stabilization is small relative to the previous frame image, the jitter of the corresponding face frame will also be small, so the user can experience a more stable face detection result.

[0125] The disclosed embodiment determines the displacement of the target object in the previous frame image and the current frame image based on the first position of the target object in the previous frame image to perform image stabilization processing, which can improve the effect of image stabilization processing on the current frame image.

[0126] In some embodiments, determining the displacement of the target object in the current frame image relative to the target object in the previous frame image based on the first position of the target object in the previous frame image includes:

[0127] Sampling pixel points in a first position area of ​​the target object in the previous frame image to obtain a first pixel point set;

[0128] Using a forward optical flow method, estimate that each pixel in the first pixel set corresponds to a second pixel set in the current frame image;

[0129] Using the inverse optical flow method, estimate that each pixel in the second pixel set corresponds to a third pixel set in the previous frame image;

[0130] Based on the first pixel point set, the second pixel point set and the third pixel point set, a displacement of the target object in the current frame image relative to the target object in the previous frame image is estimated.

[0131] In the disclosed embodiment, the first position of the target object is identified by the first face frame as an example for explanation. When the electronic device samples the pixel points in the area identified by the first face frame in the previous frame image, uniform sampling can be performed to cover the face pixel points at different positions of the face as much as possible. For example, the first pixel point set obtained by uniform sampling can be represented as pointset1. Subsequently, the electronic device performs bidirectional optical flow calculation, and estimates that each pixel point in the first pixel point set corresponds to a second pixel point set in the current frame image based on the forward optical flow method, which can be represented as pointset2. Then, the electronic device estimates that each pixel point in the second pixel point set corresponds to a third pixel point set in the previous frame image based on the reverse optical flow method, which can be represented as pointset1. *For example, the electronic device determines the tracking result pointset2 of the pixel points in pointset1 on I2 through the forward calculation of LK sparse optical flow; and then determines the tracking result pointset1 of the pixel points in pointset2 on I1 through the reverse calculation of LK sparse optical flow. * .

[0132] The electronic device can estimate the displacement of the target object in the current frame image relative to the target object in the previous frame image based on the first pixel point set, the second pixel point set, and the third pixel point set determined by the bidirectional optical flow. In some embodiments, the points in the first pixel point set can be screened based on the difference between the first pixel point set and the third pixel point set, and the displacement is determined based on the key points screened in the first pixel point set combined with the second pixel point set.

[0133] In other embodiments, for each pixel in the first pixel set, the coordinate mean of the pixel and the tracking pixel in the third pixel set is determined, and then the displacement is determined based on the coordinate mean of each pixel and the coordinate value of the tracking pixel in the second pixel set. For example, the coordinate of the central pixel can be determined based on the coordinate mean of each pixel corresponding to the first pixel set and the third pixel set, and then the difference between the coordinate of the central pixel and the coordinate of the central pixel of the second pixel set is determined as the displacement of the target object. The central pixel can also be determined based on the coordinate mean or median.

[0134] It should be noted that when performing optical flow estimation, each pixel point corresponds to a tracking point. For example, each pixel point in the first pixel point set has a corresponding tracking pixel point in the current frame image during forward optical flow estimation, and the tracking pixel point belongs to the point in the second pixel point set; and in reverse optical flow, each pixel point in the second pixel point set also has a corresponding tracking pixel point in the previous frame image, that is, a point in the third pixel point set. Based on this, each pixel point in the first pixel point set also has a corresponding tracking pixel point in the third pixel point set.

[0135] In the embodiment of the present disclosure, a second pixel point set and a third pixel point set are determined based on the initially sampled first pixel point set using a bidirectional optical flow estimation method, which facilitates the electronic device to estimate the displacement based on more optical flow tracking information, thereby improving the accuracy of the displacement estimation of the target object.

[0136] In some embodiments, estimating the displacement of the target object in the current frame image relative to the target object in the previous frame image based on the first pixel point set, the second pixel point set, and the third pixel point set includes:

[0137] Based on the difference between the first pixel point set and the third pixel point set, selecting a first key point whose difference is less than a preset difference threshold from the first pixel point set;

[0138] Filtering out a second key point corresponding to the first key point from the second pixel point set;

[0139] Preliminarily estimating a second position of the target object in the current frame image based on a first position of the target object in the previous frame image, a first key point of the previous frame image, and a second key point of the current frame image;

[0140] Based on a first position of the target object in the previous frame image and a preliminarily estimated second position of the target object in the current frame image, a displacement of the target object in the current frame image relative to the target object in the previous frame image is determined.

[0141] In the embodiment of the present disclosure, since the first pixel point set is obtained by sampling the pixel points on the previous frame image, and the third pixel point set is predicted based on the bidirectional optical flow method for the previous frame image, the difference between the first pixel point set and the third pixel point set can verify whether the key points of the initial sampling are reasonable.

[0142] In the embodiment of the present disclosure, the pixel points in the first pixel point set whose difference is smaller than the preset difference threshold are taken as the first key points, for example, represented as pointset1 efficient . It should be noted that, since the optical flow method assumes that the brightness of the same target in two adjacent frames in the time domain usually does not change significantly, and the movement distance will be very small, the difference between the first pixel point set and the third pixel point set can be characterized by the difference in pixel values ​​between the corresponding pixel points in the set, or by the difference in coordinates. Among them, the corresponding pixel points include the pixel points in the aforementioned first pixel point set and the tracking pixel points in the third pixel point set. In addition, in the embodiment of the present disclosure, the preset difference threshold can be a preset value, or it can be a value determined based on the statistical value of the difference, such as the median or mean of the coordinate difference.

[0143] In the embodiment of the present disclosure, after the electronic device selects the first key point in the previous frame image, that is, based on the first key point, the electronic device selects the corresponding second key point in the second pixel point set, which can be expressed as pointset2 efficient , the first key point and the second key point can be understood as the effective tracking point pairs in the previous and next frames. On this basis, the electronic device preliminarily estimates the second position of the target object in the current frame image based on the effective tracking point pairs and the first position of the target object, and thus determines the displacement of the target object in the current frame image relative to the target object in the previous frame image based on the first position and the preliminarily estimated second position.

[0144] It should be noted that, in the embodiment of the present disclosure, when preliminarily estimating the second position of the target object in the current frame image based on the effectively tracked point pair and the first position, for example, when estimating the second face frame in the current frame image, the electronic device may respectively estimate the four boundary values ​​of the second face frame, and when determining the displacement of the face in the current frame image relative to the face in the previous frame image based on the first face frame and the preliminarily estimated second face frame, the offset between the position center of the first face frame and the preliminarily estimated second face frame may be used as the displacement of the face. In the embodiment of the present disclosure, the second face frame preliminarily estimated by the electronic device may be expressed as

[0145] It can be understood that in the embodiment of the present disclosure, the electronic device selects a first key point with a small difference to guide the preliminary estimation of the second position of the target object in the current frame image, and determines the displacement of the target object based on the preliminary estimated second position, which can improve the accuracy of the displacement determination.

[0146] In some embodiments, based on the difference between the first pixel point set and the third pixel point set, selecting a first key point having a difference less than a preset difference threshold from the first pixel point set includes:

[0147] Determine a median value of the coordinate difference based on a coordinate difference between each pixel point in the first pixel point set and a corresponding pixel point in the third pixel point set;

[0148] A pixel point in the first pixel point set whose coordinate difference corresponding to the pixel point is less than the median value of the coordinate difference is determined as the first key point.

[0149] As mentioned above, the difference between the first pixel point set and the third pixel point set can be characterized by the coordinate difference between the corresponding pixel points in the set, and the coordinate difference can refer to the distance between the pixel points. The greater the distance, the greater the difference. The electronic device determines the median of the difference based on the coordinate difference of the corresponding pixel points, and determines the pixel point whose coordinate difference corresponding to the pixel point in the first key point set is less than the median of the difference as the first key point.

[0150] It can be understood that, on the one hand, since the coordinates of the pixel points reflect the actual positions of the pixel points, the coordinate difference can more accurately reflect the effectiveness of tracking; on the other hand, selecting the median as the threshold of the coordinate difference can reduce the influence of extreme values ​​(maximum coordinate difference and minimum coordinate difference). Based on the above two aspects, the accuracy of determining the first key point can be comprehensively improved.

[0151] In some embodiments, the method further comprises:

[0152] Based on the median of the coordinate differences, a first confidence level of the displacement of the target object is determined.

[0153] In the embodiment of the present disclosure, the electronic device further generates a pixel set based on the first pixel point set (eg, pointset1) and the third pixel point set (eg, pointset1 * ) is used to determine the first confidence of the displacement estimated by the optical flow method. When the points calculated by the forward optical flow and the backward optical flow cannot coincide well, the median of the coordinate difference can well reflect the error of the optical flow calculation. Therefore, the embodiment of the present disclosure can use the median of the coordinate difference to measure the confidence of the displacement obtained by the optical flow method, that is, the first confidence of the displacement of the target object.

[0154] Exemplarily, the electronic device may determine the first confidence level based on the following formula (2):

[0155] confidence1=(MediaError–Threshold) / Threshold (2)

[0156] Among them, MediaError is the median value of the coordinate difference, and Threshold is the preset threshold, which is generally set to 10.

[0157] As shown in formula (2), if the median value of the coordinate difference is larger, it means that the accuracy of the optical flow estimation is lower, and the first confidence level is positively correlated with the median value of the coordinate difference. At this time, the first confidence level represents the degree of unreliability.

[0158] It can be understood that in the embodiment of the present disclosure, the electronic device evaluates the accuracy of the displacement of the target object estimated based on the optical flow method based on the median of the difference in coordinates of corresponding pixel points in the first pixel point set initially sampled and the third pixel point set of bidirectional optical flow tracking. The solution is simple and effective.

[0159] In some embodiments, preliminarily estimating the second position of the target object in the current frame image based on the first position of the target object in the previous frame image, the first key point of the previous frame image, and the second key point of the current frame image includes:

[0160] Based on the coordinates of the corresponding pixel points in the first key point and the second key point, determine the median value of the coordinate translation in the horizontal direction and the median value of the coordinate translation in the vertical direction;

[0161] Preliminarily determining the coordinates of the predetermined position of the target object in the current frame image based on the coordinates of the predetermined position of the target object in the previous frame image, and the median value of the coordinate translation in the horizontal direction and the median value of the coordinate translation in the vertical direction;

[0162] Calculating the distance between any pixels in the first key points and determining a first distance median, and calculating the distance between any pixels in the second key points and determining a second distance median;

[0163] Based on the width and height of the target object in the previous frame image and the ratio of the second distance median to the first distance median, the width and height of the target object in the current frame image are preliminarily determined.

[0164] In the implementation of the present disclosure, the predetermined position can be the upper left corner point, the lower right corner point or the center point, etc., and the present disclosure embodiment does not limit this. As mentioned above, the first key point and the second key point can be understood as the effectively tracked point pairs in the previous and next frames. In the present disclosure embodiment, the electronic device counts the median of the translation amount of each point pair (i.e., each corresponding pixel point) in the predetermined coordinate direction based on the effectively tracked point pairs, so as to preliminarily determine the coordinates of the upper left corner point of the second face frame based on the median of the translation amount in the predetermined coordinate direction, for example, in combination with the coordinates of the upper left corner point of the first face frame (i.e., top, left value), that is, determine The corresponding p tl value.

[0165] Exemplarily, the electronic device determines the horizontal and vertical translation amounts dx and dy of the corresponding pixel points of the first key point and the second key point respectively, and sorts them. Then the electronic device takes the median of the translation amounts dx and dy as the translation amount of the upper left corner point. The translation coordinates of the upper left corner can be expressed as diff_box, and the coordinates of the upper left corner point of the second face frame are the sum of the coordinates of the upper left corner point of the first face frame and diff_box.

[0166] In addition, the electronic device determines the first distance median according to the distance between any pixels in the first key point, and determines the second distance median according to the distance between any pixels in the second key point, and then determines the width and height changes of the human target object based on the ratio of the distances, so as to preliminarily estimate the width and height of the target object in the current frame image. The electronic device can locate the target object in the current frame image based on the coordinates of the target object's predetermined position in the current frame image, and the width and height of the target object. For example, based on the determination of the width and height, it can be determined The corresponding p tr 、p bl 、p br value, thereby achieving the preliminary positioning of the second face frame.

[0167] If the electronic device calculates the distance between any pixel points within the first key point and sorts them to obtain a median distance value of D1; calculates the distance between any pixel points within the second key point and sorts them to obtain a median distance value of D2, then the ratio of distance change box_ratio can be determined to be D2 / D1. For example, the width of the second face frame can be the product of the width of the first face frame and box_ratio, and the height of the second face frame can be the product of the height of the first face frame and box_ratio.

[0168] It can be understood that in the embodiments of the present disclosure, on the one hand, the electronic device estimates the coordinates of the predetermined position of the target object in the current frame image and the width and height of the target object respectively, and can estimate the second position of the target object more comprehensively based on the first key point and the second key point determined by the bidirectional optical flow, thereby facilitating the determination of the displacement; on the other hand, in the process of estimating the second position of the target object, the coordinates of the predetermined position of the target object are determined based on the median of the coordinate offset, and the scaling ratio of the width and height is determined based on the ratio between the medians of the distance changes. Since the median can reduce the impact of extreme values, the accuracy of the preliminary estimate of the second position of the target object can be improved in this way.

[0169] In some embodiments, performing image stabilization processing on the current frame image based on the displacement of the target object in the previous frame image and the current frame image, and outputting the target image, includes:

[0170] Filtering the displacement of the target object in the previous frame image and the current frame image to obtain a filtered displacement;

[0171] The current frame image is subjected to image stabilization processing based on the filtered displacement, and the target image is output.

[0172] In the disclosed embodiment, the displacement of the target object in the previous frame image and the current frame image is filtered, for example, Kalman filtering, Bayesian filtering, etc. can be used to obtain the optimal estimation of the displacement. After determining the displacement after filtering, the electronic device can perform image stabilization processing on the current frame image based on the displacement after filtering, and output the target image.

[0173] In practical applications, the performance of the optical flow method is affected by many factors, such as image quality, motion speed, lighting conditions, etc. Therefore, the accuracy of determining the displacement based solely on the optical flow method is insufficient. In this regard, the embodiment of the present disclosure estimates the displacement based on the first position of the target object in the previous frame image using the optical flow method, and then further filters and adjusts the displacement, which can improve the accuracy of the displacement determination, thereby further improving the effect of image stabilization processing on the current frame image.

[0174] It should be noted that, since the displacement after filtering is obtained by filtering the displacement estimated by the optical flow method, the first confidence of the displacement estimated by the optical flow method can also be represented as the confidence of the displacement after filtering.

[0175] In some embodiments, filtering the displacement of the target object in the previous frame image and the current frame image to obtain the filtered displacement includes:

[0176] Determine a Kalman gain according to the first key point and the second key point, a first position of the target object in the previous frame image and a preliminarily estimated second position of the target object in the current frame image;

[0177] Determining, according to the first key point and the second key point, a displacement observation value of the target object in the current frame image and the previous frame image;

[0178] The displacement estimated based on the optical flow method is subjected to Kalman filtering based on the Kalman gain and the displacement observation value to obtain the filtered displacement.

[0179] In the disclosed embodiment, the electronic device may perform a priori estimation based on the first key point and the second key point, as well as the first position and the preliminarily estimated second position of the target object to determine the Kalman gain. In addition, the electronic device further determines the displacement observation value based on the first key point and the second key point, so that the electronic device can perform Kalman filtering on the displacement amount based on the Kalman gain and the displacement observation value. Exemplarily, the Kalman filter is shown in the following formula (3):

[0180] μ=diff box +K t *(diff kp_meas -diff box ) (3)

[0181] Among them, diff box is the displacement determined based on the optical flow method, diff kp_meas is the displacement observation value, K t is the Kalman gain, and the displacement after filtering is μ.

[0182] In the embodiment of the present disclosure, after determining the displacement after filtering, the electronic device can also correct the first position, so as to obtain the final position of the target object in the current frame image. Assume that the third face frame in the current frame image is finally expressed as μ is the diff box The displacement after filtering can be calculated by the following formula (4) to obtain the third face frame:

[0183]

[0184] Among them, μ.x is the horizontal displacement of the face in the image coordinate system, and μ.y is the vertical displacement of the face in the image coordinate system.

[0185] It can be understood that in the embodiment of the present disclosure, the electronic device further determines the correlation coefficient of the Kalman filter based on the screened first key point and the second key point and in combination with the first position and the preliminarily estimated second position, which can improve the accuracy of the filtering and thereby obtain a more accurate displacement after filtering.

[0186] In some embodiments, determining the Kalman gain according to the first key point and the second key point, a first position of the target object in the previous frame image and a preliminarily estimated second position of the target object in the current frame image includes:

[0187] Calculating a first center point of the first key point and a second center point of the second key point;

[0188] Calculating a first coordinate difference between the center position of the target object in the previous frame image and the first center point, and a second coordinate difference between the center position of the target object in the current frame image and the second center point that is preliminarily estimated;

[0189] Determining a covariance of the displacement observation value according to the first coordinate difference and the second coordinate difference;

[0190] The Kalman gain is determined according to the covariance of the displacement observations.

[0191] In the embodiment of the present disclosure, the key points (including the first key point and the second key point) can be represented as face land ={kp1,kp2,…,kp n}, where kp i =(x i ,y i ) is any key point. The electronic device can calculate the first center point and the second center point based on the following formula (5):

[0192]

[0193] Taking the calculation of the first center point as an example, kp in the above formula (5) is i For pointset1 efficient The coordinate values ​​of each key point in .

[0194] In the embodiment of the present disclosure, the first center point and the second center point are both the average values ​​of the coordinates in the horizontal direction and the vertical direction respectively. The first center point can be expressed as The second center point can be expressed as

[0195] After determining the first center point and the second center point, the electronic device further calculates a first coordinate difference between the center position of the target object in the previous frame image and the first center point, and a preliminarily estimated second coordinate difference between the center position of the target object in the current frame image and the second center point. Exemplarily, the electronic device may calculate the first coordinate difference and the second coordinate difference according to the following formula (6):

[0196]

[0197] Among them, face bbx For face frames (e.g. or ), kp center For the center point (e.g. or ).

[0198] In the embodiment of the present disclosure, after determining the first coordinate difference and the second coordinate difference, the electronic device can determine the covariance difference of the displacement observation value. In some embodiments, determining the covariance of the displacement observation value according to the first coordinate difference and the second coordinate difference includes:

[0199] Determining a mean value of the differences according to the first coordinate difference and the second coordinate difference;

[0200] determining a standard deviation of the differences based on the first coordinate difference, the second coordinate difference, and a mean of the differences;

[0201] The covariance of the displacement observations is determined based on a ratio of a standard deviation of the differences to a mean of the differences.

[0202] In the embodiment of the present disclosure, the electronic device may determine the mean of the differences according to the following formula (7):

[0203]

[0204] Among them, q i Represents the coordinate difference, including the first coordinate difference and the second coordinate difference. The value of n is consistent with the value of q_size, both of which are 2.

[0205] After determining the mean of the difference, the electronic device can further determine the standard deviation of the difference by combining the first coordinate difference and the second coordinate difference, and then determine the covariance of the displacement observation value. The following formula (8) is a calculation formula for the ratio of the standard deviation of the difference to the mean of the difference:

[0206]

[0207] Among them, the molecule is the standard deviation of the difference, the denominator Q mean is the mean of the difference, the present disclosure is an embodiment, dis fac It is also called discrete factor.

[0208] The electronic device can determine the covariance of the displacement observations based on the ratio of the standard deviation of the difference to the mean of the difference (discrete factor) according to the following formula (9):

[0209] R t =pow(BASE,dis fac ) (9)

[0210] In the above formula, BASE is the preset value, R t is the covariance of the displacement observations.

[0211] In the disclosed embodiment, the electronic device may determine the Kalman gain according to the covariance of the displacement observation value according to the following formula (10):

[0212] K t =P t / (P t +R t ) (10)

[0213] Among them, K t That is the Kalman gain, P t is the variance.

[0214] It should be noted that, in the embodiment of the present disclosure, when the Kalman gain is calculated for the first time, P t The value of is a preset value, and the variance P can be calculated based on the Kalman gain obtained previously according to the following formula (11). t To update:

[0215] P t =(1-K t )*(P t +Q t ) (11)

[0216] Among them, Q t is the empirical threshold and is the hyperparameter of Kalman.

[0217] In some embodiments, determining the displacement observation value of the target object in the current frame image and the previous frame image according to the first key point and the second key point includes:

[0218] The difference between the second center point and the first center point is determined as the displacement observation value.

[0219] In the embodiment of the present disclosure, the displacement observation value can be determined based on the following formula (12):

[0220]

[0221] in, is the second center point, The first center point.

[0222] It can be understood that the embodiment of the present disclosure determines the covariance of the displacement observation value based on the first coordinate difference between the center position of the target object in the previous frame image and the first center point, and the second coordinate difference between the center position of the target object in the current frame image and the second center point preliminarily estimated to further determine the Kalman gain, and determines the displacement observation value by the difference between the second center point and the first center point, so that the electronic device can filter the displacement based on the Kalman filter. Since the first center point and the second center, as well as the preliminarily estimated second position are all determined based on the effective tracking point pair determined based on the bidirectional optical flow, the correlation value of the Kalman filter determined based on the above values ​​can improve the accuracy of the filtering, so that the electronic device can obtain a more accurate filtered displacement.

[0223] In some embodiments, the method further comprises:

[0224] Correcting a first position of the target object in the previous frame image based on the filtered displacement to obtain a third position of the target object in the current frame image;

[0225] Acquire a fourth position of the target object in the current frame image detected by a target object detection method other than the optical flow method;

[0226] Based on the difference between the third position and the fourth position, a second confidence level of the displacement of the target object is determined.

[0227] In the embodiment of the present disclosure, for example, the third position of the target object in the current frame image can be obtained by the method of the above formula (4). In addition, in the embodiment of the present disclosure, the electronic device can also detect the target object in the current frame image based on the FD platform to obtain a fourth position (for example, a fourth face frame), and then determine the second confidence level based on the difference between the third position and the fourth position.

[0228] For example, the third position and the fourth position are both represented by a face frame, and the second confidence level can be determined based on the following formula (13):

[0229] confidence2=width*height / Pre_Height* Pre_Width (13)

[0230] It should be noted that the numerator width*height in formula (13) is the third person face frame The denominator Pre_Height*Pre_Width is the area of ​​the fourth face frame of the FD platform, and confidence2 is the second confidence.

[0231] Theoretically, assuming that the accuracy of the fourth face frame detected by the FD platform is high, if the accuracy of the filtered displacement estimated based on the optical flow method is high, then the accuracy of the third face frame is also high, so that the area ratio of the third face frame and the fourth face frame is close to 1, and if the area ratio is very large or very small, it means that the accuracy of the optical flow method estimation is not enough, so the above formula (13) can be used to measure the estimation result of the optical flow method. In the embodiment of the present disclosure, the second confidence level can also represent the degree of unreliability.

[0232] It can be understood that in the embodiment of the present disclosure, the electronic device combines the detection results of other target detection methods (for example, for faces, it combines the face detection results of the FD platform) and the third position of the target object determined by the filtered displacement to jointly determine the second confidence level, which can better measure the accuracy of the filtered displacement of the current target object.

[0233] In some embodiments, the method further comprises:

[0234] In response to the existence of the first confidence level and the second confidence level, a target confidence level of the displacement amount of the target object is determined based on the first confidence level and the second confidence level.

[0235] In the embodiment of the present disclosure, the electronic device may determine the target confidence based on the average of the first confidence and the second confidence, or may determine the target confidence based on the maximum value or the minimum value.

[0236] Exemplarily, the electronic device may determine the target confidence based on the following formula (14):

[0237] confidenceRatio=min(confidence1,confidence2) (14)

[0238] Among them, confidence1 can be based on the aforementioned first confidence, confidence2 can be the aforementioned second confidence, and confidenceRatio is the target confidence.

[0239] In the disclosed embodiment, the target confidence is determined based on the confidence obtained in a variety of ways, which can improve the accuracy of measuring the displacement of the target object.

[0240] In some embodiments, the method further comprises:

[0241] Acquiring a displacement of an electronic device for collecting the sequence of images determined based on motion sensor data;

[0242] Obtaining the confidence level of the displacement of the target object;

[0243] Based on the confidence level, determining a first weight of the displacement determined based on the motion sensor data and a second weight of the displacement of the target object;

[0244] The performing image stabilization processing on the current frame image and outputting a target image comprises:

[0245] Based on the displacement determined by the motion sensor data and the corresponding first weight, based on the displacement of the target object and the corresponding second weight, the current frame image is stabilized, and the target image is output.

[0246] In the embodiment of the present disclosure, the electronic device may also determine the displacement of the electronic device that collects the sequence of images based on the motion sensor data, for example, based on the gyroscope data. The confidence level of the electronic device obtaining the displacement of the target object may be the aforementioned first confidence level, second confidence level, or target confidence level, which is not limited in the embodiment of the present disclosure.

[0247] In the disclosed embodiment, the electronic device allocates a weight to the displacement determined based on the motion sensor data and a weight based on the displacement of the target object based on the confidence level of the displacement of the target object. On the one hand, since the movement of the electronic device may also cause image jitter, adding the displacement determined based on the motion sensor data to perform anti-shake during image stabilization processing can improve the image stabilization effect. On the other hand, the electronic device allocates weights to different displacement acquisition methods based on the confidence level of the displacement of the target object. Compared with the method of allocating fixed weights, the weight of the reference displacement can be dynamically determined based on the displacement of the target object, thereby helping to improve the real-time image stabilization effect.

[0248] In some embodiments, determining, based on the confidence level, a first weight of the displacement determined based on the motion sensor data and a second weight of the displacement of the target object comprises:

[0249] Obtaining a confidence difference between the confidence and a preset confidence threshold;

[0250] Based on the confidence difference, the confidence is filtered using a predetermined filter function to obtain a filtered confidence;

[0251] The first weight and the second weight are determined based on the post-filtering confidence; wherein the second weight is positively correlated with the post-filtering confidence, and the first weight is negatively correlated with the post-filtering confidence.

[0252] In the embodiment of the present disclosure, the electronic device may determine the confidence difference between the confidence and the preset confidence threshold based on the following formula (15):

[0253] delta=confidenceRatio-Threshold (15)

[0254] Among them, delta is the confidence difference, confidenceRatio is the confidence, and Threshold is the preset confidence threshold, for example, set to 3.6.

[0255] In the disclosed embodiment, the electronic device filters the confidence using a predetermined filtering function based on the confidence difference, so as to determine the second weight of the displacement of the target object based on the confidence after filtering and the first weight of the displacement determined based on the motion sensor data. It should be noted that in the disclosed embodiment, when the confidence after filtering is high, it means that the displacement of the target object is relatively reliable. At this time, the second weight can be set to be large, and the first weight can be set to be small, so that the electronic device can perform image stabilization processing based on the displacement of the target object more, thereby improving the accuracy of anti-shake; and when the confidence after filtering is low, setting the second weight to be small and the first weight to be large can enable the electronic device to perform image stabilization processing based on the displacement determined by the motion sensor data more, thereby improving the accuracy of anti-shake. In the disclosed embodiment, the second weight is positively correlated with the confidence after filtering, and the first weight is negatively correlated with the confidence after filtering.

[0256] It can be understood that in the embodiment of the present disclosure, the first weight and the second weight are dynamically determined based on the post-filtering confidence, so that the second weight is positively correlated with the post-filtering confidence, and the first weight is negatively correlated with the post-filtering confidence, which enables the electronic device to guide the weight setting based on the post-filtering confidence to improve the image stabilization processing effect.

[0257] In some embodiments, the predetermined filter function comprises:

[0258]

[0259] Among them, preConfidence is the filtered confidence corresponding to the previous frame image, exitRatio, enterRatio and enterThreshold are all preset values, delta is the confidence difference, and Confidence is the filtered confidence corresponding to the current frame image.

[0260] For example, the value of exitRatio may be 0.003, the value of enterRatio may be 0.03, and the value of enterThreshold may be 1.2. It should be noted that if the preceding frame image is the first frame image of the sequence image, the confidence of the first frame image after filtering may be set to 1.

[0261] Figure 2 This is an example diagram of a confidence filter disclosed in an embodiment of the present disclosure, such as Figure 2 As shown, L1 indicates the confidence Ratio value of the filtered displacement obtained based on the optical flow method, and L2 indicates the value of the preset confidence threshold Threshold. Based on the function of formula (16), the input confidence Ratio becomes a sine curve, and the output result Confidence changes as follows: when the confidence difference delta is greater than 0 (i.e., confidence Ratio>Threshold), Confidence decreases rapidly, and when the confidence difference is less than 0 (confidence Ratio<Threshold), Confidence rises slowly.

[0262] After filtering based on the above filter function, the values ​​of the first weight and the second weight also change with the Confidence. As mentioned above, the second weight is positively correlated with the confidence after filtering and the first weight is negatively correlated with the confidence after filtering. It can be seen that when the Confidence drops rapidly, the second weight decreases rapidly and the first weight increases rapidly, so the electronic device can quickly switch to anti-shake based on the displacement determined by the motion sensor data; and when the Confidence rises slowly, the second weight increases slowly and the first weight decreases slowly, so the electronic device can relatively smoothly combine the displacement determined by the motion sensor data and the displacement of the target object (such as the displacement before or after filtering) for image stabilization. It can be understood that the accuracy and smoothness of anti-shake can be improved in this way.

[0263] Figure 3 FIG. 1 is an example flow chart of a shooting method shown in an embodiment of the present disclosure, such as Figure 3 As shown, the following steps are included:

[0264] S21. Input data.

[0265] In the embodiment of the present disclosure, the input data may be a sequence of images including a target object acquired based on a detected shooting instruction, where the target object is a human face.

[0266] S22, whether the previous frame tracking is successful, if not, execute step S23 and continue to execute step S24; if yes, execute step S24.

[0267] S23, reset the optical flow detector.

[0268] In the embodiment of the present disclosure, the previous frame is the previous frame image. When the previous frame tracking is successful, the bidirectional optical flow tracking is continued; when the previous frame tracking fails, Figure 3 As shown, the optical flow detector is reset using the FD detection frame, that is, the first position of the face in the previous frame image is detected using an FD detection algorithm other than the optical flow method, wherein the previous frame image may be an image after image stabilization processing or an image without image stabilization processing. It should be noted that the first position of the face in the previous frame image when the previous frame tracking is successful may also be represented by the third face frame obtained by obtaining the filtered displacement based on the optical flow method.

[0269] S24, bidirectional optical flow tracking module.

[0270] In the disclosed embodiment, the bidirectional optical flow tracking module is the process of estimating the second pixel point set using the forward optical flow method and estimating the third pixel point set using the reverse optical flow method to estimate the displacement of the target object in the current frame image relative to the target object in the previous frame image.

[0271] S25. Calculate the first optical flow error.

[0272] In the disclosed embodiment, the first optical flow error is the aforementioned first confidence.

[0273] S26, smoothing the optical flow data and outputting smoothed position data.

[0274] In the disclosed embodiment, the optical flow data smoothing process outputs the smoothed position data, that is, the process of performing Kalman filtering on the displacement of the target object in the previous frame image and the current frame image to obtain the filtered displacement.

[0275] S27. Calculate the second optical flow error.

[0276] In the disclosed embodiment, calculating the second optical flow error is the process of calculating the aforementioned second confidence in combination with the FD detection frame.

[0277] S28, fast-in slow-out filter, to obtain the confidence of the position data.

[0278] In the disclosed embodiment, the fast-in slow-out filter is the aforementioned predetermined filter function, and the electronic device obtains the filtered confidence corresponding to the current frame image based on the first optical flow error and the second optical flow error using the predetermined filter function.

[0279] S29, image stabilization algorithm.

[0280] In the disclosed embodiment, the electronic device allocates a first weight of the displacement determined based on the motion sensor data and a second weight of the displacement obtained after filtering based on the optical flow method based on the confidence of the position data (confidence after filtering), and performs anti-shake processing on the current frame image in combination with the displacement determined by the motion sensor data and the displacement obtained after filtering based on the optical flow method to improve the stability of the current frame image. Based on the improvement of the stability of the current frame image, for example, during the Selfie process, the face frame in the preview screen can be displayed smoothly.

[0281] Figure 4 FIG. 1 is a diagram showing a shooting device according to an exemplary embodiment. Figure 4 , the device comprises:

[0282] The acquisition module 101 is configured to acquire a sequence of images including a target object in response to detecting a shooting instruction; wherein the sequence of images includes adjacent previous frame images and a current frame image;

[0283] A first determination module 102 is configured to determine a first position of a target object in the preceding frame image;

[0284] A second determination module 103 is configured to determine a displacement of the target object in the current frame image relative to the target object in the previous frame image based on the first position of the target object in the previous frame image;

[0285] The image stabilization module 104 is configured to perform image stabilization processing on the current frame image based on the displacement of the target object in the previous frame image and the current frame image, and output a target image.

[0286] In some embodiments, the second determination module 103 is further configured to sample pixel points within the first position area of ​​the target object in the previous frame image to obtain a first pixel point set; use a forward optical flow method to estimate that each pixel point in the first pixel point set corresponds to a second pixel point set in the current frame image; use a reverse optical flow method to estimate that each pixel point in the second pixel point set corresponds to a third pixel point set in the previous frame image; based on the first pixel point set, the second pixel point set and the third pixel point set, estimate the displacement of the target object in the current frame image relative to the target object in the previous frame image.

[0287] In some embodiments, the second determination module 103 is further configured to screen out a first key point whose difference is less than a preset difference threshold from the first pixel point set based on the difference between the first pixel point set and the third pixel point set; screen out a second key point corresponding to the first key point from the second pixel point set; preliminarily estimate the second position of the target object in the current frame image based on the first position of the target object in the previous frame image, the first key point of the previous frame image, and the second key point of the current frame image; determine the displacement of the target object in the current frame image relative to the target object in the previous frame image based on the first position of the target object in the previous frame image and the preliminarily estimated second position of the target object in the current frame image.

[0288] In some embodiments, the second determination module 103 is further configured to determine the median of the coordinate difference based on the coordinate difference between each pixel point in the first pixel point set and the corresponding pixel point in the third pixel point set; and determine the pixel point whose coordinate difference corresponding to the pixel point in the first pixel point set is less than the median of the coordinate difference as the first key point.

[0289] In some embodiments, the apparatus further comprises:

[0290] The third determination module 105 is configured to determine a first confidence level of the displacement of the target object based on the median value of the coordinate difference.

[0291] In some embodiments, the second determination module 103 is further configured to determine the median value of the coordinate translation in the horizontal direction and the median value of the coordinate translation in the vertical direction based on the coordinates of each corresponding pixel point in the first key point and the second key point; preliminarily determine the coordinates of the predetermined position of the target object in the current frame image based on the coordinates of the predetermined position of the target object in the previous frame image, and the median value of the coordinate translation in the horizontal direction and the median value of the coordinate translation in the vertical direction; calculate the distance between any pixel points in the first key point and determine the median value of the first distance, and calculate the distance between any pixel points in the second key point and determine the median value of the second distance; preliminarily determine the width and height of the target object in the current frame image based on the width and height of the target object in the previous frame image, and the ratio of the median value of the second distance to the median value of the first distance.

[0292] In some embodiments, the stabilization module 104 is further configured to filter the displacement of the target object in the previous frame image and the current frame image to obtain a filtered displacement; perform image stabilization processing on the current frame image based on the filtered displacement, and output the target image.

[0293] In some embodiments, the image stabilization module 104 is further configured to determine the Kalman gain based on the first key point and the second key point, the first position of the target object in the previous frame image and the preliminarily estimated second position of the target object in the current frame image; determine the displacement observation value of the target object in the current frame image and the previous frame image based on the first key point and the second key point; and perform Kalman filtering on the displacement estimated based on the optical flow method based on the Kalman gain and the displacement observation value to obtain the filtered displacement.

[0294] In some embodiments, the image stabilization module 104 is further configured to calculate a first center point of the first key point and a second center point of the second key point; calculate a first coordinate difference between the center position of the target object in the previous frame image and the first center point, and a second coordinate difference between the preliminarily estimated center position of the target object in the current frame image and the second center point; determine the covariance of the displacement observation value based on the first coordinate difference and the second coordinate difference; and determine the Kalman gain based on the covariance of the displacement observation value.

[0295] In some embodiments, the image stabilization module 104 is further configured to determine a mean of the difference based on the first coordinate difference and the second coordinate difference; determine a standard deviation of the difference based on the first coordinate difference, the second coordinate difference and the mean of the difference; and determine a covariance of the displacement observation value based on a ratio of the standard deviation of the difference to the mean of the difference.

[0296] In some embodiments, the image stabilization module 104 is further configured to determine the difference between the second center point and the first center point as the displacement observation value.

[0297] In some embodiments, the apparatus further comprises:

[0298] An obtaining module 106 is configured to correct the first position of the target object in the previous frame image based on the filtered displacement to obtain a third position of the target object in the current frame image;

[0299] A first acquisition module 107, configured to acquire a fourth position of the target object in the current frame image detected by a target object detection method other than an optical flow method;

[0300] The fourth determination module 108 is configured to determine a second confidence level of the displacement of the target object based on the difference between the third position and the fourth position.

[0301] In some embodiments, the apparatus further comprises:

[0302] The fifth determination module 109 is configured to determine, in response to the existence of the first confidence level and the second confidence level, a target confidence level of the displacement amount of the target object based on the first confidence level and the second confidence level.

[0303] In some embodiments, the apparatus further comprises:

[0304] A second acquisition module 110 is configured to acquire a displacement of an electronic device for collecting the sequence of images determined based on motion sensor data;

[0305] A third acquisition module 111 is configured to acquire the confidence of the displacement of the target object;

[0306] A weight determination module 112, configured to determine, based on the confidence level, a first weight of the displacement determined based on the motion sensor data and a second weight of the displacement of the target object;

[0307] The image stabilization module 104 is further configured to perform image stabilization processing on the current frame image based on the displacement determined by the motion sensor data and the corresponding first weight, based on the displacement of the target object and the corresponding second weight, and output the target image.

[0308] In some embodiments, the weight determination module 112 is further configured to obtain a confidence difference between the confidence and a preset confidence threshold; filter the confidence based on the confidence difference to obtain a filtered confidence; determine the first weight and the second weight based on the filtered confidence; wherein the second weight is positively correlated with the filtered confidence, and the first weight is negatively correlated with the filtered confidence.

[0309] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0310] Figure 5 8 is a block diagram of an electronic device according to an exemplary embodiment. The electronic device can be identified as a device 800, which can be a smart phone, a personal computer, etc.

[0311] Reference Figure 5 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0312] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0313] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0314] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0315] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0316] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0317] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0318] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0319] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as Wi-Fi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0320] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0321] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0322] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the aforementioned shooting method.

[0323] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims above.

[0324] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A shooting method, characterized in that: The method comprises: In response to detecting a shooting instruction, acquiring a sequence of images including a target object; wherein the sequence of images includes adjacent previous frame images and a current frame image; Determine a first position of the target object in the preceding frame image; Based on a first position of the target object in the previous frame image, determining a displacement of the target object in the current frame image relative to the target object in the previous frame image; Based on the displacement of the target object in the previous frame image and the current frame image, image stabilization is performed on the current frame image, and a target image is output.

2. The method according to claim 1, characterized in that The determining, based on the first position of the target object in the previous frame image, a displacement of the target object in the current frame image relative to the target object in the previous frame image includes: Sampling pixel points in a first position area of ​​the target object in the previous frame image to obtain a first pixel point set; Using a forward optical flow method, estimate that each pixel in the first pixel set corresponds to a second pixel set in the current frame image; Using the inverse optical flow method, estimate that each pixel in the second pixel set corresponds to a third pixel set in the previous frame image; Based on the first pixel point set, the second pixel point set and the third pixel point set, a displacement of the target object in the current frame image relative to the target object in the previous frame image is estimated.

3. The method according to claim 2, characterized in that The estimating, based on the first pixel point set, the second pixel point set, and the third pixel point set, a displacement of the target object in the current frame image relative to the target object in the previous frame image includes: Based on the difference between the first pixel point set and the third pixel point set, selecting a first key point whose difference is less than a preset difference threshold from the first pixel point set; Filtering out a second key point corresponding to the first key point from the second pixel point set; Preliminarily estimating a second position of the target object in the current frame image based on a first position of the target object in the previous frame image, a first key point of the previous frame image, and a second key point of the current frame image; Based on a first position of the target object in the previous frame image and a preliminarily estimated second position of the target object in the current frame image, a displacement of the target object in the current frame image relative to the target object in the previous frame image is determined.

4. The method according to claim 3, characterized in that The step of selecting, based on the difference between the first pixel point set and the third pixel point set, a first key point having a difference less than a preset difference threshold from the first pixel point set includes: Determine a median value of the coordinate difference based on the coordinate difference between each pixel point in the first pixel point set and the corresponding pixel point in the third pixel point set; A pixel point in the first pixel point set whose coordinate difference corresponding to the pixel point is less than the median value of the coordinate difference is determined as the first key point.

5. The method according to claim 4, characterized in that The method further comprises: Based on the median of the coordinate differences, a first confidence level of the displacement of the target object is determined.

6. The method according to claim 3, characterized in that The preliminarily estimating the second position of the target object in the current frame image based on the first position of the target object in the previous frame image, the first key point of the previous frame image, and the second key point of the current frame image includes: Based on the coordinates of the corresponding pixel points in the first key point and the second key point, determine the median value of the coordinate translation in the horizontal direction and the median value of the coordinate translation in the vertical direction; Preliminarily determining the coordinates of the predetermined position of the target object in the current frame image based on the coordinates of the predetermined position of the target object in the previous frame image, and the median value of the coordinate translation in the horizontal direction and the median value of the coordinate translation in the vertical direction; Calculating the distance between any pixels in the first key points and determining a first distance median, and calculating the distance between any pixels in the second key points and determining a second distance median; Based on the width and height of the target object in the previous frame image and the ratio of the second distance median to the first distance median, the width and height of the target object in the current frame image are preliminarily determined.

7. The method according to claim 3, characterized in that The step of performing image stabilization processing on the current frame image based on the displacement of the target object in the previous frame image and the current frame image, and outputting the target image, comprises: Filtering the displacement of the target object in the previous frame image and the current frame image to obtain a filtered displacement; The current frame image is subjected to image stabilization processing based on the filtered displacement, and the target image is output.

8. The method according to claim 7, characterized in that The filtering of the displacement of the target object in the previous frame image and the current frame image to obtain the filtered displacement includes: Determine a Kalman gain according to the first key point and the second key point, a first position of the target object in the previous frame image and a preliminarily estimated second position of the target object in the current frame image; Determining, according to the first key point and the second key point, a displacement observation value of the target object in the current frame image and the previous frame image; The displacement estimated based on the optical flow method is subjected to Kalman filtering based on the Kalman gain and the displacement observation value to obtain the filtered displacement.

9. The method according to claim 8, characterized in that The determining of the Kalman gain according to the first key point and the second key point, a first position of the target object in the previous frame image and a preliminarily estimated second position of the target object in the current frame image comprises: Calculating a first center point of the first key point and a second center point of the second key point; Calculating a first coordinate difference between the center position of the target object in the previous frame image and the first center point, and a second coordinate difference between the center position of the target object in the current frame image and the second center point that is preliminarily estimated; Determining a covariance of the displacement observation value according to the first coordinate difference and the second coordinate difference; The Kalman gain is determined according to the covariance of the displacement observations.

10. The method according to claim 9, characterized in that The determining the covariance of the displacement observation value according to the first coordinate difference and the second coordinate difference comprises: Determining a mean value of the differences according to the first coordinate difference and the second coordinate difference; determining a standard deviation of the differences based on the first coordinate difference, the second coordinate difference, and a mean of the differences; The covariance of the displacement observations is determined based on a ratio of a standard deviation of the differences to a mean of the differences.

11. The method according to claim 9, characterized in that The step of determining the displacement observation value of the target object in the current frame image and the previous frame image according to the first key point and the second key point includes: The difference between the second center point and the first center point is determined as the displacement observation value.

12. The method according to claim 7, characterized in that The method further comprises: Correcting a first position of the target object in the previous frame image based on the filtered displacement to obtain a third position of the target object in the current frame image; Acquire a fourth position of the target object in the current frame image detected by a target object detection method other than the optical flow method; Based on the difference between the third position and the fourth position, a second confidence level of the displacement of the target object is determined.

13. The method according to claim 5 or 12, characterized in that: The method further comprises: In response to the existence of the first confidence level and the second confidence level, a target confidence level of the displacement amount of the target object is determined based on the first confidence level and the second confidence level.

14. The method according to claim 1, characterized in that The method further comprises: Acquiring a displacement of an electronic device for collecting the sequence of images determined based on motion sensor data; Obtaining the confidence level of the displacement of the target object; Based on the confidence level, determining a first weight of the displacement determined based on the motion sensor data and a second weight of the displacement of the target object; The performing image stabilization processing on the current frame image and outputting a target image comprises: Based on the displacement determined by the motion sensor data and the corresponding first weight, and based on the displacement of the target object and the corresponding second weight, the current frame image is stabilized, and the target image is output.

15. The method according to claim 14, characterized in that The determining, based on the confidence level, a first weight of the displacement determined based on the motion sensor data and a second weight of the displacement of the target object comprises: Obtaining a confidence difference between the confidence and a preset confidence threshold; Based on the confidence difference, the confidence is filtered using a predetermined filter function to obtain a filtered confidence; The first weight and the second weight are determined based on the post-filtering confidence; wherein the second weight is positively correlated with the post-filtering confidence, and the first weight is negatively correlated with the post-filtering confidence.

16. A photographing device, characterized in that: The device includes: A collection module, configured to collect a sequence of images including a target object in response to detecting a shooting instruction; wherein the sequence of images includes adjacent previous frame images and a current frame image; A first determination module, configured to determine a first position of a target object in the preceding frame image; A second determination module is configured to determine a displacement of the target object in the current frame image relative to the target object in the previous frame image based on a first position of the target object in the previous frame image; The image stabilization module is configured to perform image stabilization processing on the current frame image based on the displacement of the target object in the previous frame image and the current frame image, and output the target image.

17. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the shooting method according to any one of claims 1 to 15.

18. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the shooting method as claimed in any one of claims 1 to 15.

Images