Method and apparatus for capturing clear image frames, and terminal device

Abstract

The application discloses a method and device for capturing a clear image frame, and a terminal device. The method comprises the following steps: collecting mobile data of a photographing device and an image frame in a frame buffer in real time; generating a mobile amplitude change curve corresponding to a current frame buffer data time period according to the mobile data; searching for an optimal flat point on the mobile amplitude change curve by rolling back from a time point when a photographing key is pressed to a time point before the time point along a time axis, to obtain a time point corresponding to the optimal flat point; and extracting an image frame corresponding to the time point from the current frame buffer as a clear image frame buffer. The application can accurately locate and capture a clear image frame in a frame buffer in real time, and is not affected by a hardware environment, and can be adapted to various system platforms.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a method and device for capturing a clear image frame and a terminal device. BACKGROUND

[0002] Image quality can be evaluated from multiple dimensions, such as color, focus, exposure, and sharpness. Sharpness is undoubtedly an important and direct evaluation dimension. If the input raw frame is not clear in image processing, a series of subsequent processing algorithms may be affected, and a clear image may not be obtained ultimately, affecting user experience.

[0003] In the prior art, a simple and direct statistical analysis of clear frames is to increase the frame buffer, cache a large number of raw frames for a long time when taking a picture, export these raw frames after taking the picture, and calculate the clearest frame offline. Then, the clearest frame is determined by the naked eye or a mathematical formula, and the sharpness is calculated and sorted. The time difference ΔT between the Keydown event of the touch screen and the clear frame is calculated by backtracking from the clearest frame. The time difference ΔT is usually calculated by a statistical method, that is, a series {ΔT1, ΔT2, ΔT3, …, ΔTn} is obtained by sampling and calculation in different scenes, and a weighted average ΔT_mean is calculated as the interval time difference between the clear frame and the Keydown event in future photography according to a certain fitting algorithm.

[0004] This method uses the weighted average ΔT as the ΔT estimation of all scenes, which obviously has a deviation and limited accuracy of the time difference, affecting the frame capture effect in real scenes. In addition, since different system platforms have different processing speeds and hardware delay characteristics, the above method has poor compatibility with different platforms, and generally needs to be recalculated for different system platforms. SUMMARY

[0005] The embodiments of the present application provide a method and device for capturing a clear image frame and a terminal device, which can accurately locate and capture a clear image frame in the frame buffer in real time, and is not affected by the hardware environment and can adapt to various system platforms.

[0006] To this end, the embodiments of the present application provide the following technical solutions:

[0007] In one aspect, the embodiments of the present application provide a method for capturing a clear image frame, which comprises:

[0008] real-time acquisition of movement data of a photographing device and image frames in a frame buffer;

[0009] generating a movement amplitude change curve corresponding to a current frame buffer data time period according to the movement data;

[0010] finding an optimal flat point on the movement amplitude variation curve by rolling back from the time when the photographing key is pressed along the time axis, and obtaining a time point corresponding to the optimal flat point;

[0011] extracting an image frame corresponding to the time point from the current frame buffer as a clear image frame.

[0012] Optionally, the real-time acquisition of the movement data of the photographing device comprises: real-time acquisition of the movement data of the photographing device by using a sensor arranged on the photographing device.

[0013] Optionally, the sensor is a gyroscope, and the movement data comprises angular accelerations in X, Y and Z axial directions.

[0014] Optionally, the generation of the movement amplitude variation curve corresponding to the time period of the current frame buffer data according to the movement data comprises:

[0015] integrating the angular accelerations in the three axial directions respectively within a sampling period, and filtering the integration results;

[0016] calculating gradient variation data in the three axial directions respectively according to the filtered data, and calculating three-axis total angular deviations corresponding to each time point by using the gradient variation data in the three axial directions;

[0017] generating the movement amplitude variation curve corresponding to the time period of the current frame buffer data according to the three-axis total angular deviations corresponding to each time point.

[0018] Optionally, the integration of the angular accelerations in the three axial directions respectively within the sampling period comprises: integration of the angular accelerations in the three axial directions respectively within the sampling period by using a gradient integration method.

[0019] Optionally, the filtering of the integration results comprises: filtering of the integration results by using a FIR band-pass filter.

[0020] Optionally, the FIR band-pass filter has an order N=32, a cutoff frequency of [8 12] Hz, and a Hamming window.

[0021] Optionally, the calculation of the three-axis total angular deviations corresponding to each time point by using the gradient variation data in the three axial directions comprises: weighted calculation of the gradient variation data in the three axial directions to obtain the three-axis total angular deviations corresponding to each time point.

[0022] Optionally, the method further comprises:

[0023] previously performing offline and / or online synchronization calibration on the clocks of the photographing device and the sensor to obtain a clock deviation;

[0024] According to the clock deviation, the sampling time corresponding to the movement data is corrected.

[0025] Optionally, the offline synchronization calibration of the clocks of the photographing device and the sensor obtains the clock deviation, and the offline synchronization calibration comprises:

[0026] A plurality of checkerboard calibration boards are photographed by using the photographing device to obtain a plurality of checkerboard images, and frame sampling information and sensor angle data corresponding to each frame are recorded;

[0027] According to the checkerboard image data of each frame and the frame sampling information, the posture angle of the photographing device is calculated.

[0028] According to the posture angle of the photographing device and the angle data of the sensor, the clock deviation is fitted.

[0029] Optionally, the calculation of the posture angle of the photographing device according to the checkerboard image data of each frame comprises: according to feature points in a single checkerboard image and an optical flow field of a plurality of checkerboard images, the posture angle of the photographing device is calculated.

[0030] In another aspect, the embodiment of the present application further provides a device for capturing a clear image frame, and the device comprises:

[0031] A data acquisition module is configured to acquire movement data of a photographing device in real time.

[0032] An image acquisition module is configured to acquire an image frame in a frame buffer in real time.

[0033] A curve generation module is configured to generate a movement amplitude change curve corresponding to a time period of current frame buffer data according to the movement data.

[0034] A search module is configured to search for a best flat point on the movement amplitude change curve by rolling back from a time point when a photographing key is pressed to a time point before the photographing key is pressed.

[0035] An image extraction module is configured to extract an image frame corresponding to the time point from the current frame buffer as a clear image frame.

[0036] Optionally, the data acquisition module is specifically configured to acquire the movement data of the photographing device in real time by using a sensor arranged on the photographing device.

[0037] Optionally, the sensor is a gyroscope, and the movement data comprises angular accelerations in X, Y and Z axial directions.

[0038] Optionally, the device further comprises an offline calibration module and / or an online calibration module and a correction module.

[0039] The offline calibration module is configured to perform offline synchronization calibration on the clocks of the photographing device and the sensor to obtain a first clock deviation.

[0040] The online calibration module is configured to perform online synchronization calibration on the clocks of the photographing device and the sensor to obtain a second clock deviation.

[0041] The correction module is configured to correct the sampling time corresponding to the movement data according to the first clock deviation and / or the second clock deviation.

[0042] In another aspect, the embodiment of the present application further provides a terminal device comprising the device for capturing a clear image frame.

[0043] Optionally, the terminal device is any one of a mobile phone, a camera, a vehicle-mounted terminal and a smart wearable device.

[0044] In another aspect, the embodiment of the present application further provides a computer readable storage medium, which is a non-volatile storage medium or a non-transient storage medium, and has stored thereon a computer program, which, when executed by a processor, performs the steps of the above method.

[0045] In another aspect, the embodiment of the present application further provides an electronic device comprising a memory and a processor, wherein the memory has stored thereon a computer program capable of being executed on the processor, and the processor, when executing the computer program, performs the steps of the above method.

[0046] The method and device for capturing a clear image frame, the terminal device provided by the embodiment of the present application capture the movement data of the photographing device and the image frame in the frame buffer in real time, generate a movement amplitude change curve corresponding to the current frame buffer data time period according to the movement data, then count back from the time when the photographing key is pressed along the time axis, find the best flat point on the movement amplitude change curve, and extract the image frame corresponding to the time point from the current frame buffer as the clear image frame. The embodiment of the present application combines the real-time movement data of the photographing device to analyze the running posture thereof, and combines the running posture of the photographing device and the photographing process to accurately locate and capture the clear image frame in real time. Since no specific clarity mathematical formula is required for calculation, the embodiment of the present application is not limited by the application scenario and the hardware processing platform, and is convenient for real-time pipeline processing. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 FIG. 1 is a schematic diagram of the process of capturing a clear image frame in the automatic photographing process and architecture;

[0048] Figure 2 FIG. 1 is a schematic diagram of the process of capturing a clear image frame in the automatic photographing process and architecture;

[0049] Figure 3 is a flow chart of generating a moving amplitude change curve in an embodiment of the present application;

[0050] Figure 4 is a schematic diagram of a moving amplitude change curve generated in an embodiment of the present application;

[0051] Figure 5 is a flow chart of offline synchronization calibration of a clock of a photographing device and a sensor in an embodiment of the present application

[0052] Figure 6 is a structural schematic diagram of an apparatus for capturing a clear image frame in an embodiment of the present application;

[0053] Figure 7 is another structural schematic diagram of an apparatus for capturing a clear image frame in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0055] In order to capture as much real-time clear image frames as possible from the frame buffer of an ISP (Image Signal Processor), it is very difficult and inaccurate to calculate the clarity of each image frame in the frame buffer according to a mathematical formula. First, the clarity is a subjective visual evaluation of the human eye, and it is difficult to measure it with a general mathematical formula. Secondly, it is very difficult to have a formula that can be applied to all scenes in theory because of the variety of scenes. Moreover, according to the existing clarity calculation algorithm, the complexity is generally high, which cannot meet the real-time pipeline processing requirements. In addition, a single or limited number of statistical calculation results or empirical values cannot adapt to all scenes.

[0056] Therefore, an embodiment of the present application provides a method and apparatus for capturing a clear image frame, and a terminal device, which analyzes the running posture of a photographing device by fusing real-time movement data of the photographing device, and accurately locates and captures a clear image frame in real time in combination with the running posture of the photographing device and the photographing process, and can adapt to various hardware environments and application scenarios. Specifically, the movement data of the photographing device can be collected in real time by using a sensor, such as a gyroscope, arranged on the photographing device.

[0057] The flow and architecture of automatic photographing are as shown in Figure 1 The architecture can be divided into an application layer, a hardware abstraction layer (HAL), an algorithm layer, and a hardware / storage layer from the outside to the inside.

[0058] The general flow of automatic photographing is as follows: The general flow of automatic photographing is as follows:

[0059] First, after turning on the camera and initializing the relevant hardware and system software, the user taps the touchscreen to enter preview mode. At this point, real-time data collection of the camera's movement begins; for example, using a gyroscope, the angular acceleration of the camera along the X, Y, and Z axes is obtained.

[0060] After the user taps the touchscreen to take a picture, the camera system performs touch debounce, analog-to-digital conversion (ADC), and generates a KeyDown event. When the user lifts the touchscreen, a KeyUp event is generated, and the system initiates a photo request event to the underlying system.

[0061] At the algorithm layer, after automatic exposure (AE) calculation and configuration of exposure gain and sensitivity (ISO) parameters on the image sensor, a series of image frames with corresponding parameter settings will be generated in the frame buffer.

[0062] The process of capturing a clear image frame involves determining the time point corresponding to the clear image frame in the current buffer during the preview hold process. Furthermore, the time difference ΔT between the time point corresponding to the clear image frame and the time the camera system receives the touchscreen Keydown event can be calculated.

[0063] The HAL layer frame extraction task can extract the corresponding image frame based on the above time point or time difference ΔT, and send it to other algorithms for post-processing.

[0064] After the algorithm layer processes the image frame through denoising, beautification, blurring and other related post-processing algorithms, it calls the JPG encoder to complete the encoding and saves it to the image library.

[0065] During the aforementioned automatic photo-taking process, after automatic focus (AF) is completed and the user presses the shutter button, the camera body shakes due to hand tremors and screen vibrations, causing motion blur or out-of-focus images to be captured. However, after autofocus is complete, the image captured by the camera should be the clearest when the camera body is at its most stable. Based on this principle, the method for capturing clear image frames provided in this embodiment of the invention utilizes real-time motion data of the camera device collected during automatic photo-taking to determine the most stable point in time within a certain period. Based on this point in time, the image frame corresponding to that point in the current buffer is captured as the clear image frame.

[0066] like Figure 2 The diagram shown is a flowchart of a method for capturing clear image frames according to an embodiment of the present invention, including the following steps:

[0067] Step 201, real-time collection of mobile data of the photographing device and image frames in the frame buffer.

[0068] The mobile data of the photographing device is collected in real time by using sensors, such as a gyroscope, arranged on the photographing device. Accordingly, the mobile data can include angular accelerations in X, Y and Z axes.

[0069] It should be noted that the sampling period of the sensor can be set to be less than or equal to the image sampling period, which is better to ensure the accuracy of the final result.

[0070] Step 202, generating a mobile amplitude variation curve corresponding to a time period of current frame buffer data according to the mobile data.

[0071] Step 203, searching for an optimal flat point on the mobile amplitude variation curve by rolling back from the time when the photographing key is pressed along the time axis, to obtain a time point corresponding to the optimal flat point.

[0072] Step 204, extracting an image frame corresponding to the time point from the current frame buffer as a clear image frame.

[0073] The following takes the gyroscope as an example to illustrate the process of generating a mobile amplitude variation curve based on the mobile data collected by the gyroscope.

[0074] As shown in Figure 3 FIG. 1 is a flowchart of generating a mobile amplitude variation curve corresponding to a time period of current frame buffer data according to real-time collected mobile data in the embodiment of the present application, including the following steps:

[0075] Step 301, performing integral operation on angular accelerations in three axes respectively in a sampling period, and filtering the integral operation results.

[0076] Since the sampling rate of the gyroscope is generally higher than that of preview or video recording (for example, a sampling rate of 100-200 Hz can be used), the gyroscope outputs angular acceleration values, so the integral operation in the sampling period is required before the angular acceleration values can be applied to the angle calculation of the photographing device. Since the X, Y and Z axis movements are relatively independent, the X, Y and Z axes are independently integrated in the present application, and the integral method simply uses a simple gradient integral method.

[0077] Similarly, in the filtering operation, the X, Y and Z axes are also separately filtered.

[0078] In a specific application, a finite impulse response (FIR) band-pass filter can be used, but is not limited to, for example, the band-pass filter parameters include: order N = 32, cutoff frequency [8 12] Hz, where 8 and 12 are the upper and lower cutoff frequencies of the band-pass filter, and a Hamming window. Of course, the parameters of the band-pass filter are not limited to this, and can be set according to the application needs, and the embodiments of the present application are not limited.

[0079] In step 302, the gradient change data of the three axes is calculated according to the filtered data, and the three-axis total angular deviation corresponding to each time point is calculated using the gradient change data of the three axes.

[0080] It is considered that in addition to the Z axis perpendicular to the screen of the photographing device, the other X and Y axes will affect the image quality to varying degrees.

[0081] Therefore, in the embodiments of the present application, one-dimensional backward difference gradient calculation is first performed on the X, Y and Z axes to obtain the gradient change data of each axis. Then, the three-axis total angular deviation corresponding to each time point is calculated according to the gradient change data of the three axes, for example, for each time point, the absolute value summation operation, or the weighted operation, or the absolute value weighted operation, etc. can be performed on the gradient change data of the three axes.

[0082] The three-axis total angular deviation reflects the motion amplitude of the photographing device.

[0083] In step 303, the motion amplitude change curve corresponding to the time period of the current frame buffer data is generated according to the three-axis total angular deviation corresponding to each time point.

[0084] Suppose the generated motion amplitude change curve is as shown in Figure 4 The horizontal axis represents time, and the vertical axis represents the related data corresponding to each time point, which mainly includes the angular change amplitudes of the X, Y and Z axes before and after photographing, i.e. the X-axis data, Y-axis data and Z-axis data shown in the figure. The absolute value summation of the gradient change of the three axes generates the motion amplitude change curve, which represents the total angular change amount of the three axes.

[0085] Figure 4 In the example shown in FIG. 8, the time point corresponding to the time when the user clicks the screen is taken as the starting point of the time, the time point corresponding to the Keydown time of the photographing key is 1146 ms, the time point corresponding to the Keyup time of the photographing key is 1278 ms, and the time point corresponding to the time when the photographing system receives the photographing request is 1370 ms.

[0086] The time difference ΔT between the time point corresponding to the clear image frame obtained based on the traditional statistical method and the time when the Keydown event of the touch screen is received by the photographing system is 380 ms, that is, the time point of the clear image frame determined by the traditional statistical method is 1370-380=990 (ms).

[0087] In the determination of the time point corresponding to the optimal flat point, the optimal flat point on the movement amplitude change curve is searched by counting back from the time when the photographing key is pressed (i.e., the time point 1146 ms), and the time point corresponding to the optimal flat point is obtained as 1050 ms in FIG. 11. The image frame corresponding to the time point in the current buffer is the clear image frame to be found. Specifically, in the search for the optimal flat point, a Gaussian-weighted optimal flat point can be found by counting back from the time point corresponding to the Keydown time as the center on the movement amplitude change curve, and the point is the optimal frame-grabbing point. Figure 4

[0088] It should be noted that, considering that there can be a certain deviation in the clock of the photographing device and the sensor (such as a gyroscope), in order to obtain more accurate measurement results, in another non-limiting embodiment of the method, the system clock of the photographing device and the sensor can also be calibrated offline and / or online in advance to obtain the clock deviation. Accordingly, the sampling time corresponding to the movement data can be corrected according to the obtained clock deviation, so that the time point corresponding to the clear image determined finally is more accurate.

[0089] In actual application, the correction of the sampling time corresponding to the movement data according to the clock deviation obtained based on offline calibration can be performed in the hardware driver of the sensor, that is, the movement data obtained by the sensor sampling is already corrected data, and it is not necessary to correct these data in the subsequent calculation process; of course, the movement data obtained by the sensor sampling can also be corrected in the calculation process of grabbing the clear image frame, which is not limited by the embodiments of the present application. The correction of the sampling time corresponding to the movement data according to the clock deviation obtained based on offline calibration needs to be performed in the calculation process of grabbing the clear image frame.

[0090] Referring to Figure 5 is a flowchart of offline synchronization calibration of the clock of the photographing device and the sensor in the embodiments of the present application, including the following steps:

[0091] Step 501, a plurality of checkerboard calibration boards are photographed by using the photographing device to obtain each frame of checkerboard image, and frame sampling information and sensor angle data corresponding to each frame are recorded.

[0092] The frame sampling information includes but is not limited to: time stamp, exposure time, shutter time, etc.​

[0093] At step 502, the attitude angle of the photographing device is calculated according to the chessboard image and the frame sampling information thereof.

[0094] In the embodiment of the present application, the attitude angle of the photographing device can be calculated according to the feature points in the single-frame chessboard image and the optical flow field of the multi-frame chessboard image. Specifically, the extrinsic parameter matrix of the photographing device between two frames of images is calculated according to the sparse optical flow field between the feature points of the single-frame image and the feature points of the multi-frame image, and the rotation and translation relationship of the photographing device, i.e., the camera attitude angle, is obtained.

[0095] At step 503, the clock deviation is fitted according to the attitude angle of the photographing device and the angle data of the sensor.

[0096] For example, the clock deviation can be calculated by using the least square fitting method, and other fitting algorithms can also be used, which are not limited in the embodiment of the present application.

[0097] In a non-limiting embodiment, the method of online synchronization calibration can include the following steps:

[0098] (1) The angle data A1 of the sensor and the time stamp T1 are stored in the first buffer queue;

[0099] (2) The continuous N (for example, 3<N<8) frames of images are stored in the second buffer queue, and at least 30 feature points are contained in the images;

[0100] (3) In the N frames of images, the attitude angle A2 of the photographing device is calculated by using the sparse optical flow field, and the image time stamp T2 is recorded;

[0101] (4) Taking the time stamp as the X axis and the angle as the Y axis, taking the T2-A2 curve of the photographing device as the reference, and moving the T1-A1 curve of the sensor according to the set step (for example, the step is set to 0.5 ms), the error between the T1-A1 curve of the sensor and the T2-A2 curve of the photographing device is calculated;

[0102] (5) The time point at which the error between the two curves is the smallest is the synchronization time point of the two, and the clock compensation amount T1x-T2x of the sensor is obtained according to the synchronization time point, wherein T1x represents the time corresponding to the synchronization time point on the T1-A1 curve, and T2x represents the time corresponding to the synchronization time point on the T2-A2 curve.

[0103] Of course, some other online synchronization calibration technologies can also be used, which are not limited in the embodiment of the present application.

[0104] In practical applications, the offline calibration and the real-time automatic calibration can work together, so that the system clock of the sensor and the photographing device can be better guaranteed to be synchronized, and thus the finally determined clear image frame can be more accurate.

[0105] The method for capturing a clear image frame provided by the embodiment of the present application can capture the movement data of the photographing device and the image frame in the frame buffer in real time, generate a movement amplitude change curve corresponding to the time period of the current frame buffer data according to the movement data, then count back from the time when the photographing key is pressed along the time axis, find the best flat point on the movement amplitude change curve, and extract the image frame corresponding to the time point from the current frame buffer as the clear image frame. The scheme of the present application analyzes the running posture of the photographing device by fusing the real-time movement data of the photographing device, and accurately locates and captures the clear image frame in real time in combination with the running posture of the photographing device and the photographing process. Since a specific sharpness mathematical formula is not required for calculation, the method is not limited by the application scene and the hardware processing platform, and is convenient for realizing real-time pipeline processing.

[0106] It should be noted that the method for capturing a clear image frame provided by the embodiment of the present application is applicable to both a zero shutter lag (ZSL) mode and a non-zero shutter lag (Non-ZSL) mode photographing system.

[0107] Correspondingly, the embodiment of the present application also provides a device for capturing a clear image frame, as shown in Figure 6 The device for capturing a clear image frame 600 includes the following modules:

[0108] The data acquisition module 601 is configured to acquire the movement data of the photographing device in real time.

[0109] The image acquisition module 602 is configured to acquire the image frame in the frame buffer in real time.

[0110] The curve generation module 603 is configured to generate a movement amplitude change curve corresponding to the time period of the current frame buffer data according to the movement data.

[0111] The finding module 604 is configured to count back from the time when the photographing key is pressed along the time axis, find the best flat point on the movement amplitude change curve, and obtain the time point corresponding to the best flat point.

[0112] The image extraction module 605 is configured to extract the image frame corresponding to the time point from the current frame buffer as the clear image frame.

[0113] The image extraction module 605 is configured to extract the image frame corresponding to the time point from the current frame buffer as the clear image frame.

[0114] The data acquisition module 601 can specifically acquire the movement data of the photographing device in real time by using a sensor arranged on the photographing device. The sensor can be a gyroscope, and the movement data includes angular acceleration of three axes X, Y and Z.

[0115] The specific manners in which the above modules achieve the respective functions can be understood from the descriptions of the method embodiments of the present application above, and thus will not be described again here.

[0116] Further, in order to obtain more accurate measurement results, the photographing device can further include a sensor for measuring the movement of the photographing device, and the sensor can be a gyroscope. Figure 7 As shown in FIG. 6, in another non-limiting embodiment of the device of the present application, the device for capturing a clear image frame 600 can further include an offline calibration module 701 and / or an online calibration module 702, and a correction module 703. Wherein:

[0117] The offline calibration module 701 is configured to perform offline synchronization calibration on the clocks of the photographing device and the sensor, to obtain a first clock deviation.

[0118] The online calibration module 702 is configured to perform offline synchronization calibration on the clocks of the photographing device and the sensor, to obtain a second clock deviation.

[0119] The correction module 703 is configured to correct the sampling time corresponding to the movement data according to the first clock deviation and / or the second clock deviation.

[0120] The processes of the offline calibration module 701 and the online calibration module 702 for performing offline synchronization calibration on the clocks of the photographing device and the sensor can be understood from the descriptions of the method embodiments of the present application above, and thus will not be described again here.

[0121] It should be noted that in another non-limiting embodiment of the device for capturing a clear image frame of the present application, the above correction module 703 can also be omitted. As described above, the correction of the sampling time corresponding to the movement data according to the first clock deviation determined by the offline calibration module 701 can be implemented in the hardware driver of the sensor, that is, the function of the above correction module 703 is added to the sensor, and the data output by the sensor (i.e. the movement data of the photographing device) is synchronized with the image frame data of the photographing device, and the movement data does not need to be corrected subsequently.

[0122] Obviously, if both offline calibration and online calibration are performed, the correction of the sampling time corresponding to the movement data by the correction module 703 is performed separately, that is, the correction processes in different processing stages are performed.

[0123] The device for capturing a clear image frame provided by the embodiment of the present application captures the movement data of the photographing device and the image frame in the frame buffer in real time, generates a movement amplitude change curve corresponding to the data time period of the current frame buffer according to the movement data, then reverses along the time axis from the time when the photographing key is pressed to find the best flat point on the movement amplitude change curve, and extracts the image frame corresponding to the time point from the current frame buffer as the clear image frame. The present application scheme analyzes the running posture of the photographing device by fusing the real-time movement data of the photographing device, and accurately locates and captures the clear image frame in real time in combination with the running posture of the photographing device and the photographing process. Since no specific clarity mathematical formula is needed for calculation, the present application is not limited by the application scene and the hardware processing platform, and is convenient for realizing real-time pipeline processing.

[0124] Correspondingly, the embodiment of the present application also provides a terminal device comprising the device for capturing a clear image frame, which can be, but is not limited to, a mobile phone, a camera, a vehicle-mounted terminal, a smart wearable device, etc.

[0125] In specific implementation, the device of each embodiment described above can correspond to a chip in a terminal device, such as a SoC (System-On-a-Chip), a baseband chip, a chip module, etc.

[0126] In specific implementation, each module / unit contained in each device and product described in the above embodiments can be a software module / unit, a hardware module / unit, or part of a software module / unit and part of a hardware module / unit.

[0127] For example, for each device, product applied to or integrated into a chip, each module / unit contained therein can be realized by hardware such as a circuit, or at least part of the modules / units can be realized by a software program running on a processor integrated in the chip, and the remaining (if any) modules / units can be realized by hardware such as a circuit; for each device, product applied to or integrated into a chip module, each module / unit contained therein can be realized by hardware such as a circuit, and different modules / units can be located in the same component (for example, a chip, a circuit module, etc.) or different components of the chip module, or at least part of the modules / units can be realized by a software program running on a processor integrated in the chip module, and the remaining (if any) modules / units can be realized by hardware such as a circuit; for each device, product applied to or integrated into a terminal, each module / unit contained therein can be realized by hardware such as a circuit, and different modules / units can be located in the same component (for example, a chip, a circuit module, etc.) or different components of the terminal, or at least part of the modules / units can be realized by a software program running on a processor integrated in the terminal, and the remaining (if any) modules / units can be realized by hardware such as a circuit.

[0128] The embodiment of the present application further provides a computer readable storage medium, which is a nonvolatile storage medium or a non-transitory storage medium, and has a computer program stored thereon, wherein the computer program is run by a processor to perform the method described above. Figure 2 Or Figure 3 Or Figure 5 The steps of the method provided by the corresponding embodiment.

[0129] The embodiment of the present application further provides another terminal device, which comprises a memory and a processor, and the memory has a computer program stored thereon, wherein the computer program is run by the processor to perform the method described above. Figure 2 Or Figure 3 Or Figure 5 The steps of the method provided by the corresponding embodiment.

[0130] The embodiment of the present application further provides an electronic device, which comprises a memory and a processor, and the memory has a computer program stored thereon, wherein the computer program is run by the processor to perform the method described above. Figure 2 Or Figure 3 Or Figure 5 The steps of the method provided by the corresponding embodiment.

[0131] In several embodiments provided by the present application, it should be understood that the disclosed method, device and system can be implemented in other manners. For example, the described device embodiments are merely schematic. For example, the division of the units is merely a logical function division. For example, there can be another division manner for the actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0132] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e. can be located in one place, or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the present application.

[0133] In addition, each function unit in the various embodiments of the present application can be integrated in a processing unit, or each unit can be a physical unit independently, or two or more units can be integrated in a unit. The integrated unit can be implemented in the form of hardware, or in the form of hardware plus software function units.

[0134] Although the present application is disclosed as above, the present application is not limited to the above. Any person skilled in the art, without departing from the spirit and scope of the present application, can make various modifications and changes, and therefore the protection scope of the present application should be subject to the scope defined by the claims.

Claims

1. A method of grabbing a clear image frame, characterized by, The method comprises: Real-time acquisition of movement data of a photographing device and image frames in a frame buffer; the movement data is acquired by a sensor on the photographing device, and a sampling period of the movement data is less than or equal to a sampling period of the image frames; the movement data comprises angular accelerations in three axial directions; Generation of a movement amplitude change curve corresponding to a current frame buffer data time period according to the movement data; Forward rollback of a time axis from a time point when a photographing key is pressed to find an optimal flat point on the movement amplitude change curve, to obtain a time point corresponding to the optimal flat point; Extraction of an image frame corresponding to the time point from the current frame buffer as a clear image frame; The generation of the movement amplitude change curve corresponding to the current frame buffer data time period according to the movement data comprises: Integration of the angular accelerations in the three axial directions respectively within a sampling period, and filtering of the integration results; Calculation of gradient change data in the three axial directions according to the filtered data, and calculation of three-axis total angular deviations corresponding to each time point by using the gradient change data in the three axial directions; Generation of the movement amplitude change curve corresponding to the current frame buffer data time period according to the three-axis total angular deviations corresponding to each time point.

2. The method of claim 1, wherein, The sensor is a gyroscope.

3. The method of claim 1, wherein, The integration of the angular accelerations in the three axial directions respectively within a sampling period comprises: Integration of the angular accelerations in the three axial directions respectively within a sampling period by using a gradient integration method.

4. The method of claim 1, wherein, The filtering of the integration results comprises: Filtering of the integration results by using a FIR band-pass filter.

5. The method of claim 4, wherein, The FIR band-pass filter has an order N=32, a cutoff frequency of [8 12] Hz, and a Hamming window.

6. The method of claim 1, wherein, The calculation of the three-axis total angular deviations corresponding to each time point by using the gradient change data in the three axial directions comprises: Weighted calculation of the gradient change data in the three axial directions to obtain the three-axis total angular deviations corresponding to each time point.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: Offline and / or online synchronization calibration of clocks of the photographing device and the sensor to obtain a clock deviation; Correction of sampling times corresponding to the movement data according to the clock deviation.

8. The method of claim 7, wherein, The offline synchronization calibration of the clocks of the photographing device and the sensor to obtain the clock deviation comprises: Photographing of multiple checkerboard calibration boards by using the photographing device to obtain multiple checkerboard images, and recording of frame sampling information and sensor angle data corresponding to each frame; Calculation of a posture angle of the photographing device according to the multiple checkerboard images and the frame sampling information; Fitting of the clock deviation according to the posture angle of the photographing device and the angle data of the sensor.

9. The method of claim 8, wherein, The calculation of the posture angle of the photographing device according to the multiple checkerboard image data comprises: Calculation of the posture angle of the photographing device according to feature points in a single checkerboard image and an optical flow field of multiple checkerboard images.

10. An apparatus for grabbing a clear image frame, characterized by The device comprises: A data collection module is configured to collect movement data of the photographing device in real time by using a sensor arranged on the photographing device, and a sampling period of the movement data is less than or equal to a sampling period of the image frame, and the movement data includes angular accelerations in three axial directions X, Y and Z. An image collection module is configured to collect the image frame in the frame buffer in real time. A curve generation module is configured to generate a movement amplitude change curve corresponding to a current frame buffer data time period according to the movement data, and the curve generation module includes the following steps: performing integral operation on the angular accelerations in the three axial directions respectively in the sampling period, and filtering the integral operation results; calculating gradient change data in the three axial directions respectively according to the filtered data, calculating total angular deviations in three axial directions corresponding to each time point by using the gradient change data in the three axial directions, and generating the movement amplitude change curve corresponding to the current frame buffer data time period according to the total angular deviations in the three axial directions corresponding to each time point. A search module is configured to search for a best flat point on the movement amplitude change curve by rolling back from a time point when a shutter button is pressed along a time axis, and obtain a time point corresponding to the best flat point. An image extraction module is configured to extract an image frame corresponding to the time point from the current frame buffer as a clear image frame.

11. The apparatus of claim 10, wherein, The sensor is a gyroscope.

12. The apparatus of claim 10 or 11, wherein, The device further includes an offline calibration module and / or an online calibration module and a correction module. The offline calibration module is configured to pre-synchronize and calibrate clocks of the photographing device and the sensor offline to obtain a first clock deviation. The online calibration module is configured to synchronize and calibrate the clocks of the photographing device and the sensor online to obtain a second clock deviation. The correction module is configured to correct a sampling time corresponding to the movement data according to the first clock deviation and / or the second clock deviation.

13. A terminal device, comprising: The device includes any one of claims 10 to 12.

14. The terminal device according to claim 13, characterized by The terminal device is any one of a mobile phone, a camera, a vehicle-mounted terminal and a smart wearable device.

15. A computer-readable storage medium, which is a non-volatile storage medium or a non-transitory storage medium, on which a computer program is stored, characterized by The computer program is executed by a processor to perform the steps of the method of any one of claims 1 to 9.

16. An electronic device comprising a memory and a processor, said memory having stored thereon a computer program operable to run on said processor, characterized in that, The processor executes the computer program to perform the steps of the method of any one of claims 1 to 9.

Images