Real-time video space-based camera image stabilization method, and corresponding storage medium

Abstract

The present application relates to the technical field of optical image stabilization, and discloses a real-time video space-based camera image stabilization method and a storage medium. The real-time video space-based camera image stabilization method comprises: determining effective IMU data of a single image frame on the basis of a preset image collection time and a preset shutter response time; dividing pixels of the single image frame into a plurality of row pixels by row, and determining the effective IMU data corresponding to each row of pixels; and performing position correction on the corresponding row of pixels by means of the effective IMU data, so as to implement an image stabilization operation for the single image frame.

Description

Camera anti-shake method based on real-time video space and corresponding storage medium TECHNICAL FIELD

[0001] The present application relates to the field of optical image stabilization, and in particular to a camera anti-shake method based on real-time video space and a storage medium. BACKGROUND

[0002] With the rapid development of image technology, users' demand for video recording clarity and stability is increasing. In the process of shooting, there is inevitably a shake; for example, the shake can be a handheld shake of the user when shooting, or the shake can be a movement of the electronic device during shooting; due to the shake during shooting, there is motion blur in the obtained video or image.

[0003] During the movement of a Webcam user, the camera lens module adopts a rolling shutter sensor during real-time shooting and collection. The real-time video frame picture, posture information IMU (Inertial Measurement Unit), and lens exposure information of an ISP (Image Signal Process) are transmitted to the Webcam application program through a transmission channel (IO, USB, network cable, etc.). Therefore, how to improve the camera anti-shake performance according to the real-time IMU data in the camera has become a technical problem to be solved at present. TECHNICAL PROBLEM

[0004] The present application provides a camera anti-shake method based on real-time video space and a storage medium to improve the camera anti-shake performance.

[0005] In a first aspect, the present application provides a camera anti-shake method based on real-time video space, which comprises:

[0006] determining effective IMU data of a single frame image based on a preset image collection time and a preset shutter response time;

[0007] dividing pixels of the single frame image into a plurality of row pixels by row, and determining the effective IMU data corresponding to each row pixel;

[0008] performing position correction on the corresponding row pixel through the effective IMU data to realize the anti-shake operation of the single frame image.

[0009] In a second aspect, the present application further provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor realizes the camera anti-shake method based on real-time video space as described above.

[0010] The application discloses a camera anti-shake method based on real-time video space and a storage medium. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] Fig. 1 is a schematic flow chart of a camera anti-shake method based on real-time video space according to an embodiment of the present application.

[0013] Fig. 2 is a structural schematic block diagram of a computer device according to an embodiment of the present application. BEST MODE FOR CARRYING OUT THE INVENTION

[0014] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0015] The flow chart shown in the drawings is only an example, not necessarily including all contents and operations / steps, and not necessarily executed in the described order. For example, some operations / steps can be decomposed, combined or partially merged, so the actual execution order can be changed according to the actual situation.

[0016] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in this specification and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0017] It is also to be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items, and that the term "at least one of' as used herein means one, or more, or any combination thereof.

[0018] Embodiments of the present application provide a camera anti-shake method based on real-time video space and a storage medium. The camera anti-shake method based on real-time video space can be applied to a server, a single frame image obtained is divided into rows in the longitudinal direction, and IMU data corresponding to each row is determined according to a calculated time, so that accurate anti-shake of different parts of the entire image is realized from local to global based on changes in the IMU data, and the anti-shake performance of the camera is improved. The server can be a stand-alone server or a server cluster.

[0019] Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0020] Please refer to FIG. 1, which is a schematic flowchart of a camera anti-shake method based on real-time video space according to an embodiment of the present application. The camera anti-shake method based on real-time video space can be applied to a server, a single frame image obtained is divided into rows in the longitudinal direction, and IMU data corresponding to each row is determined according to a calculated time, so that accurate anti-shake of different parts of the entire image is realized from local to global based on changes in the IMU data, and the anti-shake performance of the camera is improved.

[0021] As shown in FIG. 1, the camera anti-shake method based on real-time video space specifically includes steps S10 to S30.

[0022] Step S10, determining effective IMU data of a single frame image based on a preset image acquisition time and a preset shutter response time;

[0023] Step S20, dividing pixels of the single frame image into multiple row pixels by rows, and determining effective IMU data corresponding to each row pixel;

[0024] Step S30, correcting the position of the corresponding row pixel through the effective IMU data to realize the anti-shake operation of the single frame image.

[0025] In one embodiment, the inertial measurement unit is a device that measures the three-axis attitude angle (or angular rate) and acceleration of an object. Generally, an IMU contains three single-axis accelerometers and three single-axis gyroscopes. The accelerometers detect the acceleration signals of the object in the independent three-axis of the carrier coordinate system, and the gyroscopes detect the angular velocity signals of the carrier relative to the navigation coordinate system. The angular velocity and acceleration of the object in three-dimensional space are measured, and the attitude of the object is calculated based on the measurements.

[0026] Specifically, the preset IMU acquisition frequency is determined according to the shutter acquisition time and the image acquisition time. The shutter is a device used in camera equipment to control the time of light irradiation of the photosensitive element; the shutter speed is the length of time the camera shutter remains open, often referred to simply as shutter. Shutter speed is the time that controls the light into the camera. The longer the light enters, the longer the image reaction time will be. The unit of shutter is calculated in seconds, that is, 1 / 30 is 1 / 30 of a second, and 1 / 200 is 1 / 200 of a second.

[0027] Exemplarily, the image acquisition time of each frame of picture is 40ms, the shutter response time is 1 / 30s (i.e. 33ms), that is, a single frame of picture is displayed for 40ms, and the actual acquisition time of the picture is 0-33ms. Therefore, only the IMU data of 0-33ms needs to be read, and the IMU data of 33ms-40ms is discarded.

[0028] Since a single frame of image corresponds to multiple valid IMU data, for example, the single frame of picture corresponds to 34 valid IMU data, and the row pixels in the single frame of image are generated from top to bottom in sequence, therefore, the valid IMU data corresponding to each row of pixels of the single frame of picture can be determined. However, if the single frame of image has 1100 row pixels, and the number of valid IMU data is 40, then 1100 / 40=27.5, that is, each valid IMU data corresponds to 27.5 row pixels, that is, there may be a row of pixels belonging to two valid IMU data. At this time, the single valid IMU data corresponding to the row of pixels is determined according to the number of pixels in the row of pixels belonging to the two valid IMU data. For example, there are 1000 pixels in a row, of which 400 belong to IMU1 and 600 belong to IMU2, and the row of pixels is set to correspond to IMU1. 2， to realize the correction of a single row of pixels.

[0029] When the IMU acquisition rate is low, that is, the number of valid IMU data corresponding to a single frame of image is small (for example, only 2 valid IMU data), the change of the two valid IMU data is large, and the row of pixels at the boundary of the two valid IMU data may appear distorted.

[0030] To solve the above problems, in the embodiment, if the two effective IMU data corresponding to two adjacent rows of pixels change greatly, for example, the acceleration in a certain direction changes more than 10 m / s², the row pixels at the junction corresponding to the two effective IMU data are uniformly transitioned to eliminate the sharp change of the effective IMU data. For example, the 50 row pixels at the junction of the two effective IMU data can be uniformly transitioned (the specific value can be set by the user according to the total number of row pixels), of which the top 10 rows use IMU1+0 (IMU2-IMU1), the 11th to 20th rows use IMU1+0.25 (IMU2-IMU1), the middle 10 rows use IMU1+0.5 (IMU2-IMU1), the 31st to 40th rows use IMU1+0.75 (IMU2-IMU1), and the last 10 rows use IMU1+1 (IMU2-IMU1).

[0031] The embodiment discloses a camera anti-shake method based on real-time video space and a storage medium. The camera anti-shake method based on real-time video space comprises the following steps: determining effective IMU data of a single frame image based on a preset image acquisition time and a preset shutter response time; dividing pixels of the single frame image into a plurality of row pixels according to rows, and determining the effective IMU data corresponding to each row pixel; and correcting the position of the corresponding row pixel through the effective IMU data to realize the anti-shake operation of the single frame image. In the foregoing manner, the single frame image is divided into row pixels in the longitudinal direction, the effective IMU data is distributed to the row pixels, the effective IMU data corresponding to each row pixel is determined, the position of the row pixel is corrected through the effective IMU data, and different complete and accurate anti-shake operations are realized on the single frame image from the local to the global through the change of the IMU data, thereby improving the anti-shake performance of the camera.

[0032] Based on the embodiment shown in FIG. 1, in the embodiment, step S20 comprises:

[0033] The number of the effective IMU data is obtained as a first number, and the number of the total row pixels is obtained as a second number;

[0034] The second number is divided by the first number to obtain a third number;

[0035] Based on the third number, the effective IMU data corresponding to each row pixel is determined.

[0036] In specific embodiments, the number of valid IMU data corresponding to each single row of pixels can be determined by dividing the number of valid IMU data by the total number of row pixels in a single frame of image. For example, if a single frame of image includes 100 row pixels and 5 valid IMU data (IMU1, IMU2, IMU3, IMU4, and IMU5), then from top to bottom, 1-20 row pixels correspond to IMU1, 21-40 row pixels correspond to IMU2, 41-60 row pixels correspond to IMU3, 61-80 row pixels correspond to IMU4, and 81-100 row pixels correspond to IMU5.

[0037] Based on the embodiment shown in FIG. 1, in the present embodiment, step S30 includes:

[0038] Obtaining the number of pixels of the single row of pixels to be corrected, wherein the pixels in the single row of pixels to be corrected correspond to the first valid IMU data and the second valid IMU data, respectively;

[0039] Determining the target valid IMU data based on the number of pixels of the single row of pixels to be corrected included in the first valid IMU data and the second valid IMU data, respectively;

[0040] Correcting the single row of pixels to be corrected based on the target valid IMU data.

[0041] Based on the embodiment shown in FIG. 1, in the present embodiment, step S30 includes:

[0042] Uniformly transitioning the single row of pixels to be corrected corresponding to the boundary IMU data based on a preset uniform transition formula to generate corrected IMU data, wherein the boundary IMU data is a preset number of IMU data at the boundary of the two valid IMU data;

[0043] Replacing the single row of pixels to be corrected subjected to the uniform transition with the corrected IMU data to achieve the anti-shake operation of the single frame of image.

[0044] Further, before the single row of pixels to be corrected corresponding to the boundary IMU data is subjected to the uniform transition based on the preset uniform transition formula to generate the corrected IMU data, the method further includes:

[0045] Determining the preset number based on a preset ratio and the total number of row pixels.

[0046] Based on the above embodiment, in the present embodiment, the single row of pixels corresponding to the boundary IMU data is subjected to the uniform transition based on the preset uniform transition formula to generate the corrected IMU data, including:

[0047] Obtaining the third IMU data and the fourth IMU data;

[0048] generate the corrected IMU data based on the preset uniform transition formula, the third IMU data and the fourth IMU data.

[0049] More preferably, the preset uniform transition formula is IMU k= IMU1+n* (IMU2-IMU1);

[0050] wherein the IMU k is the corrected IMU data, the IMU1 is the third IMU data, IMU2 is the fourth IMU data, and n is a coefficient related to the preset number.

[0051] Based on the embodiment shown in FIG. 1, in the embodiment, step S10 comprises:

[0052] determining all IMU data of the single frame image based on the preset image acquisition time;

[0053] determining the effective IMU data from the all IMU data based on the preset shutter response time.

[0054] Specifically, the unit of shutter response time is calculated in seconds. In combination with the above embodiment, when the preset shutter response time is 1 / 30, the shutter response time is 1 / 30 second (33 ms). Correspondingly, when the preset shutter response time is 1 / 200, the shutter response time is 1 / 200 second (5 ms).

[0055] The preset image acquisition time is determined by the image acquisition frequency, and is in units of fps (Frames Per Second). For example, when the image acquisition frequency is 25 fps, 25 frames of images are acquired per second, and the acquisition time of a single frame of image is 1 / 25 second (40 ms).

[0056] In summary, when the image acquisition time of each frame of image is 40 ms, the IMU data within 40 ms is the all IMU data in the embodiment. However, due to the limitation of camera hardware (i.e. the shutter response time is less than the image acquisition time), the shutter response time of a single frame of image is 1 / 30 s (i.e. 33 ms), that is, a single frame of image is displayed for 40 ms, but only 0-33 ms of the 40 ms is the actual acquisition time of the image, so only the IMU data of 0-33 ms needs to be read as the effective IMU data, and the IMU data of 33-40 ms can be discarded.

[0057] The application method is described below in combination with the above embodiments.

[0058] When the user performs the anti-shake process on the camera, the image is captured by the camera. According to the image capture time, the entire IMU data of the single frame image is obtained, and then according to the shutter response time, the effective IMU data is determined from the entire IMU data.

[0059] Generally, the IMU data correction of the single frame image is divided into two types.

[0060] The first type is to correct a single row of pixels. The number of single row pixels corresponding to each effective IMU data is determined, and the number of effective IMU data can be divided by the total number of row pixels of the single frame image, that is, the number of row pixels corresponding to each effective IMU data can be determined.

[0061] Since the single frame image is divided into a plurality of row pixels from top to bottom, the effective IMU data of the single frame image is evenly distributed to each row pixel, and each row pixel is represented by only one effective IMU data. However, if the single frame image has 1100 row pixels and the number of effective IMU data is 40, then 1100 / 40=27.5, that is, each effective IMU data corresponds to 27.5 row pixels. That is, a certain row pixel corresponds to two effective IMU data, and the row pixel is corrected according to the pixel number of the row pixel contained in the two effective IMU data, for example, the row pixel of the row has 1000 pixels, of which 400 pixels correspond to IMU1 and 600 pixels correspond to IMU2. The row pixel of the row is set to correspond to IMU1, that is, the correction of the single row pixel is realized.

[0062] The second correction type of the IMU data is for the case that the corresponding IMU quantity in the single frame image is small, resulting in a large change of the effective IMU data in a certain direction. For example, the single frame image has 50 rows of pixels, but only corresponds to 2 effective IMU data. The 50 rows of pixels at the junction of the two effective IMU data can be uniformly transitioned (generally 1 / 10 to 1 / 5 of the total number of rows of pixels). In the 50 rows of pixels, the top 10 rows use IMU1+0 (IMU2-IMU1), the 11th to 20th rows use IMU1+0.25 (IMU2-IMU1), the middle 10 rows use IMU1+0.5 (IMU2-IMU1), the 31st to 40th rows use IMU1+0.75 (IMU2-IMU1), and the last 10 rows use IMU1+1 (IMU2-IMU1). The coefficients can be set according to the number of effective IMU data in the single frame image, that is, the IMU data in a certain range at the top and bottom of the single frame image still uses the original effective IMU data, and the junction in the middle is divided according to the number of specific effective IMU data, and different corrected IMU data are respectively corresponded, and the original effective IMU data is replaced by the corrected IMU data to realize real-time anti-shake of the camera.

[0063] Referring to FIG. 2, FIG. 2 is a structural schematic block diagram of a computer device provided by an embodiment of the present application. The computer device can be a server.

[0064] Referring to FIG. 2, the computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory can include a non-volatile storage medium and an internal memory.

[0065] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions which, when executed, can cause the processor to perform any kind of camera anti-shake method based on real-time video space.

[0066] The processor is used to provide computing and control capabilities to support the operation of the entire computer device.

[0067] The internal memory provides an environment for the running of the computer program in the non-volatile storage medium, which, when executed by the processor, can cause the processor to perform any kind of camera anti-shake method based on real-time video space.

[0068] The network interface is configured to perform network communication, such as sending the assigned task, etc. Those skilled in the art can understand that the structure shown in FIG. 2 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0069] It should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0070] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program. The computer program includes program instructions. The processor executes the program instructions to implement any one of the camera anti-shake methods based on real-time video space provided by the embodiments of the present application.

[0071] The computer readable storage medium can be an internal storage unit of the computer device, for example, a hard disk or a memory of the computer device. The computer readable storage medium can also be an external storage device of the computer device, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc.

[0072] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed in the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A camera image stabilization method based on real-time video space, comprising: Based on the preset image acquisition time and preset shutter response time, the effective IMU data of a single frame image is determined; The number of valid IMU data is taken as the first quantity, and the number of all row pixels is taken as the second quantity; Divide the second quantity by the first quantity to obtain the third quantity; Based on the third quantity, the effective IMU data corresponding to each row pixel is determined; The preset number is determined based on the preset ratio and the total number of pixels in all rows; Acquire the third IMU data and the fourth IMU data; Based on the preset uniform transition formula, the third IMU data, and the fourth IMU data, the corrected IMU data is generated, wherein the boundary IMU data is a preset number of IMU data at the junction of two effective IMU data. The image stabilization operation of the single frame is achieved by replacing the boundary pixels to be corrected in the uniform transition processing with the corrected IMU data.

2. The camera image stabilization method based on real-time video space according to claim 1, wherein the preset uniform transition formula is an IMU (In-Video Unit) algorithm. k= IMU1+n*(IMU2-IMU1) in, The IMU k The corrected IMU data refers to IMU1 being the third IMU data, IMU2 being the fourth IMU data, and n being a coefficient related to the preset quantity.

3. The camera image stabilization method based on real-time video space according to claim 1, wherein determining the effective IMU data of a single frame image based on a preset image acquisition time and a preset shutter response time includes: Based on the preset image acquisition time, determine all IMU data of the single frame image; Based on the preset shutter response time, the valid IMU data is determined from all the IMU data.

4. A camera image stabilization method based on real-time video space, comprising: Based on the preset image acquisition time and preset shutter response time, the effective IMU data of a single frame image is determined; The pixels of the single frame image are divided into multiple rows of pixels, and the valid IMU data corresponding to each row of pixels is determined. The effective IMU data is used to correct the position of the corresponding row pixels in order to achieve image stabilization for the single frame.

5. The camera stabilization method based on real-time video space according to claim 4, wherein dividing the pixels of the single-frame image into multiple rows of pixels and determining the effective IMU data corresponding to each row of pixels includes: The number of valid IMU data is taken as the first quantity, and the number of all row pixels is taken as the second quantity; Divide the second quantity by the first quantity to obtain the third quantity; Based on the third quantity, the effective IMU data corresponding to each row pixel is determined.

6. The camera image stabilization method based on real-time video space according to claim 4, wherein the row pixels include single-row pixels to be corrected, and the step of correcting the position of the corresponding row pixels using the effective IMU data to achieve image stabilization of the single frame includes: Obtain the number of pixels in the single row of pixels to be corrected, wherein the pixels in the single row of pixels to be corrected correspond to the first effective IMU data and the second effective IMU data respectively; The target valid IMU data is determined by the number of pixels of the single row of pixels to be corrected contained in the first valid IMU data and the second valid IMU data, respectively. The single row of pixels to be corrected is corrected using the target valid IMU data.

7. The camera image stabilization method based on real-time video space according to claim 4, wherein the row pixels include boundary pixels to be corrected, and the step of correcting the position of the corresponding row pixels using the effective IMU data to achieve image stabilization of the single frame includes: Based on a preset uniform transition formula, the boundary pixels to be corrected corresponding to the boundary IMU data are subjected to uniform transition processing to generate corrected IMU data, wherein the boundary IMU data is a preset number of IMU data at the junction of two valid IMU data. The image stabilization operation of the single frame is achieved by replacing the boundary pixels to be corrected in the uniform transition processing with the corrected IMU data.

8. The camera image stabilization method based on real-time video space according to claim 7, wherein before performing uniform transition processing on the boundary pixels to be corrected corresponding to the boundary IMU data based on a preset uniform transition formula to generate corrected IMU data, the method includes: The preset quantity is determined based on the preset ratio and the total number of pixels in all rows.

9. The camera image stabilization method based on real-time video space according to claim 8, wherein the boundary IMU data includes third IMU data and fourth IMU data, and the step of performing uniform transition processing on the row pixels corresponding to the boundary IMU data based on a preset uniform transition formula to generate corrected IMU data includes: Acquire the third IMU data and the fourth IMU data; The corrected IMU data is generated based on the preset uniform transition formula, the third IMU data, and the fourth IMU data.

10. The camera image stabilization method based on real-time video space according to claim 9, wherein the preset uniform transition formula is an IMU (In-Mutable Animation Model). k= IMU1+n*(IMU2-IMU1) in, The IMU k The corrected IMU data refers to IMU1 being the third IMU data, IMU2 being the fourth IMU data, and n being a coefficient related to the preset quantity.

11. The camera image stabilization method based on real-time video space according to claim 4, wherein determining the effective IMU data of a single frame image based on a preset image acquisition time and a preset shutter response time includes: Based on the preset image acquisition time, determine all IMU data of the single frame image; Based on the preset shutter response time, the valid IMU data is determined from all the IMU data.

12. A computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the camera stabilization method based on real-time video space as described in any one of claims 1.

Images