Image processing method and device, storage medium and electronic device

Abstract

The present disclosure provides an image processing method and device, electronic equipment and computer readable storage medium, and relates to the technical field of image processing. The method comprises: in a target region, determining a pixel feature value of each row in each video frame based on the gray value of each pixel point in the row; obtaining a first angular velocity and a first timestamp, and determining a first position corresponding to a target time; determining a first inter-frame pixel displacement according to the first position, determining a second inter-frame pixel displacement based on video matching, and determining a compensation pixel displacement; obtaining position data in a single direction and a second timestamp, and determining a second position corresponding to the target time; determining an inter-frame motion displacement according to the second position, and obtaining a first stabilization parameter based on the inter-frame motion displacement and the compensation pixel displacement; determining a second stabilization parameter based on the first stabilization parameter, the inter-frame motion displacement and the compensation pixel displacement, and processing the image according to the first and second stabilization parameters. The present disclosure improves the optical image stabilization effect.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and in particular, to an image processing method, an image processing apparatus, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] OIS (Optical Image Stabilization) counteracts the shaking of a camera or a camera by using special optical elements or components in the lens to keep the image stable.

[0003] In the prior art, optical image stabilization calibration is all a 2D method, and feature point detection is performed by using optical flow, but the feature point matching accuracy is reduced in a region where texture is not obvious or in a scene where the motion amplitude between the front and back frames is large, and the iterative solving process is time-consuming.

[0004] Therefore, in order to solve the above problems, the embodiments of the present disclosure provide an image processing method, an image processing apparatus, an electronic device, and a computer readable storage medium. SUMMARY

[0005] The purpose of the embodiments of the present disclosure is to provide an image processing method, an image processing apparatus, an electronic device, and a computer readable storage medium, thereby to some extent solving the problems in the related art that the feature point matching accuracy is reduced in a region where texture is not obvious or in a scene where the motion amplitude between the front and back frames is large, and the iterative solving process is time-consuming.

[0006] According to a first aspect of the present disclosure, an image processing method is provided, comprising: determining pixel point feature values of corresponding rows in a target region based on gray values of pixel points of each row of a video frame; acquiring a first angular velocity and a first timestamp by a first sensor, and determining a first position acquired by the first sensor at a target time according to the first angular velocity and the first timestamp, the target time being a time selected in a rolling shutter exposure time corresponding to the target region; determining a first inter-frame pixel displacement between adjacent video frames according to the first position, determining a second inter-frame pixel displacement between the adjacent video frames based on a video matching method according to the pixel point feature values, and determining a compensation pixel displacement according to the first inter-frame pixel displacement and the second inter-frame pixel displacement; acquiring position data in a single direction and a second timestamp by a second sensor, and determining a second position acquired by the second sensor at the target time according to the position data and the second timestamp; determining an inter-frame motion displacement of each of the adjacent video frames according to the second position, and obtaining a first stabilization parameter based on the inter-frame motion displacement and the compensation pixel displacement; determining a second stabilization parameter based on the first stabilization parameter, the inter-frame motion displacement and the compensation pixel displacement, and performing image stabilization processing according to the first stabilization parameter and the second stabilization parameter.

[0007] In an exemplary embodiment of the present disclosure, the method further comprises: constructing a loss function using the first stabilization parameter and the second stabilization parameter; and obtaining a target first stabilization parameter and a target second stabilization parameter by iteration using a global optimization algorithm based on the loss function.

[0008] In an example embodiment of the present disclosure, the constructing the loss function by using the first stabilization parameter and the second stabilization parameter comprises: constructing a pixel point array in advance, and determining a motion angle of each pixel point in the pixel point array, wherein pixel point feature values of each pixel point in the pixel point data are increasing; calculating rolling shutter exposure times of each row of pixel points in the pixel point array to obtain exposure time points of each row of pixels in a current video frame; acquiring a third timestamp and a displacement angle by using the first sensor, constructing a first interpolation function based on the third timestamp and the displacement angle, and determining a first displacement angle corresponding to the exposure time points of the each row of pixels by using the first interpolation function; acquiring a fourth timestamp by using the second sensor, constructing a second interpolation function according to the fourth timestamp, the second stabilization parameter and the inter-frame motion displacement, and determining a second displacement angle corresponding to the exposure time points of the each row of pixels based on the second interpolation function; constructing a third interpolation function according to the first displacement angle, the second displacement angle and the pixel point feature values of the each row of pixels, and acquiring pixel point feature values of each row of pixels of a next video frame based on the third interpolation function; determining pixel point feature values of the next video frame mapped to the current video frame, and constructing the loss function based on the pixel point feature values of the next video frame and the pixel point feature values of the next video frame mapped to the current video frame.

[0009] In an example embodiment of the present disclosure, the determining the pixel point feature values of the corresponding rows of pixels based on the gray values of the each row of pixel points in the target region comprises: taking a center region of each video frame as the target region, and acquiring gray values of each row of pixel points in the target region; determining an average value of the gray values of each row of pixel points, and taking the average value of the gray values as the pixel point feature value of the corresponding row.

[0010] In an example embodiment of the present disclosure, the first sensor is a gyroscope, and the target time point is a middle time point of a rolling shutter exposure time corresponding to the center region; the acquiring the first angular velocity and the first timestamp by using the first sensor, and determining the first position acquired by using the first sensor at the target time point according to the first angular velocity and the first timestamp comprises: acquiring an angular velocity and a timestamp of the gyroscope, taking the angular velocity of the gyroscope as the first angular velocity, taking the timestamp of the gyroscope as the first timestamp, and constructing a fourth interpolation function according to the first angular velocity and the first timestamp; acquiring a gyroscope position corresponding to the target time point based on the fourth interpolation function to obtain the first position.

[0011] In an example embodiment of the present disclosure, the second sensor is a Hall sensor; the position data in a single direction and the second timestamp acquired by the second sensor are determined according to the position data and the second timestamp to determine the second position acquired by the second sensor at the target time, including: acquiring the timestamp of the Hall sensor and the sensor coordinate data in a single coordinate axis direction, taking the sensor coordinate data as the position data, and taking the timestamp of the Hall sensor as the second timestamp; constructing a fifth difference function based on the position data and the second timestamp, and determining the sensor coordinate data corresponding to the target time based on the fifth difference function as the second position.

[0012] In an example embodiment of the present disclosure, the compensation pixel displacement is determined according to the first inter-frame pixel displacement and the second inter-frame pixel displacement, including: calculating the difference between the first inter-frame pixel displacement and the second inter-frame pixel displacement to obtain the compensation pixel displacement.

[0013] According to a second aspect of the present disclosure, an image processing device is provided, including: a feature value determination module configured to determine the pixel point feature value of each row in a target region based on the gray value of each pixel point of the video frame; a first position determination module configured to acquire the first angular velocity and the first timestamp by the first sensor, and determine the first position acquired by the first sensor at the target time according to the first angular velocity and the first timestamp, the target time being a time selected in the corresponding rolling shutter exposure time of the target region; a compensation pixel displacement determination module configured to determine the first inter-frame pixel displacement between adjacent video frames according to the first position, determine the second inter-frame pixel displacement between the adjacent video frames based on the pixel point feature value by a video matching method, and determine the compensation pixel displacement according to the first inter-frame pixel displacement and the second inter-frame pixel displacement; a second position determination module configured to acquire the position data in a single direction and the second timestamp by the second sensor, and determine the second position acquired by the second sensor at the target time according to the position data and the second timestamp; a first stabilization parameter determination module configured to determine the inter-frame motion displacement of each of the adjacent video frames according to the second position, and obtain the first stabilization parameter based on the inter-frame motion displacement and the compensation pixel displacement; and an image processing module configured to determine the second stabilization parameter based on the first stabilization parameter, the inter-frame motion displacement, and the compensation pixel displacement, to perform anti-shake processing on the image according to the first stabilization parameter and the second stabilization parameter.

[0014] In an example embodiment of the present disclosure, the image stabilization device further comprises a parameter optimization unit configured to: construct a loss function using the first stabilization parameter and the second stabilization parameter; and obtain a target first stabilization parameter and a target second stabilization parameter using a global optimization algorithm based on the loss function.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory storing executable instructions of the processor; wherein the processor is configured to execute the method according to any one of the preceding aspects via executing the executable instructions.

[0016] According to a fourth aspect of the present disclosure, a computer readable storage medium is provided, having stored thereon a computer program, which, when executed by a processor, implements the method according to any one of the preceding aspects.

[0017] The example embodiments of the present disclosure can have the following partial or all beneficial effects:

[0018] In the image processing method provided by the example embodiment of the present disclosure, in the target region, a pixel point feature value of each corresponding row is determined based on a gray value of each row of pixel points of the video frame; a first angular velocity and a first timestamp are obtained by a first sensor, and a first position obtained by the first sensor at a target moment is determined according to the first angular velocity and the first timestamp, the target moment being a moment selected in a corresponding rolling shutter exposure time of the target region; a first interframe pixel displacement between adjacent video frames is determined according to the first position, a second interframe pixel displacement between adjacent video frames is determined based on a video matching method according to the pixel point feature value, and a compensation pixel displacement is determined according to the first interframe pixel displacement and the second interframe pixel displacement; position data in a single direction and a second timestamp are obtained by a second sensor, and a second position obtained by the second sensor at the target moment is determined according to the position data and the second timestamp; an interframe motion displacement of each adjacent video frame is determined according to the second position, and a first stabilization parameter is obtained based on the interframe motion displacement and the compensation pixel displacement; a second stabilization parameter is determined based on the first stabilization parameter, the interframe motion displacement and the compensation pixel displacement, so as to perform anti-shake processing on the image according to the first stabilization parameter and the second stabilization parameter. The example embodiment of the present disclosure simplifies the solving model by detecting the motion of the front and rear video frames in a single direction, reduces the interference factors, improves the motion detection accuracy between the front and rear video frames, and thus improves the calibration accuracy. Meanwhile, under the same processor, the calibration along the single axis greatly reduces the time consumption of calibration.

[0019] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the disclosure. It is to be understood that the drawings are designed solely for purposes of illustration to be used in conjunction with the description. It is to be understood that the drawings are designed solely for purposes of illustration and to aid in the understanding of the disclosure, and in which:

[0021] Figure 1 A flowchart schematically illustrates an image processing method according to one embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of an OIS lens module according to one embodiment of the present disclosure is schematically illustrated;

[0023] Figure 3 A schematic diagram of a certain video frame according to one embodiment of the present disclosure is schematically illustrated;

[0024] Figure 4 An error distribution diagram of 1D OIS hall calibration parameter optimization according to one embodiment of the present disclosure is schematically illustrated;

[0025] Figure 5 A 1D OIS motion hall code variation curve according to one embodiment of the present disclosure is schematically illustrated;

[0026] Figure 6 A block diagram of an image processing apparatus according to one embodiment of the present disclosure is schematically illustrated;

[0027] Figure 7 A schematic diagram of an electronic device according to one embodiment of the present disclosure is schematically illustrated. DETAILED DESCRIPTION

[0028] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art. The described features, structures, or characteristics can be combined in one or more implementations. In the following description, numerous specific details are provided to give a thorough understanding of example implementations. One skilled in relevant art will recognize, however, that the implementations can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures have not been described in detail to avoid obscuring the understanding of this description.

[0029] In addition, the accompanying drawings are only schematic and are non-limiting illustrative of the disclosure. Identical reference signs denote identical or similar parts throughout the figures. Some of the blocks in the drawings are functional entities that may be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0030] OIS (Optical Image Stabilization) counteracts the shaking of a camera or a camcorder by using special optical elements or components in the lens to keep the image stable.

[0031] In the related art, OIS calibration is mostly 2D calibration. Exemplarily, the method can be implemented as follows: feature point matching of the front and rear frames is performed by using optical flow, RANSAC (Random Sample Consensus) is used to remove the wrong feature points, and the final feature point matching result is obtained, so that the feature point P2 of the next frame can be calculated from the feature point P1 of the previous frame; the mapping relationship of the feature points between the two frames is calculated using the gyroscope and OIS hall (Hall sensor) information, and the feature point P3 of the next frame can be calculated from the feature point P1 of the previous frame; by using an optimization algorithm, the loss = P2-P3 is constantly approaching 0, and the stable parameters are determined, so as to stabilize the image by using the stable parameters, and achieve the optical image stabilization effect.

[0032] However, in the above method, the feature point matching accuracy is reduced in areas where the texture is not obvious or in scenes where the motion amplitude between the front and rear frames is large, and the iterative solving process is time-consuming.

[0033] In order to solve the problems existing in the above method, the present example embodiment provides an image processing method, an image processing device, an electronic device and a computer readable storage medium. The technical solutions of the embodiments of the disclosure are described in detail as follows:

[0034] The present example embodiment first provides an image processing method. Referring to FIG. 1, Figure 1 The image processing method specifically includes the following steps:

[0035] Step S110: In the target region, the pixel feature value of the corresponding row is determined based on the gray value of each row of pixel points of the video frame;

[0036] Step S120: acquiring the first angular velocity and the first timestamp by the first sensor, and determining the first position acquired by the first sensor at the target moment according to the first angular velocity and the first timestamp, the target moment being a moment selected in the exposure time of the roller shutter corresponding to the target region;

[0037] Step S130: determining the first interframe pixel displacement between adjacent video frames according to the first position, determining the second interframe pixel displacement between adjacent video frames according to the pixel point feature value based on the video matching method, and determining the compensation pixel displacement according to the first interframe pixel displacement and the second interframe pixel displacement;

[0038] Step S140: acquiring the position data in a single direction and the second timestamp by the second sensor, and determining the second position acquired by the second sensor at the target moment according to the position data and the second timestamp;

[0039] Step S150: determining the interframe motion displacement of each adjacent video frame according to the second position, and obtaining the first stabilization parameter based on the interframe motion displacement and the compensation pixel displacement;

[0040] Step S160: determining the second stabilization parameter based on the first stabilization parameter, the interframe motion displacement, and the compensation pixel displacement, and performing the anti-shake processing on the image according to the first stabilization parameter and the second stabilization parameter.

[0041] In the image processing method provided in the example embodiment of the present disclosure, by detecting the motion of the front and rear video frames along a single direction, the solving model is simplified, the interference factors are reduced, and the motion detection accuracy between the front and rear video frames is improved, thereby improving the calibration accuracy. At the same time, under the same processor, the calibration along a single axis greatly reduces the time consumption of calibration.

[0042] In another embodiment, the above steps are described in more detail as follows.

[0043] In step S110, in the target region, the pixel point feature value of the corresponding row is determined based on the gray value of each row of pixel points of the video frame.

[0044] The optical anti-shake of the embodiment of the present disclosure mainly compensates the camera motion amount at the current moment by the way of panning lens, thereby playing an anti-shake effect. Limited by the hardware conditions of the lens module, the maximum compensation range of each coordinate axis is about ±1°, and the OIS lens module schematic diagram is as shown in Figure 2 In the lens module, the Hall sensor detects the OIS module position and reports the data, and at the same time, combines the tilt angle information of the gyroscope, and then drives the OIS to move in the X and Y axes to stabilize the imaging.

[0045] In the embodiments of the present disclosure, the video frame is a still image in a video, which is the smallest unit in a video sequence. The video frame is composed of pixels, each of which represents a point in the image and contains information about color and brightness, etc. In order to achieve better anti-shake effect, the target region can be the center region of each video frame, as shown in Figure 3 Figure 3 For a certain video frame, 301 is the center region of the video frame, which is the selected target region.

[0046] The gray value is a numerical value used to represent the brightness or gray intensity of an image. In the target region, the pixel point feature value of each corresponding row can be determined based on the gray values of the pixel points of each row of the video frame as follows: taking the center region of each video frame as the target region, and obtaining the gray values of the pixel points of each row in the target region; determining the average value of the gray values of each row of pixel points, and taking the average value of the gray values as the pixel point feature value of each corresponding row of pixel points. It should be noted that the above scenario is only an example, and the protection scope of the embodiments of the present disclosure is not limited thereto. For example, the pixel point feature value can also be a numerical value obtained by other processing on the gray value.

[0047] In step S120, the first angular velocity and the first timestamp are obtained by the first sensor, and the first position obtained by the first sensor at the target time is determined according to the first angular velocity and the first timestamp. The target time is a time selected in the corresponding rolling shutter exposure time of the target region.

[0048] In the embodiments of the present disclosure, the first sensor is used to obtain the angular velocity and the corresponding timestamp of the terminal device with a shooting function. For example, the first sensor can be a gyroscope, and the first angular velocity and the first timestamp are the angular velocity and the timestamp of the terminal device during shooting obtained by the gyroscope. The gyroscope is a sensor used to measure and perceive rotational motion. It is usually composed of a rotating gyroscope or other rotating mechanism, and can measure the rotational angular velocity of an object around its three axes (X-axis, Y-axis and Z-axis). The first angular velocity is a physical quantity describing the speed of rotation of an object, which is usually represented by the angle of rotation around the axis per unit time. The gyroscope measures the first angular velocity by sensing the rotational motion and converting it into an electrical signal. The first timestamp is used to mark the time point of each first angular velocity measurement. The first timestamp can represent the system time at the start or end of the measurement. Generally, the timestamp can be a counter value, which is in units of milliseconds, microseconds or nanoseconds, representing the time elapsed since a certain reference time point. Through the first timestamp information provided by the gyroscope, the change of the first angular velocity can be positioned and analyzed in time.

[0049] ​The first position acquired by the first sensor at the target moment can be achieved according to the first angular velocity and the first timestamp as follows: a fourth interpolation function is constructed according to the first angular velocity and the first timestamp acquired by the gyroscope; and a gyroscope position corresponding to the target moment is acquired based on the fourth interpolation function, which is the first position. In the embodiments of the present disclosure, each video frame is obtained by a rolling shutter exposure method, and the target moment is a moment selected in the rolling shutter exposure time corresponding to the target region. For example, the middle moment of the rolling shutter exposure time of the video frame can be selected as the target moment. For example, the process can be specifically implemented as follows: the angular displacement A1 of the gyroscope is determined according to the first angular velocity and the first timestamp acquired by the gyroscope. An interpolation function is constructed according to A1 and the gyroscope timestamp Time1, and the gyroscope position (the first position) A2 corresponding to the middle moment Time2 of each frame is calculated.

[0050] In step S130, the first interframe pixel displacement between adjacent video frames is determined according to the first position, the second interframe pixel displacement between adjacent video frames is determined according to the pixel point feature value based on the video matching method, and the compensation pixel displacement is determined according to the first interframe pixel displacement and the second interframe pixel displacement.

[0051] In the embodiments of the present disclosure, the interframe pixel displacement refers to the position change amount of a certain pixel between two adjacent frames in a video sequence. By comparing the pixel values of two consecutive frames, the displacement of the pixel in the horizontal and vertical directions can be calculated. Specifically, the first interframe pixel displacement is determined according to the first position obtained by the gyroscope information in step S120, and the interframe pixel displacement is further determined.

[0052] pixel_1=tan(ΔA2)*FocalLength

[0053] Wherein, pixel_1 is the first interframe pixel displacement, A2 is the gyroscope position corresponding to the middle moment Time2 of each frame, and FocalLength is the focal length of the camera.

[0054] The second inter-frame pixel displacement is an inter-frame pixel displacement pixel_2 determined based on video matching. The inter-frame pixel displacement based on video matching refers to a displacement of each pixel in time calculated by comparing pixel differences between two adjacent frames in video processing. The implementation can be as follows: pixel data of adjacent frames is extracted; differences of each pixel between the two frames are compared, and pixel-level differences or higher-level feature matching methods can be used; the displacement of each pixel in time is calculated according to the differences; the displacement can be further processed, such as smoothing, interpolation or application to image stabilization, according to requirements. In the embodiment of the present disclosure, the second inter-frame pixel displacement can be obtained by calculating the difference between the pixel feature values of adjacent video frames.

[0055] In the embodiment of the present disclosure, the compensation pixel displacement determined according to the first inter-frame pixel displacement and the second inter-frame pixel displacement can be implemented as follows: the difference between the first inter-frame pixel displacement and the second inter-frame pixel displacement is calculated to obtain the compensation pixel displacement. Exemplarily, the process can be specifically implemented as follows: the difference between the second inter-frame pixel displacement pixel_2 based on video matching and the first inter-frame pixel displacement pixel_1 obtained based on the gyroscope information is calculated to obtain the compensation pixel displacement pixel_3 of the OIS motion.

[0056] In step S140, the position data in a single direction and the second time stamp are obtained by the second sensor, and the second position obtained by the second sensor at the target time according to the position data and the second time stamp is obtained.

[0057] In order to solve the problems of inaccurate feature point matching, long time consumption and poor robustness in the 2D OIS hall calibration process, the embodiment of the present disclosure constructs a solving model based on the OIS motion along one axis to reduce interference factors. Specifically, the second sensor can be a hall sensor, and the 1D calibration is realized by obtaining data information of the hall sensor on a single direction coordinate axis.

[0058] The Hall sensor is a sensor based on the working principle of the Hall effect. The Hall effect refers to the generation of a certain voltage in the transverse direction when an electric current passes through a certain material under the condition of the presence of a magnetic field. The Hall sensor uses this effect to detect and measure the presence and strength of a magnetic field. The Hall sensor is usually composed of a Hall element, an amplifier, and an output stage. The Hall element is a thin sheet or chip that contains conductive materials and charge carriers inside. Under the influence of an external magnetic field, the charge carriers are deflected, generating a Hall voltage. The amplifier amplifies this small Hall voltage signal, and through the output stage, it is converted into a usable voltage or digital signal output. These parameters can be converted into coordinate data according to the requirements of specific applications, i.e., the sensor coordinate axis data mentioned above. Exemplarily, the timestamp of the sensor can be obtained by using a microcontroller or single-chip microcomputer to read the output of the Hall sensor and record the current timestamp information at each sampling, and associating the timestamp with the output of the sensor.

[0059] In the embodiments of the present disclosure, the process of acquiring position data in a single direction and a second timestamp through the second sensor, determining a second position acquired by the second sensor at a target time according to the position data and the second timestamp can be implemented as follows: acquiring the timestamp of the Hall sensor and the sensor coordinate data in a single coordinate axis direction, taking the sensor coordinate axis data as the position data, and taking the timestamp of the Hall sensor as the second timestamp; constructing a fifth difference function based on the position data and the second timestamp, and determining the sensor coordinate data corresponding to the target time as the second position based on the fifth difference function. The target time can be selected as the middle time of the rolling shutter exposure time of the video frame. Specifically, the process can be implemented as follows: constructing an interpolation function according to the Y-axis data hall_1 of the OIS Hall sensor and the timestamp hallTime, calculating the hall data hall_2 corresponding to the middle time Time_2 of each frame, and obtaining the second position of each video frame at the target time, wherein the Y-axis data of the Hall sensor is the position data in the single direction.

[0060] In step S150, the inter-frame motion displacement of each adjacent video frame is determined according to the second position, and the first stabilization parameter is obtained based on the inter-frame motion displacement and the compensation pixel displacement.

[0061] In the embodiments of the present disclosure, the inter-frame motion displacement of each adjacent video frame according to the second position can be obtained by the difference between the second positions of adjacent frames. Specifically, after the hall data hall_2 corresponding to the middle time Time_2 of each frame is calculated, the inter-frame OIS motion position hall_3 can be further calculated.

[0062] The first stabilization parameter is a parameter for stabilizing the image, and can be obtained by compensating for the ratio of the pixel displacement and the inter-frame motion displacement. Specifically, the first stabilization parameter can be an initial value of pixel / hall: PixelPerHall = pixel_3 / hall_3.

[0063] In step S160, a second stabilization parameter is determined based on the first stabilization parameter, the inter-frame motion displacement, and the compensation pixel displacement, so as to perform the anti-shake processing on the image according to the first stabilization parameter and the second stabilization parameter.

[0064] The second stabilization parameter is a parameter for stabilizing the image. For example, the process of determining the second stabilization parameter based on the first stabilization parameter, the inter-frame motion displacement, and the compensation pixel displacement can be implemented as follows: search is performed at different time steps, and the initial value HallDelay of the time delay between the OIS hall value and the video can be predicted by comparing the displacement pixel points of hall_3*PixelPerHall and pixel_3, and the initial value HallDelay is the second stabilization parameter.

[0065] After obtaining the first stabilization parameter and the second stabilization parameter, the embodiment of the present disclosure can write the first stabilization parameter and the second stabilization parameter into the device with the photographing function, so that the device processes the photographed image based on the first stabilization parameter and the second stabilization parameter, and realizes the optical anti-shake effect.

[0066] In another embodiment of the present disclosure, the first stabilization parameter and the second stabilization parameter can also be optimized. For example, the optimization process can be implemented as follows: a loss function is constructed using the first stabilization parameter and the second stabilization parameter; and the target first stabilization parameter and the target second stabilization parameter are obtained by using a global optimization algorithm based on the loss function.

[0067] The process of constructing the loss function can be implemented as follows: an increasing pixel point array is constructed, and the motion angle of each pixel point in the pixel point array is determined; a pixel point array is constructed in advance, and the motion angle of each pixel point in the pixel point array is determined, wherein the pixel point feature values of each pixel point in the pixel point data are increasing; the rolling shutter exposure time of each row of pixel points in the pixel point array is calculated to obtain the exposure time of each row of pixels in the current video frame; the third timestamp and the displacement angle are obtained through the first sensor, the first interpolation function is constructed based on the third timestamp and the displacement angle, and the first displacement angle corresponding to the exposure time of each row of pixels is determined by using the first interpolation function; the fourth timestamp is obtained through the second sensor, the second interpolation function is constructed according to the fourth timestamp, the second stable parameter and the inter-frame motion displacement, and the second displacement angle corresponding to the exposure time of each row of pixels is determined based on the second interpolation function; the third interpolation function is constructed according to the first displacement angle, the second displacement angle and the pixel point feature value of each row of pixels, and the pixel point feature value of each row of pixels of the next video frame is obtained based on the third interpolation function; the pixel point feature value of the next video frame mapped to the current video frame is determined, and the loss function is constructed based on the pixel point feature value of the next video frame and the pixel point feature value of the next video frame mapped to the current video frame.

[0068] In a specific embodiment, the construction of the loss function can include the following steps:

[0069] S1: an increasing pixel point array is constructed, P1 represents the position of each row of pixel points, and FocalLength represents the focal length, so that the angle corresponding to the motion of each pixel point can be approximately calculated:

[0070] A3 = arctan (P1 / FocalLength)

[0071] S2: the rolling shutter exposure time ROT corresponding to each row of pixel points is calculated to obtain the time of each row of pixels in this frame:

[0072] Time3 = ROT + Time2[i]

[0073] S3: an interpolation function gyroPosIntp is constructed by using the timestamp Time1 of the gyroscope and the inter-frame displacement P1 of the gyroscope, and the displacement angle of the gyroscope corresponding to the time of each row of pixels in this frame is obtained by interpolation:

[0074] A4 = gyroPosIntp (Time3)

[0075] S4: an interpolation function interpHall is constructed by using the timestamp hallTime of the OIS and the inter-frame displacement of the OIS, the displacement position of each row of pixels in this frame is calculated, and then the angle of each row of displacement is calculated:

[0076] P2 = P1 + interpHall(Time3) * PixelPerHall

[0077] A5 = arctan(P2 / FocalLength)

[0078] S5: using A4 + A5 and FrameValue (pixel feature value) to construct an interpolation function frame1DIntp to obtain the pixel value of each row of the new frame:

[0079] newFrame = frame1DIntp(A3)

[0080] S6: define the feature value of the previous frame as oldFrame, and the feature value of the next frame as newFrame. Then the feature value of the next frame mapped to the previous frame is:

[0081] NormFrame = (oldFrame - avg(oldFrame)) * (std(newFrame) / std(oldFrame)) + avg(newFrame)

[0082] avg(newFrame)

[0083] S7: construct the loss function: loss = abs(NormFrame - newFrame)

[0084] As shown in Figure 4 , after obtaining the above loss function, the Basin Hopping global optimization algorithm can be used to predict PixelPerHallLSB and HallTSDelay time based on the above loss function.

[0085] The embodiment of the present disclosure proposes a method for calibrating 1D-OIS. By detecting the feature points of the motion of the front and rear frames along a single axis, the 1D OIS motion hall code change curve is as shown in Figure 5 . This method simplifies the solution model, reduces interference factors, improves the motion detection accuracy between the front and rear frames, and thus improves the OIS hall calibration accuracy.

[0086] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. In addition or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be divided into multiple steps, etc.

[0087] Further, in the example embodiment, an image processing device is also provided, which is described with reference toFigure 6 As shown in the figure, the image processing apparatus 600 can include a feature value determination module 610, a first position determination module 620, a compensation pixel displacement determination module 630, a second position determination module 640, a first stabilization parameter determination module 650, and an image processing module 660. Among them:

[0088] The feature value determination module 610 can be configured to determine the pixel feature value of each row in the target region based on the gray value of the pixel point of each row in the video frame.

[0089] The first position determination module 620 can be configured to obtain the first angular velocity and the first timestamp through the first sensor, and determine the first position obtained through the first sensor at the target time according to the first angular velocity and the first timestamp, the target time being a time selected in the corresponding rolling shutter exposure time of the target region.

[0090] The compensation pixel displacement determination module 630 can be configured to determine the first inter-frame pixel displacement between adjacent video frames according to the first position, determine the second inter-frame pixel displacement between adjacent video frames based on the pixel feature value by using a video matching method, and determine the compensation pixel displacement according to the first inter-frame pixel displacement and the second inter-frame pixel displacement.

[0091] The second position determination module 640 can be configured to obtain the position data in a single direction and the second timestamp through the second sensor, and determine the second position obtained through the second sensor at the target time according to the position data and the second timestamp.

[0092] The first stabilization parameter determination module 650 can be configured to determine the inter-frame motion displacement of each adjacent video frame according to the second position, and obtain the first stabilization parameter based on the inter-frame motion displacement and the compensation pixel displacement.

[0093] The image processing module 660 can be configured to determine the second stabilization parameter based on the first stabilization parameter, the inter-frame motion displacement, and the compensation pixel displacement, and to perform anti-shake processing on the image according to the first stabilization parameter and the second stabilization parameter.

[0094] In the embodiments of the present disclosure, the image processing apparatus further includes a parameter optimization unit, which is configured to construct a loss function by using the first stabilization parameter and the second stabilization parameter, and obtain a target first stabilization parameter and a target second stabilization parameter by using a global optimization algorithm based on the loss function.

[0095] Specifically, the parameter optimization unit constructs the loss function using the following method: A pixel array is pre-constructed, and the motion angle of each pixel in the array is determined, where the pixel feature values ​​of each pixel in the pixel data are incremental; the shutter exposure time of each row of pixels in the pixel array is calculated to obtain the exposure time of each row of pixels in the current video frame; a third timestamp and displacement angle are obtained through a first sensor, a first interpolation function is constructed based on the third timestamp and displacement angle, and the first interpolation function is used to determine the first displacement angle corresponding to the exposure time of each row of pixels; a second sensor... The device obtains a fourth timestamp, constructs a second interpolation function based on the fourth timestamp, the second stabilization parameter, and the inter-frame motion displacement, and determines the second displacement angle corresponding to the exposure time of each row of pixels based on the second interpolation function; constructs a third interpolation function based on the first displacement angle, the second displacement angle, and the pixel feature values ​​of each row of pixels, and obtains the pixel feature values ​​of each row of pixels in the next video frame based on the third interpolation function; determines the pixel feature values ​​of the next video frame mapped to the current video frame, and constructs a loss function based on the pixel feature values ​​of the next video frame and the pixel feature values ​​of the next video frame mapped to the current video frame.

[0096] The specific implementation details of the above image processing device have been explained in detail in the corresponding section of the above image processing method, so they will not be repeated here.

[0097] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0098] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 7 It shows a schematic diagram of the structure suitable for implementing the electronic device 700 in the embodiments of this disclosure. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0099] like Figure 7As shown, the electronic device 700 can include a processing device (e.g., a central processor, a graphics processor, etc.) 701 that can perform various appropriate actions and processes to implement the image processing method of embodiments as described in the present disclosure according to programs stored in a read-only memory (ROM) 702 or loaded into a random access memory (RAM) 703 from a storage device 708. Various programs and data required by the electronic device 700 to operate are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0100] Generally, the following devices can be connected to the I / O interface 705: input devices 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 708 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 709. The communication devices 709 can allow the electronic device 700 to communicate wirelessly or wired with other devices to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or present. More or fewer devices can alternatively be implemented or present.

[0101] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts, thereby implementing the image processing method as described above. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 709, or installed from the storage devices 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0102] It should be noted that the computer-readable medium described above can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the disclosure, the computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, in which the computer-readable program code is contained. Such a propagated data signal can take any of a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the foregoing. The computer-readable signal medium can also be any computer-readable medium that is not a storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including, but not limited to, wire, cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

[0103] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0104] The computer-readable medium described above can be included in the electronic device; or exist separately from the electronic device, and not be assembled into the electronic device.

[0105] The computer-readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to:

[0106] In the target region, the pixel feature value of each corresponding row is determined based on the gray value of the pixel point of each row of the video frame;

[0107] The first angular velocity and the first timestamp are acquired by the first sensor, and the first position acquired by the first sensor at the target moment is determined according to the first angular velocity and the first timestamp, the target moment being a moment selected in the corresponding rolling shutter exposure time of the target region;

[0108] The first interframe pixel displacement between adjacent video frames is determined according to the first position, the second interframe pixel displacement between adjacent video frames is determined according to the pixel feature value based on the video matching method, and the compensation pixel displacement is determined according to the first interframe pixel displacement and the second interframe pixel displacement;

[0109] The position data in a single direction and the second timestamp are acquired by the second sensor, and the second position acquired by the second sensor at the target moment is determined according to the position data and the second timestamp;

[0110] The interframe motion displacement of each adjacent video frame is determined according to the second position, and the first stabilization parameter is obtained based on the interframe motion displacement and the compensation pixel displacement;

[0111] The second stabilization parameter is determined based on the first stabilization parameter, the interframe motion displacement and the compensation pixel displacement, and the image is subjected to anti-shake processing according to the first stabilization parameter and the second stabilization parameter.

[0112] Optionally, when the one or more programs are executed by the electronic device, the electronic device can further execute other steps described in the above embodiments.

[0113] Computer program code for carrying out operations of the present disclosure can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0114] The computer program product of the first aspect can include one or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform the operations of the first aspect. The computer program product of the first aspect can include a non-transitory computer-readable medium storing code that, when executed, causes a computer to perform operations for the first aspect.

[0115] The units described in the embodiments of the present disclosure can be implemented by software, or by hardware, or by a combination of software and hardware. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0116] The functions described in this description above can be implemented in hardware, software, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media include both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media can be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code means in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, functional

[0117] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium can include a tangible, non-transitory memory such as a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0118] The above description merely illustrates the preferred embodiments of the disclosure and a principle for applying the technologies. It is understood by those skilled in the art that the disclosed scope of the disclosure is not limited to the technical solutions formed by the specific combinations of the technical features described above, and should also cover other technical solutions formed by the combinations of the technical features described above or their equivalent features without departing from the disclosed concept. For example, the technical solutions formed by the mutual replacement of the above-described features and the technical features with similar functions disclosed in the disclosure (but not limited to) can be formed.

[0119] Further, although operations are depicted in a particular, sequential order, this should not be understood as requiring or implying that the operations are performed in the order illustrated or sequentially. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, although specific implementation details are included for the purpose of providing a thorough disclosure, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0120] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. An image processing method, characterized in that, include: In the target area, the pixel feature value of the corresponding row is determined based on the gray value of each row of pixels in the video frame; The first angular velocity and the first timestamp are obtained by the first sensor, and the first position obtained by the first sensor at the target time is determined based on the first angular velocity and the first timestamp. The target time is the time selected from the shutter exposure time corresponding to the target area. The first inter-frame pixel displacement between adjacent video frames is determined based on the first position, the second inter-frame pixel displacement between adjacent video frames is determined based on the pixel feature value using a video matching method, and the compensation pixel displacement is determined based on the first inter-frame pixel displacement and the second inter-frame pixel displacement. The second sensor acquires position data and a second timestamp in a single direction, and the second position acquired by the second sensor at the target time is determined based on the position data and the second timestamp. The inter-frame motion displacement of each adjacent video frame is determined based on the second position, and a first stabilization parameter is obtained based on the inter-frame motion displacement and the compensated pixel displacement. A second stabilization parameter is determined based on the first stabilization parameter, the inter-frame motion displacement, and the compensated pixel displacement, so as to perform image stabilization processing according to the first stabilization parameter and the second stabilization parameter.

2. The image processing method according to claim 1, characterized in that, The method further includes: A loss function is constructed using the first and second stability parameters; Based on the loss function, the first stable parameter and the second stable parameter of the target are obtained iteratively using a global optimization algorithm.

3. The image processing method according to claim 2, characterized in that, The step of constructing a loss function using the first stable parameter and the second stable parameter includes: A pixel array is pre-constructed, and the motion angle of each pixel in the pixel array is determined, wherein the pixel feature value of each pixel in the pixel data is increasing; Calculate the shutter exposure time of each row of pixels in the pixel array to obtain the exposure time of each row of pixels in the current video frame; The third timestamp and displacement angle are obtained through the first sensor, a first interpolation function is constructed based on the third timestamp and displacement angle, and the first displacement angle corresponding to the exposure time of each row of pixels is determined using the first interpolation function. A fourth timestamp is obtained through the second sensor. A second interpolation function is constructed based on the fourth timestamp, the second stabilization parameter, and the inter-frame motion displacement. The second displacement angle corresponding to the exposure time of each row of pixels is determined based on the second interpolation function. A third interpolation function is constructed based on the first displacement angle, the second displacement angle, and the pixel feature values ​​of each row of pixels, and the pixel feature values ​​of each row of pixels in the next video frame are obtained based on the third interpolation function. Determine the pixel feature values ​​that the next video frame maps to the current video frame, and construct the loss function based on the pixel feature values ​​of the next video frame and the pixel feature values ​​that the next video frame maps to the current video frame.

4. The image processing method according to claim 1, characterized in that, The step of determining the pixel feature value of the corresponding row in the target area based on the grayscale value of each row of pixels in the video frame includes: The central region of each video frame is taken as the target region, and the grayscale value of each row of pixels in the target region is obtained; Determine the average grayscale value of each row of pixels, and use the average grayscale value as the feature value of the corresponding row of pixels.

5. The image processing method according to claim 4, characterized in that, The first sensor is a gyroscope, and the target time is the midpoint of the exposure time of the roller shutter corresponding to the central region; The step of acquiring a first angular velocity and a first timestamp through a first sensor, and determining the first position acquired by the first sensor at the target time based on the first angular velocity and the first timestamp, includes: The angular velocity and timestamp of the gyroscope are obtained, the angular velocity of the gyroscope is used as the first angular velocity, the timestamp of the gyroscope is used as the first timestamp, and a fourth interpolation function is constructed based on the first angular velocity and the first timestamp. The first position is obtained by obtaining the gyroscope position corresponding to the target time based on the fourth interpolation function.

6. The image processing method according to claim 5, characterized in that, The second sensor is a Hall sensor; The second sensor acquires position data and a second timestamp in a single direction, and determines the second position acquired by the second sensor at the target time based on the position data and the second timestamp, including: The timestamp of the Hall sensor and its sensor coordinate data in a single coordinate axis direction are obtained, and the sensor coordinate axis data is used as the position data, and the timestamp of the Hall sensor is used as the second timestamp. A fifth interpolation function is constructed based on the location data and the second timestamp, and the sensor coordinate data corresponding to the target time is determined based on the fifth interpolation function, which is used as the second position.

7. The image processing method according to claim 1, characterized in that, The step of determining the compensated pixel displacement based on the first inter-frame pixel displacement and the second inter-frame pixel displacement includes: The difference between the first inter-frame pixel displacement and the second inter-frame pixel displacement is calculated to obtain the compensated pixel displacement.

8. An image processing apparatus, characterized in that, include: The feature value determination module is used to determine the feature value of the corresponding row of pixels in the target area based on the gray value of each row of pixels in the video frame; The first position determination module is used to acquire a first angular velocity and a first timestamp through a first sensor, and determine the first position acquired by the first sensor at a target time based on the first angular velocity and the first timestamp, wherein the target time is a time selected from the shutter exposure time corresponding to the target area; The compensation pixel displacement determination module is used to determine the first inter-frame pixel displacement between adjacent video frames based on the first position, determine the second inter-frame pixel displacement between adjacent video frames based on the pixel feature value using a video matching method, and determine the compensation pixel displacement based on the first inter-frame pixel displacement and the second inter-frame pixel displacement. The second position determination module is used to acquire position data and a second timestamp in a single direction through a second sensor, and determine the second position acquired by the second sensor at the target time based on the position data and the second timestamp. The first stabilization parameter determination module is used to determine the inter-frame motion displacement of each of the adjacent video frames based on the second position, and to obtain the first stabilization parameter based on the inter-frame motion displacement and the compensated pixel displacement. An image processing module is used to determine a second stabilization parameter based on the first stabilization parameter, the inter-frame motion displacement, and the compensated pixel displacement, so as to perform image stabilization processing based on the first stabilization parameter and the second stabilization parameter.

9. The image processing apparatus according to claim 8, characterized in that, The image processing device further includes a parameter optimization unit, which is used for: A loss function is constructed using the first and second stability parameters; Based on the loss function, the first stable parameter and the second stable parameter of the target are obtained iteratively using a global optimization algorithm.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

11. An electronic device, characterized in that, include: processor; Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1-7 by executing the executable instructions.

Images