Anti-shake processing method, device, electronic device and readable storage medium

Abstract

The present application discloses an anti-shake processing method, apparatus, electronic device, and readable storage medium, belonging to the field of photography technology. Among them, the method includes: obtaining motion information at a first moment, where the motion information includes: a first distance between a target object and a lens, a moving direction of the lens, and a moving speed of the lens; determining a moving speed vector of the lens at a second moment according to the moving direction and the moving speed of the lens; determining a moving distance of a user between the first moment and the second moment according to the first distance, where the second moment is a moment later than the first moment; determining first shake information of the lens at the second moment according to the moving speed vector and the moving distance; and determining target shake information according to the first shake information and second shake information detected by a sensor, where the target shake information is used to perform anti-shake processing on the lens at the second moment.

Description

Technical Field

[0001] This application belongs to the field of camera technology, and particularly relates to an anti-shake processing method, apparatus, electronic device, and readable storage medium. Background Art

[0002] With the development of information technology, more and more users use electronic devices to shoot first-person videos. For example, when a user shoots a ball game from a first-person perspective, the user fixes the electronic device on the body for shooting. During the process that the user moves following the movement of the target object, the electronic device worn by the user moves together with the user. This shooting method has high requirements for the anti-shake technology of the electronic device.

[0003] Currently, the anti-shake technologies relied on by electronic devices mainly include: optical anti-shake and electronic anti-shake. During the movement of the user, in order to clearly capture the target object, the focus of the lens often changes following the movement of the target object. In this case, the lens shakes greatly, and electronic anti-shake depends on optical anti-shake to a certain extent. Therefore, currently in a moving scenario, the anti-shake effect of the electronic device during shooting is not good. Summary of the Invention

[0004] The purpose of the embodiments of this application is to provide an anti-shake processing method, apparatus, electronic device, and readable storage medium, which can improve the anti-shake effect.

[0005] In a first aspect, the embodiments of this application provide an anti-shake processing method, which is applied to an electronic device. The electronic device is worn by a user and includes a lens and a sensor. The lens is used to shoot a target object in a moving state, and there is an associated relationship between the movement state of the user and the movement state of the target object. The method includes:

[0006] Obtain the movement information at a first moment. The movement information includes: the first distance between the target object and the lens, the moving direction of the lens, and the moving speed of the lens;

[0007] Determine the moving speed vector of the lens at a second moment according to the moving direction and the moving speed of the lens;

[0008] Determine the moving distance of the user between the first moment and the second moment according to the first distance. The second moment is a moment later than the first moment;

[0009] Determine the first shake information of the lens at the second moment according to the moving speed vector and the moving distance;

[0010] Determine the target shake information according to the first shake information and the second shake information detected by the sensor. The target shake information is used to perform anti-shake processing on the lens at the second moment.

[0011] Second aspect, an embodiment of the present application provides an anti-shake processing device, which is applied to an electronic device. The electronic device is worn by a user and includes a lens and a sensor. The lens is used to photograph a target object in a moving state, and there is an association relationship between the moving state of the user and the moving state of the target object. The device includes:

[0012] An acquisition module, configured to acquire motion information at a first moment. The motion information includes: a first distance between the target object and the lens, a moving direction of the lens, and a moving speed of the lens;

[0013] A first determination module, configured to determine a moving speed vector of the lens at a second moment according to the moving direction and the moving speed of the lens;

[0014] A second determination module, configured to determine a moving distance of the user between the first moment and the second moment according to the first distance. The second moment is a moment later than the first moment;

[0015] A third determination module, configured to determine first shake information of the lens at the second moment according to the moving speed vector and the moving distance;

[0016] A fourth determination module, configured to determine target shake information according to the first shake information and second shake information detected by the sensor. The target shake information is used to perform anti-shake processing on the lens at the second moment.

[0017] Third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory. The memory stores a program or instruction that can run on the processor. When the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented.

[0018] Fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0019] Fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is configured to run a program or instruction to implement the method described in the first aspect.

[0020] Sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0021] In an embodiment of the present application, by obtaining the motion information at the first moment, the motion information includes: the first distance between the target object and the lens, the moving direction of the lens, and the moving speed of the lens. Wherein, the lens is used to photograph the target object in a moving state, and there is an associated relationship between the motion state of the user and the motion state of the target object. According to the moving direction and moving speed of the lens, the moving speed vector of the lens at the second moment can be quickly and accurately determined. According to the first distance, the moving distance of the user between the first moment and the second moment can be effectively determined to maintain the associated relationship with the motion state of the target object; since the jitter of the lens comes from lens focusing on the one hand and user movement on the other hand, by calculating the moving speed vector representing the motion generated by lens focusing and the moving distance representing user movement, the first jitter information of the lens at the second moment can be determined by combining the lens movement situation and the user movement situation. Finally, the first jitter information and the second jitter information detected by the sensors built in the electronic device are fused to determine the target jitter information, which is used to perform anti-shake processing on the lens at the second moment, and the anti-shake effect can be improved, and the shooting experience during movement can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0023] Figure 2 is a flowchart of an anti-shake processing method provided by an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of implementing an anti-shake processing method provided by an embodiment of the present application;

[0025] Figure 4 is a structural diagram of an anti-shake processing device provided by an embodiment of the present application;

[0026] Figure 5 is one of the schematic diagrams of the hardware structure of the electronic device provided by an embodiment of the present application;

[0027] Figure 6 is the second schematic diagram of the hardware structure of the electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] The technical solutions of the embodiments of the present application will be clearly described below with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application fall within the scope of protection of the present application.

[0029] In the description and claims of this application, terms such as "first" and "second" are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of this application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the description and claims means at least one of the connected objects, and the character " / ", generally indicates an "or" relationship between the associated objects before and after.

[0030] The anti-shake processing method provided by the embodiments of this application can be applied to at least the following application scenarios, which will be described below.

[0031] With the rise of video shooting technology, more and more users use electronic devices to shoot first-person videos to record their lives. Among them, the content of short video shooting is diverse, and the lens scene usually changes greatly. Especially when users shoot ball games from the first person perspective, they will fix the electronic device on their bodies for shooting, which poses a severe challenge to the anti-shake technology requirements in the imaging module of the electronic device.

[0032] Among them, the first-person perspective refers to the perspective from which the user observes the target object, which is basically the same as the perspective of the lens shooting the target object, so as to give the viewer a immersive viewing experience when watching the video.

[0033] Such as Figure 1 shown, taking the example of a user wearing an electronic device to shoot a table tennis game from the first-person perspective. The user moves according to the movement of the target object (i.e., the table tennis ball). During the movement of the user, the electronic device worn by the user moves synchronously with the user.

[0034] Among them, the anti-shake technology of the electronic device is crucial for the video shooting effect. In order to better meet the shooting needs of users, each electronic device manufacturer is trying to optimize the anti-shake technology in the camera. The current anti-shake technologies in the electronic device cameras mainly include: optical image stabilization and electronic image stabilization. The optical image stabilization technology measures and calculates the jitter through an inertial measurement sensor, and then controls the movement of the voice coil motor to change the position of the optical element, so that the front and rear video frames are kept stable. Electronic image stabilization is to calculate the camera pose or perform frame feature point matching of the two consecutive video frames, so as to crop the edges of the video frame and maintain the stability of the central area.

[0035] That is to say, optical image stabilization stabilizes the image by adjusting optical elements, while electronic image stabilization stabilizes the image by transforming the image and cropping the edge part. On the one hand, during the movement of the user, in order to clearly capture the target, the focus of the lens optical focusing often changes following the movement of the target object. In this case, the lens shakes greatly, and relying solely on the shake detected by the sensor and then making adjustments is relatively slow, and the timeliness of image stabilization is not good. On the other hand, electronic image stabilization depends to a certain extent on optical image stabilization. In a moving scenario, if the optical focusing is inaccurate and the difference between two consecutive video frames is large, it is easy for the cropped edge to exceed the threshold, making it difficult to achieve the stability of two consecutive video frames.

[0036] Therefore, currently in a moving scenario, the anti-shake effect of electronic devices during shooting is not good.

[0037] In view of the problems in the related art, the embodiments of the present application provide an anti-shake processing method, device, electronic device and storage medium, which can solve the problem of poor anti-shake effect during shooting in the related art.

[0038] The following will combine the accompanying drawings to elaborate in detail on the anti-shake processing method provided by the embodiments of the present application through specific embodiments and their application scenarios.

[0039] Figure 2 It is a flowchart of an anti-shake processing method provided by the embodiments of the present application.

[0040] As Figure 2 shown, the anti-shake processing method may include step 210-step 250. This method is applied to an anti-shake processing device and is specifically as follows:

[0041] Step 210, obtain the motion information at the first moment. The motion information includes: the first distance between the target object and the lens, the moving direction of the lens, and the moving speed of the lens. Among them, this method can be applied to an electronic device. The electronic device is worn by the user and includes a lens and a sensor. The lens is used to photograph the target object in a moving state, and the motion state of the user is related to the motion state of the target object.

[0042] Among them, the motion state of the user is related to the motion state of the target object, which can mean that when the user observes the motion of the target object, the user makes a reaction to the motion of the target object and then moves. For example, if the target object is a table tennis ball, during the process of the user playing table tennis, the motion state of the user is related to the motion state of the table tennis ball.

[0043] In a possible embodiment, in step 210, it may specifically include the following steps:

[0044] At the first moment, acquire a first image based on the lens.

[0045] Obtain the first position information of the target object in the first image and the second position information of the target reference point in the first image, where the target reference point is the center of attention of the lens during the shooting process;

[0046] Calculate the moving direction of the lens according to the first position information and the second position information.

[0047] First, obtain the first position information of the target object in the first image, which may specifically include: based on the Hough transform, obtain the first position information of the target object in the first image. Among them, the Hough transform is a feature extraction algorithm. The Hough transform is used to identify and find the features in an object. The algorithm process of the Hough transform generally includes: given an object and the type of shape to be identified, perform voting in the parameter space to determine the shape of the object, and this is determined by the local maximum value in the accumulation space.

[0048] Secondly, since during the visual observation process of a person, the center of visual attention will always be concentrated on the center of the visible range, when using an electronic device to shoot from the first perspective, the target object, as the main focus, will always remain at the center of attention, that is, the center of attention of the shooting preview screen.

[0049] By calculating the first position information and the second position information, the difference between the sphere in the image at time T and the center of attention can be obtained According to the center of attention hypothesis, at time T+1, the position of the sphere in the image should tend to the center of attention of the image, so The moving direction of the camera lens can be accurately predicted, where T is a positive integer, the first moment can be T, and the second moment can be T+1.

[0050] Here, by calculating the first position information of the target object at the first moment and the second position information of the target reference point at the first moment, the moving direction corresponding to the movement of the lens for focusing on the target object can be effectively predicted.

[0051] Among them, in the step of calculating the moving direction of the lens according to the first position information and the second position information, it may specifically include the following steps:

[0052] Determine the third position information of the target reference point at the second moment according to the first position information and the second position information;

[0053] Calculate the moving direction of the lens according to the first position information and the third position information.

[0054] The center of attention changes with the movement of the target object. By taking a picture, the first position information of the target object in the first image captured at the first moment can be obtained. Then, by calculating with the second position information of the target reference point in the first image at the first moment, the third position information of the target reference point at the second moment can be predicted, that is, the center of attention for the camera shot at the next moment. The third position information of the target reference point at the second moment can be calculated through the following formula:

[0055] F T+1 (x,y) = (I T (x,y) + F T (x,y)) / 2 (1)

[0056] Where, I T is the first position information, that is, the coordinates of the target object in the first image captured at the first moment at time T;

[0057] F T (x, y) is the second position information of the target reference point at the first moment, that is, the set center of attention at time T; F T is a continuously iterative quantity, which will be continuously adjusted according to the actual center of the sphere image at times T and T+1. In the initial stage, F 0 = (M / 2, N / 2), where M and N are the width and height of the resolution of the captured preview image respectively.

[0058] F T+1 (x, y) is the third position information of the target reference point at the second moment, that is, the set center of attention at time T+1;

[0059] Usually, at the next moment, that is, at the second moment, the camera lens moves towards the predicted center of attention (F T+1 (x, y)), so the movement direction of the camera lens can be obtained through calculation. The movement direction of the lens can specifically be calculated through the following formula:

[0060]

[0061] In a possible embodiment, in step 210, it may specifically include the following steps:

[0062] Obtain two video frames captured by the lens. The two video frames include the video frame at the first moment and the video frame at the third moment, and the third moment is a moment earlier than the first moment;

[0063] Determine the movement speed of the lens according to the two video frames.

[0064] After determining the moving direction of the calculation lens, it is also necessary to determine the intensity of the jitter, that is, the moving speed of the lens. Since the lens moves following the movement of the target object, it may be necessary to determine the moving speed of the target object.

[0065] In the step of calculating the moving speed of the target object according to two video frames, it may specifically include: calculating the moving speed of the target object according to the optical flow information of the two video frames; determining the moving speed of the target object as the moving speed of the lens.

[0066] Based on the Lucas-Kanade optical flow estimation method, the first region U covered by the target object in the video frame (T frame) at the first moment can be obtained T (i, j) and the second region covered by the target object in the video frame (T - 1 frame) at the third moment. The optical flow records the magnitude of the displacement of the target object per unit time, and the moving speed of the target object can be easily calculated through simple calculation.

[0067]

[0068] Among them, N is the total number of pixel points in the covered region U T (i, j); P T is the optical flow information of the T frame and the T - 1 frame; is the time between two adjacent frames captured by the camera, and i and j are the horizontal and vertical coordinates of the pixel respectively.

[0069] Theoretically, for any pixel point in the covered region of the target object in the video frame, calculating the speed can obtain the moving speed of the target object, and the results calculated for all pixel points in the covered region of the target object should be the same. Here, in order to reduce errors and randomness, all pixel points in the covered region of the target object are calculated, and finally the average value of the moving speeds of all pixel points is taken as the moving speed of the target object.

[0070] In a possible embodiment, in step 210, it may specifically include the following steps:

[0071] Obtain the shooting information at the first moment, and the shooting information includes: the size information of the target object in the first image, the shooting focal length and the depth image corresponding to the first moment; the first image is the image collected by the lens at the first moment;

[0072] According to the shooting focal length and the size information, calculate the third distance between the target object and the lens;

[0073] According to the depth image, calculate the fourth distance between the target object and the lens;

[0074] According to the third distance and the fourth distance, obtain the first distance.

[0075] The distance between the target object and the lens is also a factor for predicting lens shake. To accurately estimate the distance between the target object and the lens, first, the third distance between the target object and the lens can be calculated based on the shooting focal length and size information corresponding to the first moment. Second, the dual-camera technology can also be used to calculate the depth image corresponding to the first moment to calculate the fourth distance between the target object and the lens.

[0076] Calculate the fourth distance between the target object and the lens according to the depth image corresponding to the first moment, which can be specifically implemented by the following formula:

[0077]

[0078] N DT To calculate the fourth distance between the target object and the lens according to the depth image. Where F DT is the calculated depth image, and U T (i, j) is the sphere coverage area, and N is the total number of pixels in the sphere coverage area.

[0079] To avoid contingency, the depth image of the target object coverage area can be averaged as the depth distance. Finally, the average of the third distance calculated based on optical focusing and the distance calculated based on the image can be taken as the final first distance between the target object and the lens.

[0080] D CT =(N DT +N LT ) / 2 (5)

[0081] Where, N LT is the third distance between the target object and the lens calculated according to the shooting focal length and size information; N DT is the fourth distance between the target object and the lens calculated according to the depth image; D CT is the first distance between the target object and the lens.

[0082] Step 220, predict the moving speed vector of the lens at the second moment according to the moving direction of the lens and the moving speed of the lens.

[0083] The first direction and the second direction of the moving direction of the lens are respectively multiplied by the moving speed of the lens. The moving direction of the lens only contains direction information, which is equivalent to the moving speed of the lens endowing the moving direction with magnitude information. It is also equivalent to the moving direction of the lens being the weight of the moving speed of the lens.

[0084] Among them, the first direction and the second direction can be the x direction and the y direction. According to the moving direction and moving speed of the camera, predict the moving speed vector of the camera at the second moment, which specifically can include: decompose the moving speed of the camera in the x direction and the y direction to obtain the moving speed component in the x direction and the moving speed component in the y direction, and then weight the moving speed component in the x direction and the moving speed component in the y direction according to the moving direction of the camera to obtain the moving speed vector of the camera at the second moment.

[0085] According to the moving direction and moving speed of the camera, predict the moving speed vector of the camera at the second moment, which can be calculated by the following method:

[0086]

[0087] Among them, is the moving direction of the camera; is the moving speed of the camera; is the moving speed vector of the camera at the second moment; a and b are real numbers greater than 0, which can be set by experience. Here, a = b = 0.5 and a + b = 1 can be taken.

[0088] Step 230, determine the moving distance of the user between the first moment and the second moment according to the first distance, where the second moment is a moment later than the first moment.

[0089] In actual shooting of a motion scene, in addition to rotating the camera to track the motion trajectory of the target object, in order to accurately hit the target object, the user himself will also move. Therefore, it is also necessary to predict the moving distance of the user between the first moment and the second moment to predict the moving direction of the user.

[0090] In the case where no reference object is detected, the first distance between the target object and the camera can be determined as the moving distance of the user between the first moment and the second moment;

[0091] In the case where a reference object is detected, in one possible embodiment, in step 230, it specifically can include the following steps:

[0092] Obtain the fourth position information of the reference object in the first image;

[0093] Determine the second distance between the target object and the reference object according to the first position information and the fourth position information;

[0094] Determine the moving distance of the user between the first moment and the second moment according to the first distance and the second distance.

[0095] Taking the example of a user playing table tennis, during the movement, due to the movement of the camera, the plane of the table tennis table is always changing. To accurately calculate the position of the ball relative to the net, the calculation should be performed in the same plane reference system each time. To achieve the calculation in the same coordinate system, the plane of the table tennis table can be selected as the only reference system. Since the net and the white center line on the table are perpendicular to each other and are important reference objects during the movement, a spatial rectangular coordinate system can be established with the white center line of the table as the x-axis and the net as the y-axis. That is, the reference object is the net.

[0096] When the reference object is the net, that is, when the plane where the net is located is the preset reference plane, when calculating the second distance between the target object and the target reference plane, first, the coordinate system transformation of the first position information is required to convert the first position information of the target object in the first image into the first position information under the target reference plane; then, based on the first position information and the fourth position information under the target reference plane, the second distance D between the target object and the reference object is determined. NT 。

[0097] Among them, for the coordinate system transformation of the first position information, the coordinate system in the first image needs to be converted into the spatial rectangular coordinate system established with the white center line of the table as the x-axis and the net as the y-axis.

[0098] Based on the first position information and the fourth position information, to determine the second distance between the target object and the target reference plane, it can be specifically achieved through the following formula:

[0099] D NT =|f b (i,j)-f n (i,j)| (7)

[0100] Among them, f n and f b are respectively the first position information after plane transformation and the fourth position information of the target reference plane.

[0101] Based on the first distance D CT and the second distance D NT , predict the moving distance of the user between the first moment and the second moment Specifically, it can be calculated through the following method:

[0102]

[0103] Among them, is a scalar, where c and d are real numbers greater than 0, which can be set by experience. Here, c = d = 0.5, and c + d = 1.

[0104] Used to characterize the distance result obtained by weighted fusion calculation of the first distance and the second distance, and then predict the distance that the user needs to reach within a preset time period, so as to obtain the magnitude of the user's movement at the next moment, that is, the moving distance of the user between the first moment and the second moment.

[0105] Step 240, predict the first jitter information of the lens at the second moment according to the moving speed vector and the moving distance.

[0106] After calculating the moving speed vector and the moving distance of the user the total jitter magnitude of the lens at the next moment can be calculated in the following way

[0107]

[0108] wherein, is the first jitter information of the lens at the second moment, e and f are real numbers greater than 0, which can be set by experience. Here, e = f = 0.5 and e + f = 1.

[0109] Step 250, determine the target jitter information according to the first jitter information and the second jitter information detected by the sensor, and the target jitter information is used to perform anti-shake processing on the lens at the second moment.

[0110] In a possible embodiment, determining the target jitter information according to the first jitter information and the second jitter information detected by the sensor includes:

[0111] Determine the first weighted jitter information according to the first weight and the first jitter information;

[0112] Determine the second weighted jitter information according to the second weight and the second jitter information, and the first weight is less than the second weight;

[0113] Determine the target jitter information according to the first weighted jitter information and the second weighted jitter information.

[0114] Determine the target jitter information according to the first weighted jitter information and the second weighted jitter information, and specifically can be calculated in the following way:

[0115]

[0116] wherein, is the calculated first jitter information, is the second jitter information detected by the sensor; is the target jitter information. The first weight is j and the second weight is k. Here, the acting on the control algorithm is based on the measured by the sensorMainly, the predicted direction As auxiliary, the values ​​of j and k here are j less than k, which are 0.2 and 0.8 respectively, that is, the first weight is less than the second weight.

[0117] The first jitter information and the second jitter information are weightedly fused based on the first weight and the second weight to determine the target jitter information. Finally, the fused target jitter information is used to control the voice coil motor of the camera lens, thereby obtaining a better anti-shake effect.

[0118] Generally, the general optical image stabilization technology of a camera relies on the sensor in the camera to detect the size of the lens shake in the x and y directions, and then uses the measurement results as the input of the control algorithm. The algorithm finally controls the movement of the lens in the image module after calculation to achieve stable shooting. In the embodiment of the present application, the camera shake is predicted by predicting the movement of the shooting lens and the movement of the user, and the first shake information is obtained. At the same time, it is combined with the second shake information detected by the sensor of the electronic device to determine the target shake information for anti-shake processing of the lens at the second moment, which can bring better anti-shake effect to the user when shooting and recording at the moment of motion, and greatly improve the user experience.

[0119] It should be noted that, in addition to the table tennis sport described above, the embodiments of the present application can also be applied to badminton, football, and volleyball sports.

[0120] In an embodiment of the present application, by acquiring motion information at the first moment, the motion information includes: a first distance between the target object and the lens, a moving direction of the lens, and a moving speed of the lens. Among them, the lens is used to shoot a target object in motion, and the motion state of the user is associated with the motion state of the target object. According to the moving direction of the lens and the moving speed of the lens, the moving speed vector of the lens at the second moment can be quickly and accurately determined. According to the first distance, the moving distance of the user between the first moment and the second moment in order to maintain the associated relationship with the motion state of the target object can be effectively determined; because the jitter of the lens comes from the lens focus on the one hand and the user movement on the other hand, by calculating the moving speed vector used to characterize the motion generated by the lens focus and the moving distance used to characterize the user movement, the first jitter information of the lens at the second moment can be determined in combination with the lens movement and the user movement. Finally, the first jitter information and the second jitter information detected by the sensor of the electronic device are fused to determine the target jitter information, so as to be used for anti-shake processing of the lens at the second moment through the target jitter information, which can improve the anti-shake effect and improve the shooting experience during movement.

[0121] Based on the anti-shake processing method provided in the embodiment of the present application, Figure 3 To explain:

[0122] First, during the process of the user moving following the movement of the target object, obtain the first position information of the target object at the first moment and the second position information of the target reference point at the first moment, where the target reference point is the center of attention of the lens during the shooting process; obtain two video frames captured by the lens, and the two video frames include the video frame at the first moment and the video frame at the third moment, and the third moment is a moment earlier than the first moment by a preset value.

[0123] For lens movement: First, calculate the moving direction of the lens according to the first position information and the second position information. Then, calculate the moving speed of the target object according to the video frame at the third moment and the video frame at the first moment; determine the moving speed of the target object as the moving speed of the lens. Next, predict the moving speed vector of the lens at the second moment according to the moving direction of the lens and the moving speed of the lens. Finally, predict the moving distance of the user between the first moment and the second moment according to the first distance, where the second moment is a moment later than the first moment by a preset value.

[0124] For user movement: First, obtain the first distance between the target object and the lens; determine the second distance between the target object and the reference object according to the first position information and the fourth position information. Then, predict the moving distance of the user between the first moment and the second moment according to the first distance and the second distance.

[0125] Finally, for anti-shake processing: Predict the first shake information of the lens at the second moment according to the moving speed vector and the moving distance; determine the target shake information according to the first shake information and the second shake information detected by the sensor, and the target shake information is used to be output to the controller for the controller to perform anti-shake processing on the lens at the second moment.

[0126] In the anti-shake processing method provided by the embodiment of the present application, the execution subject can be an anti-shake processing device. In the embodiment of the present application, taking the anti-shake processing device executing the anti-shake processing method as an example, the anti-shake processing device provided by the embodiment of the present application is described.

[0127] Figure 4 It is a block diagram of an anti-shake processing device provided by the embodiment of the present application. The device 400 is applied to an electronic device, the electronic device is worn on the user, the electronic device includes a lens and a sensor, the lens is used to shoot a target object in a moving state, and there is an associated relationship between the moving state of the user and the moving state of the target object. The device 400 includes:

[0128] An acquisition module 410, configured to acquire the movement information at the first moment, where the movement information includes: the first distance between the target object and the lens, the moving direction of the lens, and the moving speed of the lens.

[0129] The first determination module 420 is configured to determine the moving speed vector of the lens at the second moment according to the moving direction and moving speed of the lens.

[0130] The second determination module 430 is configured to determine the moving distance of the user between the first moment and the second moment according to the first distance, where the second moment is a moment later than the first moment.

[0131] The third determination module 440 is configured to determine the first jitter information of the lens at the second moment according to the moving speed vector and the moving distance.

[0132] The fourth determination module 450 is configured to determine the target jitter information according to the first jitter information and the second jitter information detected by the sensor, and the target jitter information is used to perform anti-shake processing on the lens at the second moment.

[0133] In a possible embodiment, the acquisition module 410 includes:

[0134] The acquisition module is configured to acquire a first image based on the lens at the first moment;

[0135] The first acquisition module is configured to acquire the first position information of the target object in the first image and the second position information of the target reference point in the first image, where the target reference point is the center of attention of the lens during shooting;

[0136] The calculation module is configured to calculate the moving direction of the lens according to the first position information and the second position information.

[0137] In a possible embodiment, the calculation module is specifically configured to:

[0138] Determine the third position information of the target reference point at the second moment according to the first position information and the second position information;

[0139] Calculate the moving direction of the lens according to the first position information and the third position information.

[0140] In a possible embodiment, the acquisition module 410 is specifically configured to:

[0141] Acquire two video frames captured by the lens, where the two video frames include the video frame at the first moment and the video frame at the third moment, and the third moment is a moment earlier than the first moment;

[0142] Determine the moving speed of the lens according to the two video frames.

[0143] In a possible embodiment, the second determination module 430 is specifically configured to:

[0144] Acquire the fourth position information of the reference object in the first image;

[0145] Determine a second distance between the target object and the reference object according to the first position information and the fourth position information;

[0146] Determine the moving distance of the user between the first moment and the second moment according to the first distance and the second distance.

[0147] In a possible embodiment, the obtaining module 410 includes:

[0148] Obtain the shooting information at the first moment, where the shooting information includes: the size information of the target object in the first image, the shooting focal length corresponding to the first moment, and the depth image; the first image is an image collected by the lens at the first moment;

[0149] Calculate a third distance between the target object and the lens according to the shooting focal length and the size information;

[0150] Calculate a fourth distance between the target object and the lens according to the depth image;

[0151] Obtain the first distance according to the third distance and the fourth distance.

[0152] In a possible embodiment, the fourth determination module 450 is specifically configured to:

[0153] Determine the first weighted jitter information according to the first weight and the first jitter information;

[0154] Determine the second weighted jitter information according to the second weight and the second jitter information, where the first weight is less than the second weight;

[0155] Determine the target jitter information according to the first weighted jitter information and the second weighted jitter information.

[0156] In an embodiment of the present application, by acquiring motion information at the first moment, the motion information includes: a first distance between the target object and the lens, a moving direction of the lens, and a moving speed of the lens. Among them, the lens is used to shoot a target object in motion, and the motion state of the user is associated with the motion state of the target object. According to the moving direction of the lens and the moving speed of the lens, the moving speed vector of the lens at the second moment can be quickly and accurately determined. According to the first distance, the moving distance of the user between the first moment and the second moment in order to maintain the associated relationship with the motion state of the target object can be effectively determined; because the jitter of the lens comes from the lens focus on the one hand and the user movement on the other hand, by calculating the moving speed vector used to characterize the motion generated by the lens focus and the moving distance used to characterize the user movement, the first jitter information of the lens at the second moment can be determined in combination with the lens movement and the user movement. Finally, the first jitter information and the second jitter information detected by the sensor of the electronic device are fused to determine the target jitter information, so as to be used for anti-shake processing of the lens at the second moment through the target jitter information, which can improve the anti-shake effect and improve the shooting experience during movement.

[0157] The anti-shake processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0158] The anti-shake processing device of the embodiment of the present application may be a device having an action system. The action system may be an Android action system, an iOS action system, or other possible action systems, which are not specifically limited in the embodiment of the present application.

[0159] The anti-shake processing device provided in the embodiments of the present application can implement each process implemented in the above method embodiments. To avoid repetition, it will not be described in detail here.

[0160] Optionally, as Figure 5 shown, the embodiments of the present application further provide an electronic device 510, including a processor 511, a memory 512, a program or instruction stored on the memory 512 and executable on the processor 511. When the program or instruction is executed by the processor 511, it implements each step of any of the above anti-shake processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described in detail here.

[0161] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0162] Figure 6 FIG. is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application.

[0163] The electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, and a processor 610, etc.

[0164] Those skilled in the art can understand that the electronic device 600 may further include a power source (such as a battery) for supplying power to each component. The power source can be logically connected to the processor 610 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. Figure 6 The structure of the electronic device shown in does not limit the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be described in detail here.

[0165] Among them, the processor 610 is used to obtain the motion information at the first moment. The motion information includes: the first distance between the target object and the lens, the moving direction of the lens, and the moving speed of the lens.

[0166] The processor 610 is used to determine the moving speed vector of the lens at the second moment according to the moving direction and moving speed of the lens.

[0167] The processor 610 is further used to determine the moving distance of the user between the first moment and the second moment according to the first distance, where the second moment is a moment later than the first moment.

[0168] The processor 610 is further used to determine the first jitter information of the lens at the second moment according to the moving speed vector and the moving distance.

[0169] The processor 610 is further configured to determine target jitter information according to the first jitter information and the second jitter information detected by the sensor, where the target jitter information is used to perform anti-shake processing on the lens at the second moment.

[0170] Optionally, the processor 610 is configured to acquire a first image based on the lens at the first moment.

[0171] The processor 610 is configured to obtain first position information of the target object in the first image and second position information of the target reference point in the first image, where the target reference point is the center of attention of the lens during shooting.

[0172] The processor 610 is further configured to calculate the moving direction of the lens according to the first position information and the second position information.

[0173] Optionally, the processor 610 is further configured to determine third position information of the target reference point at the second moment according to the first position information and the second position information;

[0174] The processor 610 is further configured to calculate the moving direction of the lens according to the first position information and the third position information.

[0175] Optionally, the processor 610 is further configured to acquire two video frames collected by the lens, where the two video frames include the video frame at the first moment and the video frame at the third moment, and the third moment is a moment earlier than the first moment;

[0176] The processor 610 is further configured to determine the moving speed of the lens according to the two video frames.

[0177] Optionally, the processor 610 is further configured to obtain fourth position information of the reference object in the first image;

[0178] The processor 610 is further configured to determine a second distance between the target object and the reference object according to the first position information and the fourth position information;

[0179] The processor 610 is further configured to determine the moving distance of the user between the first moment and the second moment according to the first distance and the second distance.

[0180] Optionally, the processor 610 is further configured to obtain shooting information at the first moment, where the shooting information includes: size information of the target object in the first image, the shooting focal length corresponding to the first moment, and a depth image; the first image is an image collected by the lens at the first moment;

[0181] The processor 610 is further configured to calculate a third distance between the target object and the lens according to the shooting focal length and the size information;

[0182] The processor 610 is further configured to calculate a fourth distance between the target object and the lens according to the depth image;

[0183] The processor 610 is further configured to obtain the first distance according to the third distance and the fourth distance.

[0184] In a possible embodiment, the processor 610 is further configured to determine first weighted jitter information according to the first weight and the first jitter information;

[0185] The processor 610 is further configured to determine second weighted jitter information according to the second weight and the second jitter information, wherein the first weight is less than the second weight;

[0186] The processor 610 is further configured to determine target jitter information according to the first weighted jitter information and the second weighted jitter information.

[0187] In an embodiment of the present application, by acquiring motion information at the first moment, the motion information includes: a first distance between the target object and the lens, a moving direction of the lens, and a moving speed of the lens. Among them, the lens is used to shoot a target object in motion, and the motion state of the user is associated with the motion state of the target object. According to the moving direction of the lens and the moving speed of the lens, the moving speed vector of the lens at the second moment can be quickly and accurately determined. According to the first distance, the moving distance of the user between the first moment and the second moment in order to maintain the associated relationship with the motion state of the target object can be effectively determined; because the jitter of the lens comes from the lens focus on the one hand and the user movement on the other hand, by calculating the moving speed vector used to characterize the motion generated by the lens focus and the moving distance used to characterize the user movement, the first jitter information of the lens at the second moment can be determined in combination with the lens movement and the user movement. Finally, the first jitter information and the second jitter information detected by the sensor of the electronic device are fused to determine the target jitter information, so as to be used for anti-shake processing of the lens at the second moment through the target jitter information, which can improve the anti-shake effect and improve the shooting experience during movement.

[0188] It should be understood that in the embodiments of the present application, the input unit 604 may include a Graphics Processing Unit (GPU) 6041 and a microphone 6042. The GPU 6041 processes the image data of static pictures or video images obtained by an image capturing device (such as a camera) in the video image capturing mode or the image capturing mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 607 includes at least one of a touch panel 6071 and other input devices 6072. The touch panel 6071 is also referred to as a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. The other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an action bar, which will not be elaborated here. The memory 609 may be used to store software programs and various data, including but not limited to application programs and operating systems. The processor 610 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interfaces, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 610.

[0189] The memory 609 can be used to store software programs and various data. The memory 609 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data. Among them, the first storage area may store an operating system, application programs or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 609 may include a volatile memory or a non-volatile memory, or the memory 609 may include both a volatile and a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synch link dynamic random access memory (SLDRAM), and a direct rambus random access memory (DRRAM). The memory 609 in the embodiments of the present application includes but is not limited to these and any other suitable types of memories.

[0190] The processor 610 may include one or more processing units; optionally, the processor 610 integrates an application processor and a modem processor. Among them, the application processor mainly processes operations related to the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above modem processor may not be integrated into the processor 610 either.

[0191] The embodiments of the present application also provide a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements each process of the above anti-shake processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0192] Among them, the processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media such as computer read-only memory ROM, random access memory RAM, magnetic disks, or optical discs.

[0193] Another embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement each process of the above anti-shake processing method embodiment, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0194] It should be understood that the chip mentioned in the embodiments of the present application can also be referred to as a system-on-chip, system chip, chip system, or system-on-chip.

[0195] The embodiments of the present application provide a computer program product. The program product is stored in a storage medium and is executed by at least one processor to implement each process of the above anti-shake processing method embodiment, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0196] It should be noted that in this article, the term "including", "comprising", or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article, or device including that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed. It may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described method may be executed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, the features described with reference to certain examples may be combined in other examples.

[0197] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present application.

[0198] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific implementation manners. The above specific implementation manners are merely illustrative rather than restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them belong to the protection scope of the present application.

Claims

1. An anti-shake processing method, characterized in that, applied to an electronic device, the electronic device is worn by a user, the electronic device includes a lens and a sensor, the lens is used to photograph a target object in a moving state, and there is an associated relationship between the moving state of the user and the moving state of the target object. The method includes: Obtaining motion information at a first moment, the motion information including: a first distance between the target object and the lens, a moving direction of the lens, and a moving speed of the lens; Determining a moving speed vector of the lens at a second moment according to the moving direction and the moving speed of the lens; Determining a moving distance of the user between the first moment and the second moment according to the first distance, where the second moment is a moment later than the first moment; Determining first jitter information of the lens at the second moment according to the moving speed vector and the moving distance; Determining target jitter information according to the first jitter information and second jitter information detected by the sensor, where the target jitter information is used to perform anti-shake processing on the lens at the second moment; Among them, determining the first jitter information of the lens at the second moment according to the moving speed vector and the moving distance includes: Calculate the first jitter information of the lens at the second moment according to the moving speed vector and the moving distance: Wherein, is the first jitter information of the lens at the second moment, is the moving speed vector, is the moving distance, and e and f are real numbers greater than 0.

2. The method according to claim 1, characterized in that, the obtaining of the motion information at the first moment includes: At the first moment, acquiring a first image based on the lens; Obtaining first position information of the target object in the first image and second position information of a target reference point in the first image, where the target reference point is the center of attention of the lens during the shooting process; Calculating the moving direction of the lens according to the first position information and the second position information.

3. The method according to claim 2, characterized in that, the calculating of the moving direction of the lens according to the first position information and the second position information includes: Determining third position information of the target reference point at a second moment according to the first position information and the second position information; Calculating the moving direction of the lens according to the first position information and the third position information.

4. The method according to claim 1, characterized in that, the obtaining of the motion information at the first moment includes: Obtaining two video frames collected by the lens, the two video frames including the video frame at the first moment and the video frame at a third moment, where the third moment is a moment earlier than the first moment; Determining the moving speed of the lens according to the two video frames.

5. The method according to claim 2, characterized in that, the determining of the moving distance of the user between the first moment and the second moment according to the first distance includes: Obtaining fourth position information of a reference object in the first image; Determining a second distance between the target object and the reference object according to the first position information and the fourth position information; Determine the moving distance of the user between the first moment and the second moment according to the first distance and the second distance.

6. The method according to claim 1, wherein, the obtaining of the motion information at the first moment includes: obtaining the shooting information at the first moment, where the shooting information includes: the size information of the target object in the first image, the shooting focal length corresponding to the first moment, and the depth image; the first image is an image collected by the lens at the first moment; calculate a third distance between the target object and the lens according to the shooting focal length and the size information; calculate a fourth distance between the target object and the lens according to the depth image; obtain the first distance according to the third distance and the fourth distance.

7. The method according to claim 1, wherein, the determining of the target jitter information according to the first jitter information and the second jitter information detected by the sensor includes: determine the first weighted jitter information according to the first weight and the first jitter information; determine the second weighted jitter information according to the second weight and the second jitter information, where the first weight is less than the second weight; determine the target jitter information according to the first weighted jitter information and the second weighted jitter information.

8. An anti-shake processing device, wherein, applied to an electronic device, the electronic device is worn on the user, the electronic device includes a lens and a sensor, the lens is used to photograph a target object in a moving state, and there is an associated relationship between the motion state of the user and the motion state of the target object, and the device includes: an obtaining module, configured to obtain motion information at the first moment, where the motion information includes: a first distance between the target object and the lens, a moving direction of the lens, and a moving speed of the lens; a first determining module, configured to determine a moving speed vector of the lens at the second moment according to the moving direction and the moving speed of the lens; a second determining module, configured to determine the moving distance of the user between the first moment and the second moment according to the first distance, where the second moment is a moment later than the first moment; a third determining module, configured to determine the first jitter information of the lens at the second moment according to the moving speed vector and the moving distance; a fourth determining module, configured to determine target jitter information according to the first jitter information and the second jitter information detected by the sensor, where the target jitter information is used to perform anti-shake processing on the lens at the second moment; wherein, determining the first jitter information of the lens at the second moment according to the moving speed vector and the moving distance includes: Calculate the first jitter information of the lens at the second moment according to the moving speed vector and the moving distance: Wherein, is the first jitter information of the lens at the second moment, is the moving speed vector, is the moving distance, and e and f are real numbers greater than 0.

9. An electronic device, wherein, includes a processor and a memory, the memory stores a program or instruction that can run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium, wherein, The program or instructions are stored on the readable storage medium, and when the program or instructions are executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images