Shooting method and electronic equipment

Abstract

The invention provides a photographing method and electronic equipment, and relates to the technical field of photographing. The electronic device collects images in a multi-frame shooting state. In the exposure period of the t-th frame image, if the electronic equipment shakes, the electronic equipment can reversely move a lens in the camera to realize intra-frame shake compensation. Afterwards, in response to the end of exposure of the t-th frame image, the electronic equipment can judge whether the position of the lens is within a first preset position range when the end of exposure of the t-th frame image, so as to judge whether the movement degree of the lens during the exposure of the t-th frame image is large or not. If the image is not in the first preset position range, it is indicated that the movement degree of the lens during exposure of the t-th frame of image is large, the electronic equipment can control the lens to move from the first position to the center point at the interval time between the t-th frame of image and the (t + 1)-th frame of image, and it is guaranteed that the lens has a large movement range during exposure of the (t + 1)-th frame of image; and the lens can be prevented from being controlled to move unnecessarily.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of photographing technology, and more particularly to a photographing method and electronic equipment. Background Art

[0002] With the development of technology, the photography function of electronic devices (such as mobile phones) has also developed rapidly. When using a mobile phone to shoot a video, the mobile phone (or described as a mobile phone camera) may shake, which may cause blurred images and result in low quality of the video.

[0003] At present, in order to improve video quality, mobile phones can use optical image stabilization (OIS) to improve video quality. During the image exposure process in the video, when the mobile phone camera shakes, the mobile phone can control the camera lens to move in the opposite direction to compensate for the shake of the mobile phone camera and achieve de-shaking.

[0004] However, due to the limited range of lens movement, when the phone shakes, the phone may not be able to compensate for the shake, resulting in blurred images, resulting in lower quality video. Summary of the invention

[0005] The embodiments of the present application provide a shooting method and an electronic device for improving the quality of shot videos.

[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, a shooting method is provided. When an electronic device is in a multi-frame shooting state, during exposure of the t-th frame image, if the electronic device shakes, in response to the shaking of the electronic device, the electronic device can move a lens in a camera of the electronic device to a first position along a first direction, wherein the multi-frame shooting state indicates a state of continuously acquiring and processing multiple frames of images, the first direction is a direction opposite to the shaking, and t is a positive integer.

[0008] Afterwards, in response to the end of exposure of the t-th frame image, the electronic device can determine whether the position of the lens when the t-th frame image is exposed is within the first preset position range. In the case that the position of the lens when the t-th frame image is exposed is not within the first preset position range, it indicates that the lens has moved to a large extent during the exposure of the t-th frame image, and the electronic device controls the lens to move from the first position to the center point within the first target time, wherein the first target time is the interval time between the t-th frame image and the t+1-th frame image, the center point is the center point of the moving range of the lens, the absolute value of the coordinate of the position point in the x direction in the first preset position range is less than the first preset coordinate value, and the absolute value of the coordinate of the position point in the y direction is less than the second preset coordinate value; the center point is within the first preset position range, that is, the first preset position range is within the moving range of the lens.

[0009] The electronic device displays the tth frame image.

[0010] In the present application, during the exposure of the t-th frame image, if the electronic device shakes, the electronic device moves the lens in the electronic device in the opposite direction to achieve intra-frame jitter compensation, so as to avoid the electronic device shaking causing the t-th frame image to be blurred. Afterwards, after the exposure of the t-th frame image is completed, it is determined whether the current position of the lens, that is, the position of the lens at the end of the exposure of the t-th frame image, is within the first preset position range, that is, whether the movement degree of the lens during the exposure of the t-th frame image is large, in other words, whether it is necessary to return to the center. If it is not within the first preset position range, it indicates that the movement degree of the lens is large, and the activity range of the lens in the next frame is small. Therefore, in order to ensure the activity range of the lens in the next frame, the electronic device can control the lens to perform the return action during the interval between the t-th frame image and the next frame image. Since the lens frame is returned to the center, it is ensured that the lens has a large activity range when the next frame image is exposed, so as to avoid the problem that the next frame image is blurred due to the inability to compensate for the jitter by moving the lens due to the limited activity range of the lens, and avoid the image (such as the image in the video) from shaking. When the multi-frame shooting state is a video state, the quality of the shot video can be improved. And by judging whether it is necessary to perform the return to center, it is possible to avoid controlling the lens to perform unnecessary return to center actions, thereby reducing waste of resources. In a possible implementation of the first aspect, if the position of the lens at the end of the exposure of the t-th frame image is within the first preset position range, it indicates that the movement degree of the lens during the exposure of the t-th frame image is small, that is, it indicates that the degree of jitter compensation of the lens during the exposure of the t-th frame image is small, and the range of movement of the lens in the next frame is still large. Therefore, the electronic device may not control the lens to move from the first position to the center point within the first target time, that is, it does not control the lens to perform the return to center action, thereby avoiding controlling the lens to perform unnecessary return to center actions, thereby reducing waste of resources.

[0011] In a possible implementation manner of the first aspect, the electronic device may control the lens to move from the first position to the center point within the frame interval of the first target time to achieve lens re-centering.

[0012] In a possible implementation of the first aspect, the above-mentioned electronic device can control the lens to move from the first position to the center point, to the target position, the target position including the center point and / or the first preset position on the movement stroke (or called stroke) of the lens during the exposure of the tth frame image, such as 1 / 4 stroke.

[0013] In a possible implementation of the first aspect, the electronic device may control the lens to perform a centering action according to a target centering speed, wherein the target centering speed may be determined by the following two implementations.

[0014] In one implementation, the electronic device may obtain the target return speed corresponding to the frame interval, thereby quickly obtaining the target return speed.

[0015] In another implementation, the electronic device can calculate the ratio between the return distance and the frame interval to obtain the target return speed, wherein the return distance represents the distance between the first coordinate of the lens and the target position, and the first coordinate of the lens represents the position of the lens at the end of exposure of the tth frame image, thereby avoiding the target return speed being too large or too small and achieving accurate determination of the target return speed.

[0016] In a possible implementation of the first aspect, after the exposure of the t+1th frame image is completed, such as in response to the completion of the exposure of the t+1th frame image, the electronic device may adjust the positions of pixels in the t+1th frame image based on the offset corresponding to the lens in combination with the EIS algorithm to obtain the t+1th frame target image, where the offset corresponding to the lens represents the offset value of the position of the lens at the end of the exposure of the t+1th frame image compared to the initial position of the lens.

[0017] Afterwards, the electronic device may display and / or save the t+1th frame of the target image.

[0018] In this application, after the lens moves (such as controlling the lens to perform a centering action between frames), the lens position will change, causing the imaging center of the electronic device camera to change, thereby changing the imaging position of the same photographed object, that is, the position of the same photographed object in two adjacent frames will change, and from the user's perception, the image will shake. Therefore, the electronic device can use the EIS algorithm to adjust the position of the pixels on the collected image (such as the t+1 frame image) to compensate for the position offset caused by the lens movement due to various reasons (such as centering) and improve the image de-shaking effect.

[0019] Among them, optionally, the offset corresponding to the above-mentioned lens represents the offset corresponding to the center of the image plane corresponding to the camera.

[0020] In a possible implementation of the first aspect, the electronic device may adjust the position of the pixel on the image based on the EIS algorithm in combination with the internal reference de-jittering method. The electronic device determines the first coordinate of the image plane center based on the initial coordinate of the image plane center and the offset corresponding to the image plane center, and the first coordinate of the image plane center represents the position of the image plane center when the exposure of the t+1 frame image ends.

[0021] Afterwards, the electronic device determines the first camera intrinsic parameter based on the first coordinate of the center of the image plane and the distance corresponding to the camera; the distance represents the distance between the image plane and the center of the lens. The distance corresponding to the camera may be the distance corresponding to the t+1th frame image, that is, the distance of the camera when the t+1th frame image is taken.

[0022] Afterwards, the electronic device adjusts the positions of the pixels in the t+1th frame image according to the first transformation matrix; the first transformation matrix is ​​obtained based on the first camera intrinsic parameter, wherein the first transformation matrix represents the conversion relationship between the world coordinate system and the pixel coordinate system.

[0023] In the present application, the offset corresponding to the center of the image plane is obtained by subtracting the position of the lens at the end of exposure of the t+1 frame image from the initial position of the lens, and the offset corresponding to the center of the image plane already includes the lens offset caused by the inter-frame return action between the t+1 frame image and the t frame image. That is, the movement of the lens (such as the movement within the frame and between frames) is converted into the change of the center of the image plane, so as to determine the latest camera internal parameters based on the coordinates of the center of the image plane after the change, so as to determine the first transformation matrix using the latest camera internal parameters. Afterwards, the electronic device can use the first transformation matrix to adjust the position of each pixel on the t+1 frame original image, which can compensate for the offset of the imaging position caused by the movement of the lens due to different reasons, ensure the stability of the image, improve the de-shaking effect, and thus improve the image quality.

[0024] Optionally, the electronic device may substitute the first camera intrinsic parameter into a transformation matrix calculation formula to obtain a first transformation matrix.

[0025] In a possible implementation manner of the first aspect, the electronic device may adjust the position of pixels on the image based on the EIS algorithm in combination with an external reference de-jittering method.

[0026] The electronic device converts the offset corresponding to the center of the image plane into an equivalent angle of the gyroscope in the camera. The equivalent angle represents the offset angle of the electronic device at the end of the exposure of the tth frame, that is, the offset angle of the gyroscope.

[0027] Afterwards, the electronic device can obtain the adjustment angle corresponding to the t+1th frame image based on the equivalent angle, and the adjustment angle represents the angle required to adjust the t+1th frame image to achieve the ideal shooting angle.

[0028] Afterwards, the electronic device may obtain a first rotation matrix based on the adjustment angle and a preset rotation matrix calculation formula;

[0029] Afterwards, the electronic device may adjust the positions of the pixels in the t+1th frame image according to the second transformation matrix; the second transformation matrix is ​​obtained based on the first rotation matrix.

[0030] In the present application, the electronic device can convert the displacement of the lens movement into the movement angle of the gyroscope, so as to determine the latest camera extrinsic parameters using the movement angle of the gyroscope, so as to optimize the image path based on the latest camera extrinsic parameters, compensate for the offset of the imaging position caused by the return center, and improve the image anti-shake effect.

[0031] In a possible implementation manner of the first aspect, the process of determining the adjustment angle corresponding to the t+1th frame image may include:

[0032] First, the electronic device can calculate the difference between the original angle corresponding to the t+1th frame image and the equivalent angle to obtain the actual angle corresponding to the t+1th frame image, wherein the original angle corresponding to the t+1th frame image represents the angle of movement of the electronic device body when the t+1th frame image is exposed, that is, the actual angle of the gyroscope, and the actual angle corresponding to the t+1th frame image represents the actual shooting angle when the lens exposes the t+1th frame image.

[0033] Afterwards, the electronic device can calculate the difference between the actual angle corresponding to the t+1th frame image and the target angle corresponding to the t+1th frame image, and obtain the adjustment angle corresponding to the t+1th frame image. The target angle corresponding to the t+1th frame image represents the ideal shooting angle corresponding to the t+1th frame image when the previous t+1 frames of images are all stable images, thereby achieving accurate calculation of the shooting adjustment angle, that is, accurate calculation of the optimized angle, so that the electronic device can use the optimized angle corresponding to the t+1th frame image to adjust the position of the pixels in the t+1th frame image and achieve image de-shaking.

[0034] In a possible implementation of the first aspect, the electronic device can calculate the ratio between the offset corresponding to the center of the image plane and the distance corresponding to the camera, and combine it with the inverse trigonometric function to obtain the equivalent angle, so as to convert the offset of the lens into the offset angle of the gyroscope, that is, convert it into the movement of the electronic device body, so that the electronic device can use the equivalent angle corresponding to the gyroscope and the actual angle of the gyroscope to determine the shooting angle of the t+1th frame image.

[0035] Optionally, the offset corresponding to the center of the image plane includes the offset of the center of the image plane in the x direction, and accordingly, the equivalent angle includes the equivalent angle in the x direction. The electronic device can use the inverse trigonometric function arctan (shift x / f) calculates the equivalent angle in the x direction, where shift x represents the offset in the x direction of the center of the image plane, and f represents the distance corresponding to the t+1th frame image.

[0036] The equivalent angle may also include an equivalent angle in the y direction and an equivalent angle in the z direction. The determination process of the equivalent angle in the y direction is similar to the determination process of the equivalent angle in the x direction. Since there is a certain correspondence between the x-axis and the z-axis, the offset in the x-direction of the center of the image plane can be converted into the offset in the z-direction, so that the offset in the z-direction can be used to determine the equivalent angle of the gyroscope in the z-direction, thereby realizing the determination of the equivalent angles in the x, y and z directions.

[0037] In a possible implementation of the first aspect, after obtaining the above-mentioned transformation matrix (such as the first transformation matrix and the second matrix), for each pixel in the t+1th frame image (that is, the t+1th frame original image), the electronic device can multiply the coordinates of the pixel by the transformation matrix to adjust the position of the pixel in the image.

[0038] In a possible implementation of the first aspect, the electronic device may use a model to adjust the position of pixels on the image. The electronic device may input the offset corresponding to the center of the image plane and the distance corresponding to the camera into a preset network model to obtain the t+1th frame target image, where the preset network model is obtained based on the EIS algorithm, and the preset network model is used to adjust the position of pixels in the image based on the offset and distance corresponding to the center of the image plane, so as to achieve rapid de-shaking of the original image, avoid changes in the optical center position (or shooting angle) of the camera when the image starts to be exposed due to lens movement, thereby causing the imaging position of the object in the previous frame of the image to shift on the next frame of the image, thereby improving image quality.

[0039] In a possible implementation of the first aspect, the multi-frame shooting state refers to a state of shooting a video, a preview state, or a state of shooting a dynamic image in a photographing state, so that the video quality, the preview image quality, or the dynamic image quality can be correspondingly improved. For example, in a video shooting state, before the electronic device exposes the t-th frame image, the electronic device needs to first receive a first operation for triggering video recording, and the electronic device responds to the first operation and starts to capture images, and anti-shake can be achieved by compensating for jitter during exposure.

[0040] In a second aspect, a shooting method is provided, the method comprising:

[0041] When the electronic device is in a multi-frame shooting state, after the exposure of the t+1 frame image is completed, the electronic device adjusts the position of the pixels in the t+1 frame image based on the offset corresponding to the lens and in combination with the EIS algorithm to obtain the t+1 frame target image; the offset corresponding to the lens represents the offset value of the position of the lens at the end of the exposure of the t+1 frame image compared to the initial position of the lens;

[0042] The electronic device displays and / or stores the target image of the t+1th frame.

[0043] The offset corresponding to the above lens represents the offset corresponding to the center of the image plane corresponding to the camera.

[0044] In this application, after the lens moves, the lens position will change, causing the imaging center of the electronic device camera to change, thereby changing the imaging position of the same photographed object, that is, the position of the same photographed object in two adjacent frames will change, and from the user's perception, the image will shake. Therefore, the electronic device can use the EIS algorithm to adjust the position of the pixels on the collected image (such as the t+1 frame image) to compensate for the offset of the imaging position caused by the lens movement and improve the image de-shaking effect.

[0045] In a possible implementation manner of the second aspect, the electronic device adjusts positions of pixels in the t+1th frame image based on an offset corresponding to the lens in combination with an electronic image stabilization technology EIS algorithm to obtain a t+1th frame target image, including:

[0046] The electronic device determines the first coordinate of the center of the image plane based on the initial coordinate of the center of the image plane and the offset corresponding to the center of the image plane; wherein the first coordinate of the center of the image plane represents the position of the center of the image plane when the exposure of the t+1th frame of image ends;

[0047] The electronic device determines a first camera intrinsic parameter based on a first coordinate of the center of the image plane and a distance corresponding to the camera; the distance represents a distance between the image plane and the center of the lens;

[0048] The electronic device adjusts the positions of pixels in the t+1th frame image according to a first transformation matrix; the first transformation matrix is ​​obtained based on the first camera intrinsic parameter.

[0049] In the present application, the offset corresponding to the center of the image plane is obtained by subtracting the position of the lens at the end of the exposure of the t+1 frame image from the initial position of the lens, that is, converting the movement of the lens, that is, the offset, into the change of the center of the image plane, thereby determining the latest camera internal parameters based on the coordinates of the center of the image plane after the change, so as to determine the first transformation matrix using the latest camera internal parameters. Afterwards, the electronic device can use the first transformation matrix to adjust the position of each pixel on the t+1 frame original image, which can compensate for the offset of the imaging position caused by the movement of the lens, ensure the stability of the image, improve the de-shaking effect, and thus improve the image quality.

[0050] In a possible implementation manner of the second aspect, the electronic device adjusts positions of pixels in the t+1th frame image based on an offset corresponding to the lens in combination with an electronic image stabilization technology EIS algorithm to obtain a t+1th frame target image, including:

[0051] The electronic device converts the offset corresponding to the center of the image plane into an equivalent angle of the gyroscope in the camera; the equivalent angle represents the offset angle of the electronic device when the exposure of the tth frame ends;

[0052] The electronic device obtains an adjustment angle corresponding to the t+1th frame image based on the equivalent angle, where the adjustment angle represents an angle that needs to be adjusted when the t+1th frame image reaches an ideal shooting angle;

[0053] The electronic device obtains a first rotation matrix based on the adjustment angle and a preset rotation matrix calculation formula;

[0054] The electronic device adjusts the positions of the pixels in the t+1th frame image according to a second transformation matrix; the second transformation matrix is ​​obtained based on the first rotation matrix.

[0055] In the present application, the electronic device can convert the displacement of the lens movement into the movement angle of the gyroscope, so as to determine the latest camera extrinsic parameters using the movement angle of the gyroscope, so as to optimize the image path based on the latest camera extrinsic parameters, compensate for the offset of the imaging position caused by the lens movement, and improve the image anti-shake effect.

[0056] In a third aspect, the present application provides an electronic device, comprising a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; the display screen is used to display an image generated by the processor, and the memory is used to store computer program code, wherein the computer program code comprises computer instructions; when the processor executes the computer instructions, the electronic device executes a shooting method as described in any one of the first aspect or the second aspect above.

[0057] In a fourth aspect, the present application provides a computer-readable storage medium, comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the shooting method as described in any one of the first aspect or the second aspect above.

[0058] In a fifth aspect, the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the shooting method as described in any one of the first aspect or the second aspect above.

[0059] It can be understood that the beneficial effects that can be achieved by the shooting method described in the second aspect, the electronic device described in the third aspect, the computer storage medium described in the fourth aspect, and the computer program product described in the fifth aspect provided above can refer to the beneficial effects in the first aspect and any possible implementation method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1A A schematic diagram of the working principle of a camera provided in an embodiment of the present application;

[0061] Figure 1B A schematic diagram of a video shooting process provided in an embodiment of the present application;

[0062] Figure 1C A schematic diagram of an image exposure process provided in an embodiment of the present application;

[0063] Figure 2A A second schematic diagram of a video shooting process provided in an embodiment of the present application;

[0064] Figure 2B A third schematic diagram of a video shooting process provided in an embodiment of the present application;

[0065] Figure 2C A video shooting process diagram provided in an embodiment of the present application Figure 4 ;

[0066] Figure 3A A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0067] Figure 3B A schematic diagram of a camera of an electronic device provided in an embodiment of the present application;

[0068] Figure 3C A second schematic diagram of a camera of an electronic device provided in an embodiment of the present application;

[0069] Figure 4 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0070] Figure 5A A flowchart of a photographing method provided in an embodiment of the present application is shown in FIG1;

[0071] Figure 5B Schematic diagram 5 of a video shooting process provided in an embodiment of the present application;

[0072] Figure 5C A schematic diagram of a lens movement range provided in an embodiment of the present application is shown first;

[0073] Figure 5D A second schematic diagram of a lens movement range provided in an embodiment of the present application;

[0074] Figure 5E A third schematic diagram of a lens movement range provided in an embodiment of the present application;

[0075] Fig. 5F A lens movement range diagram provided in an embodiment of the present application Figure 4 ;

[0076] Figure 6 A video shooting process diagram provided in an embodiment of the present application Figure 6 ;

[0077] Figure 7 A lens movement schematic diagram 1 provided in an embodiment of the present application;

[0078] Figure 8 A second flowchart of a photographing method provided in an embodiment of the present application;

[0079] Fig.9A A second schematic diagram of lens movement provided in an embodiment of the present application;

[0080] Fig. 9B A third lens movement diagram provided in an embodiment of the present application;

[0081] Fig. 9C A schematic diagram of gyroscope angular velocity provided in an embodiment of the present application;

[0082] Fig.9D A schematic diagram of a frame interval provided in an embodiment of the present application;

[0083] Fig.10 Schematic diagram 3 of a process of a photographing method provided in an embodiment of the present application;

[0084] Fig.11 A lens movement diagram provided in an embodiment of the present application Figure 4 ;

[0085] Fig.12 A schematic diagram of pixel position adjustment provided in an embodiment of the present application;

[0086] Fig.13 A schematic diagram of a photographing method provided in an embodiment of the present application Figure 4 ;

[0087] Fig.14 An equivalent conversion schematic diagram provided for an embodiment of the present application;

[0088] Fig.15 A schematic diagram of a shooting angle provided in an embodiment of the present application;

[0089] Fig.16 A schematic diagram of a screen jitter interface provided in an embodiment of the present application;

[0090] Fig.17 Schematic diagram 5 of a process of a shooting method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0091] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way. In the embodiments of the present application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following items (individuals)" or similar expressions thereof refer to any combination of these items, including any combination of single items (individuals) or plural items (individuals). For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple. In the embodiments of the present application, "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" can explicitly or implicitly include one or more of the features. In the description of the present embodiment, unless otherwise specified, the meaning of "multiple" is two or more.

[0092] For ease of understanding, the professional terms involved in this application are first introduced below.

[0093] OIS: refers to avoiding or reducing the jitter that occurs in the process of capturing light signals by moving the lens or image sensor (also called photosensitive element or sensor) in the camera, thereby improving image quality.

[0094] Electronic image stabilization (EIS) technology: It refers to the use of software algorithms to calculate the posture changes of the camera lens and crop the video image to make the video image look more stable.

[0095] Image plane: refers to the imaging plane, that is, the plane where the lens focuses on the image. Simply put, the location of the image sensor is the image plane.

[0096] The camera imaging principle is as follows: Figure 1A As shown in FIG. 1 , light is transmitted to the camera sensor (such as complementary metal-oxide-semiconductor (CMOS)) through the camera lens module (hereinafter referred to as the lens), and this process is called exposure. During the exposure process, the sensor converts the light signal collected by each pixel into an electrical signal to form a frame of image.

[0097] During the process of shooting a video with a mobile phone, if the mobile phone camera shakes during the exposure, the camera and the object being shot will have a large relative movement, and the entire movement process will be imaged on the sensor, resulting in blurred images and low video quality. For example, when a user holds a mobile phone to shoot a video, the user's hand shakes downward, causing the user's mobile phone to shake downward as well, and the content shot by the mobile phone changes (such as Figure 1B The content of the screen shown in (b) is compared with Figure 1B The image shown in (a) has changed), and the image may become blurred (such as Figure 1B The light board 10 in the picture shown in (b) is compared with Figure 1B The light sign 10 in the picture shown in (a) is blurred).

[0098] It should be noted that if the light in the shooting environment is dim, the exposure time is generally longer. For example, in a night shooting scene, the exposure time is generally 30 milliseconds (ms) to 60ms, which can easily lead to blurred images due to phone shaking.

[0099] For example, the mobile phone camera can be used as follows Figure 1CThe exposure is performed using the row-by-row exposure (or rolling shutter exposure) method shown in the figure. Row-by-row exposure means that the exposure starts from the first row, and after an interval of one row period, the second row is exposed, and so on, until the last row is exposed. After the first row is exposed, the sensor starts to read the data, and it takes one row period to read a row of data. After the data of the first row is read, the sensor can start to read the data of the second row, and so on, until the data of the last row is read. Among them, after each row is exposed, the sensor transfers the data of the photosensitive area to the storage area. After that, the data of each row read by the sensor is read from the storage area.

[0100] It should be understood that when the type of sensor in the mobile phone camera is a rolling exposure type, the sensor can be exposed in a line-by-line exposure manner.

[0101] The above introduces the terms involved in this application and the camera imaging principle. The following will begin to introduce the solution of this application.

[0102] In some embodiments, the mobile phone can use OIS to achieve video de-shaking. During the process of shooting a video, if the mobile phone detects that the mobile phone is shaking, the mobile phone can drive the camera lens to move by moving the motor in the lens (or called OIS motor) to compensate for the shaking of the mobile phone. Among them, the compensation action of driving the camera lens to move is performed within the frame, that is, during the exposure of the image frame in the video, if the mobile phone camera shakes, the mobile phone can drive the camera lens to move. For example, during the process of shooting a video, if Figure 2A As shown in FIG. 1 , after the exposure of image f0 is completed, the mobile phone camera continues to capture the next frame of image f1. During the exposure of image f1, the mobile phone detects that the mobile phone camera is shaking, such as detecting that the mobile phone moves downward. In order to compensate for the downward movement of the mobile phone, the mobile phone can drive the OIS motor to move the lens upward during the exposure of image f1 (such as Figure 2B As shown), the camera does not move relative to the photographed object, so that the obtained image f1 does not change the picture content compared to the image f0, and there is no blur problem.

[0103] However, since the OIS motor moves within a certain range, that is, the movement range of the OIS motor is limited, in other words, the movement range of the lens is limited. When the OIS motor has moved to the top, if the mobile phone camera shakes downward, the OIS motor can no longer move upward, that is, it cannot drive the lens to move upward to compensate for the downward shake of the mobile phone camera. In other words, the movement range of the lens is limited, resulting in the image captured by the mobile phone may be blurred, and the image content will change, causing the video to shake and blur.

[0104] Continuing with the above example, after image f1 is captured, the phone continues to capture the next frame of image f2. The phone camera still shakes downwards, and because the lens has moved to the top, it can no longer move upwards to compensate for the downward shake of the phone, resulting in the captured image f2 having a different picture content than image f1, and the picture is blurred (e.g. Figure 2C The light sign 11 in the image frame f2 is shown to be blurrier than the light sign 11 in the image frame f1 ).

[0105] Therefore, in view of the above problems, the present application provides a shooting method, in which when an electronic device is shaking during video shooting, the electronic device can drive the lens to move in the opposite direction by moving the OIS motor within the frame, that is, during the image exposure, the lens is driven to move in the opposite direction to compensate for the shaking of the electronic device, so that after the image exposure is completed, a clear image can be obtained, and the shooting content will not change. In order to ensure the range of movement of the lens in the next frame, the electronic device can control the lens to perform a return action during the interval between the image and the next frame. Since the lens is returned to the center between frames, it is ensured that the lens has a larger range of movement when the next frame is exposed, avoiding the problem of the next frame being blurred due to the inability to compensate for the shaking by moving the lens due to the limited range of movement of the lens, and avoiding the problem of tremor in the recorded video. Since the lens is returned to the center between frames, the position of the lens at the beginning of the exposure of the next frame is different from the position at the end of the exposure of the previous frame, resulting in different positions of the center of the image plane of the two adjacent frames, thereby causing the same shooting object to have different positions on the two adjacent frames, resulting in a shaking problem. Therefore, in order to achieve video de-shaking, electronic devices can use the EIS algorithm to optimize the position of pixels in the next frame image based on the displacement of the lens (including the offset of the lens caused by centering), so that the position of the same object in different images is the same, avoiding video jitter problems caused by centering, ensuring the quality of the video, and improving the user's shooting experience.

[0106] Exemplarily, the electronic device in the embodiments of the present application may be a mobile phone, a tablet computer, a wearable device, a personal digital assistant (PDA), a vehicle terminal, an Internet of Things device, or other device with a shooting function. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic device.

[0107] Figure 3A A schematic structural diagram of an electronic device 100 is shown.

[0108] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0109] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0110] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0111] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0112] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0113] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0114] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0115] The charging management module 140 is used to receive charging input from a charger. While the charging management module 140 is charging the battery 142 , it can also power the electronic device through the power management module 141 .

[0116] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160.

[0117] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0118] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0119] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0120] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0121] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0122] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0123] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0124] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0125] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0126] The ISP is used to process data fed back by the camera 193. For example, when taking a photo, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, and the light signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye.

[0127] The ISP can also perform algorithm optimization on the noise and brightness of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0128] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element (or image sensor). The photosensitive element can be a charge coupled device (CCD) or a CMOS phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0129] Exemplarily, the N cameras 193 may include: one or more front cameras and one or more rear cameras. For example, the electronic device 100 is a mobile phone. Figure 3B The illustrated mobile phone includes a front-facing camera, such as front-facing camera 20 . Figure 3C The mobile phone shown includes three rear cameras, such as rear cameras 21, 22 and 23. Of course, the number of cameras in the mobile phone includes but is not limited to the number described in the above example.

[0130] Among them, the above-mentioned camera 193 may include one or more of the following cameras: a main camera, a telephoto camera, a wide-angle camera, an ultra-wide-angle camera, a macro camera, a fisheye camera, a time of flight (TOF) camera, an infrared camera, a depth camera and a black and white camera.

[0131] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0132] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0133] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0134] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0135] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0136] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0137] The earphone interface 170D is used to connect a wired earphone and may be a USB interface 130 .

[0138] The above-mentioned sensor module 180 may include a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0139] Among them, the gyroscope sensor can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor. The gyroscope sensor 180B can be used for anti-shake shooting. Exemplarily, when the shutter is pressed, the gyroscope sensor detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake.

[0140] Buttons 190 include a power button, a volume button, and the like.

[0141] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.

[0142] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0143] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting or removing the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1.

[0144] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes a layered architecture system as an example to exemplify the software structure of the electronic device 100.

[0145] Figure 4 It is a structural block diagram of the electronic device 100 according to an embodiment of the present application.

[0146] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the runtime and system library, the hardware abstraction layer, and the kernel layer. Among them, the application layer, the application framework layer, the Android runtime and system library, the hardware abstraction layer, and the kernel layer all belong to the software layer.

[0147] The application layer can include a series of application packages.

[0148] like Figure 4 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0149] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0150] like Figure 4 As shown, the application framework layer may include a camera access interface, a window manager, a content provider, a view system, a resource manager, a notification manager, and the like.

[0151] The camera access interface can be used to provide an application programming interface and a programming framework for camera applications. The camera access interface can include camera management and camera devices. Camera management can be used to provide an access interface for managing cameras (or described as cameras); camera devices can be used to provide an interface for accessing cameras.

[0152] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0153] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0154] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0155] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0156] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0157] The runtime includes the core library and the virtual machine. The runtime is responsible for the scheduling and management of the system.

[0158] The core library consists of two parts: one is the function that the Java language needs to call, and the other is the core library of the system.

[0159] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0160] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0161] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0162] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0163] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0164] A 2D graphics engine is a drawing engine for 2D drawings.

[0165] The hardware abstraction layer is an interface layer located between the application framework layer and the driver layer, and provides a virtual hardware platform for the operating system. In the embodiment of the present application, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library.

[0166] Among them, the camera hardware abstraction layer can provide virtual hardware of camera devices such as camera device 1, camera device 2, etc. Camera devices such as camera device 1, camera device 2, etc. correspond to one or more cameras on the electronic device 100. The electronic device 100 can call each camera through camera devices such as camera device 1, camera device 2, etc. The camera algorithm library may include the shooting state recommendation algorithm provided in the embodiment of the present application, for example, the de-shaking algorithm corresponding to the video shooting state, such as the EIS algorithm.

[0167] The kernel layer is the layer between hardware and software. The kernel layer can also be called the driver layer, which can include camera device drivers, etc. Among them, the camera device driver is used to drive the sensor of the camera in the hardware layer to collect images and drive the image signal processor in the hardware layer to pre-process the image.

[0168] In some embodiments, the kernel layer may further include a digital signal processor driver and an image processor driver. The digital signal processor driver is used to drive the digital signal processor in the hardware layer to process images. The image processor driver is used to drive the image processor in the hardware layer to process images.

[0169] In some embodiments, the kernel layer may also include a display driver, a camera driver, an audio driver, and a sensor driver.

[0170] In the embodiment of the present application, by calling the hardware abstraction layer interface in the hardware abstraction layer, the application layer and application framework layer above the hardware abstraction layer can be connected with the driver layer and hardware layer below, thereby realizing camera data transmission and function control.

[0171] Based on the video shooting scene, combined with the above Figure 4 , which illustrates the software and hardware workflow of an electronic device.

[0172] a. When the camera application is running on the electronic device 100, in response to a first trigger operation of the user on the camera application, the camera application sends a video shooting request to the camera access interface, wherein the first trigger operation is used to trigger the electronic device 100 to record a video.

[0173] b. The camera access interface sends the above video capture request to the camera hardware abstraction layer.

[0174] c. The camera hardware abstraction layer sends the above video capture request to the camera device driver in the kernel layer.

[0175] d. The camera device driver controls the camera (such as camera 1, camera 2, TOF camera, multispectral camera, etc.) to continuously collect images based on the video shooting request.

[0176] e. After capturing a frame of image, the camera sends the image to the camera device driver.

[0177] After receiving the image, the camera device driver sends the image to the camera hardware abstraction layer.

[0178] g. After receiving the image, the camera hardware abstraction layer calls the algorithm in the camera algorithm library to process the image and sends the processed image to the camera access interface.

[0179] h. The camera access interface sends the processed image to the camera application, so that the camera application displays the processed image on the display screen of the electronic device and / or saves the processed image.

[0180] In some embodiments, in order to avoid blurring of the video image, the electronic device may also perform intra-frame compensation jitter and inter-frame centering to avoid blurring of the captured image image due to the jitter of the electronic device. The intra-frame compensation jitter process may include: i. During image exposure, the gyroscope sensor (or gyroscope) sends the collected angular velocity data to the processor. Afterwards, the processor may execute j. Based on the angular velocity data, determine whether the electronic device is shaking, and in the event of shaking, send a compensation instruction to the camera to trigger the lens in the camera to move in the opposite direction of the shaking during image exposure.

[0181] The inter-frame centering process includes: k. After the image exposure is completed, the processor sends a centering instruction to the camera to trigger the lens in the camera to perform a centering action.

[0182] It should be noted that the above-mentioned camera 1 and camera 2 can be different types of cameras. For example, sensor 1 can be a wide-angle camera, and camera 2 can be a main camera. The receiving and sending operations between modules inside the electronic device described in the embodiment of the present application actually refer to the calling of functions.

[0183] In order to avoid blurring of the camera's imaging picture due to the shaking of the electronic device's camera, the present application proposes a shooting method. In a multi-frame shooting state, during the image exposure, the electronic device compensates for the shaking of the electronic device's camera by moving the lens of the electronic device's camera, thereby realizing OIS intra-frame de-shaking and avoiding blurring of the imaging picture. In addition, in order to avoid the situation where the lens cannot continue to move to compensate for the shaking after moving to the maximum extent due to the limited range of motion of the lens, the electronic device can adopt an inter-frame return strategy so that the lens can obtain the maximum range of motion when capturing the next frame of the image, thereby continuing to compensate for the shaking of the electronic device and ensuring the imaging effect of the camera. In order to ensure that a stable image can be output, the electronic device can use the EIS algorithm to adapt the inter-frame return, that is, the electronic device can adjust the coordinates of the pixels in the image based on the movement of the lens in combination with the EIS algorithm, so that the positions of the same object on adjacent images are aligned, ensuring the anti-shake effect of the video, thereby ensuring the shooting effect of the electronic device.

[0184] The multi-frame shooting state refers to the state in which the mobile phone continuously captures multiple frames of images and processes the multiple frames of images. For example, the multi-frame shooting state refers to the state of video shooting, preview state, or shooting dynamic images in the camera state. That is to say, the shooting method provided by the present application can be applied to the scene of video shooting, the process of displaying preview images, or the scene of dynamic image shooting in the camera state. The preview state refers to the state of the mobile phone before starting to shoot in the video shooting state or the camera state.

[0185] The following will take the electronic device as a mobile phone and the application scenario as video shooting as an example to introduce the shooting method provided in the embodiment of the present application.

[0186] The embodiment of the present application provides a shooting method. In this embodiment, during the process of shooting a video by an electronic device, if it is detected that the electronic device is shaking, the electronic device can drive the camera lens to move within the frame to compensate for the shaking of the electronic device. In order to ensure that the compensation of the lens is not limited, the electronic device can control the lens to perform a centering action between frames. Specifically, Figure 5A As shown, the shooting method provided in the embodiment of the present application may include S101-S104.

[0187] S101, the mobile phone displays a shooting interface. The shooting interface is an interface displayed when the mobile phone records a video. The shooting interface includes a preview area, which is used to display images captured by the camera on the mobile phone.

[0188] The shooting interface is the interface displayed during the video recording process of the mobile phone. The shooting interface may include a preview area and a shooting button. The preview area is used to display the image collected by the camera. The video shooting button is used to trigger the mobile phone to record the video.

[0189] In some embodiments, the shooting interface may also include a stop button (such as Figure 5B The stop button 30 shown in the figure is used to trigger the mobile phone to stop recording the video and generate a video file. It should be understood that only when the mobile phone is in the recording state can the user trigger the mobile phone to stop recording.

[0190] It should be noted that the stop button and the video shooting button may be in the same position. Specifically, when the electronic device is shooting a video, the button is in a recording state, i.e., a shooting state, and the button may be used as a stop button. After receiving a click operation of the stop button by the user, the electronic device may stop recording the video. When the electronic device is in a video shooting state and is not shooting a video, the button is in a stop recording state. The button may be used as a video shooting button. After receiving a click operation of the shooting button by the user, the electronic device may start recording the video.

[0191] S102: The mobile phone receives a first trigger operation of the user on the shooting interface, and captures an image in response to the first trigger operation, wherein the first trigger operation is used to trigger video recording.

[0192] Exemplarily, the first trigger operation may be a user clicking on a video capture button. Alternatively, the first trigger operation may be a first preset gesture input by the user in the shooting interface, such as an S-shaped gesture, a √-shaped gesture, etc. The first preset gesture is used to trigger the mobile phone to record a video. In this embodiment, the shooting interface of the mobile phone may also include the video capture button or may not include the video capture button.

[0193] It should be understood that the first preset gesture can be pre-configured in the mobile phone. In addition, the mobile phone can prompt the user with the first preset gesture and the function triggered by the first preset gesture (i.e., triggering the mobile phone to shoot video) in the video recording interface.

[0194] In addition, the first trigger operation introduced above is only an example of the first operation that can trigger the mobile phone to record a video. The first operation can also be other operations input by the user. For example, the user speaks a shooting command, and the mobile phone responds to the shooting command to capture an image. Among them, the shooting command can be the default of the mobile phone or customized by the user. For another example, the user triggers the virtual shutter on the shooting interface, and the mobile phone responds to the shooting command to capture an image. For another example, the shooting object outputs a second preset gesture, and the mobile phone responds to the second preset gesture to capture an image. Among them, the second preset gesture can be the default of the mobile phone or customized by the user.

[0195] S103: When the mobile phone is exposing the t-th frame image, if the mobile phone shakes, the mobile phone moves the lens in the mobile phone in the reverse direction, where t is a positive integer.

[0196] In the embodiment of the present application, during the shaking of the mobile phone, the mobile phone exposes the t-th frame image, that is, during the exposure of the t-th frame image, the mobile phone detects that it is shaking. In order to avoid the t-th frame image being blurred due to the shaking of the mobile phone, during the exposure of the t-th frame image line by line, the mobile phone can reversely move the lens in the mobile phone by driving the OIS motor in real time based on the shaking of the mobile phone, that is, move the lens to the first position to achieve intra-frame jitter compensation. Among them, the displacement of the lens to the first position is the position of the mobile phone shaking during the exposure of the t-th frame image.

[0197] In some embodiments, the shaking of the mobile phone can be detected by a gyroscope. The mobile phone can determine the shaking displacement of the mobile phone through the angular velocity data collected by the gyroscope in the mobile phone. Afterwards, the mobile phone can control the movement of the lens in the opposite direction of the shaking, and the displacement of the reverse movement can be the shaking displacement. Among them, the angular velocity data collected by the gyroscope may include the angular velocity corresponding to different time points during the exposure of the t-th frame image, such as the angular velocity corresponding to each row of pixels. Specifically, after the movement of the camera is detected by the gyroscope, the mobile phone can calculate the displacement that needs to be compensated based on the angular velocity data reported by the gyroscope in real time or periodically to obtain the shaking displacement. Afterwards, the mobile phone can move the lens to compensate for the shaking based on the shaking displacement.

[0198] It should be understood that the motor can drive the lens to move on a two-dimensional plane, and it can move in any direction, such as up, down, left, and right. In other words, the movement range of the lens is equivalent to a two-dimensional plane (such as Figure 5C As shown). Correspondingly, the jitter displacement includes the jitter displacement in the x-direction and the jitter displacement in the y-direction. The jitter displacement in the x-direction can be a positive number, a negative number, or 0. Similarly, the jitter displacement in the y-direction can be a positive number, a negative number, or 0. For example, when the mobile phone shakes vertically upward, the jitter displacement in the y-direction is 1, and the jitter displacement in the x-direction is 0. The mobile phone can drive the lens to move vertically downward by 1 to compensate for the jitter in the y-direction. For another example, when the mobile phone shakes vertically downward, the jitter displacement in the y-direction is -1, and the jitter displacement in the x-direction is 0. The mobile phone can drive the lens to move vertically upward (as shown Figure 5D As shown in FIG. 1 , the movement amount is 1 to compensate for the jitter in the y direction. It should be understood that the movement direction of the lens (such as vertically upward and vertically downward) illustrated here is only an example. The movement direction of the lens in this application is not limited. The lens can be moved to any position in the two-dimensional plane, such as moving to Figure 5E The location of the lens shown.

[0199] It should be noted that the above Figure 5C The shape of the moving range shown (ie, a rectangle) is only an example, and the moving range may also be other shapes, such as a circle, which is not limited in the present application.

[0200] S104: When the exposure of the t-th frame image is completed, the mobile phone controls the lens to perform a re-centering action within a first target time, wherein the first target time is the interval time between the t-th frame image and the t+1-th frame image.

[0201] The end of exposure of the t-th frame image refers to the mobile phone (or the processor 110 in the mobile phone) obtaining the timestamp corresponding to the last row of pixels of the t-th frame image. It should be understood that after each row of pixels is exposed, the camera will generate a timestamp corresponding to the row of pixels. Therefore, when the processor 110 obtains the timestamp corresponding to the last row of pixels of the image, it can be considered that the exposure of the image is completed.

[0202] The above-mentioned return-to-center action means moving the lens from the current position (such as the position of the lens when the exposure of the t-th frame image ends) to the center point of the moving range, and then to the target position. The target position is a position in the moving range corresponding to the lens, and the target position can be determined in the following ways. For example, the target position is the center point of the moving range of the lens (such as the above-mentioned Figure 5C The target position may be the center point of the motor's range of motion. Fig. 5F As shown, the position of the lens at the end of exposure of the t-th frame image is (0,1), and the center point is (0,0), so the 1 / 4 stroke is 1 / 4 of the center point, that is, the lens is moved to the position of (0,1 / 4).

[0203] In the embodiment of the present application, when the exposure of the t-th frame image is completed, it indicates that the t-th frame image has been collected, and the t+1-th frame image needs to be collected next. Since the OIS motor is active within a certain range, the range of the OIS motor driving the lens to move is also limited. If the mobile phone has compensated for the shake of the mobile phone by moving the lens in a certain direction during the exposure of the previous frame image, then when exposing the next frame image, the range in which the mobile phone can move the lens in this direction will be reduced. In other words, the range of movement of the lens when exposing the next frame image is limited. Therefore, in order to avoid limiting the range of movement of the lens when exposing the next frame image, the mobile phone can control the lens to perform a return action between frames (or frame intervals), so that when the t+1-th frame image begins to be exposed, the lens still has a larger range of movement, so that even if the mobile phone still shakes, the mobile phone can still compensate for the shake by moving the lens to avoid blurring the next frame image, thereby avoiding tremor problems in the recorded video.

[0204] For example, during video recording, Figure 6 As shown in (a), the t-1th frame image is acquired and the phone shakes downward. During the exposure of the tth frame image, the phone is still shaking downward. In order to compensate for the downward shaking of the phone, the phone can drive the lens to move upward (such as Figure 6 After the exposure of the t-th frame image is completed, we get Figure 6The t-th frame image shown in (b) in the figure. Since the jitter is compensated during the exposure of the t-th frame image, the image of the t-th frame image is not blurred compared to the t-1-th frame image.

[0205] When the exposure of the tth frame image is finished, the mobile phone can control the lens to perform a return-to-center action (such as Figure 6 (c) in the figure to reserve movable space for compensation of the next frame. When exposing the t+1 frame, the phone still shakes downward. Since the lens has been centered, the phone can continue to move the lens upward to continue compensating for the shake. After the exposure of the t+1 frame is completed, the following is obtained: Figure 6 The t+1 frame image shown in (d). Since the jitter is compensated during the exposure of the t+1 frame image, the image of the t+1 frame image is still not blurred compared to the t frame image, thus avoiding the situation where the phone still jitters downward when exposing the next frame image because the lens has moved to the top of the moving range, and the phone will not be able to compensate for the jitter by moving the lens upward, resulting in a blurred image of the next frame image, thereby ensuring the quality of the next frame image.

[0206] It can be understood that the direction of the lens compensating the jitter is decoupled from the direction in which the lens is centered. Figure 6 The interface shown in (a)-(d) is combined with Figure 7 As shown in the figure, after the exposure of the t-th frame image, the lens has moved upward to the maximum position of the moving range. For the t-th frame image, the compensation direction of the lens (or motor) is upward. During the exposure of the t+1-th frame image, the compensation direction of the lens is upward. During the interval between the t-th frame image and the t+1-th frame image, the lens is controlled to move downward to return to the center. It can be seen that the direction of the lens when returning to the center can be different from the direction of the lens jitter compensation.

[0207] In some embodiments, when the shaking degree of the mobile phone is small, the degree of reverse movement of the lens to compensate for the shaking is also small, and the lens still has a large range of motion in the next frame, the mobile phone does not need to perform a re-centering action in order to make the lens have a larger range of motion in the next frame. Therefore, in order to avoid unnecessary re-centering actions, the mobile phone can determine whether to perform an inter-frame re-centering action based on the current position of the lens. Exemplary, the process of the mobile phone determining whether to perform an inter-frame re-centering action on the lens based on the position of the lens may include the following: Figure 8 S104a-S104c shown.

[0208] S104a, when the exposure of the t-th frame image is completed, the mobile phone determines whether the current position of the lens is within the first preset position range.

[0209] In the embodiment of the present application, when the exposure of the t-th frame image ends, when the current position of the lens is within the first preset position range, that is, when the position of the motor at the end of the exposure of the t-th frame image is within the first preset position range, in other words, when the position of the motor at the end of the exposure of the t-th frame image does not reach the preset position, it indicates that the lens movement is small, and the lens still has a large range of activity in the next frame, and there is no need to return the lens to the center in order to make the lens have a larger range of activity in the next frame. Therefore, the mobile phone can execute S104b. When the current position of the lens is not within the first preset position range, that is, when the position of the motor at the end of the exposure of the t-th frame image reaches the preset position, it indicates that the range of activity of the lens in a certain direction in the next frame is limited, and the mobile phone needs to control the lens to perform a return action to reserve movement space for the next frame image. Therefore, the mobile phone can execute S104c.

[0210] Wherein, the current position of the lens mentioned above indicates the current coordinate of the lens, which may include the coordinate in the x direction and the coordinate in the y direction. Correspondingly, the first preset position range may also include the coordinate range in the x direction and the coordinate range in the y direction. In other words, the position point in the first preset position range (or referred to as the preset position) may include the coordinate in the x direction and the coordinate in the y direction. The first preset position range is a pre-set partial area within the moving range of the lens. The first preset position range is an area composed of position points that are closer to the center point. The absolute value of the coordinate in the x direction of the position point in the first preset position range is less than the first preset coordinate value, and the absolute value of the coordinate in the y direction of the position point is less than the second preset coordinate value.

[0211] In some embodiments, after each frame of image is exposed, the mobile phone may determine whether the current position of the lens is within the first preset position range to determine whether it is necessary to control the lens to perform a return-to-center operation. Alternatively, after a frame of image (such as the t-th frame of image) is exposed, if the mobile phone does not shake during the exposure process of the frame of image, the mobile phone may not determine whether it is necessary to control the lens to perform a return-to-center operation after the exposure of the frame of image is completed. If the mobile phone shakes during the exposure process of the frame of image, the mobile phone may determine whether it is necessary to control the lens to perform a return-to-center operation after the exposure of the frame of image is completed.

[0212] S104b, within the first target time, the mobile phone does not control the lens to perform a return to center action.

[0213] S104c, within the first target time, the mobile phone controls the lens to perform a return to center action.

[0214] For example, the maximum values ​​of the coordinates in the y direction in the above moving range include 200 and -200. Among them, 200 represents the maximum position along the positive direction of the y axis, and -200 represents the maximum position along the negative direction of the y axis. The mobile phone detects that the mobile phone continues to shake in the y direction through the angular velocity of the gyroscope. In order to compensate for the shaking of the mobile phone, the mobile phone moves the lens in the opposite direction during the image exposure. Fig.9A As shown in FIG. 1 , when the image exposure ends (such as time t1), the lens has moved upward to the maximum position in the y direction (i.e., (0, 200)), indicating that in the next frame, it will not be possible to move the lens along the positive direction of the y axis to compensate for the shaking of the mobile phone in the negative direction of the y axis, that is, it is impossible to compensate for the downward shaking of the mobile phone. Therefore, in order to avoid being unable to compensate for the downward shaking of the mobile phone in the next frame, as shown in FIG. Fig. 9B As shown, the mobile phone can pull the lens back to 1 / 4 stroke (i.e. (0, 50)) at time t2 to reserve movement space for compensation of the next frame, so that the mobile phone can still achieve anti-shake by moving the lens in the next frame. Among them, time t2 is the interval time between the image and the next frame.

[0215] Afterwards, if Fig. 9C As shown, after time t2, when the mobile phone shakes upward, the mobile phone moves the lens downward. Fig. 9B At t3 time shown in the figure, the lens has moved to the maximum position in the y direction (i.e. (0, -200)), indicating that in the next frame, it will not be possible to move the lens along the negative direction of the y axis to compensate for the shaking of the mobile phone in the positive direction of the y axis, that is, it is impossible to compensate for the upward shaking of the mobile phone. Therefore, in order to avoid being unable to compensate for the upward shaking of the mobile phone in the next frame, as shown in the figure, Fig. 9B As shown, the mobile phone can pull the lens back to the 1 / 4 stroke position ((0, -50)) at time t4, where time t4 is the interval between the image and the next frame.

[0216] It is understood that the positive and negative of the above coordinates represent the direction, not the size. For example, -200 represents the maximum position on the negative direction of the y-axis.

[0217] In some embodiments, the mobile phone can control the lens to perform a return-to-center action in the interval between the t-th frame image and the t+1-th frame image, that is, the frame interval in the first target time. Fig.9DAs shown, the first target time represents the time from the end of the exposure of the first row of pixels in the t-th frame image to the start of the exposure of the first row of pixels in the t+1-th frame image. The first target time includes the frame gap and the data reading time. The data reading time refers to the time when the sensor reads each row of pixel data (such as electrical signal data) after the exposure of each row of pixels is completed. In other words, the sensor reads each row of pixel data of the image during the data reading time, such as reading from the storage area. The frame gap represents the idle time of the sensor. The sensor does not expose the image or read the data during the frame gap. For example, assuming the frame rate is 30, the first target time, that is, the frame interval, is 33ms. The data reading time is 13ms, and accordingly, the frame gap is 33-13=20ms, that is, the lens needs to perform the return action within these 20ms.

[0218] Among them, optionally, the mobile phone can determine the target return speed according to the frame gap, so as to use the target return speed to control the lens to perform the return action. In one example, the mobile phone can obtain the preset return speed corresponding to the frame gap and use it as the target return speed. When the frame gap is shorter, it means that the time available for the return action is shorter, the return speed needs to be faster, and accordingly, the return accuracy is lower. In other words, after the return action is completed, the position of the lens may differ from the target position. When the frame gap is longer, it means that the time available for the return action is longer, the return speed can be slower, and accordingly, the return accuracy is higher. In other words, after the return action is completed, the position of the lens may differ from the target position.

[0219] In another example, the mobile phone can obtain the first coordinate of the lens, which represents the position of the lens at the end of the exposure of the t-th frame image. Afterwards, the mobile phone can determine the return distance based on the first coordinate of the lens and the target position, that is, the return distance represents the distance between the first coordinate of the lens and the target position. Afterwards, the mobile phone can calculate the ratio of the return distance to the frame interval to obtain the target return speed, so as to accurately determine the speed of the lens return action.

[0220] Optionally, the target return speed may be determined during a frame gap, such as when the frame gap is reached, the target return speed is determined.

[0221] The above describes how, when a mobile phone shakes during video recording, the mobile phone moves the lens within the frame to compensate for the shaking of the mobile phone, and how the mobile phone re-centers the lens between frames. The following will continue to describe the process of using EIS adaptive re-centering to avoid shaking in the video content.

[0222] An embodiment of the present application provides a shooting method. In this embodiment, due to the movement of the lens (such as after the lens is controlled to perform a centering action between frames), the lens position will change, causing the imaging center of the mobile phone camera to change, thereby causing the imaging position of the same photographed object to change, that is, causing the position of the same photographed object in two adjacent frames to change. From the user's perception, the video has shaken. Therefore, in order to ensure the de-shaking effect of the video, the mobile phone can use the EIS algorithm to adjust the position of the pixels on the captured image to compensate for the position offset caused by the movement of the lens (such as centering).

[0223] Exemplarily, the process of adjusting the position of the pixel on the image using the EIS algorithm can be adjusted by camera intrinsic parameters or camera extrinsic parameters. Among them, the process of adjusting the position of the pixel on the image using camera intrinsic parameters to achieve de-shaking can be introduced in the following part a. The process of adjusting the position of the pixel on the image using camera extrinsic parameters to achieve de-shaking can be introduced in the following part b.

[0224] a. Use camera internal parameters to remove jitter

[0225] In the embodiment of the present application, the mobile phone can control the displacement of the lens during the re-centering action to be converted into the offset of the center of the image plane, so as to determine the latest camera internal parameters using the offset of the center of the image plane, so that the latest camera internal parameters are adapted to the re-centering action, thereby accurately adjusting the position of the pixels on the image based on the latest camera internal parameters, compensating for the offset of the imaging position caused by the movement of the lens (such as re-centering), and improving the anti-shake effect of the video. Specifically, Fig.10 As shown, the process of de-shaking using camera internal parameters may include S201-S208.

[0226] S201, the mobile phone obtains the t+1th frame of the original image captured by the camera.

[0227] The original image may be an image captured by a camera, that is, an image sensor, which represents an image in which pixel coordinates are not adjusted.

[0228] S202, the mobile phone determines the offset corresponding to the center of the image plane based on the first coordinate and the second coordinate of the lens, wherein the first coordinate of the lens represents the initial position of the lens, and the second coordinate of the lens represents the position of the lens when the exposure of the t+1th frame image ends.

[0229] Among them, the initial position of the lens refers to the initial position of the lens without movement. The lens can be driven by a motor, and accordingly, the coordinates of the lens (such as the first coordinate and the second coordinate) can be represented by the coordinates of the motor. In other words, the initial position of the lens can also be the initial position of the motor. Generally speaking, the motor is initially located at the center, such as the center point of the moving range of the motor. Accordingly, the initial position of the motor can be (0,0). The second coordinate of the lens can also represent the position of the motor when the exposure of the t+1 frame image (i.e., the t+1 frame original image) is completed.

[0230] The above-mentioned image plane center refers to the center of the image plane coordinate system.

[0231] In the embodiment of the present application, the mobile phone calculates the difference between the second coordinate and the first coordinate of the lens to obtain the offset of the lens relative to the initial position when the exposure of the t+1 frame image ends. Generally speaking, the center of the image plane and the optical center (i.e., the center of the lens of the lens, in simple terms, the center of the lens) are on the same straight line. Therefore, the offset corresponding to the lens can also be used as the offset corresponding to the center of the image plane.

[0232] Exemplarily, since the lens can move on a two-dimensional plane, the first coordinate and the second coordinate of the lens can include coordinates in the x-direction and coordinates in the y-direction. The above offset can include an offset in the x-direction and an offset in the y-direction. For example, if the lens moves vertically, that is, moves along the y-axis, the offset in the x-direction of the lens is 0, and the offset in the y-direction is not 0, such as Fig.11 As shown, the lens moves vertically downward, and there is an offset in the y direction. For another example, the lens moves horizontally to the right, that is, moves along the positive x direction, and the offset of the lens in the y direction is 0.

[0233] S203: The mobile phone determines the first coordinates of the center of the image plane based on the initial coordinates of the center of the image plane and the offset corresponding to the center of the image plane.

[0234] In the embodiment of the present application, since the camera intrinsic parameters need to use the coordinates of the center of the image plane, the mobile phone needs to calculate the difference between the initial coordinates of the center of the image plane and the offset corresponding to the center of the image plane to obtain the first coordinates of the center of the image plane. The first coordinates of the center of the image plane represent the position of the center of the image plane when the exposure of the t+1 frame image ends.

[0235] Exemplarily, the offset corresponding to the center of the image plane may include an offset in the x direction and an offset in the y direction. The initial coordinate of the center of the image plane may include an initial coordinate in the x direction and an initial coordinate in the y direction. Correspondingly, the first coordinate of the center of the image plane also includes a first coordinate in the x direction and a first coordinate in the y direction. The mobile phone may use Formula 1: C' x =Cx -shift x , calculate the first coordinate of the x-direction of the center of the image plane. Among them, C' x The first x-coordinate of the center of the image plane, C x Indicates the initial x-coordinate of the center of the image plane, shift x Indicates the x-direction offset of the center of the image plane.

[0236] The mobile phone can be based on formula 2: C' y =C y -shift y , calculate the first coordinate of the image plane center in the y direction. Among them, C' y The first coordinate in the y direction of the center of the image plane, C y Indicates the initial coordinate of the image plane in the y direction, shift y Indicates the y-direction offset of the center of the image plane.

[0237] The initial coordinates of the center of the image plane represent the center of the image plane when the lens does not move initially. For example, if the resolution of the image is 4000*3000 and the upper left corner is generally used as the origin of the coordinates, the initial coordinates of the center of the image plane can be (2000, 1500). x , shift y ) can be 0, a positive number, or a negative number.

[0238] S204: The mobile phone determines a first camera intrinsic parameter based on the first coordinate of the center of the image plane and the distance corresponding to the original image of the t+1th frame.

[0239] The distance (also called image distance) indicates the distance between the image plane and the center of the lens. The distance corresponding to the t+1th frame image may be sent by the camera, which indicates the distance when the camera captured the t+1th frame original image (or t+1th frame image).

[0240] For example, after obtaining the first coordinate of the center of the image plane, the mobile phone can substitute the first coordinate of the center of the image plane and the distance corresponding to the t+1th frame image into the camera intrinsic parameter calculation formula to obtain the first camera intrinsic parameter (i.e., the latest camera intrinsic parameter). The camera intrinsic parameter calculation formula is: Specifically, after obtaining the first coordinate of the center of the image plane, the mobile phone can calculate the first coordinate of the center of the image plane in the x direction, that is, C' x Substitute into the camera internal parameter calculation formula C x At the position, the first coordinate of the y direction of the center of the image plane, that is, C' y Substitute into the above camera internal parameter calculation formula Cy , to obtain the first camera intrinsic parameter, that is, the latest camera intrinsic parameter.

[0241] It can be understood that the distance corresponding to the t+1th frame image and the distance corresponding to the tth frame image may be the same or different. The distance corresponding to the tth frame image refers to the distance when the camera captures the tth frame image.

[0242] After determining the internal parameters, the mobile phone needs to use the internal parameters to adjust the coordinates of the pixels in the original image of the t+1th frame to improve image stability. There are many ways to adjust the coordinates of the pixels in the original image of the t+1th frame using the camera internal parameters. The following will introduce a possible adjustment method through S205-S206.

[0243] S205: The mobile phone determines a first transformation matrix based on the first camera intrinsic parameter, wherein the first transformation matrix represents a conversion relationship between a world coordinate system and a pixel coordinate system.

[0244] Exemplarily, the mobile phone can substitute the first camera intrinsic parameter into the transformation matrix calculation formula to obtain the first transformation matrix. The transformation matrix calculation formula can be

[0245] Among them, K represents the camera internal parameter, K -1 represents the inverse matrix of K, and R represents the rotation matrix. Specifically, after obtaining the intrinsic parameters of the first camera, the mobile phone can substitute the intrinsic parameters of the first camera into the K in the transformation matrix calculation formula, that is, through The first transformation matrix is ​​calculated.

[0246] S206: The mobile phone adjusts the coordinates of the pixels in the t+1th frame original image based on the first transformation matrix to obtain the t+1th frame target image.

[0247] Exemplarily, for each pixel in the t+1th frame original image, the mobile phone can multiply the coordinates of the pixel by the first transformation matrix to adjust the position of the pixel, thereby obtaining the adjusted t+1th frame image, that is, the t+1th frame target image.

[0248] In the embodiment of the present application, the offset corresponding to the center of the image plane is obtained by subtracting the position of the lens at the end of exposure of the t+1 frame image from the initial position of the lens, and the offset corresponding to the center of the image plane already includes the lens offset caused by the inter-frame centering action between the t+1 frame image and the t frame image. That is, the movement of the lens (such as the movement within the frame and between frames) is converted into the change of the center of the image plane, so as to determine the latest camera internal parameters based on the coordinates of the center of the image plane after the change, so as to determine the first transformation matrix using the latest camera internal parameters. Afterwards, the mobile phone can use the first transformation matrix to adjust the position of each pixel on the original image of the t+1 frame, which can not only compensate for the imaging position offset caused by the inter-frame centering, but also compensate for the imaging position offset caused by the movement of the lens within the frame, thereby ensuring the stability of the image and the de-shaking effect of the video.

[0249] In some embodiments, in order to improve the de-jittering efficiency, the mobile phone may select multiple target coordinates from the original image of the t+1th frame. Afterwards, for each target coordinate, the mobile phone may multiply the target coordinate by the first transformation matrix to obtain an adjusted target coordinate to adjust a portion of the position on the image. Afterwards, the mobile phone may determine the adjusted coordinates of the pixels on the original image of the t+1th frame based on the adjusted target coordinates in combination with the interpolation method to adjust the pixel coordinates on the original image of the t+1th frame.

[0250] For example, the target coordinates may be the coordinates of the grid points in the original image of the t+1th frame. Fig.12 The coordinates of the grid points before deformation (i.e., the coordinates of the grid points before deformation) shown in FIG. 1 are used as the target coordinates. Afterwards, for each grid point before deformation, the mobile phone can multiply the coordinates of the grid point before deformation by the first transformation matrix to adjust the coordinates of the grid point before deformation, and obtain the following: Fig.12 The deformed coordinates of the grid points shown (i.e., the coordinates of the deformed grid points). Afterwards, the mobile phone can obtain the coordinates of the surrounding pixel points of the deformed grid points based on the coordinates of the deformed grid points in combination with the interpolation method, thereby obtaining the t+1th frame of the initial adjustment image. Afterwards, the mobile phone can crop the t+1th frame of the initial adjustment image to obtain the t+1th frame of the target image.

[0251] Of course, the above target coordinates may also be determined by other methods, such as the above target coordinates are the coordinates of pixels randomly selected by the mobile phone from the t+1th frame of the original image.

[0252] S207: The mobile phone receives a second trigger operation of the user on the shooting interface.

[0253] S208. The mobile phone stops recording the video in response to the second trigger operation, and generates a video file based on all the target images.

[0254] The second trigger operation may be a user clicking the stop button (such as a single click). The mobile phone may receive a user clicking the stop button at any time during the video recording process. In response to the clicking operation, the mobile phone ends the recording and generates a recorded video file. The video file includes the target image processed by the mobile phone during the video recording process.

[0255] In some embodiments, if no centering action is performed between frames, the mobile phone may not need to adjust the position of pixels on the next frame image, then the video file may also include an image captured by the camera on the mobile phone, which represents the image that has not been de-jittered using the first transformation matrix.

[0256] b. De-shaking using camera extrinsics

[0257] In the embodiment of the present application, the mobile phone can convert the displacement of the lens movement (such as performing a centering action) into the movement angle of the gyroscope, so as to use the movement angle of the gyroscope to determine the latest camera extrinsic parameters, so that the latest camera extrinsic parameters are adapted to the centering action, thereby optimizing the image path based on the latest camera extrinsic parameters, compensating for the offset of the imaging position caused by the centering, and improving the anti-shake effect of the video. Specifically, Fig.13 As shown, the process of de-shaking using camera extrinsic parameters may include S301-S309.

[0258] S301, the mobile phone obtains the t+1th frame of the original image captured by the camera.

[0259] S302: The mobile phone determines an offset corresponding to the center of the image plane based on the first coordinate and the second coordinate of the lens.

[0260] The process of determining the offset corresponding to the center of the image plane may refer to the corresponding content of S202 above, which will not be described in detail here.

[0261] S303, the mobile phone converts the offset corresponding to the center of the image plane into an equivalent angle of the gyroscope.

[0262] In the embodiment of the present application, the mobile phone can obtain the equivalent angle of the gyroscope based on the ratio between the offset and the distance corresponding to the center of the image plane, combined with the inverse trigonometric function, that is, convert the offset of the lens into the offset angle of the gyroscope, that is, the movement of the mobile phone body. In other words, the equivalent angle of the gyroscope represents the offset angle of the mobile phone body at the end of the exposure of the t+1 frame image. In other words, it can represent the offset angle of the lens in the mobile phone relative to the initial position at the end of the exposure of the t+1 frame image. Among them, the equivalent angle includes equivalent angles in the three directions of x, y and z, that is, the equivalent angle in the x direction, the equivalent angle in the y direction and the equivalent angle in the z direction.

[0263] The mobile phone can use the ratio of the x-direction offset and the distance of the center of the image plane, combined with the inverse trigonometric function, to determine the equivalent angle of the gyroscope in the x-direction. Fig.14 As shown, the mobile phone can be based on the inverse trigonometric function arctan (shift x / f), calculate the equivalent angle of the gyroscope in the x direction, which is an equivalent angle, not the actual angle of the gyroscope. Similarly, the mobile phone can use the ratio of the offset in the y direction of the image plane center and the distance, combined with the inverse trigonometric function, to determine the equivalent angle of the gyroscope in the y direction. Since the image plane center moves on a two-dimensional plane and it does not have a z-axis, the mobile phone needs to convert the two-dimensional coordinates into three-dimensional coordinates. There is a certain correspondence between the x-axis and the z-axis, so the offset in the x-direction of the image plane center can be converted into the offset in the z-direction, so that the offset in the z-direction can be used to determine the equivalent angle of the gyroscope in the z-direction.

[0264] S304, the mobile phone obtains an adjustment angle corresponding to the t+1th frame of the original image based on the equivalent angle, where the adjustment angle indicates the angle required to adjust the t+1th frame of the original image to achieve an ideal shooting angle.

[0265] In some embodiments, the mobile phone can determine the adjustment angle using the equivalent angle of the gyroscope corresponding to the t+1th frame of the original image (or the t+1th frame of the image), the original angle corresponding to the gyroscope, and the target angle (or the ideal shooting angle) corresponding to the t+1th frame of the image. Among them, the original angle represents the actual angle of the gyroscope when the t+1th frame of the image is exposed, that is, the angle of the movement of the mobile phone body (or the moving angle), which is obtained by integrating the angular velocity data collected by the gyroscope in the mobile phone during the exposure of the t+1th frame of the image. The target angle can represent the angle of the gyroscope when the t+1th frame of the image obtained by the mobile phone does not have a jitter when shooting an image based on the target angle, that is, the angle of the gyroscope when the mobile phone is shooting steadily. In other words, the target angle corresponding to the t+1th frame of the image represents the ideal shooting angle corresponding to the t+1th frame of the image when the previous t+1 frames of the image are all stable images. The mobile phone can use an optimization algorithm (such as a filtering algorithm) to optimize the path of the t+1th frame of the image in combination with the previous t frames of the image to obtain the target angle corresponding to the t+1th frame of the image.

[0266] It is understandable that when the OIS motor does not move, the angle of the gyroscope is consistent with the angle of the lens, then the angle of the gyroscope is the shooting angle of the lens, that is, the original angle corresponding to the above-mentioned t-th frame image is the shooting angle of the lens. If the OIS motor moves, the shooting angle of the lens is equal to the angle of the gyroscope superimposed on the moving angle of the lens. Alternatively, since the angle (such as the equivalent angle mentioned above) is signed, the difference in angles can also be replaced by addition, such as the shooting angle of the lens is equal to the angle of the gyroscope minus the moving angle of the lens, but it is just a change in the direction corresponding to the pre-defined positive and negative signs.

[0267] Exemplarily, the implementation process of the above S304 may include: first, the mobile phone may obtain the original angle corresponding to the t+1th frame image.

[0268] Afterwards, since the lens in the mobile phone may move between frames or within frames, it is necessary to use the lens offset to determine the shooting angle of the lens. That is, the mobile phone can calculate the difference between the original angle corresponding to the t+1th frame image and the above equivalent angle to obtain the actual angle corresponding to the t+1th frame image. The actual angle is the shooting angle after compensating for the lens offset, that is, the actual angle represents the actual shooting angle when the lens exposes the t+1th frame image. The actual angle can be simply understood as the shooting angle that compensates for the lens offset between frames, that is, the shooting angle corresponding to the tth frame image.

[0269] Afterwards, the mobile phone calculates the difference between the actual angle corresponding to the t+1th frame image and the target angle corresponding to the t+1th frame image to obtain the adjustment angle. The adjustment angle represents the angle that should be adjusted in order to make the shooting angle of the lens reach the target angle.

[0270] For example, the t+1th frame image is the second frame image. The mobile phone uses the angular velocity data collected by the gyroscope during the exposure of the second frame image to obtain the angle of the gyroscope when the second frame image is exposed, and obtains the original angle corresponding to the second frame image. Moreover, when the exposure of the second frame image is completed, the mobile phone can convert the offset of the lens relative to the initial position to the offset corresponding to the center of the image plane based on the current position of the lens (or described as coordinates) and the initial position, so as to use the offset corresponding to the center of the image plane and the distance corresponding to the second frame image to determine the movement angle of the lens, that is, to obtain the equivalent angle of the gyroscope.

[0271] Afterwards, the mobile phone can subtract the equivalent angle from the original angle to obtain the shooting angle when the lens exposes the second frame of the image, that is, to obtain the actual angle corresponding to the second frame of the image, thereby accurately determining the shooting angle.

[0272] Afterwards, in order to ensure the stability of the video image, the mobile phone can calculate the difference between the actual angle corresponding to the second frame image and the target angle corresponding to the second frame image, and obtain the adjustment angle, so as to use the adjustment angle to adjust the position of the pixels in the second frame image. It can not only compensate for the imaging position offset caused by inter-frame centering, but also compensate for the imaging position offset caused by the movement of the lens within the frame, thereby ensuring image stability.

[0273] S305: The mobile phone obtains a first rotation matrix based on the above adjustment angle and a preset rotation matrix calculation formula.

[0274] Exemplarily, the mobile phone can substitute the adjustment angle into the preset rotation matrix calculation formula to obtain the first rotation matrix. The preset rotation matrix calculation formula can be the Rodriguez rotation matrix formula, that is, Wherein, R represents the rotation matrix, x represents the adjustment angle of the gyroscope in the x direction, y represents the adjustment angle of the gyroscope in the y direction, and z represents the adjustment angle of the gyroscope in the z direction.

[0275] After determining the first rotation matrix, the mobile phone needs to use the first rotation matrix to adjust the coordinates of the pixels in the original image of the t+1th frame to improve image stability. There are many ways to adjust the coordinates of the pixels in the original image of the t+1th frame using the first rotation matrix. A possible adjustment method will be introduced below through S306-S307.

[0276] S306: The mobile phone determines a second transformation matrix based on the first rotation matrix.

[0277] S307: The mobile phone adjusts the coordinates of the pixels in the above-mentioned t+1th frame original image based on the second transformation matrix to obtain the t+1th frame target image.

[0278] Exemplarily, the mobile phone substitutes the first rotation matrix into the above transformation matrix calculation formula to obtain the second transformation matrix. Since the angular velocity data of the gyroscope is in the world coordinate system, and the position of the pixel on the image is in the pixel coordinate system, it is necessary to establish a relationship between the relevant angles of the gyroscope, such as the above adjustment angle, and the pixel coordinate system. In an embodiment of the present application, the above adjustment angle is converted from the world coordinate system to the pixel coordinate system through the second transformation matrix, and finally mapped to the adjustment of the position of the pixel on the image, so as to realize the smoothing of the gyroscope angle, that is, the shooting angle, and avoid the offset of the imaging position of the same photographed object in different frames, so as to realize the optimization processing of the image path, thereby realizing the de-shaking processing of the video.

[0279] S308, the mobile phone receives a second trigger operation of the user on the shooting interface.

[0280] Exemplarily, the second trigger operation may be a user clicking a stop button on the shooting interface. The second trigger operation may also be other operations, such as a gesture operation, which is not limited in the present application.

[0281] Of course, the second trigger operation introduced above is only an example of the second operation that can trigger the mobile phone to stop recording video. The second operation can also be other operations input by the user, and this application does not limit it.

[0282] S309: The mobile phone stops recording the video in response to the second trigger operation, and generates a video file based on all the target images.

[0283] In the embodiment of the present application, during the video recording process, if Fig.15 As shown, the angles of the five frames of images f0, f1, f2, f3, f4, and f5 taken by the mobile phone are angle a0, angle a1, angle a2, angle a3, and angle a4, respectively. It can be seen that the shooting angles of these five frames of images vary greatly, and the mobile phone has shaken. In order to avoid the blurring of the imaging picture caused by the shaking of the mobile phone, the mobile phone can move the camera lens to compensate for the shaking of the mobile phone, and perform a back-centering action, so that the angles of the five frames of images f0, f1, f2, f3, f4, and f5 taken by the mobile phone are equivalent to angle b0, angle b1, angle b2, angle b3, and angle b4. Although the angles of angle b0, angle b1, angle b2, angle b3, and angle b4 vary less than angle a0, angle a1, angle a2, angle a3, and angle a4, there are still large differences.

[0284] Therefore, in order to achieve de-shaking, the mobile phone can adapt the camera extrinsic parameters to the centering action, so as to use the adapted camera extrinsic parameters to adjust the position of the pixels on the image, that is, the movement angle of the fuselage itself when taking the image is compensated by the angle of deviation between the lens and the initial position due to the movement of the lens (that is, the equivalent angle corresponding to the image), and the actual angle of the lens when taking the image is obtained. Therefore, by adjusting the actual angle, the five frames of images f0, f1, f2, f3, f4, and f5 are equivalent to being shot based on angles c0, c1, c2, c3, and c4, so as to achieve smooth processing of the shooting angle, which can not only compensate for the deviation of the lens caused by inter-frame jitter, but also compensate for the deviation of the lens caused by intra-frame jitter, that is, the change in shooting angle, so as to avoid the large difference in the position of the same object in different frames due to large changes in shooting angles (such as Fig.16 shown in the figure).

[0285] It can be understood that the above angles a0, a1, a2, a3 and a4 are equivalent to the original angles corresponding to the image, that is, the angles of the gyroscope. The above angles b0, b1, b2, b3 and b4 are equivalent to the actual angles corresponding to the image. The above angles c0, c1, c2, c3 and c4 are equivalent to the target angles corresponding to the image.

[0286] In some embodiments, the mobile phone can choose to use the camera intrinsic parameter de-shaking method or the camera extrinsic parameter de-shaking method to adjust the position of the pixels on the image according to the actual situation. For example, the mobile phone can randomly select the camera intrinsic parameter de-shaking method or the camera extrinsic parameter de-shaking method to adjust the position of the pixels on the image. For another example, the method of using the camera intrinsic parameter de-shaking only regards the movement of the lens as the translation of the center of the image plane, which cannot be used for large-angle perspective distortion and image rotation. Therefore, in the case of large-angle perspective distortion and image rotation, the mobile phone can use the camera extrinsic parameter de-shaking method to adjust the position of the pixels on the image.

[0287] It should be noted that in the above-mentioned de-shaking using camera intrinsic parameters, it is the camera intrinsic parameters that are changed based on the movement of the lens, and the camera extrinsic parameters, that is, the calculation of the rotation matrix will not take into account the movement of the lens, but the rotation matrix can also be calculated according to the Rodriguez rotation matrix formula, except that the gyroscope adjustment angle is equal to the difference between the original angle corresponding to the image and the target angle corresponding to the image. Similarly, in the above-mentioned de-shaking using camera extrinsic parameters, it is the camera extrinsic parameters that are changed based on the movement of the lens, and the calculation of the camera intrinsic parameters will not take into account the movement of the lens, but directly substitute the initial coordinates of the center of the image plane into the calculation formula of the camera intrinsic parameters.

[0288] The above describes the process of using the EIS algorithm to adjust the position of pixels in the collected image after the mobile phone controls the lens to perform the re-centering action. The mobile phone can also use the EIS algorithm alone to adjust the position of pixels in the collected image to improve the image quality. In other words, even if the mobile phone does not control the lens to perform the re-centering action between frames to achieve de-shaking, the mobile phone can still use the above EIS algorithm to adjust the position of pixels in the collected image to achieve de-shaking of the image, thereby improving the image quality.

[0289] In some embodiments, the mobile phone can directly use the trained network model to process the t+1th frame of the original image to obtain the t+1th frame of the target image. Figure 4 , the process of obtaining the target image of the t+1th frame by using the network model is introduced. Fig.17 As shown, the process may include S401-S417.

[0290] S401, the camera application in the mobile phone receives a click operation of a user on a video shooting button.

[0291] S402, the camera application sends a video capture request to the camera hardware abstraction layer in the mobile phone in response to a click operation on the video capture button, wherein the video capture request is used to trigger the mobile phone camera to capture a video.

[0292] S403: The camera hardware abstraction layer sends a video capture request to the mobile phone camera.

[0293] Among them, the specific process of the camera application sending a video shooting request to the mobile phone camera can refer to the process of the corresponding application sending a video shooting request to the camera through the camera access interface, the camera hardware abstraction layer, and the camera device driver introduced above.

[0294] S404, the camera receives the video shooting request.

[0295] S405, the camera captures an image in response to the video shooting request.

[0296] S406, the gyroscope in the mobile phone sends angular velocity data to the processor and camera hardware abstraction layer in the mobile phone.

[0297] For example, the gyroscope in the mobile phone detects the movement of the mobile phone body, and can collect angular velocity data in real time or periodically, and save the collected angular velocity data to a queue. Other modules or processes (such as the camera hardware abstraction layer, the processor) can read the angular velocity data from the queue.

[0298] S407: The processor sends a compensation instruction to the camera based on the angular velocity data of the gyroscope.

[0299] The processor may be the above Figure 3A A processor 110 is shown.

[0300] S408: The camera controls the lens to move in the reverse direction in response to the compensation instruction.

[0301] For example, the processor can determine whether the mobile phone is shaking based on the angular velocity data of the gyroscope. If shaking occurs, the lens is controlled to move in the opposite direction by driving the motor to compensate for the shaking of the mobile phone. If shaking does not occur, there is no need to control the lens to move in the opposite direction.

[0302] In some embodiments, during the exposure of the t-th frame image (i.e., the t-th frame original image), the gyroscope in the camera will send multiple angular velocities, that is, send the angular velocities corresponding to different moments. When the angular velocities corresponding to different moments are slightly different, such as the difference between adjacent angular velocities is less than a preset difference, it indicates that the mobile phone is not shaking. When the angular velocities corresponding to different moments are significantly different, such as the difference between adjacent angular velocities is greater than or equal to a preset difference, it indicates that the mobile phone is shaking.

[0303] S409, after obtaining the timestamp corresponding to the last row of pixels of the t-th frame image, the processor sends a return-to-center instruction to the camera, wherein the return-to-center instruction includes a target position and a target return-to-center speed.

[0304] When the mobile phone obtains the timestamp corresponding to the last row of pixels of the t-th frame image, it determines that the exposure of the t-th frame image is completed.

[0305] S410, the camera receives a return instruction.

[0306] S411, the camera responds to the above return-to-center instruction and controls the lens to move to the target position at a target return-to-center speed.

[0307] For example, when the exposure of the tth frame image is completed, the processor can send a re-centering instruction to the camera to trigger the camera to perform a re-centering action. The camera can move the lens to the target position by driving the motor, such as (51, 175). During the movement of the motor, the motor can report the coordinates of the motor, such as (53, 172), to the camera application abstraction layer and the processor in real time or periodically.

[0308] S412, when the interval time between the t-th frame image and the t+1-th frame image ends, the camera starts to expose the t+1-th frame image.

[0309] S413. After obtaining the timestamp corresponding to the last row of pixels of the t+1th frame image, the camera sends the second coordinate of the motor, the t+1th frame original image and the parameters corresponding to the t+1th frame original image to the camera hardware abstraction layer.

[0310] When the mobile phone obtains the timestamp corresponding to the last row of pixels of the t+1th frame image, it determines that the exposure of the t+1th frame image is completed. The above-mentioned t+1th frame original image includes the data of the pixel points in the t+1th frame image (such as the position of the pixel points). The parameters corresponding to the above-mentioned t+1th frame original image represent the parameters when the camera captures the image. Exemplarily, the image parameters corresponding to the t+1th frame original image may include one or more of resolution, field of view (FOV) magnification, exposure time, and rolling shutter time.

[0311] It should be understood that the interaction between the camera and the camera hardware layer can be implemented through the camera device driver. For example, the camera can send the angular velocity data of the gyroscope to the camera hardware abstraction layer through the camera device driver. Similarly, the camera hardware abstraction layer can send compensation instructions to the camera through the camera device driver.

[0312] S414, the camera hardware abstraction layer uses the angular velocity data, the first coordinate of the motor, the second coordinate of the motor, the camera parameters, the original image of the t+1th frame, and the parameters corresponding to the original image of the t+1th frame as inputs of a preset network model, runs the preset network model, and obtains the target image of the t+1th frame.

[0313] The preset network model is used to adjust the coordinates of the pixels in the image based on the original image of the t+1th frame, combined with the intrinsic reference de-shaking method or the extrinsic reference de-shaking method, to obtain the target image. The intrinsic reference de-shaking method can refer to the relevant content of using the camera intrinsic reference de-shaking described above, and the extrinsic reference de-shaking method can refer to the relevant content of using the camera extrinsic reference de-shaking described above.

[0314] The above-mentioned camera parameters include the distance (such as the distance corresponding to the original image of the t+1th frame) and the initial coordinates of the optical center. The camera parameters can be sent by the camera or pre-stored. In addition, the distance can also be included in the parameters corresponding to the original image of the t+1th frame, and the camera parameters do not need to include the distance. Similarly, the first coordinate of the motor, that is, the initial position of the motor can also be sent by the camera or pre-stored.

[0315] It should be understood that the image data corresponding to the above-mentioned t+1th frame of the original image can be used to align the angular velocity data of the gyroscope at the moment. For example, the angular velocity data of the gyroscope collected during the exposure of the t+1th frame of the original image is corresponded to the t+1th frame of the image; for another example, each row of pixels in the t+1th frame of the original image is corresponded to the angular velocity data of the gyroscope collected during the exposure of the t+1th frame of the original image; for another example, each pixel in the t+1th frame of the original image is corresponded to the angular velocity data of the gyroscope collected during the exposure of the t+1th frame of the original image, so as to avoid using the wrong angular velocity data of the gyroscope for calculation. For example, the angular velocity data of the gyroscope collected previously cannot be applied to the current frame of the image, resulting in errors in the calculation of related data corresponding to the current frame of the image, such as the original angle.

[0316] In addition, the original angle of the gyroscope used in the process of determining the target image of the t+1th frame may be obtained by integrating the angular velocity data of the gyroscope corresponding to the t+1th frame image.

[0317] S415. The camera hardware abstraction layer sends the t+1th frame of the target image to the camera application.

[0318] Exemplarily, the camera hardware abstraction layer may send the t+1th frame of the target image to the camera application via the camera access interface. After receiving the t+1th frame of the target image, the camera application may save and / or display it, such as displaying it in a preview area.

[0319] It should be understood that after the above-mentioned camera captures the t-th frame of the original image, it can also perform relevant processing on the t-th frame of the original image to obtain the t-th frame of the target image. That is, the mobile phone can not only de-jitter the image captured after the back-to-center action, but also de-jitter the image captured without the back-to-center action. In other words, the mobile phone can de-jitter each frame of the original image captured by the camera to ensure the de-jitter effect of the video.

[0320] S416: The camera application receives a click operation on a stop button by the user.

[0321] S417: In response to a click operation on the stop button, the camera application stops recording the video and generates a video file based on all the target images.

[0322] In an embodiment of the present application, during the video recording process, after receiving the image captured by the camera (or described as the original image), the hardware abstraction layer can call the EIS algorithm in the camera algorithm library, that is, input the image and the image data corresponding to the image, the coordinates of the motor and the camera parameters into the preset network model, so that the preset network model adjusts the coordinates of the pixels in the image through the internal parameter de-jittering method or the external parameter de-jittering method, obtains and outputs the target image, realizes rapid de-jittering of the original image, and avoids the optical center position (or shooting angle) of the camera when it starts to expose the image due to the return action, thereby causing the imaging position of the object in the previous frame image to shift on the next frame image.

[0323] In some embodiments, the mobile phone is fixed on a vibration table, and videos can be recorded at different frame rates under the same lighting environment, exposure time, and amplitude to obtain videos at different frame rates. If the tremor degree, that is, the blur degree, of the videos at different frame rates is similar, it can be determined that the mobile phone uses an inter-frame centering solution.

[0324] Optionally, when the mobile phone adopts the inter-frame centering solution, it can be determined whether the image frame in the video is offset, that is, a jitter problem. If no jitter problem occurs, the mobile phone may adopt the EIS algorithm proposed in this application to adapt the centering action.

[0325] In some embodiments, the present application provides a computer-readable storage medium, including computer instructions, which, when executed on a Bluetooth device, enable the electronic device to execute the text processing method as described above.

[0326] In some embodiments, the present application provides a computer program product, which, when executed on a Bluetooth device, enables the electronic device to execute the text processing method as described above.

[0327] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0328] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0329] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0330] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0331] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0332] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A shooting method, characterized in that: Applied to an electronic device, the electronic device includes a camera; the method includes: When the electronic device is in a multi-frame shooting state, during the exposure of the t-th frame image, in response to the electronic device shaking, the electronic device moves the lens in the camera to a first position in a first direction, wherein the multi-frame shooting state represents a state of continuously acquiring and processing multiple frames of images, the first direction is a direction opposite to the shaking, and t is a positive integer; In response to the end of exposure of the t-th frame image, if the position of the lens at the end of exposure of the t-th frame image is not within a first preset position range, the electronic device controls the lens to move from the first position to the center point within a first target time, wherein the first target time is the interval time between the t-th frame image and the t+1-th frame image; the center point is the center point of the moving range of the lens; the absolute value of the coordinate of the position point in the x direction in the first preset position range is less than the first preset coordinate value, and the absolute value of the coordinate of the position point in the y direction is less than the second preset coordinate value; the center point is within the first preset position range; The electronic device displays the tth frame image.

2. The method according to claim 1, characterized in that The method further comprises: When the position of the lens at the end of exposure of the t-th frame image is within a first preset position range, the electronic device does not control the lens to move from the first position to the center point within the first target time.

3. The method according to claim 1, characterized in that The first target time includes a frame gap; Controlling the lens to move from the first position to the center point within the first target time includes: In the frame interval of the first target time, the electronic device controls the lens to move from the first position to the center point and to the target position.

4. The method according to claim 3, characterized in that: The electronic device controls the lens to move from the first position to the center point, including: The electronic device acquires the target return speed corresponding to the frame interval; The electronic device controls the lens to move from the first position to the center point at a target return speed corresponding to the frame interval; or, The electronic device calculates a ratio between a return-to-center distance and the frame interval to obtain a target return-to-center speed, wherein the return-to-center distance represents a distance between a first coordinate of the lens and the target position, and the first coordinate of the lens represents a position of the lens when the exposure of the t-th frame image ends; The electronic device controls the lens to move from the first position to the center point according to the target return-to-center speed.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: After the exposure of the t+1 frame image is completed, the electronic device adjusts the position of the pixels in the t+1 frame image based on the offset corresponding to the lens and in combination with the electronic image stabilization technology EIS algorithm to obtain the t+1 frame target image; the offset corresponding to the lens represents the offset value of the position of the lens at the end of the exposure of the t+1 frame image compared to the initial position of the lens; The electronic device displays and / or stores the target image of the t+1th frame.

6. The method according to claim 5, characterized in that The offset corresponding to the lens represents the offset corresponding to the center of the image plane corresponding to the camera; The electronic device adjusts the position of the pixel in the t+1 frame image based on the offset corresponding to the lens and in combination with the electronic image stabilization technology EIS algorithm to obtain the t+1 frame target image, including: The electronic device determines the first coordinate of the center of the image plane based on the initial coordinate of the center of the image plane and the offset corresponding to the center of the image plane; wherein the first coordinate of the center of the image plane represents the position of the center of the image plane when the exposure of the t+1th frame of image ends; The electronic device determines a first camera intrinsic parameter based on a first coordinate of the center of the image plane and a distance corresponding to the camera; the distance represents a distance between the image plane and the center of the lens; The electronic device adjusts the positions of pixels in the t+1th frame image according to a first transformation matrix; the first transformation matrix is ​​obtained based on the first camera intrinsic parameter.

7. The method according to claim 5, characterized in that The offset corresponding to the lens represents the offset corresponding to the center of the image plane corresponding to the camera; The electronic device adjusts the position of the pixel in the t+1 frame image based on the offset corresponding to the lens and in combination with the electronic image stabilization technology EIS algorithm to obtain the t+1 frame target image, including: The electronic device converts the offset corresponding to the center of the image plane into an equivalent angle of the gyroscope in the camera; the equivalent angle represents the offset angle of the electronic device when the exposure of the tth frame ends; The electronic device obtains an adjustment angle corresponding to the t+1th frame image based on the equivalent angle, where the adjustment angle represents an angle that needs to be adjusted when the t+1th frame image reaches an ideal shooting angle; The electronic device obtains a first rotation matrix based on the adjustment angle and a preset rotation matrix calculation formula; The electronic device adjusts the positions of the pixels in the t+1th frame image according to a second transformation matrix; the second transformation matrix is ​​obtained based on the first rotation matrix.

8. The method according to claim 7, characterized in that The electronic device obtains the adjustment angle corresponding to the t+1th frame image based on the equivalent angle, including: The electronic device calculates the difference between the original angle corresponding to the t+1th frame image and the equivalent angle to obtain the actual angle corresponding to the t+1th frame image; wherein the original angle corresponding to the t+1th frame image represents the angle of movement of the electronic device when the t+1th frame image is exposed; and the actual angle corresponding to the t+1th frame image represents the actual shooting angle of the lens when exposing the t+1th frame image; The electronic device calculates the difference between the actual angle corresponding to the t+1th frame image and the target angle corresponding to the t+1th frame image to obtain the adjustment angle corresponding to the t+1th frame image; the target angle corresponding to the t+1th frame image represents the ideal shooting angle corresponding to the t+1th frame image when the previous t+1 frames of images are all stable images.

9. The method according to claim 7 or 8, characterized in that: The converting the offset corresponding to the center of the image plane into an equivalent angle of a gyroscope in the camera includes: The electronic device obtains the equivalent angle based on the ratio between the offset corresponding to the center of the image plane and the distance corresponding to the camera in combination with an inverse trigonometric function.

10. The method according to claim 9, characterized in that The offset corresponding to the center of the image plane includes the offset of the center of the image plane in the x direction, and the equivalent angle includes the equivalent angle in the x direction; The equivalent angle in the x direction is obtained by the inverse trigonometric function arctan (shift x / f) calculated, the shift x represents the offset of the center of the image plane in the x direction, and f represents the distance.

11. The method according to any one of claims 7 to 10, characterized in that The electronic device adjusts the positions of pixels in the t+1th frame image according to the second transformation matrix, including: For each pixel in the t+1th frame image, the electronic device multiplies the coordinates of the pixel by the second transformation matrix.

12. The method according to claim 5, characterized in that The offset corresponding to the lens represents the offset corresponding to the center of the image plane corresponding to the camera; The electronic device adjusts the position of the pixel in the t+1 frame image based on the offset corresponding to the lens and in combination with the electronic image stabilization technology EIS algorithm to obtain the t+1 frame target image, including: The electronic device uses the offset corresponding to the center of the image plane and the distance corresponding to the camera as input parameters of a preset network model, runs the preset network model, and obtains the t+1th frame target image; the preset network model is obtained based on the EIS algorithm, and the preset network model is used to adjust the position of pixels in the image based on the offset corresponding to the center of the image plane and the distance.

13. The method according to any one of claims 1 to 12, characterized in that The electronic device controls the lens to move from the first position to the center point within a first target time, including: The electronic device controls the lens to move from the first position to the center point to the target position within a first target time, and the target position includes the center point and / or a first preset position of the lens on the movement range during the exposure of the tth frame image.

14. The method according to any one of claims 5 to 13, characterized in that The multi-frame shooting state includes a video shooting state; Before the exposure period of the t-th frame image, the method further includes: The electronic device receives a first operation, where the first operation is used to trigger video recording; After the electronic device obtains the t+1th frame of the target image, the method further includes: The electronic device receives a second operation; In response to the second operation, the electronic device stops recording the video and generates a video file based on all the obtained target images.

15. An electronic device, characterized in that: The electronic device includes a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; the display screen is used to display an image generated by the processor, and the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the method described in any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 14.

Images