Photographing method and electronic device

Abstract

The present application relates to the technical field of photographing. Provided are a photographing method and an electronic device. The electronic device collects an image in a multi-frame photographing state. During the exposure of a t-th image frame, if the electronic device shakes, the electronic device can reversely move a lens in a camera, so as to realize intra-frame jitter compensation; then, in response to the end of the exposure of the t-th image frame, the electronic device can determine whether the position of the lens at the end of the exposure of the t-th image frame is within a first preset position range, so as to determine whether the degree of movement of the lens during the exposure of the t-th image frame is relatively great; and if the position of the lens is not within the first preset position range, it is indicated that the degree of movement of the lens during the exposure of the t-th image frame is relatively great, the electronic device can control, on the basis of an interval time between the t-th image frame and a (t+1)th image frame, the lens to move from a first position to a center point, thereby ensuring that the lens has a relatively large activity range during the exposure of the (t+1)th image frame, and the lens being controlled to perform unnecessary movement can be avoided.

Description

Shooting method and electronic equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on July 29, 2023, with application number 202310957013.7 and invention name “A Shooting Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of photographing technology, and in particular to a photographing method and electronic device. Background Art

[0003] With the development of technology, the photography function of electronic devices (such as mobile phones) has also developed rapidly. When using a mobile phone to shoot videos, the mobile phone (or described as a mobile phone camera) may shake, which may cause blurred images and result in low video quality.

[0004] Currently, mobile phones can use optical image stabilization (OIS) to improve video quality. During the video exposure process, if the mobile phone camera shakes, the phone can control the camera lens to move in the opposite direction to compensate for the shake, thus achieving de-shaking.

[0005] However, due to the limited range of lens movement, when the phone shakes, the phone may not be able to compensate for the shake, resulting in blurred imaging, resulting in low quality of the captured video.

[0006] Summary of the Invention

[0007] Embodiments of the present application provide a shooting method and electronic device for improving the quality of shot videos.

[0008] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0009] On the first aspect, a shooting method is provided. When an electronic device is in a multi-frame shooting state, during the exposure of the t-th frame image, if the electronic device shakes, in response to the shaking of the electronic device, the electronic device can move the lens in the camera of the electronic device to a first position in a first direction, wherein the multi-frame shooting state indicates a state of continuously capturing and processing multiple frames of images, the first direction is a direction opposite to the shaking, and t is a positive integer.

[0010] Afterwards, in response to the completion of exposure of the t-th frame image, the electronic device may determine whether the position of the lens at the time of exposure of the t-th frame image is within a first preset position range. If the position of the lens at the completion of exposure of the t-th frame image is not within the first preset position range, it indicates that the lens has moved significantly during the exposure of the t-th frame image. The electronic device controls the lens to move from the first position to the center point within a first target time, where the first target time is the interval between the t-th frame image and the t+1-th frame image, the center point is the center point of the lens's movement range, the absolute value of the x-coordinate of the position point within the first preset position range is less than the first preset coordinate value, and the absolute value of the y-coordinate of the position point is less than the second preset coordinate value; if the center point is within the first preset position range, then the first preset position range is within the lens's movement range.

[0011] The electronic device displays the t-th frame image.

[0012] In the present application, if the electronic device experiences jitter during the exposure of the t-th frame, the electronic device moves the lens in the opposite direction to compensate for intra-frame jitter, thereby preventing blur in the t-th frame caused by the jitter. After the exposure of the t-th frame, the current position of the lens—that is, the position of the lens at the end of the exposure of the t-th frame—is determined to be within a first preset position range. Specifically, it is determined whether the lens has moved significantly during the exposure of the t-th frame. In other words, it is determined whether recentering is necessary. If it is not within the first preset position range, it indicates that the lens has moved significantly and its range of motion in the next frame is small. Therefore, to ensure the range of motion of the lens in the next frame, the electronic device can control the lens to perform a recentering action during the interval between the t-th frame and the next frame. Because the lens is recentered between frames, the lens has a larger range of motion when the next frame is exposed, avoiding the problem of blur in the next frame due to the limited range of motion of the lens, which makes it impossible to compensate for jitter by moving the lens, and thus avoids shaky images (such as those in a video). When the multi-frame capture state is a video state, the quality of the captured video can be improved. Furthermore, by determining whether re-centering is necessary, unnecessary re-centering of the lens can be avoided, thereby reducing resource waste. In one possible implementation of the first aspect, if the position of the lens at the end of exposure of the aforementioned t-th frame image is within a first preset position range, this indicates that the degree of movement of the lens during the exposure of the t-th frame image is small, i.e., the degree of lens shake compensation during the exposure of the t-th frame image is small, and the range of movement of the lens in the next frame is still large. Therefore, the electronic device may not control the lens to move from the first position to the center point within the first target time, i.e., it may not control the lens to re-center, thereby avoiding unnecessary re-centering of the lens and reducing resource waste.

[0013] In a possible implementation of the first aspect, the electronic device may control the lens to move from the first position to the center point within the frame interval of the first target time to achieve lens re-centering.

[0014] In a possible implementation of the first aspect, the above-mentioned electronic device can control the lens to move from the first position to the center point, and then to the target position, where the target position includes the center point and / or a first preset position on the movement stroke (or called stroke) of the lens during the exposure of the t-th frame image, such as 1 / 4 stroke.

[0015] In a possible implementation of the first aspect, the electronic device may control the lens to perform a centering action according to a target centering speed. The target centering speed may be determined by the following two implementations.

[0016] In one implementation, the electronic device may obtain the target centering speed corresponding to the frame interval, thereby quickly obtaining the target centering speed.

[0017] In another implementation, the electronic device can calculate the ratio between the return distance and the frame gap to obtain the target return speed, where the return distance represents the distance between the first coordinate of the lens and the target position, and the first coordinate of the lens represents the position of the lens at the end of exposure of the t-th frame image, thereby avoiding the target return speed from being too large or too small, and achieving accurate determination of the target return speed.

[0018] In a possible implementation of the first aspect, after the exposure of the t+1th frame image is completed, such as in response to the completion of the exposure of the t+1th frame image, the electronic device can adjust the positions of the pixels in the t+1th frame image based on the offset corresponding to the lens in combination with the EIS algorithm to obtain the t+1th frame target image, where the offset corresponding to the lens represents the offset value of the position of the lens at the end of the exposure of the t+1th frame image compared to the initial position of the lens.

[0019] Afterwards, the electronic device may display and / or save the t+1th frame of the target image.

[0020] In this application, due to lens movement (such as controlling the lens to perform a re-centering action between frames), the lens position will change, causing the imaging center of the electronic device camera to change, thereby changing the imaging position of the same photographed object, that is, the position of the same photographed object in two adjacent frames will change. From the user's perspective, the image is jittered. Therefore, the electronic device can use the EIS algorithm to adjust the position of the pixels on the collected image (such as the t+1 frame image) to compensate for the position offset caused by lens movement due to various reasons (such as re-centering) and improve the image de-shaking effect.

[0021] Optionally, the offset corresponding to the lens represents the offset corresponding to the center of the image plane corresponding to the camera.

[0022] In one possible implementation of the first aspect, the electronic device may adjust the positions of pixels in the image based on an EIS algorithm in combination with an internal reference de-jittering method. The electronic device determines a first coordinate of the image plane center based on the initial coordinate of the image plane center and an offset corresponding to the image plane center. The first coordinate of the image plane center represents the position of the image plane center at the end of exposure of the t+1th frame of image.

[0023] The electronic device then determines a first camera intrinsic parameter based on the first coordinate of the center of the image plane and the distance corresponding to the camera. The distance represents the distance between the image plane and the center of the lens. The distance corresponding to the camera can be the distance corresponding to the t+1th frame image, that is, the distance between the cameras when the t+1th frame image was captured.

[0024] Afterwards, the electronic device adjusts the position of the pixels in the t+1th frame image according to the first transformation matrix; the first transformation matrix is ​​obtained based on the first camera intrinsic parameter, wherein the first transformation matrix represents the conversion relationship between the world coordinate system and the pixel coordinate system.

[0025] In this application, the offset corresponding to the image plane center is obtained by subtracting the lens position at the end of exposure of the t+1 frame from the initial position of the lens. This offset corresponding to the image plane center already includes the lens offset caused by the inter-frame re-centering movement between the t+1 frame and the t frame. In other words, the movement of the lens (such as movement within a frame or between frames) is converted into the change of the image plane center, and the latest camera intrinsic parameters are determined based on the coordinates of the changed image plane center, which are then used to determine the first transformation matrix. The electronic device can then use the first transformation matrix to adjust the position of each pixel in the t+1 frame original image, compensating for the offset of the imaging position caused by lens movement due to various reasons, ensuring image stability, improving the de-shaking effect, and thus improving image quality.

[0026] Optionally, the electronic device may substitute the first camera intrinsic parameter into a transformation matrix calculation formula to obtain a first transformation matrix.

[0027] In a possible implementation manner of the first aspect, the electronic device may adjust the positions of pixels on the image based on an EIS algorithm in combination with an external reference de-jittering method.

[0028] The electronic device converts the offset corresponding to the center of the image plane into an equivalent angle of the gyroscope in the camera. The equivalent angle represents the offset angle of the electronic device at the end of the exposure of the t-th frame, that is, the offset angle of the gyroscope.

[0029] Afterwards, the electronic device can obtain the adjustment angle corresponding to the t+1th frame image based on the equivalent angle, and the adjustment angle represents the angle required to adjust the t+1th frame image to reach the ideal shooting angle.

[0030] Afterwards, the electronic device may obtain a first rotation matrix based on the adjustment angle and a preset rotation matrix calculation formula;

[0031] Afterwards, the electronic device may adjust the positions of the pixels in the t+1th frame image according to the second transformation matrix; the second transformation matrix is ​​obtained based on the first rotation matrix.

[0032] In this application, the electronic device can convert the displacement of the lens movement into the movement angle of the gyroscope, so as to use the movement angle of the gyroscope to determine the latest camera extrinsic parameters, so that the image path can be optimized based on the latest camera extrinsic parameters, compensate for the offset of the imaging position caused by the return center, and improve the image anti-shake effect.

[0033] In a possible implementation manner of the first aspect, the process of determining the adjustment angle corresponding to the t+1th frame image may include:

[0034] First, the electronic device can calculate the difference between the original angle corresponding to the t+1th frame image and the equivalent angle to obtain the actual angle corresponding to the t+1th frame image, where the original angle corresponding to the t+1th frame image represents the angle of movement of the electronic device body when the t+1th frame image is exposed, that is, the actual angle of the gyroscope, and the actual angle corresponding to the t+1th frame image represents the actual shooting angle when the lens exposes the t+1th frame image.

[0035] Afterwards, the electronic device can calculate the difference between the actual angle corresponding to the t+1th frame image and the target angle corresponding to the t+1th frame image to obtain the adjustment angle corresponding to the t+1th frame image. The target angle corresponding to the t+1th frame image represents the ideal shooting angle corresponding to the t+1th frame image when the previous t+1 frames are all stable images, thereby achieving accurate calculation of the shooting adjustment angle, that is, accurate calculation of the optimized angle, so that the electronic device can use the optimized angle corresponding to the t+1th frame image to adjust the position of the pixels in the t+1th frame image to achieve image de-shaking.

[0036] In a possible implementation of the first aspect, the electronic device can calculate the ratio between the offset corresponding to the center of the image plane and the distance corresponding to the camera, and combine it with the inverse trigonometric function to obtain the equivalent angle, so as to convert the offset of the lens into the offset angle of the gyroscope, that is, convert it into the movement of the electronic device body, so that the electronic device can use the equivalent angle corresponding to the gyroscope and the actual angle of the gyroscope to determine the shooting angle of the t+1 frame image.

[0037] Optionally, the offset corresponding to the center of the image plane includes the offset in the x direction of the center of the image plane. Accordingly, the equivalent angle includes the equivalent angle in the x direction. The electronic device can calculate the equivalent angle by the inverse trigonometric function arctan (shift x / f) calculates the equivalent angle in the x direction, where shift x represents the x-direction offset of the center of the image plane, and f represents the distance corresponding to the t+1th frame image.

[0038] The equivalent angles can also include equivalent angles in the y-direction and equivalent angles in the z-direction. The process for determining the equivalent angle in the y-direction is similar to that for determining the equivalent angle in the x-direction. Because there is a certain correspondence between the x-axis and the z-axis, the x-direction offset of the image plane center can be converted into a z-direction offset. The z-direction offset can then be used to determine the equivalent angle of the gyroscope in the z-direction, thereby achieving the determination of equivalent angles in the x, y, and z directions.

[0039] In a possible implementation of the first aspect, after obtaining the above-mentioned transformation matrix (such as the first transformation matrix and the second matrix), for each pixel in the t+1 frame image (that is, the t+1 frame original image), the electronic device can multiply the coordinates of the pixel by the transformation matrix to adjust the position of the pixel in the image.

[0040] In a possible implementation of the first aspect, the electronic device may use a model to adjust the positions of pixels on an image. The electronic device may input an offset corresponding to the center of the image plane and a distance corresponding to the camera into a preset network model to obtain the target image of the t+1th frame. The preset network model is obtained based on an EIS algorithm. The preset network model is used to adjust the positions of pixels in the image based on the offset and distance corresponding to the center of the image plane, thereby achieving rapid de-jittering of the original image and avoiding changes in the optical center position (or shooting angle) of the camera at the beginning of image exposure due to lens movement, thereby causing the imaging position of the object in the previous frame of the image to shift in the next frame of the image, thereby improving image quality.

[0041] In one possible implementation of the first aspect, the multi-frame capture state refers to a video capture state, a preview state, or a state of capturing dynamic images in a photo capture state, thereby correspondingly improving video quality, preview image quality, or dynamic image quality. For example, in the video capture state, before exposing the t-th frame image, the electronic device needs to first receive a first operation for triggering video recording. In response to the first operation, the electronic device begins capturing images, and anti-shake can be achieved by compensating for jitter during the exposure period.

[0042] In a second aspect, a shooting method is provided, the method comprising:

[0043] When the electronic device is in a multi-frame shooting state, after the exposure of the t+1 frame image is completed, the electronic device adjusts the positions of pixels in the t+1 frame image based on the offset corresponding to the lens in combination with the EIS algorithm to obtain a t+1 frame target image; the offset corresponding to the lens represents an offset value of the lens position at the end of the exposure of the t+1 frame image compared to the initial position of the lens;

[0044] The electronic device displays and / or stores the t+1th frame target image.

[0045] The offset corresponding to the lens mentioned above represents the offset corresponding to the center of the image plane corresponding to the camera.

[0046] In this application, since the lens position changes after lens movement, the imaging center of the electronic device camera changes, which in turn changes the imaging position of the same photographed object. In other words, the position of the same photographed object in two adjacent frames of images changes, which, from the user's perspective, causes the image to shake. Therefore, the electronic device can use the EIS algorithm to adjust the position of the pixels on the collected image (such as the t+1 frame image) to compensate for the offset of the imaging position caused by lens movement and improve the image de-shaking effect.

[0047] In a possible implementation of the second aspect, the electronic device adjusts positions of pixels in the t+1th frame image based on an offset corresponding to the lens in combination with an electronic image stabilization (EIS) algorithm to obtain the t+1th frame target image, including:

[0048] The electronic device determines a first coordinate of the image plane center based on the initial coordinate of the image plane center and the offset corresponding to the image plane center; wherein the first coordinate of the image plane center represents the position of the image plane center when the exposure of the t+1th frame image is completed;

[0049] The electronic device determines a first camera intrinsic parameter based on a first coordinate of the center of the image plane and a distance corresponding to the camera; the distance represents a distance between the image plane and the center of the lens;

[0050] The electronic device adjusts the positions of pixels in the t+1th frame image according to a first transformation matrix; the first transformation matrix is ​​obtained based on the first camera intrinsic parameter.

[0051] In this application, the offset corresponding to the center of the image plane is obtained by subtracting the position of the lens at the end of exposure of the t+1 frame image from the initial position of the lens. In other words, the movement of the lens, that is, the offset, is converted into the change of the image plane center. Based on the coordinates of the changed image plane center, the latest camera intrinsic parameters are determined, and the first transformation matrix is ​​determined using the latest camera intrinsic parameters. Afterwards, the electronic device can use the first transformation matrix to adjust the position of each pixel on the t+1 frame original image, which can compensate for the offset of the imaging position caused by the lens movement, ensure image stability, improve the de-shaking effect, and thus improve image quality.

[0052] In a possible implementation of the second aspect, the electronic device adjusts positions of pixels in the t+1th frame image based on an offset corresponding to the lens in combination with an electronic image stabilization (EIS) algorithm to obtain the t+1th frame target image, including:

[0053] The electronic device converts the offset corresponding to the center of the image plane into an equivalent angle of the gyroscope in the camera; the equivalent angle represents the offset angle of the electronic device at the end of the exposure of the t-th frame;

[0054] The electronic device obtains an adjustment angle corresponding to the t+1th frame image based on the equivalent angle, where the adjustment angle represents an angle that needs to be adjusted when the t+1th frame image reaches an ideal shooting angle;

[0055] The electronic device obtains a first rotation matrix based on the adjustment angle and a preset rotation matrix calculation formula;

[0056] The electronic device adjusts the positions of the pixels in the t+1th frame image according to a second transformation matrix; the second transformation matrix is ​​obtained based on the first rotation matrix.

[0057] In this application, the electronic device can convert the displacement of the lens movement into the movement angle of the gyroscope, so as to use the movement angle of the gyroscope to determine the latest camera extrinsic parameters, so that the image path can be optimized based on the latest camera extrinsic parameters, compensate for the offset of the imaging position caused by the lens movement, and improve the image anti-shake effect.

[0058] In a third aspect, the present application provides an electronic device, comprising a display screen, a memory, and one or more processors; the display screen, the memory, and the processor are coupled; the display screen is used to display an image generated by the processor, and the memory is used to store computer program code, wherein the computer program code comprises computer instructions; when the processor executes the computer instructions, the electronic device executes a shooting method as described in any one of the first aspect or the second aspect above.

[0059] In a fourth aspect, the present application provides a computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the shooting method as described in any one of the first and second aspects above.

[0060] In a fifth aspect, the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the shooting method as described in any one of the first and second aspects above.

[0061] It can be understood that the beneficial effects that can be achieved by the shooting method described in the second aspect, the electronic device described in the third aspect, the computer storage medium described in the fourth aspect, and the computer program product described in the fifth aspect provided above can refer to the beneficial effects in the first aspect and any possible implementation method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] FIG1A is a schematic diagram of the working principle of a camera provided in an embodiment of the present application;

[0063] FIG1B is a first schematic diagram of a video shooting process according to an embodiment of the present application;

[0064] FIG1C is a schematic diagram of an image exposure process provided in an embodiment of the present application;

[0065] FIG2A is a second schematic diagram of a video shooting process provided by an embodiment of the present application;

[0066] FIG2B is a third schematic diagram of a video shooting process provided by an embodiment of the present application;

[0067] FIG2C is a fourth schematic diagram of a video shooting process provided by an embodiment of the present application;

[0068] FIG3A is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0069] FIG3B is a first schematic diagram of a camera of an electronic device provided in an embodiment of the present application;

[0070] FIG3C is a second schematic diagram of a camera of an electronic device provided in an embodiment of the present application;

[0071] FIG4 is a structural block diagram of an electronic device provided in an embodiment of the present application;

[0072] FIG5A is a first flow chart of a photographing method according to an embodiment of the present application;

[0073] FIG5B is a fifth schematic diagram of a video shooting process provided by an embodiment of the present application;

[0074] FIG5C is a first schematic diagram of a lens movement range provided in an embodiment of the present application;

[0075] FIG5D is a second schematic diagram of a lens movement range provided in an embodiment of the present application;

[0076] FIG5E is a third schematic diagram of a lens movement range provided in an embodiment of the present application;

[0077] FIG5F is a fourth schematic diagram of a lens movement range provided in an embodiment of the present application;

[0078] FIG6 is a sixth schematic diagram of a video shooting process provided by an embodiment of the present application;

[0079] FIG7 is a first schematic diagram of lens movement provided in an embodiment of the present application;

[0080] FIG8 is a second flow chart of a photographing method provided in an embodiment of the present application;

[0081] FIG9A is a second schematic diagram of lens movement provided in an embodiment of the present application;

[0082] FIG9B is a third schematic diagram of lens movement provided in an embodiment of the present application;

[0083] FIG9C is a schematic diagram of gyroscope angular velocity according to an embodiment of the present application;

[0084] FIG9D is a schematic diagram of a frame interval provided in an embodiment of the present application;

[0085] FIG10 is a third flow chart of a photographing method provided in an embodiment of the present application;

[0086] FIG11 is a fourth schematic diagram of lens movement provided in an embodiment of the present application;

[0087] FIG12 is a schematic diagram of pixel position adjustment provided in an embodiment of the present application;

[0088] FIG13 is a fourth flow chart of a photographing method provided in an embodiment of the present application;

[0089] FIG14 is a schematic diagram of an equivalent conversion provided by an embodiment of the present application;

[0090] FIG15 is a schematic diagram of a shooting angle provided in an embodiment of the present application;

[0091] FIG16 is a schematic diagram of a screen jitter interface provided by an embodiment of the present application;

[0092] FIG17 is a fifth flow chart of a shooting method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0093] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described in the present application as "exemplary" or "for example" should not be interpreted as being more preferred or advantageous than other embodiments or design. To be precise, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way. In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple. In the embodiments of the present application, "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" can explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, the meaning of "multiple" is two or more.

[0094] For ease of understanding, the professional terms involved in this application are introduced below.

[0095] OIS: refers to avoiding or reducing jitter in the process of capturing light signals by moving the lens or image sensor (also called photosensitive element or sensor) in the camera, thereby improving image quality.

[0096] Electronic image stabilization (EIS) technology: It uses software algorithms to calculate the changes in the camera lens's posture and crop the video image to make the video image look more stable.

[0097] Image plane: refers to the imaging plane, that is, the plane where the lens focuses the image. Simply put, the location of the image sensor is the image plane.

[0098] The principle of camera imaging is as shown in Figure 1A. Light passes through the camera's lens module (hereinafter referred to as the lens) and is transmitted to the camera's sensor (such as a complementary metal-oxide-semiconductor (CMOS)). This process is called exposure. During the exposure process, the sensor converts the light signal collected by each pixel into an electrical signal, forming an image frame.

[0099] During the process of shooting a video with a mobile phone, if the mobile phone camera shakes during the exposure period, the camera and the object being shot will have a large relative motion, and the entire motion process will be imaged on the sensor, resulting in blurred imaging and lower video quality. For example, when a user is holding a mobile phone to shoot a video, the user's hand shakes downward, causing the user's holding the mobile phone to shake downward, and the content of the mobile phone shooting changes (the content of the picture shown in Figure 1B (b) has changed compared to the picture shown in Figure 1B (a)), and the image may become blurred (the light sign 10 in the picture shown in Figure 1B (b) is blurred compared to the light sign 10 in the picture shown in Figure 1B (a)).

[0100] It should be noted that if the shooting environment is dim, the exposure time is generally longer. For example, in night shooting scenes, the exposure time is generally 30 milliseconds (ms) to 60ms, which can easily cause blurred images due to phone shaking.

[0101] For example, the mobile phone camera can be exposed using a row-by-row exposure (or rolling shutter exposure) method as shown in FIG1C . Row-by-row exposure means starting exposure from the first row, and then starting exposure of the second row after a row period, and so on, until the last row is exposed. After the first row is exposed, the sensor starts reading data, and reading a row of data requires a row period. After the first row of data is read, the sensor can start reading the second row of data, and so on, until the last row of data is read. After each row of exposure, the sensor transfers the data of the photosensitive area to the storage area. Afterwards, the sensor reads the data of each row from the storage area.

[0102] It should be understood that when the type of sensor in the mobile phone camera is a rolling shutter exposure type, the sensor can be exposed in a row-by-row exposure manner.

[0103] The above introduces the terms involved in this application and the camera imaging principle. Now we will begin to introduce the solution of this application.

[0104] In some embodiments, the mobile phone can use OIS to achieve video de-shaking. During the video shooting process, if the mobile phone detects that the mobile phone is shaking, the mobile phone can drive the camera lens to move by moving the motor in the lens (or called OIS motor) to compensate for the shaking of the mobile phone. Among them, the compensation action of driving the camera lens to move is performed within the frame, that is, during the exposure of the image frame in the video, if the mobile phone camera shakes, the mobile phone can drive the camera lens to move. For example, during the video shooting process, as shown in Figure 2A, after the exposure of image f0 is completed, the mobile phone camera continues to capture the next frame of image f1. During the exposure of image f1, the mobile phone detects that the mobile phone camera is shaking. For example, if it detects that the mobile phone moves downward, in order to compensate for the downward movement of the mobile phone, the mobile phone can drive the OIS motor to move the lens upward during the exposure of image f1 (as shown in Figure 2B), so that the camera does not move relative to the object being photographed, so that the obtained image f1 does not change the picture content compared to image f0, and there is no blur problem.

[0105] However, because the OIS motor moves within a certain range, its range of movement is limited. In other words, the range of movement of the lens is limited. If the phone camera shakes downward when the OIS motor has reached the top, the OIS motor can no longer move upward, which means it cannot drive the lens upward to compensate for the downward shake of the phone camera. In other words, the range of movement of the lens is limited, resulting in the possibility of blurring the image captured by the phone and causing changes in the image content, resulting in jitter and blur in the video.

[0106] Continuing with the above example, after image f1 is captured, the phone continues to capture the next frame, image f2. The phone's camera is still shaking downward, and because the lens has already moved to the top, it can no longer move upward to compensate for the downward shaking. As a result, the captured image f2 has different content compared to image f1 and is blurred (as shown in Figure 2C, light sign 11 in image frame f2 is blurrier than light sign 11 in image frame f1).

[0107] Therefore, to address the above-mentioned issues, the present application provides a shooting method. When an electronic device is shooting a video and shakes, the electronic device can drive the lens to move in the opposite direction within the frame by moving the OIS motor. That is, during the image exposure period, the lens is driven to move in the opposite direction to compensate for the shake of the electronic device. As a result, after the image exposure is completed, a clear image with no changes in the captured content can be obtained. In order to ensure the range of movement of the lens in the next frame, the electronic device can control the lens to perform a re-centering action during the interval between the current image and the next frame. Since the lens is re-centered between frames, the lens has a larger range of movement when the next frame is exposed, avoiding the problem of the next frame being blurred due to the inability to compensate for the shake by moving the lens due to the limited range of movement of the lens, and avoiding the problem of trembling in the recorded video. Since the lens is re-centered between frames, the position of the lens at the beginning of the exposure of the next frame is different from the position at the end of the exposure of the previous frame, resulting in different positions of the center of the image plane of the two adjacent frames, which causes the same photographed object to be in different positions in the two adjacent frames, resulting in a shaking problem. Therefore, in order to achieve video de-shaking, electronic devices can use the EIS algorithm to optimize the position of pixels in the next frame image based on the displacement of the lens (including the offset of the lens caused by centering), so that the position of the same shooting object is the same in different images, avoiding video jitter problems caused by centering, ensuring video quality, and improving the user's shooting experience.

[0108] For example, the electronic device in the embodiments of the present application may be a mobile phone, a tablet computer, a wearable device, a personal digital assistant (PDA), a vehicle-mounted terminal, an Internet of Things device, or other device with a shooting function. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic device.

[0109] FIG3A shows a schematic structural diagram of the electronic device 100 .

[0110] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0111] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0112] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0113] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0114] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0115] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0116] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0117] The charging management module 140 is configured to receive charging input from a charger. While charging the battery 142 , the charging management module 140 can also provide power to the electronic device through the power management module 141 .

[0118] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160.

[0119] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0120] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0121] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0122] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0123] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0124] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0125] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0126] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0127] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0128] The ISP is used to process data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then transmitted to the ISP for processing and converted into an image visible to the naked eye.

[0129] The ISP can also perform algorithmic optimization on image noise and brightness. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0130] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element (or image sensor). The photosensitive element can be a charge coupled device (CCD) or a CMOS phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0131] Exemplarily, the N cameras 193 may include one or more front-facing cameras and one or more rear-facing cameras. For example, assuming the electronic device 100 is a mobile phone, the mobile phone shown in FIG3B includes one front-facing camera, such as front-facing camera 20. The mobile phone shown in FIG3C includes three rear-facing cameras, such as rear-facing cameras 21, 22, and 23. Of course, the number of cameras in the mobile phone includes but is not limited to the number described in the above example.

[0132] Among them, the above-mentioned camera 193 may include one or more of the following cameras: main camera, telephoto camera, wide-angle camera, ultra-wide-angle camera, macro camera, fisheye camera, time of flight (TOF) camera, infrared camera, depth camera and black and white camera.

[0133] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0134] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0135] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0136] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0137] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0138] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0139] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 .

[0140] The sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0141] Among them, the gyroscope sensor can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake.

[0142] The buttons 190 include a power button, a volume button, and the like.

[0143] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts and can also be used for touch vibration feedback.

[0144] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0145] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and disconnected from the electronic device 100 by inserting or removing the SIM card into or from the SIM card interface 195. The electronic device 100 may support one or N SIM card interfaces, where N is a positive integer greater than one.

[0146] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes a layered architecture system as an example to exemplify the software structure of the electronic device 100.

[0147] FIG4 is a structural block diagram of the electronic device 100 according to an embodiment of the present application.

[0148] A layered architecture divides software into several layers, each with distinct roles and divisions of labor. Layers communicate with each other through software interfaces. In some embodiments, the system is divided into five layers: application layer, application framework layer, runtime and system libraries, hardware abstraction layer, and kernel layer, from top to bottom. The application layer, application framework layer, Android runtime and system libraries, hardware abstraction layer, and kernel layer are all software layers.

[0149] The application layer can include a series of application packages.

[0150] As shown in FIG4 , the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.

[0151] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0152] As shown in FIG4 , the application framework layer may include a camera access interface, a window manager, a content provider, a view system, a resource manager, a notification manager, and the like.

[0153] The camera access interface can be used to provide an application programming interface and programming framework for camera applications. The camera access interface can include camera management and camera devices. Camera management can be used to provide an access interface for managing cameras (or described as cameras); camera devices can be used to provide an interface for accessing cameras.

[0154] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0155] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0156] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0157] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0158] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0159] The runtime includes the core library and the virtual machine. The runtime is responsible for system scheduling and management.

[0160] The core library consists of two parts: one is the functional functions that need to be called by the Java language, and the other is the core library of the system.

[0161] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0162] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0163] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0164] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0165] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0166] A 2D graphics engine is a drawing engine for 2D drawings.

[0167] The hardware abstraction layer is an interface layer located between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system. In the embodiment of the present application, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library.

[0168] The camera hardware abstraction layer can provide virtual hardware for camera devices such as camera device 1 and camera device 2. Camera devices such as camera device 1 and camera device 2 correspond to one or more cameras on electronic device 100. Electronic device 100 can call each camera through camera device 1, camera device 2, and other camera devices. The camera algorithm library can include the shooting state recommendation algorithm provided in the embodiments of the present application, for example, a de-shaking algorithm corresponding to the video shooting state, such as the EIS algorithm.

[0169] The kernel layer is the layer between hardware and software. It can also be called the driver layer, and it includes, for example, camera device drivers. These drivers drive the camera sensor in the hardware layer to capture images and the image signal processor in the hardware layer to pre-process the images.

[0170] In some embodiments, the kernel layer may further include a digital signal processor driver and an image processor driver. The digital signal processor driver is used to drive the digital signal processor in the hardware layer to process images. The image processor driver is used to drive the image processor in the hardware layer to process images.

[0171] In some embodiments, the kernel layer may further include a display driver, a camera driver, an audio driver, and a sensor driver.

[0172] In an embodiment of the present application, by calling the hardware abstraction layer interface in the hardware abstraction layer, the application layer and application framework layer above the hardware abstraction layer can be connected with the driver layer and hardware layer below, thereby realizing camera data transmission and function control.

[0173] The following is an example of the software and hardware workflow of the electronic device based on a video shooting scenario and in combination with FIG4 .

[0174] a. When the camera application is running on the electronic device 100, in response to a first trigger operation of the user on the camera application, the camera application sends a video capture request to the camera access interface. The first trigger operation is used to trigger the electronic device 100 to record a video.

[0175] b. The camera access interface sends the above video capture request to the camera hardware abstraction layer.

[0176] c. The camera hardware abstraction layer sends the above video capture request to the camera device driver in the kernel layer.

[0177] d. The camera device driver controls the cameras (such as camera 1, camera 2, TOF camera, multispectral camera, etc.) to continuously capture images based on the video capture request.

[0178] e. After capturing a frame of image, the camera sends the image to the camera device driver.

[0179] After receiving the image, the camera device driver sends the image to the camera hardware abstraction layer.

[0180] g. After receiving the image, the camera hardware abstraction layer calls the algorithm in the camera algorithm library to process the image and sends the processed image to the camera access interface.

[0181] h. The camera access interface sends the processed image to the camera application, so that the camera application displays the processed image on the display screen of the electronic device and / or saves the processed image.

[0182] In some embodiments, to avoid video blur, the electronic device may also perform intra-frame jitter compensation and inter-frame re-centering to prevent blurring of the captured image due to jitter of the electronic device. The intra-frame jitter compensation process may include: i. During image exposure, the gyroscope sensor (or gyroscope) sends collected angular velocity data to the processor. The processor may then perform j. Based on the angular velocity data, determine whether the electronic device is shaking. If shaking occurs, send a compensation instruction to the camera to trigger the camera lens to move in the opposite direction of the jitter during image exposure.

[0183] The inter-frame re-centering process includes: k. After the image exposure is completed, the processor sends a re-centering instruction to the camera to trigger the lens in the camera to perform a re-centering action.

[0184] It should be noted that the above-mentioned camera 1 and camera 2 can be different types of cameras. For example, sensor 1 can be a wide-angle camera, and camera 2 can be a main camera. The receiving and sending operations between modules within the electronic device described in the embodiments of the present application actually refer to function calls.

[0185] In order to avoid blurring of the camera's imaging picture due to the shaking of the electronic device's camera, the present application proposes a shooting method. In a multi-frame shooting state, during the image exposure period, the electronic device compensates for the shaking of the electronic device's camera by moving the lens of the electronic device's camera, thereby realizing intra-frame de-shaking of OIS and avoiding blurring of the imaging picture. In addition, in order to avoid the situation where the lens cannot continue to move to compensate for the shaking after moving to the maximum extent due to the limited range of movement of the lens, the electronic device can adopt an inter-frame return strategy so that the lens can obtain the maximum range of movement when capturing the next frame of image, thereby continuing to compensate for the shaking of the electronic device and ensuring the imaging effect of the camera. In order to ensure that a stable image can be output, the electronic device can adopt an EIS algorithm to adapt the inter-frame return, that is, the electronic device can adjust the coordinates of the pixels in the image based on the movement of the lens in combination with the EIS algorithm, so that the positions of the same object on adjacent images are aligned, thereby ensuring the anti-shake effect of the video and thus ensuring the shooting effect of the electronic device.

[0186] The multi-frame shooting state refers to the state in which the mobile phone continuously captures and processes multiple frames of images. For example, the multi-frame shooting state refers to the state of video shooting, previewing, or shooting dynamic images in the photo shooting state. In other words, the shooting method provided in this application can be applied to video shooting scenarios, the process of displaying preview images, or dynamic image shooting scenarios in the photo shooting state. The preview state refers to the state of the mobile phone before starting to shoot in the video shooting state or the photo shooting state.

[0187] The following will take the electronic device as a mobile phone and the application scenario as video shooting as an example to introduce the shooting method provided in the embodiment of the present application.

[0188] An embodiment of the present application provides a shooting method. In this embodiment, when an electronic device is shooting a video, if shaking of the electronic device is detected, the electronic device can drive the camera lens to move within the frame to compensate for the shaking of the electronic device. To ensure that the lens compensation is not limited, the electronic device can control the lens to perform a re-centering action between frames. Specifically, as shown in Figure 5A, the shooting method provided in the embodiment of the present application may include S101-S104.

[0189] S101: The mobile phone displays a shooting interface. The shooting interface is the interface displayed when the mobile phone is recording a video. The shooting interface includes a preview area, which is used to display images captured by the camera on the mobile phone.

[0190] The capture interface is the interface displayed when the phone is recording a video. This interface can include a preview area and a capture button. The preview area is used to display the image captured by the camera. The capture button is used to trigger the phone to record the video.

[0191] In some embodiments, the shooting interface may further include a stop button (stop button 30 as shown in FIG5B ), which is used to trigger the mobile phone to stop recording the video and generate a video file. It should be understood that the user can only trigger the mobile phone to stop recording when the mobile phone is in the recording state.

[0192] It should be noted that the stop button and the video capture button can be located in the same location. Specifically, when the electronic device is capturing a video, the button is in the recording state, i.e., the capture state. The button can be used as a stop button. Upon receiving a user click on the stop button, the electronic device can stop recording the video. When the electronic device is in the video capture state but not capturing a video, the button is in the stop recording state. The button can be used as a video capture button. Upon receiving a user click on the capture button, the electronic device can start recording the video.

[0193] S102: The mobile phone receives a first trigger operation on a shooting interface by a user, and captures an image in response to the first trigger operation, wherein the first trigger operation is used to trigger video recording.

[0194] Exemplarily, the first triggering operation can be a user clicking a video capture button. Alternatively, the first triggering operation can be a user inputting a first preset gesture on the capture interface, such as an S-shaped gesture or a √-shaped gesture. This first preset gesture is used to trigger the phone to record a video. In this embodiment, the phone's capture interface may or may not include the video capture button.

[0195] It should be understood that the first preset gesture can be pre-configured in the mobile phone. In addition, the mobile phone can prompt the user with the first preset gesture and the function triggered by the first preset gesture (i.e., triggering the mobile phone to shoot video) in the video shooting interface.

[0196] In addition, the first trigger operation described above is only an example of the first operation that can trigger the mobile phone to record a video. The first operation can also be other operations input by the user. For example, the user speaks a shooting command, and the mobile phone responds to the shooting command to capture an image. The shooting command can be the default of the mobile phone or customized by the user. For another example, the user triggers the virtual shutter on the shooting interface, and the mobile phone responds to the shooting command to capture an image. For another example, the subject outputs a second preset gesture, and the mobile phone responds to the second preset gesture to capture an image. The second preset gesture can be the default of the mobile phone or customized by the user.

[0197] S103: When the mobile phone is exposing the t-th frame image, if the mobile phone shakes, the mobile phone moves the lens in the mobile phone in the reverse direction, where t is a positive integer.

[0198] In an embodiment of the present application, during the process of mobile phone shaking, the mobile phone exposes the tth frame of image. That is, during the exposure of the tth frame of image, the mobile phone detects that it is shaking. To avoid blurring of the tth frame of image due to the shaking of the mobile phone, during the process of progressively exposing the tth frame of image, the mobile phone can reversely move the lens in the mobile phone by driving the OIS motor in real time based on the shaking of the mobile phone. That is, the lens is moved to the first position to achieve intra-frame shake compensation. The displacement of the lens to the first position is the position of the mobile phone shaking during the exposure of the tth frame of image.

[0199] In some embodiments, the shaking of the mobile phone can be detected by a gyroscope. The mobile phone can determine the shaking displacement of the mobile phone through the angular velocity data collected by the gyroscope in the mobile phone. Afterwards, the mobile phone can control the movement of the lens in the direction opposite to the shaking, and the displacement of the reverse movement can be the shaking displacement. Among them, the angular velocity data collected by the gyroscope may include the angular velocity corresponding to different time points during the exposure of the t-th frame image, such as the angular velocity corresponding to each row of pixels. Specifically, after the gyroscope detects that the camera has moved, the mobile phone can calculate the displacement that needs to be compensated based on the angular velocity data reported by the gyroscope in real time or periodically to obtain the shaking displacement. Afterwards, the mobile phone can move the lens based on the shaking displacement to compensate for the shaking.

[0200] It should be understood that the motor can drive the lens to move on a two-dimensional plane, and it can move in any direction, such as up, down, left, and right. In other words, the range of movement of the lens is equivalent to a two-dimensional plane (as shown in Figure 5C). Accordingly, the jitter displacement includes the jitter displacement in the x-direction and the jitter displacement in the y-direction. The jitter displacement in the x-direction can be positive, negative, or 0. Similarly, the jitter displacement in the y-direction can be positive, negative, or 0. For example, if the phone shakes vertically upward, the jitter displacement in the y-direction is 1, and the jitter displacement in the x-direction is 0. The phone can compensate for the jitter in the y-direction by driving the lens vertically downward by a movement of 1. For another example, if the phone shakes vertically downward, the jitter displacement in the y-direction is -1, and the jitter displacement in the x-direction is 0. The phone can compensate for the jitter in the y-direction by driving the lens vertically upward (as shown in Figure 5D) by a movement of 1. It should be understood that the movement direction of the lens illustrated here (such as vertically upward, vertically downward) is only an example. The movement direction of the lens in this application is not limited, and the lens can be moved to any position within the two-dimensional plane, such as moving to the position of the lens shown in Figure 5E.

[0201] It should be noted that the shape of the moving range shown in FIG5C (ie, a rectangle) is only an example, and the moving range may also be other shapes, such as a circle, which is not limited in this application.

[0202] S104: When the exposure of the t-th frame image is completed, the mobile phone controls the lens to perform a re-centering action within a first target time, wherein the first target time is the interval between the t-th frame image and the t+1-th frame image.

[0203] The term "exposure completion of the tth frame image" refers to the time when the mobile phone (or the processor 110 in the mobile phone) obtains the timestamp corresponding to the last row of pixels in the tth frame image. It should be understood that after each row of pixels is exposed, the camera generates a timestamp corresponding to that row of pixels. Therefore, when the processor 110 obtains the timestamp corresponding to the last row of pixels in the image, it can be considered that the image exposure is completed.

[0204] The above-mentioned return-to-center action means moving the lens from its current position (e.g., the position of the lens at the end of exposure of the t-th frame image) toward the center point of the movement range, and thus to the target position. The target position is a position within the corresponding movement range of the lens, and the target position can be determined in a variety of ways. For example, the target position is the center point of the lens's movement range (e.g., the center point shown in FIG5C ), which represents the center point of the motor's movement range. For another example, the target position can be at 1 / 4 of the travel distance. As shown in FIG5F , the position of the lens at the end of exposure of the t-th frame image is (0, 1), and the center point is (0, 0). Therefore, the 1 / 4 of the travel distance is 1 / 4 of the center point, meaning that the lens is moved to the position (0, 1 / 4).

[0205] In the embodiment of the present application, when the exposure of the t-th frame image is completed, it indicates that the t-th frame image has been collected and the t+1-th frame image needs to be collected next. Since the OIS motor is active within a certain range, the range of movement of the lens driven by the OIS motor is also limited. If the mobile phone has compensated for the shake of the mobile phone by moving the lens in a certain direction during the exposure of the previous frame image, then when exposing the next frame image, the range in which the mobile phone can move the lens in this direction will be reduced. In other words, the range of movement of the lens when exposing the next frame image is limited. Therefore, in order to avoid the range of movement of the lens being limited when exposing the next frame image, the mobile phone can control the lens to perform a re-centering action between frames (or called frame intervals), so that when the exposure of the t+1-th frame image begins, the lens still has a larger range of movement. Therefore, even if the mobile phone is still shaking, the mobile phone can still compensate for the shake by moving the lens to avoid blurring the next frame image, thereby avoiding the problem of trembling in the recorded video.

[0206] For example, during video capture, as shown in Figure 6(a), after the acquisition of the t-1 frame image is complete, the phone shakes downward. During the exposure of the t-th frame image, the phone is still shaking downward. To compensate for the downward shaking of the phone, the phone can drive the lens upward (as shown in Figure 6(b)). After the exposure of the t-th frame image is completed, the t-th frame image is obtained as shown in Figure 6(b). Because the shaking was compensated during the exposure of the t-th frame image, the t-th frame image is not blurred compared to the t-1-th frame image.

[0207] At the end of the exposure of the t-th frame, the phone can control the lens to perform a re-centering action (as shown in Figure 6(c)) during the interval between the t-th and t+1-th frames, reserving movable space for compensation of the next frame. When exposing the t+1-th frame, the phone still experiences downward jitter. Since the lens has already re-centered, the phone can continue to move the lens upward to compensate for the jitter. After the exposure of the t+1-th frame, the resulting image is the t+1-th frame shown in Figure 6(d). Because the jitter was compensated during the exposure of the t+1-th frame, the t+1-th frame is still not blurred compared to the t-th frame. This avoids the situation where the phone still jitters downward when exposing the next frame, and cannot compensate for the jitter by moving the lens upward due to the lens having moved to the top of the movement range, resulting in a blurred image in the next frame. This ensures the quality of the next frame.

[0208] It can be understood that the direction of lens shake compensation is decoupled from the direction of lens recentering. Based on the interfaces shown in (a)-(d) of Figure 6 above, combined with the content shown in Figure 7, after the exposure of the t-th frame image, the lens has moved upward to the maximum position of its movement range. For the t-th frame image, the lens (or motor) compensation direction is upward. During the exposure of the t+1-th frame image, the lens compensation direction is upward. In the interval between the t-th and t+1-th frames, the lens is controlled to move downward for recentering. It can be seen that the direction of lens recentering and the direction of lens shake compensation can be different.

[0209] In some embodiments, when the degree of mobile phone shake is small, the degree of reverse lens movement required to compensate for the shake is also small, and the lens still has a large range of motion in the next frame, the mobile phone does not need to perform a re-centering action to increase the lens's range of motion in the next frame. Therefore, to avoid unnecessary re-centering actions, the mobile phone can determine whether to perform an inter-frame re-centering action based on the current position of the lens. Exemplarily, the process of determining whether to perform an inter-frame re-centering action based on the lens's position may include steps S104a-S104c as shown in FIG8 .

[0210] S104a: When the exposure of the t-th frame image is completed, the mobile phone determines whether the current position of the lens is within a first preset position range.

[0211] In the embodiment of the present application, at the end of exposure of the t-th frame image, if the current position of the lens is within the first preset position range, that is, if the position of the motor at the end of exposure of the t-th frame image is within the first preset position range, in other words, if the position of the motor at the end of exposure of the t-th frame image has not reached the preset position, it indicates that the degree of lens movement is small, and the lens still has a large range of movement in the next frame. There is no need to re-center the lens to enable a larger range of movement in the next frame. Therefore, the mobile phone can execute S104b. If the current position of the lens is not within the first preset position range, that is, if the position of the motor at the end of exposure of the t-th frame image has reached the preset position, it indicates that the range of movement of the lens in a certain direction in the next frame is limited. The mobile phone needs to control the lens to perform a re-centering action to reserve movement space for the next frame image. Therefore, the mobile phone can execute S104c.

[0212] The current position of the lens mentioned above represents the current coordinates of the lens, which may include coordinates in the x-direction and coordinates in the y-direction. Correspondingly, the first preset position range may also include a coordinate range in the x-direction and a coordinate range in the y-direction. In other words, a position point in the first preset position range (or referred to as a preset position) may include coordinates in the x-direction and coordinates in the y-direction. The first preset position range is a pre-set partial area within the moving range of the lens. The first preset position range is an area composed of position points that are closer to the center point. The absolute value of the x-direction coordinate of the position point in the first preset position range is smaller than the first preset coordinate value, and the absolute value of the y-direction coordinate of the position point is smaller than the second preset coordinate value.

[0213] In some embodiments, after each frame of image exposure is completed, the mobile phone may determine whether the current position of the lens is within a first preset position range to determine whether the lens needs to be controlled to perform a re-centering operation. Alternatively, after a frame of image exposure (e.g., the t-th frame of image) is completed, if the mobile phone does not shake during the exposure process of the frame of image, the mobile phone may not determine whether the lens needs to be controlled to perform a re-centering operation after the exposure of the frame of image is completed. If the mobile phone shakes during the exposure of the frame of image, the mobile phone may determine whether the lens needs to be controlled to perform a re-centering operation after the exposure of the frame of image is completed.

[0214] S104b: Within the first target time, the mobile phone does not control the lens to perform the re-centering action.

[0215] S104c: Within the first target time, the mobile phone controls the lens to perform a return-to-center action.

[0216] For example, the maximum values ​​of the y-axis coordinates within the aforementioned range of motion include 200 and -200. 200 represents the maximum position along the positive y-axis, and -200 represents the maximum position along the negative y-axis. The phone detects continuous y-axis vibrations through the angular velocity of the gyroscope. To compensate for this vibration, the phone moves the lens in the opposite direction during image exposure. As shown in FIG9A , at the end of image exposure (e.g., at time t1), the lens has moved upward to its maximum y-axis position (i.e., (0, 200)). This indicates that in the next frame, it will not be possible to compensate for any vibrations in the negative y-axis by moving the lens along the positive y-axis. This means that downward vibrations cannot be compensated. Therefore, to avoid this inability to compensate for downward vibrations in the next frame, as shown in FIG9B , the phone can retract the lens to a quarter-stroke position (i.e., (0, 50)) at time t2, reserving movement space for compensation in the next frame. This allows the phone to still achieve anti-shake by moving the lens in the next frame. Time t2 is the interval between the current image and the next frame.

[0217] Afterwards, as shown in FIG9C , after time t2, when the phone experiences an upward shake, the phone moves the lens downward. When the image exposure ends (as shown in FIG9B at time t3), the lens has moved to its maximum position in the y-direction (i.e., (0, -200)), indicating that in the next frame, it will not be possible to compensate for the phone's shake in the positive y-direction by moving the lens in the negative y-direction, i.e., it will not be possible to compensate for the phone's upward shake. Therefore, to avoid being unable to compensate for the phone's upward shake in the next frame, as shown in FIG9B , the phone can pull the lens back to 1 / 4 of its travel ((0, -50)) at time t4. Among them, time t4 is the interval between the image and the next frame.

[0218] It is understood that the positive and negative of the above coordinates indicate direction, not size. For example, -200 indicates the maximum position on the negative direction of the y-axis.

[0219] In some embodiments, the mobile phone can control the lens to perform a re-centering action during the interval between the tth frame image and the t+1th frame image, i.e., the interframe interval within the first target time. As shown in FIG9D , the first target time represents the period from the end of exposure of the first row of pixels in the tth frame image to the start of exposure of the first row of pixels in the t+1th frame image. The first target time includes the interframe interval and the data read time. The data read time refers to the time it takes for the sensor to read the pixel data (e.g., electrical signal data) for each row of pixels after exposure. In other words, the sensor reads the pixel data for each row of the image, such as from a storage area, during the data read time. The interframe interval represents the idle time of the sensor; during the interframe interval, the sensor neither exposes the image nor reads data. For example, assuming a frame rate of 30, the first target time, i.e., the interframe interval, is 33 ms. The data read time is 13 ms, and accordingly, the interframe interval is 33 - 13 = 20 ms. In other words, the lens needs to perform the re-centering action within these 20 ms.

[0220] Among them, optionally, the mobile phone can determine the target return speed based on the frame gap, so as to use the target return speed to control the lens to perform the return action. In one example, the mobile phone can obtain a preset return speed corresponding to the frame gap and use it as the target return speed. When the frame gap is shorter, it means that the time available for the return action is shorter, the return speed needs to be faster, and accordingly, the return accuracy is lower. In other words, after the return action is completed, the position of the lens and the target position may differ more. When the frame gap is longer, it means that the time available for the return action is longer, the return speed can be slower, and accordingly, the return accuracy is higher. In other words, after the return action is completed, the position of the lens and the target position may differ less.

[0221] In another example, the phone can obtain the first coordinate of the lens, which represents the lens position at the end of exposure for the tth frame. The phone can then determine the re-centering distance based on the first coordinate and the target position. Specifically, the re-centering distance represents the distance between the first coordinate and the target position. The phone can then calculate the ratio of the re-centering distance to the interframe interval to obtain the target re-centering speed, enabling accurate determination of the lens' re-centering speed.

[0222] Optionally, the target return speed may be determined during a frame gap, such as when the frame gap is reached.

[0223] The above describes how, when a phone shakes during video recording, the phone moves the lens within the frame to compensate for the shake, and how the phone re-centers the lens between frames. Next, we'll describe how to use EIS adaptive re-centering to avoid jitter in video content.

[0224] An embodiment of the present application provides a shooting method. In this embodiment, due to lens movement (such as after controlling the lens to perform a centering action between frames), the lens position will change, causing the imaging center of the mobile phone camera to change, thereby causing the imaging position of the same shooting object to change, that is, the position of the same shooting object in two adjacent frames of images will change. From the user's perception, the video has shaken. Therefore, in order to ensure the de-shaking effect of the video, the mobile phone can use the EIS algorithm to adjust the position of the pixels on the captured image to compensate for the position offset caused by lens movement (such as centering).

[0225] For example, the process of adjusting the positions of pixels on an image using the EIS algorithm can be performed using camera intrinsic parameters or camera extrinsic parameters. The process of adjusting the positions of pixels on an image using camera intrinsic parameters to achieve de-shaking is described in Section a below. The process of adjusting the positions of pixels on an image using camera extrinsic parameters to achieve de-shaking is described in Section b below.

[0226] a. Using camera internal parameters to de-shake

[0227] In an embodiment of the present application, the mobile phone can control the lens to perform a re-centering action, converting the displacement of the lens into an offset of the center of the image plane. This offset can then be used to determine the latest camera intrinsic parameters, allowing the latest camera intrinsic parameters to adapt to the re-centering action. This allows accurate adjustment of the position of pixels on the image based on the latest camera intrinsic parameters, compensating for the offset of the imaging position caused by lens movement (e.g., re-centering), and improving the video stabilization effect. Specifically, as shown in FIG10 , the process of using camera intrinsic parameters for de-shaking may include S201-S208.

[0228] S201, the mobile phone obtains the t+1th frame of the original image captured by the camera.

[0229] The original image may be an image captured by a camera, that is, an image sensor, which represents an image in which pixel coordinates have not been adjusted.

[0230] S202: The mobile phone determines an offset corresponding to the center of the image plane based on the first coordinate and the second coordinate of the lens, wherein the first coordinate of the lens represents the initial position of the lens, and the second coordinate of the lens represents the position of the lens at the end of exposure of the t+1th frame image.

[0231] The initial position of the lens represents the initial position of the lens without movement. The lens may be driven by a motor, and accordingly, the coordinates of the lens (such as the first coordinate and the second coordinate) may be represented by the coordinates of the motor. In other words, the initial position of the lens may also be the initial position of the motor. Generally speaking, the motor is initially located at the center, such as the center point of the motor's range of movement. Accordingly, the initial position of the motor may be (0,0). The second coordinate of the lens may also represent the position of the motor at the end of exposure of the t+1th frame image (i.e., the t+1th frame of the original image).

[0232] The image plane center mentioned above refers to the center of the image plane coordinate system.

[0233] In this embodiment of the present application, the mobile phone calculates the difference between the second coordinate and the first coordinate of the lens to obtain the offset of the lens relative to the initial position at the end of exposure of the t+1 frame image. Generally speaking, the center of the image plane and the optical center (i.e., the center of the lens of the lens, in simpler terms, the center of the lens) are on the same straight line. Therefore, the offset corresponding to the lens can also be used as the offset corresponding to the center of the image plane.

[0234] For example, since the lens can move on a two-dimensional plane, the first and second coordinates of the lens may include coordinates in the x-direction and coordinates in the y-direction. The offset may include an offset in the x-direction and an offset in the y-direction. For example, when the lens moves vertically, that is, along the y-axis, the x-direction offset of the lens is 0, while the y-direction offset is not 0. As shown in FIG11 , when the lens moves vertically downward, there is an offset in the y-direction. For another example, when the lens moves horizontally to the right, that is, along the positive x-direction, the y-direction offset of the lens is 0.

[0235] S203: The mobile phone determines the first coordinates of the center of the image plane based on the initial coordinates of the center of the image plane and the offset corresponding to the center of the image plane.

[0236] In this embodiment of the present application, since the camera intrinsic parameters require the coordinates of the image plane center, the mobile phone needs to calculate the difference between the initial coordinates of the image plane center and the offset corresponding to the image plane center to obtain the first coordinate of the image plane center. The first coordinate of the image plane center represents the position of the image plane center at the end of the exposure of the t+1 frame image.

[0237] Exemplarily, the offset corresponding to the center of the image plane may include an offset in the x-direction and an offset in the y-direction. The initial coordinate of the center of the image plane may include an initial coordinate in the x-direction and an initial coordinate in the y-direction. Correspondingly, the first coordinate of the center of the image plane also includes a first coordinate in the x-direction and a first coordinate in the y-direction. The mobile phone can calculate the value of C' according to Formula 1: x =C x -shiftx , calculate the first coordinate of the x-direction of the center of the image plane. x The first x-coordinate of the center of the image plane, C x Indicates the initial x-coordinate of the center of the image plane, shift x Indicates the x-axis offset of the image plane center.

[0238] The mobile phone can be calculated according to formula 2: C' y =C y -shift y , calculate the first coordinate of the y direction of the center of the image plane. y The first coordinate in the y direction of the center of the image plane, C y Indicates the initial coordinate of the y direction of the center of the image plane, shift y Indicates the y-direction offset of the image plane center.

[0239] The initial coordinates of the image plane center represent the center of the image plane when the lens does not move initially. For example, if the resolution of the image is 4000*3000 and the upper left corner is generally used as the origin of the coordinates, the initial coordinates of the image plane center can be (2000, 1500). x , shift y ) can be 0, a positive number, or a negative number.

[0240] S204 : The mobile phone determines a first camera intrinsic parameter based on the first coordinate of the center of the image plane and the distance corresponding to the original image of the t+1th frame.

[0241] The distance (also called the image distance) represents the distance between the image plane and the center of the lens. The distance corresponding to the t+1th frame image may be sent by the camera and represents the distance when the camera captured the t+1th frame original image (or the t+1th frame image).

[0242] For example, after obtaining the first coordinate of the image plane center, the mobile phone can substitute the first coordinate of the image plane center and the distance corresponding to the t+1 frame image into the camera intrinsic parameter calculation formula to obtain the first camera intrinsic parameter (i.e., the latest camera intrinsic parameter). The camera intrinsic parameter calculation formula is Specifically, after obtaining the first coordinate of the center of the image plane, the mobile phone can use the first coordinate of the center of the image plane in the x direction, i.e. C′ x Substitute into the camera internal parameter calculation formula C x At the position of the image plane, the first coordinate in the y direction of the center, that is, C' y Substitute into the above camera internal parameter calculation formula C y, to obtain the first camera intrinsic parameter, that is, the latest camera intrinsic parameter.

[0243] It is understandable that the distance corresponding to the t+1th frame image may be the same as or different from the distance corresponding to the tth frame image. The distance corresponding to the tth frame image refers to the distance when the camera captures the tth frame image.

[0244] After determining the intrinsic parameters, the phone uses them to adjust the coordinates of the pixels in the original image of frame t+1 to improve image stability. There are various ways to adjust the coordinates of the pixels in the original image of frame t+1 using the camera's intrinsic parameters. One possible adjustment method is described below in steps S205-S206.

[0245] S205: The mobile phone determines a first transformation matrix based on the first camera intrinsic parameter, wherein the first transformation matrix represents a conversion relationship between a world coordinate system and a pixel coordinate system.

[0246] For example, the mobile phone can substitute the first camera intrinsic parameter into the transformation matrix calculation formula to obtain the first transformation matrix. The transformation matrix calculation formula can be

[0247] Among them, K represents the camera internal parameter, K -1 Represents the inverse matrix of K, and R represents the rotation matrix. Specifically, after obtaining the first camera intrinsic parameter, the mobile phone can substitute the first camera intrinsic parameter into K in the transformation matrix calculation formula, that is, through The first transformation matrix is ​​calculated.

[0248] S206 : The mobile phone adjusts the coordinates of the pixels in the t+1th frame original image based on the first transformation matrix to obtain the t+1th frame target image.

[0249] For example, for each pixel in the t+1th frame original image, the mobile phone can multiply the coordinates of the pixel by the first transformation matrix to adjust the position of the pixel, thereby obtaining the adjusted t+1th frame image, that is, the t+1th frame target image.

[0250] In an embodiment of the present application, the offset corresponding to the center of the image plane is obtained by subtracting the position of the lens at the end of exposure of the t+1 frame image from the initial position of the lens. The offset corresponding to the center of the image plane already includes the lens offset caused by the inter-frame centering action between the t+1 frame image and the t frame image. That is, the movement of the lens (such as the movement within the frame and between frames) is converted into the change of the image plane center, and the latest camera intrinsic parameters are determined based on the coordinates of the changed image plane center, so as to use the latest camera intrinsic parameters to determine the first transformation matrix. Afterwards, the mobile phone can use the first transformation matrix to adjust the position of each pixel on the t+1 frame original image, which can not only compensate for the imaging position offset caused by inter-frame centering, but also compensate for the imaging position offset caused by the movement of the lens within the frame, thereby ensuring the stability of the image and the de-shaking effect of the video.

[0251] In some embodiments, to improve de-jittering efficiency, the mobile phone may select multiple target coordinates from the original image of frame t+1. The mobile phone may then multiply each target coordinate by the first transformation matrix to obtain an adjusted target coordinate to adjust a portion of the image. The mobile phone may then determine the adjusted coordinates of the pixels in the original image of frame t+1 based on the adjusted target coordinates and interpolate the pixels to adjust the pixel coordinates in the original image of frame t+1.

[0252] Exemplarily, the above-mentioned target coordinates may be the coordinates of the grid points in the original image of the t+1th frame. The mobile phone may use the coordinates of the grid points before deformation (i.e., the coordinates of the grid points before deformation) as the target coordinates as shown in FIG12. Afterwards, for each grid point before deformation, the mobile phone may multiply the coordinates of the grid point before deformation by the first transformation matrix to adjust the coordinates of the grid point before deformation, and obtain the deformed coordinates of the grid point as shown in FIG12 (i.e., the coordinates of the grid point after deformation). Afterwards, the mobile phone may obtain the coordinates of the pixel points surrounding the deformed grid point based on the coordinates of the deformed grid point in combination with the interpolation method, thereby obtaining the initial adjusted image of the t+1th frame. Afterwards, the mobile phone may crop the initial adjusted image of the t+1th frame to obtain the target image of the t+1th frame.

[0253] Of course, the above target coordinates may also be determined by other means, such as the above target coordinates being the coordinates of pixels randomly selected by the mobile phone from the t+1th frame of the original image.

[0254] S207: The mobile phone receives a second triggering operation of the user on the shooting interface.

[0255] S208. The mobile phone stops recording the video in response to the second trigger operation, and generates a video file based on all the target images.

[0256] The second triggering operation may be a user clicking the stop button (e.g., a single click). The mobile phone may receive a user clicking the stop button at any time during video recording. In response to the click, the mobile phone ends recording and generates a recorded video file. The video file includes the target image processed by the mobile phone during the video recording process.

[0257] In some embodiments, if no re-centering action is performed between frames, the mobile phone may not need to adjust the position of the pixels on the next frame image, then the video file may also include an image captured by the camera on the mobile phone, which represents the image that has not been de-jittered using the first transformation matrix.

[0258] b. Deshake using camera extrinsics

[0259] In an embodiment of the present application, the mobile phone can convert the displacement of the lens movement (such as performing a centering action) into the movement angle of the gyroscope, so as to use the movement angle of the gyroscope to determine the latest camera extrinsic parameters, so that the latest camera extrinsic parameters are adapted to the centering action. Therefore, the image path can be optimized based on the latest camera extrinsic parameters, compensating for the offset of the imaging position caused by the centering action, and improving the anti-shake effect of the video. Specifically, as shown in Figure 13, the process of using camera extrinsic parameters to remove jitter can include S301-S309.

[0260] S301, the mobile phone obtains the t+1th frame of the original image captured by the camera.

[0261] S302: The mobile phone determines an offset corresponding to the center of the image plane based on the first coordinate and the second coordinate of the lens.

[0262] The process of determining the offset corresponding to the center of the image plane may refer to the corresponding content of S202 above, which will not be described in detail here.

[0263] S303: The mobile phone converts the offset corresponding to the center of the image plane into an equivalent angle of the gyroscope.

[0264] In the embodiment of the present application, the mobile phone can obtain the equivalent angle of the gyroscope based on the ratio between the offset and the distance corresponding to the center of the image plane, combined with the inverse trigonometric function, that is, convert the offset of the lens into the offset angle of the gyroscope, that is, the movement of the mobile phone body. In other words, the equivalent angle of the gyroscope represents the offset angle of the mobile phone body at the end of the exposure of the t+1 frame image. In other words, it can represent the offset angle of the lens in the mobile phone relative to the initial position at the end of the exposure of the t+1 frame image. Among them, the equivalent angle includes the equivalent angles in the x, y and z directions, that is, the equivalent angle in the x direction, the equivalent angle in the y direction and the equivalent angle in the z direction.

[0265] The mobile phone can use the ratio of the x-direction offset and the distance of the center of the image plane, combined with the inverse trigonometric function, to determine the equivalent angle of the gyroscope in the x-direction. Specifically, as shown in Figure 14, the mobile phone can use the inverse trigonometric function arctan (shift x / f), the equivalent angle of the gyroscope in the x-direction is calculated. This equivalent angle is an equivalent angle, not the actual angle of the gyroscope's movement. Similarly, the mobile phone can use the ratio of the y-direction offset of the image plane center and the distance, combined with inverse trigonometric functions, to determine the equivalent angle of the gyroscope in the y-direction. Because the image plane center moves on a two-dimensional plane and has no z-axis, the mobile phone needs to convert the two-dimensional coordinates into three-dimensional coordinates. There is a certain correspondence between the x-axis and the z-axis. Therefore, the x-direction offset of the image plane center can be converted to the z-direction offset, and the z-direction offset can be used to determine the equivalent angle of the gyroscope in the z-direction.

[0266] S304: The mobile phone obtains an adjustment angle corresponding to the t+1th frame original image based on the equivalent angle. The adjustment angle indicates the angle required to adjust the t+1th frame original image to achieve an ideal shooting angle.

[0267] In some embodiments, the mobile phone can determine the adjustment angle using the equivalent angle of the gyroscope corresponding to the t+1th original image (or the t+1th frame), the original angle corresponding to the gyroscope, and the target angle (or ideal shooting angle) corresponding to the t+1th frame. The original angle represents the actual angle of the gyroscope when the t+1th frame is exposed, i.e., the angle of movement of the mobile phone body (or the movement angle). This is obtained by integrating the angular velocity data collected by the gyroscope in the mobile phone during the exposure of the t+1th frame. The target angle can represent the gyroscope angle when the mobile phone captures the t+1th frame without any jitter when capturing the image based on the target angle, i.e., the gyroscope angle when the phone is shooting smoothly. In other words, the target angle corresponding to the t+1th frame represents the ideal shooting angle corresponding to the t+1th frame if the previous t+1 frames are all stable. The mobile phone can use an optimization algorithm (e.g., a filtering algorithm) to optimize the path of the t+1th frame in combination with the previous t frames to obtain the target angle corresponding to the t+1th frame.

[0268] It is understandable that when the OIS motor does not move, the angle of the gyroscope is consistent with the angle of the lens. In this case, the angle of the gyroscope is the shooting angle of the lens, that is, the original angle corresponding to the aforementioned frame t is the shooting angle of the lens. If the OIS motor moves, the shooting angle of the lens is equal to the angle of the gyroscope plus the movement angle of the lens. Alternatively, since angles (such as the equivalent angles mentioned above) are signed, the difference in angles can also be described as addition, such as the shooting angle of the lens is equal to the angle of the gyroscope minus the movement angle of the lens, but the direction corresponding to the pre-defined positive and negative signs is changed.

[0269] Exemplarily, the implementation process of the above S304 may include: first, the mobile phone may obtain the original angle corresponding to the t+1th frame image.

[0270] Since the phone's lens may move between and within frames, lens offset is used to determine the lens's shooting angle. Specifically, the phone calculates the difference between the original angle corresponding to the t+1th frame and the equivalent angle described above to obtain the actual angle corresponding to the t+1th frame. This actual angle is the shooting angle after compensating for lens offset, meaning it represents the actual shooting angle at the time the lens exposed the t+1th frame. The actual angle can be simply understood as the shooting angle after compensating for lens offset between frames, i.e., the shooting angle corresponding to the tth frame.

[0271] The phone then calculates the difference between the actual angle corresponding to the t+1th frame image and the target angle corresponding to the t+1th frame image to obtain the adjustment angle. The adjustment angle represents the angle that should be adjusted to achieve the target angle.

[0272] For example, the aforementioned frame t+1 is the second frame. The phone uses the angular velocity data collected by the gyroscope during the exposure of the second frame to obtain the gyroscope angle at the time of exposure of the second frame, thereby obtaining the original angle corresponding to the second frame. Furthermore, at the end of exposure of the second frame, the phone can convert the lens offset relative to the initial position based on the lens's current position (or described as coordinates) and the initial position into an offset corresponding to the center of the image plane. This offset corresponding to the center of the image plane and the distance corresponding to the second frame can then be used to determine the lens's movement angle, thereby obtaining the equivalent angle of the gyroscope.

[0273] Afterwards, the mobile phone can subtract the equivalent angle from the original angle to obtain the shooting angle when the lens exposes the second frame of image, that is, the actual angle corresponding to the second frame of image, thereby accurately determining the shooting angle.

[0274] Afterwards, in order to ensure the stability of the video image, the mobile phone can calculate the difference between the actual angle corresponding to the second frame image and the target angle corresponding to the second frame image to obtain the adjustment angle, so as to use the adjustment angle to adjust the position of the pixels in the second frame image. This can not only compensate for the imaging position offset caused by inter-frame centering, but also compensate for the imaging position offset caused by the movement of the lens within the frame, thereby ensuring image stability.

[0275] S305: The mobile phone obtains a first rotation matrix based on the above adjustment angle and a preset rotation matrix calculation formula.

[0276] For example, the mobile phone can substitute the adjustment angle into the preset rotation matrix calculation formula to obtain the first rotation matrix. The preset rotation matrix calculation formula can be the Rodriguez rotation matrix formula, that is, Where R represents the rotation matrix, x represents the adjustment angle of the gyroscope in the x direction, y represents the adjustment angle of the gyroscope in the y direction, and z represents the adjustment angle of the gyroscope in the z direction.

[0277] After determining the first rotation matrix, the mobile phone uses the first rotation matrix to adjust the coordinates of the pixels in the original image of frame t+1 to improve image stability. There are various ways to adjust the coordinates of the pixels in the original image of frame t+1 using the first rotation matrix. One possible adjustment method is described below in steps S306-S307.

[0278] S306: The mobile phone determines a second transformation matrix based on the first rotation matrix.

[0279] S307 : The mobile phone adjusts the coordinates of the pixels in the t+1th frame original image based on the second transformation matrix to obtain the t+1th frame target image.

[0280] Exemplarily, the mobile phone substitutes the first rotation matrix into the above-mentioned transformation matrix calculation formula to obtain the second transformation matrix. Since the angular velocity data of the gyroscope is in the world coordinate system, and the position of the pixel on the image is in the pixel coordinate system, it is necessary to establish a relationship between the relevant angles of the gyroscope, such as the above-mentioned adjustment angle, and the pixel coordinate system. In an embodiment of the present application, the above-mentioned adjustment angle is converted from the world coordinate system to the pixel coordinate system through the second transformation matrix, and finally mapped to the adjustment of the position of the pixel on the image, so as to realize the smoothing of the gyroscope angle, that is, the shooting angle, and avoid the offset of the imaging position of the same shooting object in different frames, and realize the optimization processing of the image path, thereby realizing the de-shaking processing of the video.

[0281] S308: The mobile phone receives a second triggering operation of the user on the shooting interface.

[0282] Exemplarily, the second trigger operation may be a user clicking a stop button on the shooting interface. The second trigger operation may also be other operations, such as a gesture operation, which is not limited in this application.

[0283] Of course, the second trigger operation introduced above is only an example of a second operation that can trigger the mobile phone to stop recording video. The second operation can also be other operations input by the user, and this application does not limit it.

[0284] S309: The mobile phone stops recording the video in response to the second trigger operation, and generates a video file based on all the target images.

[0285] In an embodiment of the present application, during video recording, as shown in FIG15 , the angles at which the mobile phone captures the five frames of images f0, f1, f2, f3, f4, and f5 are angles a0, a1, a2, a3, and a4, respectively. It can be seen that the shooting angles of these five frames vary significantly, indicating that the phone is shaking. To prevent the image from being blurred due to the shaking of the phone, the phone can move the camera lens to compensate for the shaking of the phone and perform a centering action, so that the angles at which the mobile phone captures the five frames of images f0, f1, f2, f3, f4, and f5 are equivalent to angles b0, b1, b2, b3, and b4. Although the angles b0, b1, b2, b3, and b4 vary less than angles a0, a1, a2, a3, and a4, there is still a significant difference.

[0286] Therefore, in order to achieve de-shaking, the mobile phone can adapt the camera extrinsic parameters to the centering action, so as to use the adapted camera extrinsic parameters to adjust the position of the pixels on the image, that is, the movement angle of the body itself when taking the image is compensated by the angle of offset between the lens and the initial position due to the movement of the lens (that is, the equivalent angle corresponding to the image), and the actual angle of the lens when taking the image is obtained. By adjusting the actual angle, the five frames of images f0, f1, f2, f3, f4, and f5 are equivalent to being shot based on angles c0, c1, c2, c3, and c4, thereby achieving smoothing of the shooting angle, which can not only compensate for the offset of the lens caused by inter-frame jitter, but also compensate for the offset of the lens caused by intra-frame jitter, that is, the change in shooting angle, and avoid the situation where the position of the same object in different frames is greatly different due to large changes in shooting angles (as shown in Figure 16).

[0287] It can be understood that the angles a0, a1, a2, a3, and a4 correspond to the original angles corresponding to the image, that is, the angles of the gyroscope. The angles b0, b1, b2, b3, and b4 correspond to the actual angles corresponding to the image. The angles c0, c1, c2, c3, and c4 correspond to the target angles corresponding to the image.

[0288] In some embodiments, the mobile phone can choose to use the camera intrinsic parameter de-shaking method or the camera extrinsic parameter de-shaking method to adjust the position of the pixels on the image according to the actual situation. For example, the mobile phone can randomly select the camera intrinsic parameter de-shaking method or the camera extrinsic parameter de-shaking method to adjust the position of the pixels on the image. For another example, the method of using the camera intrinsic parameter de-shaking only regards the movement of the lens as the translation of the center of the image plane, which cannot be used for large-angle perspective distortion and image rotation. Therefore, in the case of large-angle perspective distortion and image rotation, the mobile phone can use the camera extrinsic parameter de-shaking method to adjust the position of the pixels on the image.

[0289] It should be noted that in the above-mentioned de-shaking using camera intrinsic parameters, it is the camera intrinsic parameters that are changed based on lens movement. The camera extrinsic parameters, that is, the calculation of the rotation matrix, do not take lens movement into account. However, the rotation matrix can also be calculated according to the Rodriguez rotation matrix formula, except that the gyroscope adjustment angle is equal to the difference between the original angle corresponding to the image and the target angle corresponding to the image. Similarly, in the above-mentioned de-shaking using camera extrinsic parameters, it is the camera extrinsic parameters that are changed based on lens movement. The calculation of the camera intrinsic parameters does not take lens movement into account. Instead, the initial coordinates of the center of the image plane are directly substituted into the calculation formula of the camera intrinsic parameters.

[0290] The above describes how the EIS algorithm can be used to adjust the pixel positions in the captured image after the phone controls the lens to perform a re-centering action. The phone can also use the EIS algorithm independently to adjust the pixel positions in the captured image to improve image quality. In other words, even if the phone does not control the lens to perform a re-centering action between frames to achieve image de-jitter, the phone can still use the EIS algorithm to adjust the pixel positions in the captured image to achieve image de-jitter and improve image quality.

[0291] In some embodiments, the mobile phone can directly use the trained network model to process the original image of frame t+1 to obtain the target image of frame t+1. The following describes the process of the mobile phone using the network model to obtain the target image of frame t+1, in conjunction with Figure 4 above. As shown in Figure 17, this process may include steps S401-S417.

[0292] S401: The camera application in the mobile phone receives a click operation on a video shooting button by a user.

[0293] S402: In response to a click operation on the video capture button, the camera application sends a video capture request to the camera hardware abstraction layer in the mobile phone, wherein the video capture request is used to trigger the mobile phone camera to capture a video.

[0294] S403: The camera hardware abstraction layer sends a video capture request to the mobile phone camera.

[0295] Among them, the specific process of the camera application sending a video shooting request to the mobile phone camera can refer to the process introduced above of the corresponding application sending a video shooting request to the camera through the camera access interface, camera hardware abstraction layer, and camera device driver.

[0296] S404: The camera receives the video shooting request.

[0297] S405: The camera captures an image in response to the video capture request.

[0298] S406: The gyroscope in the mobile phone sends angular velocity data to the processor and camera hardware abstraction layer in the mobile phone.

[0299] For example, a gyroscope in a mobile phone detects the movement of the phone body and can collect angular velocity data in real time or periodically, and save the collected angular velocity data to a queue. Other modules or processes (such as the camera hardware abstraction layer and the processor) can read the angular velocity data from the queue.

[0300] S407: The processor sends a compensation instruction to the camera based on the angular velocity data of the gyroscope.

[0301] The processor may be the processor 110 shown in FIG. 3A .

[0302] S408: The camera controls the lens to move in the reverse direction in response to the compensation instruction.

[0303] For example, the processor can determine whether the phone is shaking based on the angular velocity data of the gyroscope. If shaking occurs, the motor is driven to control the lens to move in the opposite direction to compensate for the shaking of the phone. If shaking does not occur, there is no need to control the lens to move in the opposite direction.

[0304] In some embodiments, during the exposure of the t-th frame (i.e., the t-th original image), the gyroscope in the camera sends multiple angular velocities, that is, sends the angular velocities corresponding to different moments. If the difference between the angular velocities corresponding to different moments is small, such as the difference between adjacent angular velocities is less than a preset difference, it indicates that the phone is not shaking. If the difference between the angular velocities corresponding to different moments is large, such as the difference between adjacent angular velocities is greater than or equal to the preset difference, it indicates that the phone is shaking.

[0305] S409: After obtaining the timestamp corresponding to the last row of pixels of the t-th frame image, the processor sends a return-to-center instruction to the camera, wherein the return-to-center instruction includes a target position and a target return-to-center speed.

[0306] When the mobile phone obtains the timestamp corresponding to the last row of pixels of the t-th frame image, it determines that the exposure of the t-th frame image is completed.

[0307] S410: The camera receives a return instruction.

[0308] S411 , the camera responds to the above-mentioned return-to-center instruction and controls the lens to move to the target position according to the target return-to-center speed.

[0309] For example, when the exposure of the tth frame image is completed, the processor can send a re-centering instruction to the camera to trigger the camera to re-center. The camera can move the lens to the target position by driving the motor, such as (51, 175). During the movement of the motor, the motor can report the motor coordinates, such as (53, 172), to the camera application abstraction layer and the processor in real time or periodically.

[0310] S412 , when the interval between the t-th frame image and the t+1-th frame image ends, the camera starts exposing the t+1-th frame image.

[0311] S413. After obtaining the timestamp corresponding to the last row of pixels of the t+1th frame image, the camera sends the second coordinate of the motor, the t+1th frame original image, and the parameters corresponding to the t+1th frame original image to the camera hardware abstraction layer.

[0312] When the mobile phone obtains the timestamp corresponding to the last row of pixels of the t+1th frame image, it determines that the exposure of the t+1th frame image is completed. The above-mentioned t+1th frame original image includes the data of the pixel points in the t+1th frame image (such as the position of the pixel points). The parameters corresponding to the above-mentioned t+1th frame original image represent the parameters when the camera captured the image. Exemplarily, the image parameters corresponding to the t+1th frame original image may include one or more of resolution, field of view (FOV) magnification, exposure time, and rolling shutter time.

[0313] It should be understood that the aforementioned interaction between the camera and the camera hardware layer can be implemented through the camera device driver. For example, the camera can send gyroscope angular velocity data to the camera hardware abstraction layer through the camera device driver. Similarly, the camera hardware abstraction layer can send compensation instructions to the camera through the camera device driver.

[0314] S414. The camera hardware abstraction layer uses the angular velocity data, the first coordinate of the motor, the second coordinate of the motor, the camera parameters, the original image of the t+1th frame, and the parameters corresponding to the original image of the t+1th frame as inputs of a preset network model, runs the preset network model, and obtains the target image of the t+1th frame.

[0315] The preset network model is used to adjust the coordinates of the pixels in the original image of frame t+1, combined with either an intrinsic reference de-jittering method or an extrinsic reference de-jittering method, to obtain the target image. For the intrinsic reference de-jittering method, refer to the above description of using camera intrinsic reference de-jittering. For the extrinsic reference de-jittering method, refer to the above description of using camera extrinsic reference de-jittering.

[0316] The above-mentioned camera parameters include the distance (such as the distance corresponding to the original image of the t+1th frame) and the initial coordinates of the optical center. The camera parameters can be sent by the camera or pre-stored. In addition, the distance can also be included in the parameters corresponding to the original image of the t+1th frame, and the camera parameters do not need to include this distance. Similarly, the first coordinate of the motor, that is, the initial position of the motor, can also be sent by the camera or pre-stored.

[0317] It should be understood that the image data corresponding to the above-mentioned t+1th frame original image can be used to align the angular velocity data of the gyroscope at all times. For example, the angular velocity data of the gyroscope collected during the exposure of the t+1th frame original image is corresponded to the t+1th frame image; for another example, each row of pixels in the t+1th frame original image is corresponded to the angular velocity data of the gyroscope collected during the exposure of the t+1th frame original image; for another example, each pixel in the t+1th frame original image is corresponded to the angular velocity data of the gyroscope collected during the exposure of the t+1th frame original image, so as to avoid using the wrong angular velocity data of the gyroscope for calculation. For example, the angular velocity data of the gyroscope collected previously cannot be applied to the current frame image, resulting in incorrect calculation of the relevant data corresponding to the current frame image, such as the original angle.

[0318] In addition, the original angle of the gyroscope used in the process of determining the target image of the t+1th frame may be obtained by integrating the angular velocity data of the gyroscope corresponding to the t+1th frame image.

[0319] S415. The camera hardware abstraction layer sends the t+1th frame of target image to the camera application.

[0320] For example, the camera hardware abstraction layer can send the target image of the t+1th frame to the camera application through the camera access interface. After receiving the target image of the t+1th frame, the camera application can save and / or display it, such as displaying it in a preview area.

[0321] It should be understood that after the above-mentioned camera captures the original image of the tth frame, it can also perform relevant processing on the original image of the tth frame to obtain the target image of the tth frame. That is, the mobile phone can not only de-jitter the image captured after the back-centering action, but also de-jitter the image captured after the back-centering action is not performed. In other words, the mobile phone can de-jitter each frame of the original image captured by the camera to ensure the de-jitter effect of the video.

[0322] S416: The camera application receives a click operation on the stop button by the user.

[0323] S417 : In response to the click operation on the stop button, the camera application stops recording the video and generates a video file based on all the target images.

[0324] In an embodiment of the present application, during the video recording process, after receiving the image captured by the camera (or described as the original image), the hardware abstraction layer can call the EIS algorithm in the camera algorithm library, that is, input the image and the image data corresponding to the image, the coordinates of the motor and the camera parameters into the preset network model, so that the preset network model adjusts the coordinates of the pixels in the image through the internal parameter de-jittering method or the external parameter de-jittering method, obtains and outputs the target image, realizes rapid de-jittering of the original image, and avoids the optical center position (or shooting angle) of the camera at the beginning of exposing the image due to the return action. Changes, thereby causing the imaging position of the object in the previous frame image to shift on the next frame image.

[0325] In some embodiments, a mobile phone can be mounted on a vibration table and recorded at different frame rates under the same lighting conditions, exposure time, and amplitude. If the degree of judder, or blur, is similar across the videos at different frame rates, it can be determined that the mobile phone is using an inter-frame centering solution.

[0326] Optionally, if the mobile phone uses an inter-frame re-centering solution, it can determine whether the image frames in the video are offset, that is, a jitter problem. If no jitter problem occurs, the mobile phone may use the EIS algorithm proposed in this application to adapt the re-centering action.

[0327] In some embodiments, the present application provides a computer-readable storage medium including computer instructions. When the computer instructions are executed on a Bluetooth device, the electronic device executes the text processing method described above.

[0328] In some embodiments, the present application provides a computer program product, which, when executed on a Bluetooth device, enables the electronic device to execute the text processing method described above.

[0329] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0330] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0331] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0332] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0333] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0334] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A shooting method, characterized in that: Applied to an electronic device, the electronic device includes a camera; the method includes: When the electronic device is in a multi-frame shooting state, during the exposure of the t-th frame image, in response to the electronic device shaking, the electronic device moves the lens in the camera to a first position in a first direction, wherein the multi-frame shooting state represents a state of continuously acquiring and processing multiple frames of images, the first direction is a direction opposite to the shaking, and t is a positive integer; In response to the end of exposure of the t-th frame image, if the position of the lens at the end of exposure of the t-th frame image is not within a first preset position range, the electronic device controls the lens to move from the first position to the center point within a first target time, wherein the first target time is the interval time between the t-th frame image and the t+1-th frame image; the center point is the center point of the moving range of the lens; the absolute value of the coordinate of the position point in the x direction in the first preset position range is less than the first preset coordinate value, and the absolute value of the coordinate of the position point in the y direction is less than the second preset coordinate value; the center point is within the first preset position range; The electronic device displays the tth frame image.

2. The method according to claim 1, characterized in that The method further comprises: When the position of the lens at the end of exposure of the t-th frame image is within a first preset position range, the electronic device does not control the lens to move from the first position to the center point within the first target time.

3. The method according to claim 1, characterized in that The first target time includes a frame gap; Controlling the lens to move from the first position to the center point within the first target time includes: In the frame interval of the first target time, the electronic device controls the lens to move from the first position to the center point and to the target position.

4. The method according to claim 3, characterized in that The electronic device controls the lens to move from the first position to the center point, including: The electronic device acquires the target return speed corresponding to the frame interval; The electronic device controls the lens to move from the first position to the center point at a target return speed corresponding to the frame interval; or, The electronic device calculates a ratio between a return-to-center distance and the frame interval to obtain a target return-to-center speed, wherein the return-to-center distance represents a distance between a first coordinate of the lens and the target position, and the first coordinate of the lens represents a position of the lens when the exposure of the t-th frame image ends; The electronic device controls the lens to move from the first position to the center point according to the target return-to-center speed.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: After the exposure of the t+1 frame image is completed, the electronic device adjusts the position of the pixels in the t+1 frame image based on the offset corresponding to the lens and in combination with the electronic image stabilization technology EIS algorithm to obtain the t+1 frame target image; the offset corresponding to the lens represents the offset value of the position of the lens at the end of the exposure of the t+1 frame image compared to the initial position of the lens; The electronic device displays and / or stores the target image of the t+1th frame.

6. The method according to claim 5, characterized in that The offset corresponding to the lens represents the offset corresponding to the center of the image plane corresponding to the camera; The electronic device adjusts the position of the pixel in the t+1 frame image based on the offset corresponding to the lens and in combination with the electronic image stabilization technology EIS algorithm to obtain the t+1 frame target image, including: The electronic device determines the first coordinate of the center of the image plane based on the initial coordinate of the center of the image plane and the offset corresponding to the center of the image plane; wherein the first coordinate of the center of the image plane represents the position of the center of the image plane when the exposure of the t+1th frame of image ends; The electronic device determines a first camera intrinsic parameter based on a first coordinate of the center of the image plane and a distance corresponding to the camera; the distance represents a distance between the image plane and the center of the lens; The electronic device adjusts the positions of pixels in the t+1th frame image according to a first transformation matrix; the first transformation matrix is ​​obtained based on the first camera intrinsic parameter.

7. The method according to claim 5, characterized in that The offset corresponding to the lens represents the offset corresponding to the center of the image plane corresponding to the camera; The electronic device adjusts the position of the pixel in the t+1 frame image based on the offset corresponding to the lens and in combination with the electronic image stabilization technology EIS algorithm to obtain the t+1 frame target image, including: The electronic device converts the offset corresponding to the center of the image plane into an equivalent angle of the gyroscope in the camera; the equivalent angle represents the offset angle of the electronic device when the exposure of the tth frame ends; The electronic device obtains an adjustment angle corresponding to the t+1th frame image based on the equivalent angle, where the adjustment angle represents an angle that needs to be adjusted when the t+1th frame image reaches an ideal shooting angle; The electronic device obtains a first rotation matrix based on the adjustment angle and a preset rotation matrix calculation formula; The electronic device adjusts the positions of the pixels in the t+1th frame image according to a second transformation matrix; the second transformation matrix is ​​obtained based on the first rotation matrix.

8. The method according to claim 7, characterized in that The electronic device obtains the adjustment angle corresponding to the t+1th frame image based on the equivalent angle, including: The electronic device calculates the difference between the original angle corresponding to the t+1th frame image and the equivalent angle to obtain the actual angle corresponding to the t+1th frame image; wherein the original angle corresponding to the t+1th frame image represents the angle of movement of the electronic device when the t+1th frame image is exposed; and the actual angle corresponding to the t+1th frame image represents the actual shooting angle of the lens when exposing the t+1th frame image; The electronic device calculates the difference between the actual angle corresponding to the t+1th frame image and the target angle corresponding to the t+1th frame image to obtain the adjustment angle corresponding to the t+1th frame image; the target angle corresponding to the t+1th frame image represents the ideal shooting angle corresponding to the t+1th frame image when the previous t+1 frames of images are all stable images.

9. The method according to claim 7 or 8, characterized in that: The converting the offset corresponding to the center of the image plane into an equivalent angle of a gyroscope in the camera includes: The electronic device obtains the equivalent angle based on the ratio between the offset corresponding to the center of the image plane and the distance corresponding to the camera in combination with an inverse trigonometric function.

10. The method according to claim 9, characterized in that The offset corresponding to the center of the image plane includes the offset of the center of the image plane in the x direction, and the equivalent angle includes the equivalent angle in the x direction; The equivalent angle in the x direction is obtained by the inverse trigonometric function arctan (shift x / f) calculated, the shift x represents the offset of the center of the image plane in the x direction, and f represents the distance.

11. The method according to any one of claims 7 to 10, characterized in that The electronic device adjusts the positions of pixels in the t+1th frame image according to the second transformation matrix, including: For each pixel in the t+1th frame image, the electronic device multiplies the coordinates of the pixel by the second transformation matrix.

12. The method according to claim 5, characterized in that The offset corresponding to the lens represents the offset corresponding to the center of the image plane corresponding to the camera; The electronic device adjusts the position of the pixel in the t+1 frame image based on the offset corresponding to the lens and in combination with the electronic image stabilization technology EIS algorithm to obtain the t+1 frame target image, including: The electronic device uses the offset corresponding to the center of the image plane and the distance corresponding to the camera as input parameters of a preset network model, runs the preset network model, and obtains the t+1th frame target image; the preset network model is obtained based on the EIS algorithm, and the preset network model is used to adjust the position of pixels in the image based on the offset corresponding to the center of the image plane and the distance.

13. The method according to any one of claims 1 to 12, characterized in that The electronic device controls the lens to move from the first position to the center point within a first target time, including: The electronic device controls the lens to move from the first position to the center point to the target position within a first target time, and the target position includes the center point and / or a first preset position of the lens on the movement range during the exposure of the tth frame image.

14. The method according to any one of claims 5 to 13, characterized in that The multi-frame shooting state includes a video shooting state; Before the exposure period of the t-th frame image, the method further includes: The electronic device receives a first operation, where the first operation is used to trigger video recording; After the electronic device obtains the t+1th frame of the target image, the method further includes: The electronic device receives a second operation; In response to the second operation, the electronic device stops recording the video and generates a video file based on all the obtained target images.

15. An electronic device, characterized in that: The electronic device includes a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; the display screen is used to display an image generated by the processor, and the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the method described in any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 14.

Images