A video zooming method and related device

Abstract

The application provides a video zooming method and related equipment. According to the method, when the electronic device is recording a video, if the user triggers zooming, different methods are adopted for preview stream and video stream to realize the alignment of the FOV center. On the one hand, for the video stream, the electronic device can correct the parallax corresponding to the video frame to be output based on multiple groups of double-stream images, so as to obtain a more accurate spatial alignment matrix, and then more accurately realize the alignment of the FOV center. In this way, the video picture obtained by the electronic device is smoother. On the other hand, for the preview stream, the electronic device can determine the spatial alignment matrix of the preview frame to be output based on calibration information, distance information and double-stream images, which not only ensures a certain accuracy, but also realizes high real-time performance.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a video zoom method and related equipment. Background Technology

[0002] Currently, mobile phones, tablets, and other electronic devices generally include multiple physical cameras (simply referred to as cameras / cameras, or camera modules / camera modules). Different cameras can have different focal lengths. During video recording, the electronic device can switch cameras based on user input to achieve zoom. For example, when zooming to 1x or less, an ultra-wide-angle camera can be used, while zooming to 1x or more, a wide-angle camera can be used. During the process of switching cameras to achieve zoom, due to differences in camera mounting positions, such as differences in the position of the lenses within different cameras, the center point of the image output by the camera before and after the switch will shift. Summary of the Invention

[0003] This application provides a video zoom method and related equipment. According to the method, during the recording of video by an electronic device, in response to the zoom operation, during the acquisition of video frames, the electronic device can determine the parallax corresponding to the target dual-stream image through multiple sets of dual-stream images output by the original camera and the target camera. This results in a more accurate parallax, better subsequent image alignment, and higher quality video frames.

[0004] In a first aspect, this application provides a video zoom method. This method can be applied to an electronic device including a first camera and a second camera. The method may include: during video recording using the first camera, in response to a first operation, the electronic device can output dual-stream images via the first and second cameras, the first operation being used to trigger zoom; during the acquisition of video frames based on the dual-stream images, the electronic device can determine a first target parallax based on multiple sets of dual-stream images, the multiple sets of dual-stream images including a target dual-stream image; the electronic device can also process the target dual-stream image based on the first target parallax to obtain a target video frame.

[0005] In the solution provided in this application, the user can trigger zoom during video recording using an electronic device. In response to this zoom operation, the electronic device can acquire multiple sets of dual-stream images output by the original camera and the target camera for the video stream. Based on these multiple sets of dual-stream images, the FOV center movement trajectory of the original camera is optimized. This means that the FOV alignment effect of each set of dual-stream images can be improved based on more global information. This avoids parallax anomalies caused by differences in a single image during FOV center alignment, which would lead to an uneven overall movement trajectory and thus prevent images with obvious jumps in the subsequently recorded video. Furthermore, the parallax acquired based on the dual-stream images is used as the standard during video frame acquisition. Compared to parallax obtained based on calibration information, the parallax acquired based on the dual-stream images is more accurate, resulting in higher quality video frames.

[0006] In addition, since the above scheme reserves more global information, it can provide greater possibilities for the algorithm's performance. For example, it can align the image content of images output from different cameras when zooming, perform AI scene alignment, and image AI fusion.

[0007] In conjunction with the first aspect, in one possible implementation, the aforementioned multiple sets of dual-stream images may include at least one set of dual-stream images output by the original camera and the target camera before outputting the target dual-stream image, and / or at least one set of dual-stream images output by the original camera and the target camera after outputting the target dual-stream image.

[0008] In the solution provided in this application, the electronic device can optimize parallax not only by combining future image information, but also by combining historical image information. This allows for the optimization of the movement trajectory of the FOV center based on more global information, making the acquired video frames more coherent and smooth with the video frames before and after them.

[0009] In conjunction with the first aspect, in one possible implementation, the electronic device determines a first target disparity based on multiple sets of dual-stream images. Specifically, the electronic device can determine multiple disparities based on the multiple sets of dual-stream images, and the multiple disparities can include a first disparity, which is a disparity determined based on the target dual-stream image. If the first disparity satisfies a first condition, the electronic device can process the first disparity based on the multiple disparities to obtain the first target disparity.

[0010] In the solution provided in this application, the electronic device can obtain multiple parallaxes based on multiple sets of dual-stream images output by the original camera and the target camera. Based on these multiple parallaxes, it can determine whether the parallax determined based on the target dual-stream image is an unreasonable parallax. If the parallax is determined to be unreasonable, the electronic device can process the parallaxes determined based on other dual-stream images in the multiple sets of dual-stream images to obtain a reasonable parallax. Based on this reasonable parallax, the target dual-stream image is then processed to obtain the target video frame. This allows for the filtering out of calculated unreasonable parallaxes by combining multiple sets of dual-stream images, improving parallax accuracy and thus improving the accuracy of FOV center alignment. This makes the movement trajectory of the FOV center smoother, resulting in a smoother video after subsequent recording and preventing screen jumps.

[0011] In conjunction with the first aspect, in one possible implementation, if the first disparity does not satisfy the first condition, the first target disparity can be the first disparity.

[0012] In the solution provided in this application, if the electronic device determines that the disparity determined based on the target dual-stream image is a reasonable disparity based on the multiple disparities, it can directly use the disparity for FOV center alignment without performing other processing, which can simplify the process and save energy to a certain extent.

[0013] In conjunction with the first aspect, in one possible implementation, the electronic device processes a first disparity based on multiple disparities to obtain a first target disparity. Specifically, this may include: the electronic device filtering out the first disparity and determining a first average value, where the first average value is the average value of a first type of disparity among the multiple disparities, and the first type of disparity refers to disparities that do not satisfy a first condition. The first target disparity can be the first average value.

[0014] In the solution provided in this application, if the electronic device determines that the disparity determined based on the target dual-stream image is an unreasonable disparity based on the multiple disparities, the electronic device can use the average value of the disparities calculated based on other dual-stream images as the reasonable disparity, that is, combine global information to determine the disparity. In this way, a more accurate disparity can be obtained, thereby optimizing the movement trajectory of the FOV center, making the video obtained after subsequent recording smoother and avoiding the occurrence of screen jumps.

[0015] In conjunction with the first aspect, in one possible implementation, the first disparity satisfies the first condition, which may specifically include: the difference between the first disparity and the second average value is greater than the first threshold, and the second average value is the average of the aforementioned multiple disparities.

[0016] In the solution provided in this application, during the process of determining whether the disparity determined based on the target dual-stream image is an unreasonable disparity based on the multiple disparities, the disparity determined based on the target dual-stream image can be determined based on the degree of difference between the multiple disparities. If the disparity determined based on the target dual-stream image is significantly different from other disparities, the electronic device can determine that the disparity determined based on the target dual-stream image is an unreasonable disparity (or abnormal disparity). This may be due to abnormal jitter or stuttering, etc. In this way, these unreasonable disparities can be screened out in time to ensure the accuracy of subsequent FOV center alignment.

[0017] In conjunction with the first aspect, in one possible implementation, the method may further include: during the acquisition of video frames based on dual-stream images, the electronic device may determine a disparity based on each set of dual-stream images output by the first and second cameras, store the obtained disparity in a first queue, and update the first queue based on the acquisition status of the video frame. Wherein, the number of disparities in the first queue is greater than a second threshold.

[0018] In the solution provided in this application, the electronic device can store multiple disparities determined based on multiple sets of dual-stream images in the form of a queue, and control the number of elements in the queue and the enqueue and dequeue rules, which can more effectively realize the management of the multiple disparities and the subsequent updates of more disparities.

[0019] In conjunction with the first aspect, in one possible implementation, the first queue may include multiple disparities. The electronic device determines the first target disparity based on multiple sets of dual-stream images. Specifically, the electronic device may filter the multiple disparities in the first queue to obtain a processed first queue, which may include the first target disparity.

[0020] In the solution provided in this application, after the electronic device acquires a new disparity based on a new set of dual-stream images, it can put it into a queue and filter the disparities in the queue if the number of queue elements meets the requirements, thereby eliminating unreasonable disparities. In this way, unreasonable disparities can be eliminated directly by managing and maintaining the queue, which is more efficient.

[0021] In conjunction with the first aspect, in one possible implementation, the method further includes: during the process of acquiring a preview frame based on the dual-stream image, the electronic device can determine a second target disparity based on calibration information and / or the target dual-stream image; the electronic device can also process the target dual-stream image based on the second target disparity to obtain the target preview frame.

[0022] In the solution provided in this application, the user can trigger zoom during video recording using an electronic device. In response to the zoom operation, for the preview stream, the electronic device can determine the parallax based on calibration information to ensure timely acquisition of preview frames and real-time display of the image. Furthermore, when dual-stream images can be acquired in a timely manner, the parallax determined based on the dual-stream images can be adjusted to the parallax determined based on the calibration information, ensuring the accuracy of the parallax to a certain extent while maintaining real-time performance.

[0023] In a second aspect, this application provides an electronic device including one or more memories and one or more processors; the one or more memories may be coupled to the one or more processors, the memories being used to store computer program code including computer instructions, the one or more processors calling the computer instructions to cause the electronic device to perform the method as described in the first aspect or any implementation thereof.

[0024] Thirdly, this application provides a computer storage medium. The computer storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect or any implementation thereof.

[0025] Fourthly, embodiments of this application provide a chip. This chip can be applied to an electronic device, and the chip includes one or more processors. The processors are used to invoke computer instructions to cause the electronic device to perform the methods described in the first aspect or any implementation thereof.

[0026] Fifthly, embodiments of this application provide a chip system. This chip system can be applied to an electronic device. The chip system includes at least one processor and an interface for receiving instructions and transmitting them to the at least one processor; the at least one processor executes the received instructions, causing the electronic device to perform the method described in the first aspect or any implementation thereof.

[0027] In some embodiments of this application, the chip system may be an application processor (AP) or a system on chip (SoC) including an AP, and the method described in the first aspect or any implementation thereof may be implemented by an AP.

[0028] In some other embodiments of this application, the chip system may include an access point (AP) and other modules. These other modules may be a modem (also known as a baseband processor).

[0029] Sixthly, embodiments of this application provide a computer program product including instructions. When the computer program product is run on an electronic device, it causes the electronic device to perform the method described in the first aspect or any implementation thereof.

[0030] It is understood that the electronic device provided in the second aspect, the computer storage medium provided in the third aspect, the chip provided in the fourth aspect, the chip system provided in the fifth aspect, and the computer program product provided in the sixth aspect are all used to execute the method described in the first aspect or any implementation thereof. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects of any possible implementation of the first aspect, which will not be repeated here. Attached Figure Description

[0031] Figure 1 A schematic diagram of FOV variation provided in an embodiment of this application;

[0032] Figure 2 This is a schematic diagram illustrating a screen transition caused by switching cameras, provided as an embodiment of this application.

[0033] Figure 3 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0034] Figure 4 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0035] Figure 5 A flowchart illustrating a video zoom method provided in this application embodiment;

[0036] Figure 6 This application provides an embodiment of an image processing diagram for a video zoom process.

[0037] Figure 7A A flowchart illustrating a video zoom method provided in this application embodiment;

[0038] Figure 7B A flowchart illustrating yet another video zoom method provided in this application embodiment;

[0039] Figure 8A A schematic diagram of video stream processing provided in an embodiment of this application;

[0040] Figure 8B A schematic diagram of a preview stream processing provided for an embodiment of this application;

[0041] Figure 9 A schematic diagram of an aligned FOV provided for an embodiment of this application;

[0042] Figure 10This is a schematic diagram illustrating the change in the position of the subject during a video zoom process, provided as an embodiment of this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0044] In this application, the use of singular pronouns to denote "one or more" rather than "one and only one," unless otherwise specified. Furthermore, "at least one," "one or more of the following," or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c; a and b; a and c; b and c; or a and b and c. Here, a, b, and c can be single or multiple. Additionally, in the description of embodiments in this application, "multiple" means two or more.

[0045] It is understood that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, rather than to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0046] In the description of this application, the words "exemplary" or "for example" are used to indicate that something is an example, illustration, or illustration. Any embodiment or design that is described as "exemplary," "for example," or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of the words "exemplary," "for example," or "for example" is intended to present the relevant concepts in a specific manner.

[0047] It is understood that in the description of this application, "when," "under what circumstances," "if," and "if" all refer to the device performing a corresponding action under certain objective circumstances, and are not time-limited, nor do they require the device to perform a judgment action when it is implemented, nor do they imply any other limitations. Specifically, the device performing a corresponding action under certain objective circumstances includes: satisfying the objective circumstances, i.e., being able to perform the corresponding action; or satisfying both the objective circumstances and other circumstances, in order to perform the corresponding action.

[0048] It is understood that the term "user interface" in the specification, claims, and drawings of this application refers to the medium interface through which an application or operating system interacts and exchanges information with the user. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of a terminal device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0049] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0050] First, some of the terms and related technologies used in this application will be explained to facilitate understanding by those skilled in the art.

[0051] 1. Internal and external references

[0052] (1) Internal Reference

[0053] Intrinsic parameters describe the camera's internal geometry and optical characteristics and are primarily used to map points in the camera coordinate system to the pixel coordinate system.

[0054] The intrinsic parameter matrix is ​​a 3×3 matrix, usually represented as:

[0055] Among them, f x and f y These are the focal lengths of the image in the x and y directions (in pixels), reflecting the relationship between the physical focal length and the pixel size. f is the physical focal length, and dx and dy are the physical dimensions of a pixel. f can be in millimeters, while dx and dy can be in millimeters per pixel. x and c y It is the coordinate of the principal point (the intersection of the optical axis and the imaging plane) in the pixel coordinate system, usually close to the center of the image. w and h are the width and height of the image (in pixels).

[0056] It is understood that the camera / camera involved in this application may include lenses, image sensors, etc. In some embodiments of this application, the camera / camera in the electronic device may also include other processing modules, such as depth sensors, and this application does not impose any limitations on this.

[0057] (2) External reference

[0058] Extrinsic parameters describe the camera's position and orientation in the world coordinate system. Camera extrinsic parameters can include camera pose.

[0059] Camera pose typically consists of two parts: position (translation) and orientation (rotation). It refers to the camera's location and orientation in three-dimensional space. The position mentioned here refers to the camera's position in the world coordinate system, or the position of the camera's coordinate system relative to the world coordinate system. This is usually represented by a three-dimensional vector, specifically the translation vector t = [t...]. x ,t y ,t z The direction mentioned here refers to the camera's orientation, or the orientation of the camera's coordinate system relative to the world coordinate system, which can be represented by a rotation matrix R, a quaternion, or a rotation vector.

[0060] The pose of a camera can be described as a transformation of a rigid body (i.e., an object whose shape and size remain unchanged during motion and after being subjected to forces, and whose relative positions of all points inside remain unchanged), usually represented by a homogeneous transformation matrix T, in the following form: Where R is a 3×3 matrix and t is a 3×1 vector.

[0061] 2. World coordinate system, camera coordinate system, and pixel coordinate system

[0062] (1) World Coordinate System

[0063] The world coordinate system is a global, fixed reference coordinate system used to describe the position and orientation of all objects and cameras in the entire scene. It applies to the entire scene, is independent of any specific camera or object, and does not change with camera movement or object changes. The world coordinate system is a unified reference frame, typically choosing a fixed point in the scene as its origin.

[0064] The world coordinate system, as a reference coordinate system, consists of an origin, X, and N. w Axis, Y w Axis and Z w Composed of axes. Understandably, X... w Axis, Y w Axis and Z w The axes are orthogonal to each other, and the point where these three axes intersect is the origin. The world coordinate system can be used to describe the location of a scene in the real world.

[0065] (2) Camera coordinate system

[0066] The camera coordinate system is a local coordinate system centered on the camera, used to describe the position and orientation of objects within the camera's field of view.

[0067] The camera coordinate system has its origin at the camera's focal center (optical center) and its coordinate system around the optical axis (Z). c A three-dimensional Cartesian coordinate system is established using the axes. The other two axes of the camera coordinate system are X, Y, and Z. c axis and Y c The axes are parallel to the x-axis and y-axis of the image physical coordinate system (i.e., the imaging plane coordinate system), respectively.

[0068] (3) Pixel coordinate system

[0069] A pixel coordinate system is a two-dimensional coordinate system used to describe the position of each pixel in an image. Its origin is usually located at the top left corner of the image, with the horizontal direction being the x-axis (or u-axis) and the vertical direction being the y-axis (or v-axis). The unit of the pixel coordinate system is the pixel.

[0070] Using the camera's intrinsic parameter matrix K, points in the camera coordinate system can be converted to points in the pixel coordinate system. The conversion formula is:

[0071] 3. Camera Calibration

[0072] Electronic devices can define a fixed coordinate system (i.e., the world coordinate system) and calibrate each of their cameras to obtain their intrinsic parameters (e.g., focal length, principal point, etc.) and extrinsic parameters (e.g., pose).

[0073] 4. Parallax and Depth

[0074] Parallax is a core concept in stereo vision, used to describe the difference in position of an object in the same scene across images taken from different viewpoints (e.g., images taken from left and right cameras). Specifically, parallax refers to the horizontal position offset (i.e., the difference in horizontal coordinates) of the same object in the left and right images. If the coordinates of a point in the left image are (x...)... L The coordinates of the corresponding point in the right image are (x, y), R If x, y), then the disparity d is defined as: d = xL -x R .

[0075] Parallax reflects the difference in position of an object in the left and right images, and is inversely proportional to the object's depth (distance between the object and the camera). The greater the parallax, the closer the object is to the camera; the smaller the parallax, the farther the object is from the camera. The relationship between parallax and object depth can be expressed as: Where Z is the depth of the object, i.e. the straight-line distance between the object and the camera (e.g., the camera), f is the focal length of the camera, and B is the horizontal distance between the two cameras.

[0076] 5. Scale alignment

[0077] Scale alignment refers to adjusting the size of two images so that they have a consistent proportional relationship in terms of physical size or pixel size. The purpose of scale alignment is to ensure that the two images are consistent in spatial scale in order to facilitate subsequent processing (e.g., image fusion, stitching, or hybrid zoom).

[0078] 6. Field of view

[0079] The field of view (FOV) refers to the angle of the scene that a camera or sensor can capture. It is determined by the camera's intrinsic parameters (such as focal length and sensor size).

[0080] 7. FOV alignment

[0081] FOV alignment refers to adjusting the field of view of two images so that they capture the same range of the scene. This is commonly used in multi-camera systems, such as aligning the field of view of images output from wide-angle and telephoto lenses.

[0082] 8. Zoom magnification

[0083] The zoom factor mentioned in this application can characterize the degree of change in the focal length of the camera. Correspondingly, the zoom factor can also characterize the degree of change in the field of view and the size change of the subject in the image (i.e., the image displayed on the screen of an electronic device). Here, the subject refers to the object being photographed by the camera.

[0084] Understandably, when the distance between the subject and the camera is the same, the proportion of the subject in an image captured by a telephoto camera will be larger compared to an image captured by a short-focal-length camera. Specifically, when the focal length changes to n times its original focal length, the size of the subject in the frame also changes to n times its original size. For example, if the camera's focal length is increased to twice its original focal length, the size of the subject in the frame will also increase to twice its original size.

[0085] Furthermore, the field of view changes with the focal length. Generally, the shorter the focal length, the larger the field of view. For example, a focal length of 15 mm results in a field of view of approximately 8 degrees; 10 mm results in approximately 12 degrees; 7.5 mm results in approximately 14 degrees; 5 mm results in approximately 23 degrees; and 2.5 mm results in approximately 45 degrees.

[0086] The zoom factor can be expressed as nx. It means the focal length changes to n times the original focal length. Here, n is a positive number. As mentioned above, the larger n is, the longer the focal length, the smaller the field of view, and the larger the subject in the image captured by the camera. It is understood that this application does not limit the specific value of n. For example, n can be 1, 2, or 10.

[0087] In some embodiments of this application, the original focal length may refer to the focal length of the main camera (hereinafter referred to as the main camera). In this case, the zoom factor can be used to describe the relative relationship between the focal length of different cameras and the focal length of the main camera. Simply put, the main camera is the core camera (i.e., the core camera) in the camera system of an electronic device, and it usually has the highest pixel count, the largest aperture, and the best image quality. The focal length of the main camera is usually close to the human eye's field of view, suitable for most shooting scenarios. In still some embodiments of this application, the original focal length refers to the focal length of a standard camera. A standard camera refers to a camera with a field of view generally in the range of 35 degrees to 50 degrees, and its focal length is approximately equal to the diagonal length of the captured image.

[0088] In some embodiments of this application, the electronic device may include multiple cameras. In this case, in response to a user triggering an operation to adjust the zoom level, the electronic device can achieve the target zoom level by switching between cameras with different focal lengths. That is, the electronic device can meet the user's zoom needs by switching between cameras with different focal lengths.

[0089] For example, an electronic device may include an ultra-wide-angle lens, a wide-angle main camera, and a telephoto lens. When the electronic device uses the wide-angle main camera to take a picture, the zoom level is 1x. In response to a user triggering an operation to adjust the zoom level to 2x, the electronic device can switch from the wide-angle main camera to the telephoto lens.

[0090] As mentioned above, electronic devices can switch cameras based on user input to achieve zoom. However, during the process of switching cameras to achieve zoom, differences in camera mounting positions, such as differences in the position of lenses in different cameras, cause a shift in the center point of the image output by the camera before and after the switch. It should be noted that the center point of the image output by the camera mentioned in this application refers to the center point of the field of view (FOV) of the camera's output image.

[0091] For ease of understanding and description, this application refers to the cameras before and after the above-mentioned switching as the original camera and the target camera, respectively. That is, the camera used before zooming (i.e. the camera used before switching the camera) is referred to as the original camera, and the camera used after zooming (i.e. the camera used after switching the camera) is referred to as the target camera.

[0092] In some embodiments of this application, such as Figure 1 As shown, after the user triggers zoom, the electronic device can control both the original camera and the target camera to simultaneously output images (referred to as image output). A spatial alignment transform (SAT) algorithm is used to align the images output by the original and target cameras, causing the field of view (FOV) center of the original camera's output image to continuously shift, thereby aligning the FOV center of the original camera's output image with that of the target camera's output image. Once the FOV center of the original camera's output image and the target camera's output image are aligned, the electronic device can switch to outputting the image from the target camera and display that image.

[0093] It should be noted that after the electronic device switches to the target camera, it will process and display the images captured by the target camera, instead of continuing to capture images through the original camera.

[0094] It should also be noted that "switching cameras" in this application refers to replacing the display camera. A display camera, in a multi-camera system, is the camera whose image data is directly used for display. Typically, the image data from the display camera is processed before being sent to the display screen for display. In some embodiments of this application, while the display camera is outputting an image, the electronic device can also control other cameras to output images, that is, use these other cameras as auxiliary paths to assist the display camera in calculations. In other words, the image data from these other cameras is not directly used for display, but rather for auxiliary calculations, such as depth estimation, image fusion, and motion detection.

[0095] For example, the zoom scenario mentioned above involves switching the display camera. Before switching the display camera (i.e., switching from the original camera to the target camera), the electronic device can control both the original camera and the target camera to output images. In this case, the display camera is still the original camera, while the target camera acts as an auxiliary path to assist in the calculation (such as image alignment).

[0096] In one possible implementation, such as Figure 1 As shown, after the electronic device switches to the target camera for display (or switches to the target camera as the display camera), it no longer needs to control the original camera for image output, that is, the original camera is not needed as an auxiliary path to assist in the calculation.

[0097] In another possible implementation, after the electronic device switches to displaying the target camera, it can also control the original camera to output the image. In this case, the target camera displays the image, while the original camera acts as an auxiliary path to assist in the calculation (such as subsequent image fusion).

[0098] For example, when an electronic device zooms to 1x or less, the display is sent through the ultra-wide-angle camera, and the main camera (such as the wide-angle camera) acts as an auxiliary path to assist in the calculation. When the electronic device zooms to 1x or more, the display is sent through the main camera, and the ultra-wide-angle camera acts as an auxiliary path to assist in the calculation.

[0099] However, the above methods have many drawbacks. For example, to avoid affecting zoom latency, the SAT algorithm needs to complete the calculation within 1 to 2 frames. This means that when the screen displayed on the electronic device changes significantly or the zoom is rapid, the SAT algorithm may not be able to guarantee the calculation effect of each frame, resulting in severe screen jumps when switching cameras. For instance, when zooming is rapid, the existing SAT algorithm cannot guarantee the calculation effect of each frame, leading to inaccurate FOV center shift of the original camera. Figure 2 As shown, when an electronic device switches from the original camera to the target camera, the position of the subject in the image changes significantly. Furthermore, switching cameras for image fusion consumes considerable resources.

[0100] Based on the above, this application provides a video zoom method and related equipment. According to this method, when an electronic device is recording video, if the user triggers zoom, different image alignment schemes are adopted for the preview stream and the video stream. On one hand, for the video stream, the electronic device can more accurately obtain the spatial alignment matrix of the video frame to be output based on multiple sets of dual-stream images, thereby more accurately shifting the FOV center of the target camera. This results in a smoother zoom effect in the video acquired by the electronic device. On the other hand, for the preview stream, the electronic device can determine the spatial alignment matrix of the preview frame to be output based on calibration information and distance information. Furthermore, it can combine the images output by the original camera and the target camera to make corrections during the process of determining the spatial alignment matrix, and shift the FOV center of the target camera based on the spatial alignment matrix. This enables fast image alignment and higher real-time performance.

[0101] The apparatus involved in the embodiments of this application will be introduced first below.

[0102] The electronic devices involved in this application can be terminal devices, specifically mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), or dedicated cameras (e.g., SLR cameras, point-and-shoot cameras), etc. The embodiments of this application do not impose any restrictions on the specific type of electronic devices.

[0103] The hardware structure of the electronic device involved in the embodiments of this application is described below.

[0104] Please see Figure 3 , Figure 3 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0105] like Figure 3 As shown, the electronic device may include: a processor, an external memory interface, internal memory, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, antenna 1, antenna 2, a mobile communication module, a wireless communication module, a sensor module, buttons, a motor, an indicator, a camera, a display screen, an audio module, and a subscriber identity module (SIM) card slot, etc. The audio module may include a speaker, receiver, microphone, headphone jack, etc., and the sensor module may include pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, proximity sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0106] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. It is understood that the illustrated components can be implemented in hardware, software, or a combination of both. In some embodiments of this application, the electronic device may include more components than illustrated. For example, the electronic device may include other types of sensors. In still other embodiments of this application, the electronic device may include fewer components than illustrated, or combine some components, or split some components, or arrange different components. The interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device.

[0107] A processor may include one or more processing units, such as an application processor (AP), a modem (also known as a baseband processor), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a neural network processing unit (NPU). The AP is responsible for running the operating system and applications. The modem is responsible for handling various communication protocols.

[0108] The wireless communication function of an electronic device can be implemented through antenna 1, antenna 2, a mobile communication module, a wireless communication module, and a modem. The modem can interact with the base station through antennas (e.g., antenna 1, antenna 2, etc.). In some embodiments, antenna 1 of the electronic device is coupled to the mobile communication module, and antenna 2 is coupled to the wireless communication module, enabling the electronic device to communicate with networks and other devices through wireless communication technology.

[0109] Electronic devices can achieve display functions through GPUs, displays, and application processors.

[0110] A GPU is a microprocessor for image processing, connected to a display screen and an application processor. A GPU performs mathematical and geometric calculations for graphics rendering. A processor may include one or more GPUs, which execute program instructions to generate or modify display information. A display screen is used to display images, videos, etc. In some embodiments, an electronic device may include one or more displays screens.

[0111] A camera is used to capture still images or videos. An ISP (Image Signal Processor) processes the data fed back from the camera. Light is transmitted through the lens to the camera's photosensitive element, where the light signal is converted into an electrical signal. The camera's photosensitive element then transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. Electronic devices may include one or more cameras.

[0112] Internal memory may include one or more volatile memory devices (VMs) and one or more non-volatile memory devices (NVMs). Volatile memory devices are those whose data is lost upon power failure or system restart. These devices are typically used for temporary data storage to improve system performance. Volatile memory devices may include random access memory (RAM). RAM can be directly read and written by the processor and can be used to store executable programs (e.g., machine instructions) of the operating system or other running programs, as well as user and application data.

[0113] Non-volatile storage devices are storage devices whose data is not lost after a power outage or system restart. These devices are typically used for persistent data storage, ensuring data remains unchanged even after power failure. Non-volatile memory can also store executable programs and user / application data, which can be pre-loaded into random access memory for direct read / write operations by the processor. Non-volatile storage devices can include read-only memory (ROM), flash memory, universal flash storage (UFS), or embedded multimedia cards (eMMC), etc.

[0114] In this embodiment, the code implementing the method described in this embodiment can be stored in non-volatile memory. When the electronic device runs a camera application, the electronic device can load the executable code stored in the non-volatile memory into random access memory.

[0115] External memory interfaces can be used to connect to external non-volatile memory, thereby expanding the storage capacity of electronic devices.

[0116] Electronic devices can implement audio functions through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.

[0117] The operating system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture of the Android operating system as an example to exemplify the software structure of an electronic device. It should be noted that although this application uses the Android operating system (which can be simply referred to as the Android system) as an example, its basic principles are equally applicable to electronic devices based on operating systems such as iOS or Windows.

[0118] Figure 4 This is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application.

[0119] The software architecture of electronic devices adopts a layered architecture, dividing the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. Taking the Android system running on an application processor (AP) as an example, in some embodiments of this application, the Android system's software architecture is divided into five layers, from top to bottom: the application layer, the framework layer, the Android runtime and system libraries, the Hardware Abstraction Layer (HAL), and the kernel layer.

[0120] The application layer can include a series of application packages. These application packages may include applications such as camera, gallery, SMS, call, music, navigation, calendar, map, WLAN, Bluetooth, and video. The application layer may also include the system user interface (systemUI). The systemUI is used to manage and display the interface of the electronic device, such as rendering and displaying user interface elements like the device's home screen, notification bar, status bar, and settings menu; handling user input (such as touch, click, and swipe); and passing this input to the corresponding applications or system services.

[0121] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The framework layer can include predefined functions. For example, it may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc. The phone manager provides call functionality for electronic devices, such as managing call status (including connection and disconnection).

[0122] The runtime is responsible for system scheduling and management. The runtime includes the core libraries and the virtual machine. The core libraries consist of two parts: one part contains the functionalities that the programming language (e.g., Java) needs to call, and the other part contains the system's core libraries. The application layer and framework layer run in the virtual machine. The virtual machine executes the programming files (e.g., Java files) of the application layer and framework layer into binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0123] A system library can include multiple functional modules. Examples include a surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), and 2D graphics engines (e.g., SGL). The specific meanings and functions of these modules can be found in relevant technical documentation and will not be elaborated upon here.

[0124] The Hardware Abstraction Layer (HAL) is an interface layer located between the operating system kernel and upper-level software, its purpose being to abstract away hardware. The HAL is an abstract interface for device kernel drivers, providing application programming interfaces (APIs) that allow access to the underlying device to higher-level Java API frameworks. The HAL can provide a standard interface that exposes device hardware functionality to higher-level Java API frameworks. The HAL contains several library modules. When the system framework layer API requests access to the portable device's hardware, the operating system loads the library module for that hardware component.

[0125] like Figure 4 As shown, HAL may include a multi-stream data processing module, a first image alignment module, a second image alignment module, a first image processing module, and a second image processing module.

[0126] The multi-stream data processing module can be used to obtain image data output by the original camera and image data output by the target camera from image data output by multiple cameras, and input them into the first image alignment module and the second image alignment module.

[0127] The first image alignment module and the first image processing module are used to process the preview data (or preview stream, i.e., the image data stream captured by the camera and displayed in real time in the user interface). Specifically, the first image alignment module can be used to align the image data output by the original camera and the target camera, and input the aligned image data into the first image processing module. The first image processing module can be used to process (e.g., image fusion) the image data output by the first image alignment module (i.e., the aligned image data mentioned above), and send the processed image data to the upper-layer modules (e.g., modules in the framework layer and application layer). The image data output by the first image processing module to the upper-layer modules is the preview data. In this way, the application layer can obtain the image data output by the first image processing module and render the image data onto the screen through a display interface (e.g., Surface) (such as ImageReader).

[0128] The second image alignment module and the second image processing module are used to process video data (or video stream, i.e., a continuous sequence of video frames encoded and transmitted at a certain rate). Specifically, the second image alignment module can be used to align the image data output by the original camera and the target camera, and input the aligned image data into the second image processing module. The second image processing module can be used to process the image data output by the second image alignment module (i.e., the aligned image data mentioned above) (e.g., image fusion), and send the processed image to upper-layer modules (e.g., modules in the frame layer and application layer). The image data output by the second image processing module to the upper-layer modules is video data. Thus, the application layer can obtain the image data output by the second image processing module, encode the image data using an encoder (e.g., MediaCodec), convert it into a format suitable for transmission, and then save the encoded image data to memory through the file system.

[0129] The kernel layer is the foundation of the Android system. It is responsible for hardware drivers, networking, power management, system security, and memory management. The kernel layer acts as an intermediary between hardware and software, relaying application requests to the hardware. It can include display drivers and camera drivers. Of course, the kernel layer can also include other drivers, such as memory drivers (or simply storage drivers) and encoder drivers (or simply encoding drivers), but this application does not impose any limitations on this.

[0130] It should be noted that the application provides Figure 4 The illustrated software architecture diagram of the electronic device is merely an example and does not limit the specific module divisions within different layers of the Android system. For details, please refer to the descriptions of the Android system software architecture in conventional technologies. Furthermore, the method provided in this application can also be implemented on other operating systems, which will not be listed here.

[0131] The following describes a video zoom method provided in this application.

[0132] Please see Figure 5 , Figure 5 A flowchart illustrating a video zoom method provided in this application embodiment. This method may include, but is not limited to, the following steps:

[0133] S101: During the recording of video using the original camera by the electronic device, in response to the first operation, the electronic device outputs dual-stream images through the original camera and the target camera.

[0134] The first operation can be that the user triggers the zoom from the original zoom level to the target zoom level.

[0135] During video recording using the original camera on an electronic device, the user can trigger zoom. Based on this zoom trigger, the electronic device can determine the target camera and output dual-stream images from both the original and target cameras. The original camera can be any camera in the electronic device's multi-camera system. The target camera is the camera corresponding to the critical zoom level closest to the target zoom level. The critical zoom level is the minimum or maximum zoom level required to switch cameras.

[0136] It should be noted that if an electronic device includes only two cameras, there is only one critical zoom magnification. If an electronic device includes more than two cameras, there are at least two critical zoom magnifications. For example, if an electronic device includes only an ultra-wide-angle camera and a wide-angle camera, the critical zoom magnification can be 1x. As another example, if an electronic device includes an ultra-wide-angle camera, a wide-angle camera, and a telephoto camera, the critical zoom magnification between the ultra-wide-angle camera and the wide-angle camera is 1x, while the critical zoom magnification between the wide-angle camera and the telephoto camera is 5x.

[0137] For ease of understanding and description, this application designates the critical zoom factor closest to the target zoom factor as the first critical zoom factor. The first critical zoom factor is the minimum or maximum zoom factor required to trigger a switch from the original camera to the target camera for video recording. If the zoom factor using the original camera is less than the zoom factor using the target camera, the first critical zoom factor is the minimum zoom factor required to trigger a switch from the original camera to the target camera for recording; if the zoom factor using the original camera is greater than the zoom factor using the target camera, the first critical zoom factor is the maximum zoom factor required to trigger a switch from the original camera to the target camera for recording.

[0138] It is understandable that the difference between the target zoom level and the first critical zoom level is smaller than the difference between the original zoom level and the first critical zoom level.

[0139] It should be noted that dual-stream imaging refers to two image streams, meaning both the source camera and the target camera output image data. In one possible implementation, the source camera and the target camera output image data at the same frame rate. In another possible implementation, the time difference between each frame output by the source camera and the target camera is less than a certain threshold. It is understood that this threshold can be determined based on one or more factors such as the synchronization method of the source and target cameras, the hardware performance of the electronic equipment, etc., and this application does not impose any restrictions on it. For example, this threshold can be at the microsecond level, such as 2 microseconds.

[0140] S102: During the process of acquiring video frames based on dual-stream images, the electronic device determines a first target parallax based on multiple sets of dual-stream images, including the target dual-stream image.

[0141] In some embodiments of this application, each set of dual-stream images may include a frame output by the original camera and a frame output by the target camera. In one possible implementation, the frame output by the original camera and the frame output by the target camera refer to a frame output by the original camera and the target camera at the same time. In another possible implementation, the time difference between the frame output by the original camera and the frame output by the target camera is less than a certain threshold, as detailed in the above description.

[0142] In some embodiments of this application, each set of dual-stream images may include multiple frames of images output by the original camera and multiple frames of images output by the target camera.

[0143] It is understood that the multiple sets of dual-stream images may include at least one set of dual-stream images output before the target dual-stream image, and / or at least one set of dual-stream images output after the target dual-stream image. That is, the electronic device can determine the first target disparity not only by combining historical dual-stream images, but also by combining future dual-stream images. This allows for the filtering out of unreasonable disparities (such as abnormal disparities caused by abnormal jitter) by combining multiple disparities determined based on multi-frame image data.

[0144] In some embodiments of this application, the electronic device can determine a disparity for each set of dual-stream images. That is, the electronic device can determine multiple disparities based on the multiple sets of dual-stream images, and the multiple disparities may include the disparity determined based on the target dual-stream image, i.e., the first disparity.

[0145] In some embodiments of this application, if the first parallax meets the first condition, the electronic device can process the first parallax based on the plurality of parallaxes to obtain the first target parallax.

[0146] In one possible implementation, if the first disparity satisfies the first condition, the electronic device can filter out the first disparity and other disparities among the plurality of disparities that satisfy the first condition, and determine the average value (i.e., the first average value) of the other disparities (i.e., the disparities among the plurality of disparities that do not satisfy the first condition), which is the first target disparity.

[0147] In another possible implementation, if the first disparity satisfies the first condition, the electronic device can filter out the first disparity, as well as other disparities among the plurality of disparities that satisfy the first condition, and determine the median of the other disparities, which is the first target disparity.

[0148] Of course, electronic devices can also obtain the first target parallax based on parallax that does not meet the first condition in other ways, and this application does not limit this.

[0149] In some embodiments of this application, if the first disparity does not meet the first condition, the electronic device may also determine the first target disparity in the manner described above.

[0150] In some other embodiments of this application, if the first disparity does not meet the first condition, the first target disparity may be the first disparity.

[0151] In some embodiments of this application, the first disparity satisfies a first condition, which may specifically include: the difference between the first disparity and the second average value is greater than a first threshold, wherein the second average value is the average value of the plurality of disparities.

[0152] In one possible implementation, the first threshold can be threshold 2, which is mentioned below.

[0153] In some embodiments of this application, the first disparity satisfies a first condition, which may specifically include: the absolute value of the difference between the first disparity and the maximum or minimum value among the plurality of disparities is greater than a certain threshold. It is understood that this threshold can be set according to actual needs, and this application does not impose any restrictions on it.

[0154] Of course, the first condition may include other content, and this application does not limit this.

[0155] In some embodiments of this application, in response to the first operation, during the acquisition of video frames based on the dual-stream images, the electronic device can obtain a disparity for each set of dual-stream images, and can also place the disparity into a first queue, updating the first queue based on the acquisition status of the video frames. This first queue is the disparity queue mentioned below. Furthermore, the number of disparities in the first queue is greater than a second threshold.

[0156] In one possible implementation, the second threshold can be threshold 1, as mentioned below.

[0157] In some embodiments of this application, the electronic device updates the first queue based on the acquisition status of the video frame. Specifically, this may include: if the electronic device acquires a video frame based on the parallax in the first queue, it can remove the parallax from the first queue, and after the parallax obtained based on the subsequently output dual-stream image is added to the first queue, it acquires the next video frame based on the first queue. In this case, the first queue includes the parallax determined based on the target dual-stream image, and the parallax determined based on the dual-stream image output after the target dual-stream image.

[0158] In some other embodiments of this application, if the electronic device updates the first queue based on the acquisition of the video frame, it may specifically include: if the electronic device acquires a video frame based on the parallax in the first queue, it may not delete the parallax from the first queue, but instead directly acquire the next video frame based on the next parallax in the first queue. In this case, the first queue may include parallax determined based on the dual-stream image output before the target dual-stream image, parallax determined based on the target dual-stream image, and parallax determined based on the dual-stream image output after the target dual-stream image.

[0159] In some other embodiments of this application, if the electronic device updates the first queue based on the acquisition status of the video frame, it may specifically include: after the electronic device acquires a video frame based on the parallax in the first queue, it may not delete the parallax from the first queue, but instead acquire the next video frame based on the first queue after the parallax obtained based on the subsequently output dual-stream image is added to the first queue. In this case, the updated first queue is the same as the original queue until a video frame is acquired based on the parallax that was last added to the first queue. That is, after acquiring a video frame based on the parallax that was last added to the first queue, the electronic device may determine the parallax based on the subsequently output dual-stream image and add it to the first queue, thereby acquiring the next video frame based on the newly added parallax in the first queue.

[0160] In some embodiments of this application, the first queue may include the aforementioned plurality of disparities. In this case, the electronic device may filter the plurality of disparities in the first queue to obtain a processed first queue. The processed first queue includes the target disparity. In one possible implementation, the filtering process may include, as mentioned above, filtering out disparities in the first queue that satisfy the first condition, and determining a reasonable disparity to replace the filtered disparity. Specific implementation details can be found above, and will not be repeated here.

[0161] After obtaining a disparity based on each set of dual-stream images, the electronic device can determine whether the number of disparities in the first queue is greater than a second threshold. If the number of disparities in the first queue is not greater than the second threshold, the electronic device does not process that disparity, but continues to obtain a disparity based on the next set of dual-stream images, and so on, until the number of disparities in the first queue is greater than the second threshold. For example, the second threshold can be m-1. After the electronic device obtains the disparity based on the m-th dual-stream image and puts it into the first queue, the number of disparities in the first queue is greater than m for the first time. In this case, the electronic device can determine whether the disparity determined based on the 1-th dual-stream image (i.e., the disparity that was first enqueued in the current first queue) needs to be filtered out based on the multiple disparities in the first queue. If the disparity determined based on the 1-th dual-stream image does not need to be filtered out, the electronic device can directly process the 1-th image data based on the disparity to obtain the first video frame after receiving the first operation. If the disparity determined based on the 1-th dual-stream image needs to be filtered out, the electronic device can determine a new disparity based on the multiple disparities and process the 1-th image data based on the new disparity to obtain the first video frame after receiving the first operation.

[0162] In some embodiments of this application, the multiple sets of dual-stream images may include a first set of dual-stream images to a m-th set of dual-stream images. In response to the first operation, after the original camera and the target camera output dual-stream images, the electronic device can determine a first target parallax based on the first set of dual-stream images (i.e., the first frame image data output by the original camera and the first frame image data output by the target camera after receiving the first operation) to the m-th set of dual-stream images, and process the first set of dual-stream images based on the first target parallax to obtain the first frame video data acquired after receiving the first operation.

[0163] S103: The electronic device processes the dual-stream image of the target based on the parallax of the first target to obtain the target video frame.

[0164] The electronic device can allocate the first target disparity and the first rotation amount based on the target zoom magnification to obtain the target displacement and target rotation amount. Based on these target displacement and rotation amounts, a target spatial alignment matrix can be obtained. The first rotation amount is the rotation of the relative pose of the original camera and the target camera determined based on calibration information. Furthermore, the electronic device can process the target dual-stream image based on the target spatial alignment matrix to obtain the target video frame.

[0165] In some embodiments of this application, the electronic device allocates the first target parallax based on the target zoom magnification to obtain a target spatial alignment matrix. Specifically, this may include: the electronic device determining the target displacement based on the ratio of a first difference to a second difference. The ratio of the target displacement to the first target parallax is equal to the ratio of the first difference to the second difference. The first difference is the difference between the target zoom magnification and the original zoom magnification, and the second difference is the difference between a first critical zoom magnification and the original zoom magnification.

[0166] Of course, the electronic device can also allocate the first target parallax and the first rotation amount based on other methods, as detailed below, which will not be elaborated here.

[0167] It should be noted that, in response to the first operation, the electronic device can calculate a preview frame based on the dual-stream images output by the original camera and the target camera.

[0168] In some embodiments of this application, during the process of acquiring a preview frame based on the dual-stream image, the electronic device can determine a second target parallax based on calibration information and / or the target dual-stream image, and process the target dual-stream image based on the second target parallax to obtain the target preview frame.

[0169] As you can understand, the specific method for obtaining the preview frame can be found below, and will not be elaborated here.

[0170] The following is combined with Figure 4 The software structure of the electronic device shown is used to illustrate one implementation of the above embodiments.

[0171] As mentioned above, during video recording on an electronic device, in response to a user's zoom-in operation, the electronic device can control both the original camera and the target camera to simultaneously output images. Correspondingly, such as... Figure 6 As shown, the electronic device can send image data output from the original camera and the target camera to the multi-stream data processing module of the HAL layer of the electronic device through the kernel layer (e.g., the camera driver in the kernel layer). It can be understood that, as... Figure 6 As shown, the electronic device can also control more cameras to output images to meet other needs (needs beyond zoom). That is, the core layer of the electronic device can send image data from multiple cameras to the multi-stream processing module of the HAL layer. The electronic device can then use this module to select the original camera's output image data and the target camera's output image data from these multiple cameras. Furthermore, the electronic device can use the multi-stream processing module to send the original camera's output image data and the target camera's output image data to a first image alignment module, and to send them to a second image alignment module.

[0172] On one hand, the electronic device can process the image data output by the original camera and the image data output by the target camera through the first image alignment module to obtain aligned image data, and then send the aligned image data to the first image processing module. Furthermore, the electronic device can process the aligned image data through the first image processing module (e.g., image fusion, adding filters, etc.) to obtain video data, and then transmit the video data upwards to the framework layer and application layer.

[0173] In some embodiments of this application, the aligned image data output by the first image alignment module can be a frame of image data, which is generated by aligning the image data output by the original camera and the image data output by the target camera into the same coordinate system.

[0174] In some embodiments of this application, the aligned image data output by the first image alignment module may include aligned image data corresponding to the original camera and aligned image data corresponding to the target camera. In one possible implementation, the first image alignment module may align the image data output by the original camera to the camera coordinate system corresponding to the target camera. In this case, the aligned image data output by the first image alignment module may only include the aligned image data corresponding to the original camera. That is, the first image alignment module may send the aligned image data corresponding to the original camera and the image data output by the target camera to the first image processing module.

[0175] Furthermore, the camera application in the electronic device application layer can receive the video data uploaded by the first image processing module and send it to the encoder (e.g., a hardware encoder or a software encoder) for encoding through the encoding driver to obtain the encoded video data.

[0176] In one possible implementation, the electronic device can directly store the encoded video data into memory using an encoder. In another possible implementation, the electronic device can upload the encoded video data to a camera application via an encoder driver. The electronic device can then further process the encoded video data using the camera application to obtain processed video data, and finally store the processed video data into memory using a storage driver.

[0177] On the other hand, the electronic device can process the image data output by the original camera and the image data output by the target camera through the second image alignment module to obtain aligned image data, and then send the aligned image data to the second image processing module. Furthermore, the electronic device can process the aligned image data through the second image processing module (e.g., image fusion, adding filters, etc.) to obtain preview data, and then transmit the preview data upwards to the framework layer and the application layer.

[0178] In some embodiments of this application, the aligned image data output by the second image alignment module can be a frame of image data, which is generated by aligning the image data output by the original camera and the image data output by the target camera into the same coordinate system.

[0179] In some embodiments of this application, the aligned image data output by the second image alignment module may include aligned image data corresponding to the original camera and aligned image data corresponding to the target camera. In one possible implementation, the second image alignment module may align the image data output by the original camera to the camera coordinate system corresponding to the target camera. In this case, the aligned image data output by the second image alignment module may only include the aligned image data corresponding to the original camera. That is, the second image alignment module may send the aligned image data corresponding to the original camera and the image data output by the target camera to the second image processing module.

[0180] It should be noted that the aligned image data output by the first image alignment module and the aligned image data output by the second image alignment module are image data obtained based on different processing methods. For details, please refer to [link / reference needed]. Figure 7A and Figure 7B The content shown. The aligned image data output by the first image alignment module and the aligned image data output by the second image alignment module can be the same or different.

[0181] Furthermore, the camera application at the electronic device's application layer can receive the preview data uploaded by the second image processing module and display the preview data on the screen via the display driver. In one possible implementation, after receiving the preview data, the camera application at the electronic device's application layer can process it to obtain processed preview data, which the electronic device can then display on the screen via the display driver.

[0182] In some embodiments of this application, during video recording by the electronic device, when the user does not trigger zoom, the multi-stream data processing module can also send multiple image data streams to the first image alignment module and the second image alignment module for processing. The processing procedure of the first image alignment module for this multi-stream image data differs from its processing procedure after the user triggers zoom. Similarly, the processing procedure of the second image alignment module for this multi-stream image data differs from its processing procedure after the user triggers zoom.

[0183] In some embodiments of this application, during video recording by the electronic device, when the user does not trigger zoom, the multi-stream processing module can send only the original image data output by the camera to the first image alignment module and the second image alignment module for processing. Similarly, the subsequent processing of the processed image data by the first and second image processing modules differs from their processing after the user triggers zoom; for example, image fusion is not required, which reduces resource consumption.

[0184] In some embodiments of this application, when the electronic device is not recording video (i.e., the electronic device is in a non-recording preview state, such as the preview state in video recording mode or the preview state in photo taking mode), the second image processing module does not perform image fusion processing on the input image data, which can reduce resource consumption.

[0185] The following is combined with Figure 7A and Figure 7B This section describes a specific implementation of the above embodiments.

[0186] 1. Image alignment is achieved through the first image alignment module to output video data. Figure 7A and Figure 8A )

[0187] Please see Figure 7A , Figure 7A A flowchart illustrating another video zoom method provided in this application embodiment. This method may include, but is not limited to, the following steps:

[0188] S201: In response to the first operation, the multi-stream processing module sends the image data stream output by the original camera and the image data stream output by the target camera to the first image alignment module.

[0189] In response to the first operation, the multi-stream processing module can select the image data stream output by the original camera and the image data stream output by the target camera from the image data streams output by multiple cameras (i.e., multiple image data streams), and send them to the first image alignment module. The first operation can be the zoom-triggered operation mentioned above, i.e., triggering a zoom from the original zoom level to the target zoom level, such as a click or slide operation on the zoom level control. This application does not limit the specific implementation method.

[0190] It should be noted that the image data stream is a continuous sequence of image data, typically captured and transmitted at a fixed frame rate (e.g., 30 frames per second). This means that as the original camera and the target camera output image data at a fixed frame rate, the multi-stream processing module can continuously send the image data output by the original camera and the target camera to the first image alignment module. Specifically, the multi-stream processing module can successively send the first frame, second frame, ..., nth frame of image data output by the original camera after zoom triggering to the first image alignment module, and also send the first frame, second frame, ..., nth frame of image data output by the target camera after zoom triggering to the first image alignment module.

[0191] In some embodiments of this application, the first frame image data output by the original camera and the first frame image data output by the target camera after triggering zoom can be the first set of dual-stream images mentioned above. Similarly, the second frame image data output by the original camera and the second frame image data output by the target camera after triggering zoom can be the second set of dual-stream images. And so on, the nth frame image data output by the original camera and the nth frame image data output by the target camera after triggering zoom can be the nth set of dual-stream images.

[0192] S202: The first image alignment module calculates the disparity based on the image data output by the original camera and the image data output by the target camera, and puts the calculated disparity into the disparity queue.

[0193] Each time the first image alignment module receives one frame of image data output from the original camera and one frame of image data output from the target camera, it can calculate the disparity based on these two frames and add the calculated disparity to the disparity queue. In other words, the calculated disparity is the disparity corresponding to this frame of image data output from the original camera and this frame of image data output from the target camera, or the disparity corresponding to this set of dual-stream images.

[0194] Understandably, the parallax queue is used to store the parallax obtained based on the dual-stream information (the image data stream output by the original camera and the image data stream output by the target camera).

[0195] In some embodiments of this application, such as Figure 8A As shown, the first image alignment module can perform FOV alignment based on the image data output by the original camera and the image data output by the target camera, obtaining two frames of image data after FOV alignment. Then, feature point registration is performed on the two frames of image data after FOV alignment to find corresponding feature point pairs. Optionally, the first image alignment module can also filter the found feature point pairs, removing incorrectly matched feature point pairs and retaining high-quality feature point pairs. Furthermore, the first image alignment module can calculate disparity based on the matched feature point pairs.

[0196] It should be noted that after FOV alignment, the image data output by the original camera and the image data output by the target camera have the same FOV size (i.e., the angle range of the image that the camera can receive is the same, but the specific scenes involved may differ), but the FOV center does not change. In other words, the two frames of image data after FOV alignment cover the same spatial range of the shooting scene, but because their FOV centers are not aligned, there is a difference in their FOV centers, resulting in differences in the specific shooting content involved in the two frames of image data after FOV alignment.

[0197] For example, the FOV of the original camera can be larger than the FOV of the target camera. In this case, such as Figure 9 As shown, after FOV alignment of the image data output from the original camera and the image data output from the target camera, the aligned original camera image data can be obtained. Compared to the original image data output from the original camera, the FOV size of the aligned original camera image data changes, becoming the same as the FOV size of the target camera image data, but its FOV center remains unchanged. This means that the FOV center of the aligned original camera image data is the same as the FOV center of the target camera image data (i.e., the FOV center of the original camera image data is the same as the FOV center of the target camera image data). Figure 9 The original camera's FOV center is different from that of the target camera. Therefore, the content of the subject in the original camera's image data after FOV alignment differs from the content of the subject in the target camera's output image data. Figure 9 As shown, in the original camera image data after FOV alignment, the subject is located to the upper right of the center of its FOV, while in the image data output by the target camera, the subject is located at the center of its FOV. In other words, the position of the subject in the original camera image data after FOV alignment is different from the position in the image data output by the target camera.

[0198] In one possible implementation, feature point registration may include the following steps: feature point detection (e.g., extracting feature points from an image using a feature detection algorithm), feature description (e.g., calculating a descriptor for each detected feature point), feature matching (e.g., comparing the differences between feature point descriptors in two images using a similarity metric to determine the correspondence), and transformation model estimation (e.g., estimating the geometric transformation model between images based on the matched feature point pairs using the least squares method or the iterative nearest point algorithm).

[0199] In one possible implementation, the first image alignment module may perform feature point selection based on angle, random sampling consistency, depth information, or semantic segmentation.

[0200] In one possible implementation, the first image alignment module can extract the coordinates in the left and right images from the matched feature point pairs, and determine the difference in horizontal coordinates between the left and right images as the disparity.

[0201] It is understood that the first image alignment module may also use other methods to perform feature point registration, feature point filtering, and parallax, which will not be elaborated here.

[0202] S203: The first image alignment module determines whether the number of disparities in the disparity queue is greater than the threshold 1.

[0203] After the first image alignment module adds the disparity calculated based on the image data output by the original camera and the image data output by the target camera to the disparity queue, it can determine whether the number of disparities in the disparity queue is greater than the threshold 1. That is, each time the first image alignment module adds the disparity calculated based on one frame of image data output by the original camera and one frame of image data output by the target camera to the disparity queue, it can determine whether the number of disparities in the disparity queue is greater than the threshold 1. If the number of disparities in the disparity queue is greater than the threshold 1, the first image alignment module can execute subsequent steps (steps S204-S208). If the number of disparities in the disparity queue is less than or equal to the threshold 1, the first image alignment module can execute steps S202 and S203 again. That is, the first image alignment module can calculate the disparity based on the next frame of image data output by the original camera and the next frame of image data output by the target camera, add the disparity to the disparity queue, and then determine whether the number of disparities in the disparity queue is greater than the threshold 1.

[0204] It is understood that the threshold 1 can be set according to actual needs, and this application does not impose any restrictions on it. For example, the threshold 1 can be 5. As another example, the threshold 1 can be 20.

[0205] For example, threshold 1 can be 2. In response to a user-triggered zoom operation, the multi-stream processing module continuously sends image data output by the original camera and image data output by the target camera to the first image alignment module. After receiving image data A1 from the original camera and image data B1 from the target camera, the first image alignment module can calculate disparity C1 based on image data A1 and image data B1, and add disparity C1 to the disparity queue. That is, disparity C1 is the first element in the disparity queue. Further, if the first image alignment module determines that the disparity queue contains only one disparity, i.e., the number of disparities in the disparity queue is less than 2, then the first image alignment module can continue to calculate disparity C2 based on image data A2 from the original camera and image data B2 from the target camera, and add disparity C2 to the disparity queue. That is, disparity C2 is the second element in the disparity queue. Furthermore, if the first image alignment module determines that the disparity queue includes two disparities (i.e., the number of disparities in the disparity queue is less than two), then the first image alignment module can continue to calculate disparity C3 based on the image data A3 output by the original camera and the image data B3 output by the target camera, and add disparity C3 to the disparity queue. That is, disparity C3 is the third element in the disparity queue. Further, if the first image alignment module determines that the disparity queue includes three disparities (i.e., the number of disparities in the disparity queue is greater than two), then the first image alignment module can continue to execute subsequent steps, such as steps S204-S208.

[0206] In this application, image data A1, image data A2, and image data A3 are three consecutive frames of image data output by the original camera. That is, after the original camera outputs image data A1, the next frame output is image data A2, and after the original camera outputs image data A2, the next frame output is image data A3. Similarly, image data B1, image data B2, and image data B3 are three consecutive frames of image data output by the target camera. That is, after the target camera outputs image data B1, the next frame output is image data B2, and after the target camera outputs image data B2, the next frame output is image data B3. In some embodiments of this application, image data A1 and image data B1 can be a set of dual-stream images, image data A2 and image data B2 can be a set of dual-stream images, and image data A3 and image data B3 can be a set of dual-stream images.

[0207] S204: The first image alignment module performs filtering on the disparity queue to obtain the processed disparity queue.

[0208] If the number of disparities in the disparity queue exceeds a threshold of 1, the first image alignment module can filter the disparities in the disparity queue to obtain a processed disparity queue. This removes unreasonable disparity values ​​from the disparity queue and improves its reliability.

[0209] In some embodiments of this application, the first image alignment module can filter the disparities in the disparity queue. Specifically, the first image alignment module can determine whether there are unreasonable disparities in the disparity queue. If there are no unreasonable disparities in the disparity queue, the processed disparity queue is the original disparity queue. If there are unreasonable disparities in the disparity queue, the first image alignment module can determine a reasonable disparity value to replace the unreasonable disparities.

[0210] In one possible implementation, the absolute value of the difference between unreasonable disparity and the average disparity is greater than a threshold of 2. Here, the average disparity is the average of all disparities in the disparity queue. The threshold of 2 can be set according to actual needs, and this application does not impose any restrictions on it.

[0211] For example, the threshold 2 can be 20. If the disparity queue is {11, 10, 20, 50, 9}, then the average disparity is 20. The difference between 50 in this disparity queue and the average disparity is greater than 20, meaning that 50 in this disparity queue is an unreasonable disparity. In this case, the disparity queue can calculate the average of the other four disparities besides 50, which is 12.5, and replace 50 in the disparity queue with 12.5. That is, the processed disparity queue is {11, 10, 20, 12.5, 9}.

[0212] In another possible implementation, the absolute value of the difference between the unreasonable parallax and the average parallax is greater than the average parallax.

[0213] For example, the disparity queue could be {21, 15, 15, 50, 19}, then the average disparity is 24. The difference between 50 and 24 in this disparity queue is greater than 24, meaning 50 in the disparity queue is an unreasonable disparity. In this case, the disparity queue can calculate the average of the other four disparities besides 50, which is 17.5, and replace 50 in the disparity queue with 17.5. That is, the processed disparity queue is {21, 15, 15, 17.5, 19}.

[0214] It is understood that the first image alignment module can also filter the disparity in the disparity queue based on other filtering methods (e.g., median filtering, Gaussian filtering, bilateral filtering, etc.), and this application does not limit this.

[0215] S205: The first image alignment module determines the disparity corresponding to the current video frame to be output from the processed disparity queue.

[0216] It is understood that the disparity queue can include not only the disparity corresponding to the current frame of video data to be output (i.e., the video frame to be output), but also the disparity corresponding to already output video frames and / or other video frames to be output (i.e., the disparity determined based on historical dual-stream images and / or the disparity determined based on future dual-stream images mentioned above). In this case, after the first image alignment module performs filtering on the disparity queue, it can determine the disparity corresponding to the current video frame to be output from the processed disparity queue.

[0217] The disparity corresponding to the current video frame to be output is calculated based on the original image corresponding to the current video frame to be output. The original image corresponding to the current video frame to be output refers to the image data output by the original camera and the image data output by the target camera required to obtain the current video frame to be output.

[0218] In some embodiments of this application, the disparity queue may include not only the disparity corresponding to the current video frame to be output, but also the disparity corresponding to the image data of the previous m1 frames (i.e., the m1 video frames preceding the current video frame). Here, m1 is an integer, and m1 can be set according to actual needs; this application does not impose any restrictions on it. For example, m1 can be 5.

[0219] In some embodiments of this application, the disparity queue may include not only the disparity corresponding to the current video frame to be output, but also the disparity corresponding to the image data of the m2 frames following the current video frame to be output (i.e., the m2 frames to be output after the current video frame to be output). Here, m2 is an integer, and m2 can be set according to actual needs; this application does not impose any restrictions on it. For example, m2 can be 10.

[0220] In some embodiments of this application, the disparity queue may include not only the disparity corresponding to the current video frame to be output, but also the disparity corresponding to the m1 video frames preceding the current video frame to be output, and the disparity corresponding to the m2 video frames following the current video frame to be output.

[0221] For example, the disparity queue can be {11, 10, 20, 30, 12}. Here, 11 and 10 represent the disparities corresponding to already output video frames, meaning the first image alignment module has already output the corresponding video frames based on these two disparities and the corresponding image data output by the original camera and the target camera. 20, 30, and 12 represent the disparities corresponding to the video frames to be output, meaning the first image alignment module has not yet output the corresponding video frames based on these three disparities and the corresponding image data output by the original camera and the target camera. Among the image data required to calculate these three disparities, the image data required to calculate 20 is the earliest image data output by the original camera and the target camera, while the image data required to calculate 12 is the latest image data output by the original camera and the target camera. In this case, the disparity corresponding to the current video frame to be output is 20.

[0222] S206: The first image alignment module processes the calibration information and the disparity corresponding to the video frame to be output based on the target zoom magnification to obtain the spatial alignment matrix corresponding to the current video frame to be output.

[0223] For ease of understanding and description, this application denotes the zoom factor as x, the displacement of the spatial alignment matrix as y, the original zoom factor as a, the critical zoom factor for switching to the target camera as b, and the parallax corresponding to the video frame to be output as c. Here, the original zoom factor refers to the zoom factor used by the electronic device before the user triggers zoom. The critical zoom factor for switching to the target camera refers to the critical zoom factor that triggers the switch from the original camera to the target camera (such as the first critical zoom factor mentioned above). In other words, if the user triggers zoom to this critical zoom factor, the electronic device can switch from the original camera to the target camera. For example, if the user triggers zoom below 1x, the electronic device can use the ultra-wide-angle camera; if the user triggers zoom above 1x, the electronic device can use the wide-angle camera. In other words, when the original camera is an ultra-wide-angle camera and the target camera is a wide-angle camera, the critical zoom level for switching to the target camera is 1x. Similarly, when the original camera is a wide-angle camera and the target camera is an ultra-wide-angle camera, the critical zoom level for switching to the target camera is also 1x.

[0224] As stated above, when the electronic device uses the original zoom level (i.e., zoom is not triggered), the spatial alignment matrix is ​​not yet determined, meaning the displacement of the spatial alignment matrix is ​​0. However, when the electronic device zooms to the critical zoom level required to switch to the target camera (i.e., zoom is triggered to that critical zoom level), the displacement of the spatial alignment matrix is ​​the parallax calculated above. In other words, when x = a, y = 0; when x = b, y = c.

[0225] In some embodiments of this application, y and x can have a linear relationship. In this case, y = c / (ba)*(xa).

[0226] In some other embodiments of this application, y and x can have a non-linear relationship, specifically representing an exponential function, a logarithmic function, or a trigonometric function, etc. In one possible implementation, the first image alignment module can determine the relationship between y and x through data fitting or curve fitting.

[0227] As described above, the first operation is used to trigger zooming from the original zoom level to the target zoom level. It should be noted that the target zoom level is not necessarily equal to the critical zoom level for switching to the target camera. For example, a user can trigger the electronic device to zoom from 0.5x to 0.8x. In this case, the critical zoom level for switching to the target camera is 1x, while the target zoom level is 0.8x.

[0228] Understandably, the first image alignment module can substitute the target zoom factor into the relationship between y and x, and the resulting y is the displacement corresponding to the current video frame to be output. For example, the parallax corresponding to the current video frame to be output is 100 pixels, and the original zoom factor is 0.5. If the target zoom factor is 0.8, then the displacement corresponding to the current video frame to be output is: 100 / (1-0.5)*(0.8-0.5), which is 60 pixels.

[0229] The first image alignment module can determine the relative pose of the original camera and the target camera based on calibration information. The relative pose can include translation and rotation.

[0230] For ease of understanding and description, this application denotes the rotation amount of the spatial alignment matrix as z.

[0231] Similarly, z and x can be linear or nonlinear. The specific relationship between z and x can be found in the method for determining the relationship between y and x described above, which will not be repeated here.

[0232] The first image alignment module can substitute the target zoom factor into the relationship between z and x, and the resulting z is the rotation amount corresponding to the current video frame to be output.

[0233] In this way, after the first image alignment module determines the displacement and rotation of the current video frame to be output, it can use the displacement of the current video frame as the displacement in the spatial alignment matrix and the rotation of the current video frame as the rotation in the spatial alignment matrix. The resulting spatial alignment matrix is ​​the spatial alignment matrix corresponding to the current video frame to be output.

[0234] S207: The first image alignment module processes the image data output by the original camera based on the spatial alignment matrix corresponding to the current video frame to be output, and obtains the aligned image data.

[0235] After the first image alignment module determines the spatial alignment matrix corresponding to the current video frame to be output, it can process the image data output by the original camera based on the spatial alignment matrix. For example, it can use the spatial alignment matrix to map each pixel in the image data output by the original camera to the position in the image data output by the target camera, and resample to generate aligned image data.

[0236] S208: The first image alignment module sends the aligned image data and the image data output by the target camera to the first image processing module.

[0237] After the first image alignment module processes and obtains the aligned image data, it can send the aligned image data and the image data output by the target camera to the first image processing module.

[0238] S209: The first image processing module processes the aligned image data and the image data output by the target camera to obtain the current video frame to be output.

[0239] After receiving the aligned image data and the image data output by the target camera, the first image processing module processes them to obtain processed image data, which is then sent to the upper-layer module. This processed image data is the current video frame to be output. For example, the first image processing module can fuse the aligned image data and the image data output by the target camera to obtain the fused image data, which is the current video frame to be output. Furthermore, the first image processing module can further process the fused image data, such as cropping, scaling, or adding filters, to obtain the current video frame to be output.

[0240] As described above, after the first image processing module sends the current video frame to be output to the upper-layer module, the upper-layer module can encode the current video frame to be output using hardware encoding or software encoding to meet the video recording requirements, and then save the encoded video frame.

[0241] By using the above method, information from subsequent frames can be obtained in advance. Based on global information, trajectory optimization can be performed, avoiding the overall trajectory unevenness caused by individual image differences during the process, resulting in smoother zoom effects in the video. For example, Figure 10 As shown, in the video saved by the electronic device, the position of the subject in the image gradually changes before switching from the original camera to the target camera. However, the change in the subject's position is less pronounced when switching from the original camera to the target camera. Figure 2 The positional changes shown are obvious, meaning the zoom effect is smoother and the subject's position does not jump.

[0242] 2. Image alignment is achieved through the second image alignment module to output preview data. Figure 7B and Figure 8B )

[0243] Please see Figure 7B , Figure 7B A flowchart illustrating another video zoom method provided in this application embodiment. This method may include, but is not limited to, the following steps:

[0244] S301: In response to the first operation, the multi-stream processing module sends the image data stream output by the original camera and the image data stream output by the target camera to the second image alignment module.

[0245] It is understood that the specific implementation of step S301 can be referred to the relevant description of step S201, and this application will not repeat it here.

[0246] S302: The second image alignment module calculates the disparity based on the calibration information and distance information to obtain offset1.

[0247] Understandably, calibration information can include camera intrinsic parameters (e.g., focal length, principal point, etc.) and camera extrinsic parameters (e.g., pose), and distance information can include the straight-line distance between the object (or subject) and the target camera. Furthermore, the second image alignment module can determine the relative pose between the original camera and the target camera. The displacement of this relative pose is the distance in the world coordinate system (e.g., distance in millimeters), and the second image alignment module can convert this displacement into a displacement in the pixel coordinate system (i.e., pixel value) based on the camera intrinsic parameters. Specifically, as... Figure 8B As shown, according to the above-mentioned The second image alignment module can substitute the displacement of the relative pose into B in the formula, the focal length of the target camera into f in the formula, and the straight-line distance between the object and the camera into Z in the formula, so as to obtain the parallax d, i.e., offset1.

[0248] S303: The second image alignment module calculates the parallax based on the image data output by the original camera and the image data output by the target camera to obtain offset2.

[0249] It is understood that the specific implementation of step S303 can be referred to the relevant description of step S202, and this application will not repeat it here.

[0250] In some embodiments of this application, such as Figure 8BAs shown, the second image alignment module can perform FOV alignment based on the image data output by the original camera and the image data output by the target camera, obtaining two frames of image data after FOV alignment. Then, feature point registration is performed on these two frames to find corresponding feature point pairs. Optionally, the second image alignment module can also filter the found feature point pairs, removing incorrectly matched pairs and retaining high-quality ones. Furthermore, the second image alignment module can calculate the disparity based on the matched feature point pairs to obtain offset2.

[0251] It is understood that step S303 is an optional step.

[0252] S304: The second image alignment module performs scale alignment based on offset1 and / or offset2 to obtain offset3.

[0253] In some embodiments of this application, the second image alignment module may perform scale alignment based solely on offset1 to obtain offset3.

[0254] In some embodiments of this application, the second image alignment module can perform scale alignment based on offset1 and offset2 based on weights to obtain offset3. For example, the second image alignment module can perform scale alignment based on (e1*offset1+e2*offset2) to obtain offset3.

[0255] Where e1 is not less than 0 and not greater than 1, and e2 is not less than 0 and not greater than 1. e1 and e2 can be set according to actual needs, and this application does not impose any restrictions on them. For example, e1 can be 0.8 and e2 can be 0.1. For another example, e1 can be 1 and e2 can be 0. For yet another example, e1 can be 0 and e2 can be 1.

[0256] In some embodiments of this application, the scale alignment involved includes coordinate system alignment. Because this application involves multiple cameras (e.g., a primary camera and a target camera), each camera has its own camera coordinate system. Scale alignment can unify the coordinate systems of the multiple cameras, ensuring that the calculated pixel coordinate values ​​are consistent. In some embodiments of this application, the scale alignment involved may also include image size alignment (e.g., image width and length).

[0257] It should be noted that after the second image alignment module obtains offset3 by performing scale alignment based on parallax, it can directly apply offset3 to the image to be displayed for offset.

[0258] S305: The second image alignment module processes the calibration information and offset3 based on the target zoom level to obtain the spatial alignment matrix corresponding to the preview frame to be output.

[0259] It is understood that the specific implementation of step S305 can be referred to the relevant description of step S206, and this application will not repeat it here.

[0260] S306: The second image alignment module processes the image data output by the original camera based on the spatial alignment matrix corresponding to the preview frame to be output, and obtains the aligned image data.

[0261] After the second image alignment module determines the spatial alignment matrix corresponding to the preview frame to be output, it can process the image data output by the original camera based on the spatial alignment matrix. For example, it can use the spatial alignment matrix to map each pixel in the image data output by the original camera to the position in the image data output by the target camera, and resample to generate aligned image data.

[0262] S307: The second image alignment module sends the aligned image data and the image data output by the target camera to the second image processing module.

[0263] S308: The second image processing module processes the aligned image data and the image data output by the target camera to obtain the preview frame to be output.

[0264] After receiving the aligned image data and the image data output by the target camera, the second image processing module processes them to obtain preview data, which is then sent to the upper-layer module. This preview data is the preview frame to be output. For example, the second image processing module can fuse the aligned image data and the image data output by the target camera to obtain the fused image data, which is the preview frame to be output. Furthermore, the second image processing module can further process the fused image data, such as cropping, scaling, or adding filters, to obtain the preview frame to be output.

[0265] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A video zoom method, characterized in that, The method is applied to an electronic device including a first camera and a second camera, and the method includes: During the recording of video using the first camera in the electronic device, in response to a first operation, dual-stream images are output through the first camera and the second camera; the first operation is used to trigger zoom. During the process of acquiring video frames based on the dual-stream images, a first target parallax is determined based on multiple sets of dual-stream images, including the target dual-stream images. The target dual-stream image is processed based on the first target parallax to obtain the target video frame.

2. The method as described in claim 1, characterized in that, The multiple sets of dual-stream images include at least one set of dual-stream images output before the target dual-stream image, and / or at least one set of dual-stream images output after the target dual-stream image.

3. The method as described in claim 1 or 2, characterized in that, The determination of the first target disparity based on multiple sets of dual-stream images includes: Multiple disparities are determined based on the multiple sets of dual-stream images; the multiple disparities include a first disparity, which is a disparity determined based on the target dual-stream image; If the first disparity satisfies the first condition, the first disparity is processed based on the plurality of disparities to obtain the first target disparity.

4. The method as described in claim 3, characterized in that, If the first disparity does not meet the first condition, the first target disparity is the first disparity.

5. The method as described in claim 3 or 4, characterized in that, The step of processing the first disparity based on the multiple disparities to obtain the first target disparity includes: Filter out the first disparity and determine a first average value, which is the average value of the first type of disparity among the plurality of disparities, and the first type of disparity is the disparity that does not meet the first condition; Wherein, the first target disparity is the first average value.

6. The method according to any one of claims 3-5, characterized in that, The first disparity satisfies a first condition, including: the difference between the first disparity and the second average value is greater than a first threshold value, where the second average value is the average value of the plurality of disparities.

7. The method according to any one of claims 3-6, characterized in that, The method further includes: During the process of acquiring video frames based on the dual-stream images, a parallax is determined based on each set of dual-stream images output by the first camera and the second camera. The obtained parallax is stored in the first queue, and the first queue is updated based on the acquisition status of the video frames. The number of disparities in the first queue is greater than the second threshold.

8. The method as described in claim 7, characterized in that, The first queue includes the plurality of disparities; determining the first target disparity based on multiple sets of dual-stream images includes: The multiple disparities in the first queue are filtered to obtain a processed first queue, which includes the first target disparity.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: During the process of acquiring a preview frame based on the dual-stream image, the second target disparity is determined based on the calibration information and / or the target dual-stream image; The target dual-stream image is processed based on the second target parallax to obtain a target preview frame.

10. An electronic device, characterized in that, The electronic device includes one or more memories and one or more processors; the one or more memories are coupled to the one or more processors, the memories are used to store computer program code, the computer program code includes computer instructions, and the processor calls the computer instructions to perform the method of any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, Used to store computer instructions that, when executed on an electronic device, cause the electronic device to perform the method described in any one of claims 1-9.

Images