Image alignment method, electronic device, chip system, and storage medium

Abstract

This application provides an image alignment method, electronic device, chip system, and storage medium for multiple cameras. In this method, the alignment of the FOV areas of the multiple cameras in the electronic device is achieved in two stages: first, a coarse alignment is performed in the IFE module processing stage, and then a fine alignment is performed in the IPE module processing stage. Thus, in a shooting preview scenario, the alignment of the FOV areas of the multiple cameras is no longer limited by the size of the margin area reserved when the IFE module crops the image. This not only ensures a smooth transition in the preview image during camera switching but also does not affect the clarity of the preview image on the electronic device, thereby improving the user experience.

Description

Technical Field

[0001] This application relates to the field of smart terminal technology, and in particular to an image alignment method, electronic device, chip system and storage medium. Background Technology

[0002] With the development of electronic devices such as mobile phones and tablets, the camera function has become increasingly important to users. Taking mobile phones as an example, in order to provide users with a better photography experience, electronic devices are usually equipped with multiple cameras with different focal lengths to capture objects at different distances.

[0003] When shooting objects at different distances, users can switch to the camera displayed on the electronic device using zoom to improve the shooting effect. The smoother the preview transition when switching cameras, the less jitter there will be. However, to ensure a smooth preview transition, the clarity of the preview image may be affected, thus impacting the user's visual experience. Summary of the Invention

[0004] This application provides an image alignment method, an electronic device, a chip system, and a storage medium. In this method, the alignment of the FOV (Field of View) areas of multiple cameras during camera switching is no longer limited by the size of the margin area reserved when the IFE module crops the image. This not only ensures a smooth transition of the preview screen during camera switching but also does not affect the clarity of the preview screen of the electronic device.

[0005] In a first aspect, embodiments of this application provide an image alignment method between multiple cameras. This method is applied in an electronic device, which includes at least two cameras and an image signal processor (ISP) chip. The ISP chip includes a first image processing module and a second image processing module. The method includes: the electronic device increasing its current zoom ratio to a first value in response to a first operation, and acquiring a first image captured by the first camera; the electronic device determining a first cropping frame in the first image based on the first value and prior information; wherein the first cropping frame is not centered on the first image and is offset towards the field of view (FOV) direction of the second camera in the first image; the focal length of the first camera is smaller than that of the second camera; the electronic device controlling the first image processing module to crop the first image according to the first cropping frame to obtain a second image; the electronic device calculating a first warp matrix and a second cropping frame corresponding to the second image; the electronic device controlling the second image processing module to process the second image according to the first warp matrix to obtain a third image, and cropping the third image according to the second cropping frame to obtain a fourth image; the fourth image being used to generate a preview image for display.

[0006] The first image processing module refers to the IFE module in the ISP chip mentioned below, and the second image processing module refers to the IPE module in the ISP chip mentioned below.

[0007] For example, the first operation can be a zoom operation implemented by clicking or sliding the zoom control, or a zoom operation implemented by sliding two fingers back to back. The first value belongs to the zoom magnification range of the first camera.

[0008] Among them, the edge (margin) area is smaller in the image area cropped according to the first cropping frame.

[0009] In this embodiment, the electronic device calculates a warp matrix corresponding to the second image, so that after the second image is processed based on the warp matrix, the FOV area of ​​the first camera can be aligned with the FOV area of ​​the second camera and located in the center region of the image. The second cropping frame is a central cropping frame, which is center-aligned with the third image after offset and rotation processing based on the warp matrix.

[0010] In this embodiment, the alignment of the FOV areas of the first and second cameras is achieved in two stages. First, a coarse alignment is performed during the IFE module processing stage, and then a fine alignment is performed during the IPE module processing stage. This way, in the shooting preview scenario, the alignment of the FOV areas of the multiple cameras is no longer limited by the size of the margin area reserved when the IFE module crops the image. This not only ensures a smooth transition in the preview image during camera switching but also does not affect the clarity of the preview image on the electronic device, thereby improving the user experience.

[0011] According to the first aspect, the electronic device determines a first cropping box in a first image based on a first value and prior information, including: the electronic device determines a third cropping box in the first image based on the FOV area of ​​the first camera corresponding to the first value; wherein the third cropping box is aligned with the center of the first image; the electronic device calculates a target offset of the third cropping box based on the first value and prior information; the electronic device performs offset processing on the third cropping box based on the target offset of the third cropping box to obtain the first cropping box.

[0012] Thus, during the IFE module processing stage, the IFE module performs non-center cropping on the first image to reduce the size of the margin region in the cropped image.

[0013] According to the first aspect, or any implementation of the above first aspect, the electronic device calculates the target offset of the third cropping frame based on the first value and the prior information, including: the electronic device calculates the total FOV center offset between the first camera and the second camera according to the prior information; the electronic device allocates the total FOV center offset according to the first value to obtain the second offset of the second cropping frame.

[0014] Exemplarily, the rough FOV center offset between the first camera and the second camera is offset_12, the zoom magnification range sent by camera 1 is [m x, n x), the current zoom magnification value (i.e., the first value) is p x, m < p < n, then the second offset of the second cropping frame is: Offset = [(p - m) / (n - m)] * offset_12.

[0015] According to the first aspect, or any implementation of the above first aspect, the method further includes: the electronic device obtains the fifth image collected by the second camera in response to the first operation. Among them, the electronic device calculates the first warp matrix and the second cropping frame corresponding to the second image, including: the electronic device determines the FOV area of the second camera in the fifth image according to the first value and the fifth image; the electronic device performs spatial transformation alignment processing according to the FOV area of the second camera in the fifth image and the second image to obtain the first warp matrix and the second cropping frame corresponding to the second image.

[0016] In this way, in the case where both the first camera and the second camera are streaming, the electronic device can calculate the fine FOV area offset required to be implemented in the IPE processing stage based on the image spatial transformation alignment processing. <>

[0017] According to the first aspect, or any implementation of the above first aspect, the second camera is not started. The electronic device calculates the first warp matrix and the second cropping frame corresponding to the second image, including: the electronic device calculates the total FOV center offset between the first camera and the second camera according to the calibration information of the first camera and the second camera; the electronic device calculates the offset to be offset of the FOV area of the first camera in the second image according to the total FOV center offset and the target offset corresponding to the first cropping frame; the electronic device calculates the first warp matrix and the second cropping frame corresponding to the second image according to the offset to be offset.

[0018] That is, subtracting the offset corresponding to the first cropping frame from the total FOV center offset is the offset to be offset of the FOV area of the first camera in the second image. Among them, the offset corresponding to the first cropping frame is the target offset of the third cropping frame mentioned above.

[0019] In this way, when only the first camera is in operation, the electronic device can use the calibration data of the first and second cameras to calculate the total offset of the FOV center between the first and second cameras, and then calculate the fine FOV area offset required for the IPE processing stage.

[0020] According to the first aspect, or any implementation of the first aspect above, the method further includes: the electronic device increasing the current zoom magnification value of the electronic device to a second value in response to the first operation, and switching the display camera of the electronic device from the first camera to the second camera.

[0021] After switching the display camera of the electronic device from the first camera to the second camera, the image captured by the second camera is used to generate the preview image. Since the FOV center of the first camera is always aligned with the FOV center of the second camera before the display camera switch, the field of view of the two preview images displayed by the electronic device (the first frame is the preview image generated by the image captured by the first camera sensor, and the second frame is the preview image generated by the image captured by the second camera sensor) does not change much when the display camera is switched from the first camera to the second camera, and the preview screen transitions smoothly.

[0022] According to the first aspect, or any implementation of the first aspect above, the method further includes: the electronic device reducing its current zoom ratio to a third value in response to a second operation, acquiring a sixth image captured by a first camera and a seventh image captured by a second camera; the seventh image being used to generate a preview image for display; the electronic device determining a fourth cropping box in the sixth image based on the third value and prior information; wherein the fourth cropping box is not centered aligned with the sixth image and is offset towards the FOV direction of the second camera in the sixth image; the electronic device controlling a first image processing module to crop the sixth image according to the fourth cropping box to obtain an eighth image; the electronic device determining the FOV region of the second camera in the seventh image based on the third value and the seventh image; the electronic device performing spatial transformation alignment processing based on the FOV region of the second camera in the seventh image and the eighth image to obtain a second warp matrix and a fifth cropping box corresponding to the eighth image; the electronic device controlling a second image processing module to process the eighth image according to the second warp matrix to obtain a ninth image, and cropping the ninth image according to the fifth cropping box to obtain a tenth image.

[0023] The fifth cropping frame is centered with the ninth image.

[0024] For example, the second operation can be a zoom-out operation achieved by clicking or sliding the zoom control, or it can be a zoom-out operation achieved by sliding two fingers back to back. The third value belongs to the zoom magnification range of the second camera.

[0025] When both the first and second cameras are active, although the second camera is the display camera, the FOV area of ​​the first camera is still aligned to the FOV area of ​​the second camera in two stages. The electronic device can calculate the precise FOV area offset required in the IPE processing stage based on image space transformation alignment processing. Thus, in the shooting preview scenario, the alignment of the FOV areas of multiple cameras when switching display cameras is no longer limited by the size of the margin area reserved when the IFE module crops the image. This not only ensures a smooth transition in the preview image during camera switching but also does not affect the clarity of the preview image on the electronic device, thereby improving the user experience.

[0026] According to the first aspect, or any implementation of the first aspect above, the method further includes: the electronic device reducing the current zoom ratio of the electronic device to a fourth value in response to the second operation, switching the display camera of the electronic device from the second camera to the first camera, and using the tenth image to generate a preview image for display.

[0027] After switching the display camera of the electronic device from the second camera to the first camera, the image captured by the first camera is used to generate the preview image. Since the FOV center of the first camera is aligned with the FOV center of the second camera before the display camera switch, the field of view of the two preview images displayed by the electronic device (the first frame is the preview image generated by the image captured by the second camera sensor, and the second frame is the preview image generated by the image captured by the first camera sensor) does not change much when the display camera is switched from the second camera to the first camera, and the preview screen transitions smoothly.

[0028] According to the first aspect, or any implementation of the first aspect above, the electronic device determines a fourth cropping box in the sixth image based on the third value and prior information, including: the electronic device determines a sixth cropping box in the sixth image based on the FOV area of ​​the first camera corresponding to the third value; wherein the sixth cropping box is aligned with the center of the sixth image; the electronic device calculates the target offset of the sixth cropping box based on the third value and prior information; the electronic device performs offset processing on the sixth cropping box based on the target offset of the sixth cropping box to obtain the fourth cropping box.

[0029] According to the first aspect, or any implementation of the first aspect above, the prior information includes:

[0030] The extrinsic parameter matrix between the first and second cameras, and the intrinsic parameter matrix between the first and second cameras.

[0031] According to the first aspect, or any implementation of the first aspect above, the first camera is an ultra-wide-angle camera and the second camera is a wide-angle camera; or, the first camera is a wide-angle camera and the second camera is a telephoto camera.

[0032] Secondly, embodiments of this application provide an electronic device. The electronic device includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when executed by the one or more processors, cause the electronic device to perform the image alignment method between multiple cameras as described in the first aspect and any one of the first aspects.

[0033] The second aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the second aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0034] Thirdly, embodiments of this application provide a computer-readable storage medium. This computer-readable storage medium includes a computer program that, when run on an electronic device, causes the electronic device to perform the image alignment method between multiple cameras as described in the first aspect and any one of the first aspects.

[0035] The third aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the third aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0036] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when run, causes a computer to execute an image alignment method between multiple cameras as described in the first aspect or any one of the first aspects.

[0037] The fourth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the fourth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0038] Fifthly, this application provides a chip including a processing circuit and transceiver pins. The transceiver pins and the processing circuit communicate with each other via an internal connection path. The processing circuit executes an image alignment method between multiple cameras as described in the first aspect or any one of the first aspects, controls the receiving pin to receive signals, and controls the transmitting pin to transmit signals.

[0039] The fifth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the fifth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0040] In a sixth aspect, this application provides a chip system applied in an electronic device, the chip system including instructions and at least one processor, the at least one processor executing instructions to cause the electronic device to perform an image alignment method between multiple cameras as described in the first aspect or any one of the first aspects.

[0041] The sixth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the sixth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here. Attached Figure Description

[0042] Figure 1a This is an exemplary schematic diagram of the focal length switching of a display camera on an electronic device.

[0043] Figure 1b This is an illustrative diagram of an electronic device zooming scenario;

[0044] Figure 2 This is an example illustrating how the preview screen changes when an electronic device switches between camera feeds;

[0045] Figure 3 The process of an electronic device, as exemplarily shown, processes images captured by a camera to generate a preview image;

[0046] Figure 4a This is a schematic diagram illustrating the offset clipping operation performed by the IPE module in an ISP chip, as an example.

[0047] Figure 4b This is an example illustration of selecting the center cutout frame;

[0048] Figure 4cThis example illustrates the difference in how the size of the margin area affects the sharpness of the preview image.

[0049] Figure 5 A schematic diagram of the hardware structure of an electronic device as an example;

[0050] Figure 6 A schematic diagram of the software structure of an electronic device as an example;

[0051] Figure 7 This is an example of a module interaction diagram;

[0052] Figure 8 This is a schematic diagram illustrating the IFE module in an ISP chip performing clipping based on a non-central clipping frame;

[0053] Figure 9 This is an example illustrating how the preview screen changes when an electronic device switches between camera feeds;

[0054] Figure 10 This is an example illustrating how the preview screen changes when an electronic device switches cameras. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0057] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0058] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0059] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0060] To meet users' shooting needs, it has become an essential feature of electronic devices to include cameras with different focal lengths within the same device. Users can switch to the camera displayed on the device's screen (hereinafter referred to as the display camera) via zoom to improve the shooting results.

[0061] In one possible implementation, the electronic device includes an ultra-wide-angle camera, a wide-angle camera, and a telephoto camera. The ultra-wide-angle camera has a larger field of view (FOV) than the wide-angle camera, and the wide-angle camera has a larger FOV than the telephoto camera. The electronic device selects one of these cameras as the display camera based on the user's currently set zoom ratio (or zoom scale). (See reference...) Figure 1a For example, if the user adjusts the zoom ratio to within the zoom ratio range [0.6x, 1.0x), the electronic device selects the ultra-wide-angle camera as the display camera; if the user adjusts the zoom ratio to within the zoom ratio range [1.0x, 3.5x), the electronic device selects the wide-angle camera as the display camera; and if the user adjusts the zoom ratio to a value greater than or equal to 3.5x (e.g., a zoom ratio range of [3.5x, 5x]), the electronic device selects the telephoto camera as the display camera. The above zoom ratio ranges are merely illustrative examples and are not intended to limit the scope of this embodiment.

[0062] In other words, in the zoom-in scenario, the user's zoom operation can switch the electronic device's display camera from an ultra-wide-angle camera to a wide-angle camera, or from a wide-angle camera to a telephoto camera, to magnify the scene (i.e., bring the scene closer). In the zoom-out scenario, the user's zoom operation can switch the electronic device's display camera from a telephoto camera to a wide-angle camera, or from a wide-angle camera to an ultra-wide-angle camera, to shrink the scene (i.e., pull the scene further away).

[0063] It should be noted that, in order to ensure a smoother switching between the display camera and the electronic device, the electronic device usually activates the target camera (i.e., the display camera after the switch) in advance to begin capturing images (referred to as activating the target camera). Continue to refer to... Figure 1aIn the zoom-out scenario, when the zoom ratio is close to 1.0x, such as 0.8x, the electronic device will activate the wide-angle camera, meaning it controls the exposure of the wide-angle camera's image sensor to capture the image (output image). When the zoom ratio is close to 3.5x, such as 3.2x, the electronic device will activate the telephoto camera, meaning it controls the exposure of the telephoto camera's sensor to output the image. Similarly, in the zoom-out scenario, when the zoom ratio is close to 3.5x, such as 3.7x, the electronic device will activate the wide-angle camera, meaning it controls the exposure of the wide-angle camera's sensor to output the image. When the zoom ratio is close to 1.0x, such as 1.2x, the electronic device will activate the ultra-wide-angle camera, meaning it controls the exposure of the ultra-wide-angle camera's sensor to output the image.

[0064] The aforementioned electronic devices may be mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), or specialized cameras (such as SLR cameras and point-and-shoot cameras), etc. This application does not impose any restrictions on the specific types of the aforementioned electronic devices.

[0065] The following explanation uses a mobile phone as an example of an electronic device.

[0066] Figure 1b An example application scenario is illustrated. The phone detects a user action (such as a touch / click) on the camera app icon, and in response to this action, it can display, for example... Figure 1b The shooting interface 10 shown in Figure (1) can be the user interface of the default shooting mode of a camera application, on which the user can take a picture. A camera application is an image shooting application on an electronic device, and this application does not limit the name of the application. That is, the user can open the shooting interface 10 of the camera application by clicking the camera application icon. It is understood that the default camera in the default shooting mode is not limited to the rear camera, and the default camera can also be the front camera.

[0067] like Figure 1bAs shown in Figure (1), the shooting interface 10 may include a parameter adjustment area 101, a preview area 102, a camera mode option area 103, a gallery shortcut control 105, a camera flip control 104, a shutter control 106, and a zoom control 107. The controls in the parameter adjustment area 101 are used to adjust corresponding shooting parameters, including but not limited to: flash setting control, AI recognition switch setting control, color standard setting control, and more detailed camera setting controls. The preview area 102 can be used to display a preview image, which is an image captured in real-time by the mobile phone through the display camera. The mobile phone can refresh the display content in the preview area 102 in real time so that the user can view the image captured by the display camera promptly. The camera mode option 103 may display one or more shooting mode options. These one or more shooting mode options may include, but are not limited to: aperture mode option, night scene mode option, portrait mode option, photo mode option, video mode option, professional option mode, and more options. Understandably, these one or more shooting mode options can be represented on the interface as text information, such as "Aperture", "Night Scene", "Portrait", "Photo", "Video", "Pro", "More", or as icons or other forms of interactive elements (IE). This application does not impose any restrictions on this.

[0068] The zoom control 107 is used to trigger the phone to adjust the zoom ratio, thereby adjusting the FOV of the phone's shooting preview interface 102. Users can adjust the phone's current zoom ratio by clicking or sliding the zoom control 107, thus zooming in or out on the scene in the shooting preview interface. Optionally, users can also perform a two-finger relative swipe or a two-finger back-to-back swipe on the preview area 102 to adjust the phone's current zoom ratio, thereby zooming in or out on the scene in the shooting preview interface.

[0069] Continue to refer to Figure 1b As shown in (1), taking the user performing a two-finger back-to-back zoom operation in the preview area 102 as an example. In response to the user performing a two-finger back-to-back zoom operation in the preview area 102, the phone increases the zoom level and magnifies the scene in the shooting preview interface. This can be referred to... Figure 1b As shown in (2).

[0070] Specifically, if a user's zoom operation keeps the phone's zoom level within the same camera's zoom range, the zoom operation will not trigger the electronic device to switch the display camera. However, if the user's zoom operation changes the phone's zoom level from one camera's zoom range to another camera's zoom range, then the user's zoom operation will trigger the electronic device to switch the display camera. For example, if the user's zoom operation changes the phone's zoom level from 1.0x to 3.5x (see reference...), then... Figure 1b As shown in the figure, 1.0x is within the zoom range of a wide-angle camera, and 3.5x is within the zoom range of a telephoto camera. Therefore, the user's zoom operation will trigger the electronic device to switch the display camera from the wide-angle camera to the telephoto camera.

[0071] However, when the display camera of an electronic device switches, that is, when the output sensor of the two consecutive preview images switches, the preview screen of the electronic device will jump due to the different positions of the different cameras in the electronic device. Figure 2 An example is shown where the preview screen jumps when the phone switches between camera feeds. Specifically, Figure 2 (1) is the preview screen before switching the display camera. Figure 2 (2) is the preview screen after switching the camera. (Comparison) Figure 2 As can be seen from (1) and (2), when the display camera of the electronic device switches, the field of view of the preview image changes, that is, the subject being photographed shifts in the preview frame, causing the user to visually perceive a significant jump in the preview image. Obviously, the switching of the preview image is quite abrupt, and the closer the shooting distance, the more abrupt the image switching, and the more obvious the visual jump in the user's perception.

[0072] To enhance the user's visual experience and smooth out transitions in the shooting preview, electronic devices typically align the center point of the field of view when switching between multiple cameras, effectively smoothing the preview transitions. This smoothing can be achieved using a field of view (FOV) offset based on Spatial Alignment Transform (SAT). However, SAT-based FOV offset processing relies on the image's margin area; only when the margin area is sufficiently large can the electronic device perform SAT-based FOV offset processing.

[0073] The following explanation uses a zoom scenario as an example, where the display camera of an electronic device is about to switch from a wide-angle camera to a telephoto camera. Assume that at the current zoom level, both the wide-angle and telephoto camera sensors of the electronic device are output images, with the wide-angle camera being the display camera.

[0074] like Figure 3 As shown, the processing flow of the ISP (Image Signal Processor) chip in electronic devices for images output from wide-angle camera sensors can be roughly divided into the IFE (Image Signal Processing Front End) module processing stage and the IPE (Image Signal Processing Post End) module processing stage.

[0075] The SAT module calculates the center crop (or clipping) bounding box corresponding to the image captured by the wide-angle camera sensor based on the current zoom level, and sends this center crop bounding box to the IFE module. The center crop bounding box can be understood as a cropping box aligned with the center of the image captured by the wide-angle camera sensor. (See reference...) Figure 3 During the IFE module processing stage, the IFE module crops the image captured by the wide-angle camera sensor according to the central cropping frame, obtaining an IFE cropped image. In addition to the image region corresponding to the wide-angle camera's display FOV, the IFE cropped image also includes edge regions (e.g., the margin 1 between the upper boundary of the image region corresponding to the wide-angle camera's display FOV and the upper boundary of the IFE cropped image, and the margin 2 between the left boundary of the image region corresponding to the wide-angle camera's display FOV and the left boundary of the IFE cropped image). These edge regions prevent black areas (i.e., areas without image) from appearing in the image due to warping during the IPE module processing stage. After obtaining the IFE cropped image, the IFE module also performs downsampling processing on the IFE cropped image to obtain a downsampled image, ensuring that the image size of the downsampled image matches the preset display size.

[0076] The SAT module can also determine the telephoto camera's field of view (FOV) in the image captured by the telephoto camera sensor based on the current zoom ratio. It then performs spatial alignment transformation on the telephoto and wide-angle camera FOV image regions, calculating the warp matrix corresponding to the downsampled image and a cropping box aligned with the image center. Finally, it sends the warp matrix and cropping box to the IPE module. (Continue referring to...) Figure 3During the IPE module processing stage, the IPE module performs offset and rotation processing on the downsampled image output by the IFE module based on the warp matrix, obtaining the offset and rotated image (i.e., the downsampled image). In the offset and rotated downsampled image, the telephoto camera's display FOV area (i.e., the display area aligned with the center of the telephoto camera's display FOV) is located in the center of the image. Subsequently, the IPE module can crop the offset and rotated image based on the center cropping box output by the SAT module, obtaining the IPE cropped image. The cropping box output by the SAT module is center-aligned with the offset and rotated image. After the IPE module completes the cropping operation, it can perform upsampling processing on the IPE cropped image to obtain an upsampled image that matches the preset size (i.e., the display size), which is the display preview image captured by the wide-angle camera.

[0077] It should be noted that the SAT module may include an ROI (Region of Interest) translation module and a SAT algorithm module. The ROI translation module can be used to calculate the center cropping box corresponding to the image acquired by the sensor based on the current zoom level, while the SAT algorithm module can be used to calculate the FOV center offset between cameras based on the SAT algorithm. This embodiment does not limit the functional module division of the SAT module in the ISP chip.

[0078] It should be noted that if, at the current zoom level, only the wide-angle camera's sensor outputs an image while the telephoto camera's sensor is not active, the SAT module can calculate the warp matrix and cropping box based on the calibration data of the telephoto and wide-angle cameras (such as the distance between the cameras, intrinsic parameter matrix, extrinsic parameter matrix, etc.).

[0079] Continue to refer to Figure 4a The IPE module performs offset and rotation operations on the downsampled image based on the warp matrix calculated by the SAT module to center the telephoto camera's display FOV. Then, the IPE module can crop the telephoto camera's display FOV from the downsampled image using the center cropping box calculated by the SAT module, resulting in the IPE cropped image. Therefore, the offset operation of the display FOV in the IPE module processing stage relies on the margin area in the IFE cropped image. In close-up shooting scenarios, the FOV center offset between multiple cameras is generally large. Thus, to ensure FOV center alignment during multi-camera switching and a smooth transition in the preview, the margin area in the IFE cropped image needs to be increased to allow the IPE module to successfully complete the offset cropping operation and avoid black border areas (i.e., areas without image) in the IPE cropped image.

[0080] Continue to refer to Figure 4b As shown, in the image captured by the wide-angle camera sensor, the FOV area of ​​the wide-angle camera is the actual image area that needs to be displayed for preview, while the FOV area of ​​the telephoto camera is the actual image area that needs to be displayed for preview after switching from the telephoto camera. To ensure a smooth transition when switching cameras, the electronic device needs to perform spatial transformation alignment processing on the FOV image areas of the wide-angle and telephoto cameras. Therefore, the IFE (In-Frame Edge) cropped image must have a margin area to ensure that it simultaneously covers both the FOV image areas of the wide-angle and telephoto cameras. In close-up shooting scenarios, the FOV center offset between the wide-angle and telephoto cameras is relatively large, inevitably leading to a larger margin area in the IFE cropped image. Figure 4b The dimensions of Margin1 and Margin2 are relatively large.

[0081] However, an increase in the margin area within an IFE-cropped image will inevitably reduce the clarity of the preview image on the electronic device. The following explains why the size of the margin area in an IFE-cropped image affects the clarity of the preview image.

[0082] Figure 4c An exemplary diagram illustrates the impact of margin area size on the sharpness of the preview image. (Refer to...) Figure 4cAs shown, IFE-cropped image 201 is an image with a larger margin region obtained by the IFE module based on the center cropping frame, and IFE-cropped image 202 is an image with a smaller margin region obtained by the IFE module based on the center cropping frame. For example, Margin_1 is greater than Margin_3, and Margin_2 is greater than Margin_4. The IFE module downsamples IFE-cropped image 201 to a preset size (i.e., the display image size) to obtain downsampled image 202, and the IFE module downsamples IFE-cropped image 301 to a preset size to obtain downsampled image 302. The image sizes of downsampled images 202 and 302 are the same. Obviously, since the size of IFE-cropped image 201 is larger than the size of IFE-cropped image 301, the downsampling ratio of IFE-cropped image 201 is greater than the downsampling ratio of IFE-cropped image 301. Comparing downsampled images 202 and 302, it can be seen that the FOV (Field of View) area of ​​the wide-angle camera occupies a larger proportion in downsampled image 202, while it occupies a smaller proportion in downsampled image 302. The IPE module, based on the warp matrix and cropping box calculated by the SAT module, offsets and rotates downsampled image 202 before cropping, obtaining IPE cropped image 203 corresponding to the FOV area of ​​the telephoto camera. Similarly, the IPE module, based on the warp matrix and cropping box calculated by the SAT module, offsets and rotates downsampled image 302 before cropping, obtaining IPE cropped image 303 corresponding to the FOV area of ​​the telephoto camera. A comparison shows that the image size of IPE cropped image 303 is larger than that of IPE cropped image 203. Subsequently, the IPE module can upsample IPE cropped image 203 to obtain upsampled image 204, which is the FOV image 1 of the wide-angle camera. Similarly, the IPE module can upsample the IPE cropped image 303 to obtain the upsampled image 304, which is the display image 2 from the wide-angle camera. Since the upsampled image 204 and the upsampled image 304 have the same size, the upsampling ratio corresponding to the IPE cropped image 203 will necessarily be greater than the upsampling ratio corresponding to the IPE cropped image 303. The larger the upsampling ratio, the lower the sharpness of the resulting display image. That is, the image sharpness of the display image 1 from the wide-angle camera is lower than the image sharpness of the display image 2 from the wide-angle camera.

[0083] This application provides an image alignment method between multiple cameras. In this method, the electronic device splits the offset of the display FOV region during the IPE module processing stage into two parts. First, the IPE module uses a coarsely offset off-center cropping box to complete the cropping operation when cropping the image acquired by the sensor. Then, the IPE module performs a fine offset on the display FOV region before cropping the downsampled image, thus completing the alignment of the display FOV region images between multiple cameras.

[0084] In this way, during camera preview scenarios, the alignment of the displayed FOV area image is no longer constrained by the margin area size in the IFE cropped image. This means the smooth transition of the preview image during camera switching is no longer limited by the margin area size in the IFE cropped image. This image alignment method not only ensures a smooth transition in the preview image during camera switching but also does not affect the clarity of the electronic device's preview image, thereby improving the user experience.

[0085] Figure 5 A schematic diagram of the structure of the electronic device 100 is shown. It should be understood that... Figure 5 The electronic device 100 shown is merely an example of an electronic device, and the electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. Figure 5 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0086] Electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0087] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP) (or ISP chip), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0088] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0089] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0090] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0091] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0092] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0093] The charging management module 140 receives charging input from the charger. The charger can be a wireless charger or a wired charger. Figure 2 As shown, in some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0094] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0095] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0096] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0097] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0098] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0099] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with networks and other devices through wireless communication technology.

[0100] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0101] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel.

[0102] Electronic device 100 can perform shooting functions through ISP chip, camera 193, video codec, GPU, display screen 194 and application processor.

[0103] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0104] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0105] In one implementation, the electronic device 100 may include three cameras: an ultra-wide-angle camera, a wide-angle camera, and a telephoto camera, each with its own image sensor. For example, the ultra-wide-angle camera has a zoom range of [0.6, 1.0), the wide-angle camera has a zoom range of [1.0, 3.5), and the telephoto camera has a zoom range of [3.5, 5].

[0106] In some implementations, the camera can be a TOF (Time of Flight) camera (or lens). The TOF camera is used to acquire TOF data. In some implementations, the TOF camera may include a TOF sensor, a TOF sensor controller, a TOF light source, and a TOF light source controller.

[0107] In some implementations, the TOF light source controller is controlled by the TOF sensor controller to control the TOF light source. Under the control of the TOF light source controller, the TOF light source emits infrared (IR) light. The TOF sensor is used to sense the infrared light reflected from an object (e.g., a human face) to acquire TOF data. Both the TOF sensor controller and the TOF light source controller can communicate with the processor 110. The processor 110 can also be used to generate TOF images, including infrared images and depth images, based on the TOF data acquired by the TOF camera.

[0108] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 is in frequency selection mode, the DSP is used to perform Fourier transforms on the frequency energy, etc.

[0109] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0110] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0111] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0112] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0113] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0114] A pressure sensor is used to sense pressure signals and convert them into electrical signals. In some embodiments, the pressure sensor may be located on the display screen 194. There are many types of pressure sensors, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to the pressure sensor, the capacitance between the electrodes changes. The electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to the display screen 194, the electronic device 100 detects the intensity of the touch operation based on the pressure sensor. The electronic device 100 may also calculate the touch position based on the detection signal from the pressure sensor. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities may correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS message is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS message is executed.

[0115] An accelerometer can detect the magnitude of acceleration of an electronic device 100 in various directions (typically three axes). When the electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of the electronic device and is applied to applications such as screen orientation switching and pedometers.

[0116] A touch sensor, also known as a "touch panel," can be located on the display screen 194. The touch sensor and display screen 194 together form a touchscreen, also called a "touch screen." The touch sensor detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor may also be located on the surface of the electronic device 100, in a different position than the display screen 194.

[0117] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0118] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback.

[0119] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0120] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0121] Figure 6 This is a software structure block diagram of the electronic device 100 according to an embodiment of this application.

[0122] The layered architecture of the electronic device 100 divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the hardware abstraction layer (HAL), and the kernel layer (also known as the driver layer).

[0123] Understandable. Figure 6For example, between the application framework layer and the HAL layer, there could also be an Android runtime and a library layer. The Android Runtime includes core libraries and a virtual machine, responsible for scheduling and managing the Android system. System libraries can include multiple functional modules, such as a surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), and 2D graphics engines (e.g., SGL).

[0124] The application layer can include a series of application packages. For example... Figure 6 As shown, an application package can include applications such as camera apps, gallery apps, and apps with camera functionality. The application package can also include applications such as calling, calendar, maps, navigation, music, video, and text messaging.

[0125] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0126] like Figure 6 As shown, the application framework layer can include a camera service, which can be called by camera applications to implement shooting-related functions. In addition, the application framework layer may also include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0127] The window manager is used to manage window applications. It can obtain the screen size, determine if a status bar is present, lock the screen, and capture screenshots, among other things.

[0128] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0129] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0130] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0131] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0132] The notification manager allows applications to display notifications in the status bar. These notifications can be used to convey informational messages and can disappear automatically after a short pause, requiring no user interaction. Examples include notifications of download completion and message alerts. Notifications can also appear as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Other notification methods include text messages displayed in the status bar, sound alerts, vibrations from electronic devices, and flashing indicator lights.

[0133] It should be noted that the camera application may also call the content provider, resource manager, notification manager, window manager, view system, etc., according to actual business needs, and this application embodiment does not impose any restrictions on this.

[0134] The kernel layer is the layer between hardware and software. The kernel layer includes at least a camera driver and an ISP driver. The camera driver can be used to drive hardware modules with shooting capabilities, such as the image sensor in a camera. In other words, the camera driver is responsible for data interaction with the camera. The ISP driver can be used to drive the ISP chip, specifically to send control commands to the ISP chip or transmit image data. For example, the ISP driver can control the IFE and IPE modules in the ISP chip to perform related image processing. Of course, the kernel layer may also include display drivers, audio drivers, sensor drivers, etc., and this application embodiment does not impose any limitations on this.

[0135] Furthermore, the HAL layer can encapsulate drivers in the kernel layer and provide an interface for calls to the application framework layer, shielding it from the implementation details of the low-level hardware. For example... Figure 6 As shown, the HAL layer described above may include a Camera HAL.

[0136] Camera HAL is the core software framework of Camera, which may include interface modules, Sensornode, ROI translator, SAT algorithm module, multi-camera decision module, etc.

[0137] The Sensor node and interface module are components in the image data and control command transmission pipeline of Camera HAL, each with different functions. For example, a Sensor node can be a control node for the camera sensor, controlling it through a camera driver. Similarly, an interface module can be a software interface for the application framework layer, used for data interaction with it. Exemplarily, the interface module can also interact with other modules in Camera HAL (such as the multi-camera decision module, Sensor node, ROI translator, and SAT algorithm module).

[0138] The multi-camera decision module can determine the camera sensor for display and output (or streaming) based on the application scenario, such as the front camera sensor or the rear camera sensor, and further, the ultra-wide-angle camera sensor, wide-angle camera sensor, and telephoto camera sensor within the rear camera setup. The camera application can pass the user-selected camera mode, zoom parameters, and other information to the camera service in the application framework layer, which then passes this information to the multi-camera decision module via the HAL layer interface module.

[0139] The multi-camera decision-making module can also predict the target camera to be switched to based on the user's zoom operation, and activate the target camera's camera sensor in advance via the camera driver. Additionally, the multi-camera decision-making module can also select the camera to display among the multiple cameras with their camera sensors activated.

[0140] In this embodiment, the ROI translator can be used to calculate the IFE (In-Frame Extent) crop box (hereinafter referred to as the non-center crop box) that is not aligned with the center of the image acquired by the sensor. Specifically, the ROI translator can calculate the IFE crop box (hereinafter referred to as the center crop box) that is aligned with the center of the image acquired by the sensor based on the zoom ratio, calculate the coarse offset between multiple cameras based on prior information, and calculate the offset corresponding to the center crop box based on the zoom ratio. Then, the offset is applied to the center crop box to obtain the non-center crop box.

[0141] In this embodiment, the SAT algorithm module can be used to spatially align image data according to the zoom ratio and the preset SAT algorithm, and calculate the warp matrix and cropping box for use by the IPE module in the ISP chip.

[0142] Understandable, Figure 6The layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer layers than illustrated, and each layer may include more or fewer components, or combine some components, or split some components, or have different component arrangements; this application does not impose any limitations.

[0143] It is understood that, in order to implement the image alignment method among multiple cameras in the embodiments of this application, the electronic device includes hardware and / or software modules that perform the respective functions. Based on the algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0144] The following uses a zoomin scenario as an example to explain the image alignment method between multiple cameras provided in this application embodiment. In this zoomin scenario, the display camera of the electronic device is a wide-angle camera, and the sensor of the telephoto camera has been activated and started to acquire images.

[0145] like Figure 7 The diagram shows the interaction between each module. (Refer to...) Figure 7 The process of the image alignment method between multiple cameras provided in this application embodiment specifically includes:

[0146] S401, in response to a zoom operation performed by the user in the shooting preview interface, the camera application sends a zoom ratio value to the camera service.

[0147] The shooting preview interface can be either a photo preview interface or a video preview interface; this embodiment does not limit this. In the zoom-in scenario, the zoom operation is an operation to increase the zoom magnification. For example, this zoom operation can be an operation where the user clicks or slides the zoom control, or an operation where two fingers slide back and forth, etc.; this embodiment does not limit this. Correspondingly, in the zoom-out scenario, the zoom operation is an operation to decrease the zoom magnification.

[0148] When a user zooms in, the phone's touch module generates a touch signal and sends it to the camera application. Upon receiving the touch signal, the camera application converts the corresponding coordinates into a zoom ratio value and sends it to the camera service within the application framework layer.

[0149] S402, the camera service sends a preview frame request to the interface module in the Camera HAL.

[0150] The camera service writes the received zoom ratio value into a preview frame request (or preview image frame request) and sends the preview frame request to the interface module in the Camera HAL.

[0151] S403, the interface module passes the preview frame request to the sensor node, and the sensor node sends the image output command to the camera driver based on the preview frame request.

[0152] For example, the image output command (or image acquisition command) may include, but is not limited to, the identifier of the camera sensor to be output. In this scenario, the image output command is used to instruct the wide-angle camera sensor and the telephoto camera sensor to output images simultaneously. The image acquired by the wide-angle camera sensor is the image that needs to be displayed.

[0153] It's important to note that due to the round-robin buffering mechanism between the application framework layer and the HAL layer in the Android architecture, when a user opens the camera application, the camera service first sends multiple (e.g., 8) preview frame requests to the Camera HAL consecutively. After receiving a preview image frame from the Camera HAL, it sends another preview image frame request to the Camera HAL. In the Camera HAL, the sensor node sequentially controls the camera sensor to perform exposure and image output operations based on these multiple preview frame requests, through the camera driver. In other words, for any given preview frame request, the sensor node will only control the camera sensor to perform exposure and image output operations after all previous preview frame requests have been processed and exposed by the sensor node.

[0154] S404, the camera driver drives the wide-angle camera sensor to expose the image and acquires the image captured by the wide-angle camera sensor.

[0155] S405, the camera driver drives the telephoto camera sensor to expose and output an image, and acquires the image captured by the telephoto camera sensor.

[0156] This embodiment does not limit the timing of S404 and S405.

[0157] S406, the camera driver sends the image captured by the wide-angle camera sensor to the IFE module in the ISP chip through the ISP driver.

[0158] S407, the camera driver sends the image captured by the telephoto camera sensor to the SAT algorithm module in the Camera HAL.

[0159] This embodiment does not limit the timing of S406 and S407.

[0160] In S408, the interface module in the Cameral HAL sends the zoom ratio value to the ROI translator.

[0161] S409, the interface module in the Camera HAL sends the zoom ratio value to the SAT algorithm module.

[0162] This embodiment does not limit the timing order of S408 and S409. Similarly, this embodiment does not limit the timing order of S403 with S408 and S409.

[0163] In S410, the ROI translator in Cameral HAL calculates the center clipping box based on the current zoom ratio value.

[0164] The center cropping frame is center-aligned with the image captured by the wide-angle camera sensor.

[0165] In this embodiment, the central cropping frame is determined based on the display FOV of the wide-angle camera, and does not need to include the display FOV of the wide-angle camera. That is to say, when cropping the image captured by the wide-angle camera sensor according to the central cropping frame, the margin area is relatively small.

[0166] For example, such as Figure 8 As shown in (1), the ROI translator in Cameral HAL calculates the center cropping box corresponding to the FOV of the wide-angle camera based on the current zoom ratio. In the image region corresponding to this center cropping box, the margin area is relatively small, that is, the size of Margin1 and Margin2 is small.

[0167] In S411, the ROI translator in Cameral HAL calculates a rough FOV center offset based on prior information and calculates the offset of the center clipping frame based on the current zoom magnification value.

[0168] In the zoom-in scenario, assuming that the user's zoom operation drives the display camera of the electronic device to switch from Camera 1 to Camera 2, the ROI translator in the Camera HAL needs to calculate the approximate FOV center offset between Camera 1 and Camera 2 based on prior information. Subsequently, the ROI translator calculates the offset of the central cropping frame according to the current zoom magnification value and the approximate FOV center offset between Camera 1 and Camera 2.

[0169] Assume that the approximate FOV center offset between Camera 1 and Camera 2 is offset_12, the display zoom magnification range of Camera 1 is [m x, n x), and the current zoom magnification value is p x, where m < p < n. Then the offset of the central cropping frame is: Offset = [(p - m) / (n - m)] * offset_12.

[0170] In the zoom-in scenario described in this process, since the user's zoom operation drives the display camera of the electronic device to switch from the wide-angle camera to the telephoto camera, the ROI translator in the Camera HAL needs to calculate the approximate FOV center offset between the wide-angle camera and the telephoto camera based on prior information. Subsequently, the ROI translator can calculate the offset of the central cropping frame according to the current zoom magnification value and the approximate FOV center offset between the wide-angle camera and the telephoto camera.

[0171] Exemplarily, the approximate FOV center offset between the wide-angle camera and the telephoto camera is offset_wt, the display zoom magnification range of the wide-angle camera is [1.0x, 3.5x), and the current zoom magnification value is 3.3x. Then the offset of the central cropping frame is: Offset = [(3.3 - 1.0) / (3.5 - 1.0)] * offset_wt.

[0172] It should be noted that the approximate FOV center offset between the cameras is a two-dimensional vector, so the offset of the central cropping frame is a two-dimensional vector.

[0173] As an optional implementation, the ROI translator can calculate the approximate FOV center offset between Camera 1 and Camera 2 based on the calibration data of Camera 1 and Camera 2. Among them, the focal length of Camera 1 is less than that of Camera 2. For example, the ROI translator can calculate the approximate FOV center offset between Camera 1 and Camera 2 according to the extrinsic matrix [R, t] between Camera 1 and Camera 2, the intrinsic matrix K1 of Camera 1, and the intrinsic matrix K2 of Camera 2. Among them, R is the rotation matrix and t is the translation vector.

[0174] Suppose that a point p2 is selected in the image captured by camera 2, and point p2 corresponds to a 3D point P in the world coordinate system. The projection point of 3D point P in the image captured by camera 1 is p1. Then, the approximate FOV center offset between camera 1 and camera 2 is:

[0175] offset_12 = p1 - p2.

[0176] Where p1 = K1*[R,t]*Z*K2 -1 *p2, Z represents the shooting distance, which can be obtained through external TOF or AF (Automatic Focus) data.

[0177] Among them, the intrinsic parameter matrix of the camera c x c y f represents the coordinates of the origin of the image coordinate system in the pixel coordinate system. x =f / dx, f y = f / dy, where f is the focal length of the camera, dx is the image length of a single pixel in the x-direction, and dy is the image length of a single pixel in the x-direction.

[0178] In S412, the ROI translator in Cameral HAL offsets the center crop box based on its offset to obtain a non-center crop box, and then sends the non-center crop box to the IFE module in the ISP chip.

[0179] For example, such as Figure 8 As shown in (2), applying the offset of the center crop box to the center crop box will yield a corresponding non-center crop box. The "non-center" in the non-center crop box means that the crop box is not aligned with the center of the image captured by the wide-angle camera sensor.

[0180] For example, the ROI translator in Cameral HAL can send non-central clipping boxes to the IFE module in the ISP chip via the ISP driver.

[0181] In S413, the IFE module in the ISP chip crops the image captured by the wide-angle camera sensor according to the non-center cropping frame, obtains the IFE cropped image, and sends the IFE cropped image to the SAT algorithm module in the Camera HAL.

[0182] Continue to refer to Figure 8 As shown in (3), the IFE module crops the wide-angle sensor output image according to the non-center cropping box to obtain the IFE cropped image.

[0183] For example, the IFE module can send the IFE-cropped image to the SAT algorithm module in Cameral HAL via the ISP driver.

[0184] In this way, the IFE module crops the image captured by the wide-angle camera sensor based on a non-centered cropping frame, achieving initial alignment between the center point of the wide-angle camera's FOV and the center point of the telephoto camera's FOV. Moreover, the margin area in the IFE cropped image obtained by the IFE module is relatively small, so the clarity of the displayed preview image is not compromised.

[0185] S414, the IFE module in the ISP chip downsamples the IFE-cropped image and sends the downsampled image to the IPE module.

[0186] For example, the IFE module calculates the downsampling ratio based on the size of the IFE cropped image and the preset size of the image to be displayed, and performs downsampling processing on the IFE cropped image according to the downsampling ratio to obtain a downsampled image.

[0187] In the S415, the SAT algorithm module in the Camera HAL calculates the warp matrix and the center cropping box based on the IFE-cropped image, the current zoom ratio, and the image captured by the telephoto camera sensor, and sends the warp matrix and the center cropping box to the IPE module in the ISP chip.

[0188] The SAT algorithm module can determine the field of view (FOV) of the telephoto camera from the image captured by the telephoto camera sensor based on the current zoom ratio. Then, based on the preset SAT algorithm, it performs spatial transformation and alignment processing on the IFE-cropped image and the telephoto camera's FOV image to obtain the FOV center offset corresponding to the IFE-cropped image. Subsequently, it generates a warp matrix corresponding to the downsampled image and a central cropping bounding box corresponding to the downsampled image based on this FOV center offset. The area covered by the central cropping bounding box is the FOV region to be displayed.

[0189] The calculation methods for the warp matrix and the center cropping box corresponding to the downsampled IFE cropped image can be found in existing techniques and will not be elaborated here.

[0190] For example, the SAT algorithm module can send the warp matrix and center cropping box corresponding to the downsampled IFE cropped image to the IPE module in the ISP chip through the ISP driver.

[0191] This embodiment does not limit the timing of S414 and S415.

[0192] In S416, the IPE module in the ISP chip performs offset rotation processing on the downsampled image based on the warp matrix, and then performs cropping processing on the offset rotation downsampled image based on the center cropping frame to obtain the IPE cropped image.

[0193] Because the SAT algorithm module calculates the precise offset between the FOV centers of the wide-angle camera and the telephoto camera, and generates a warp matrix based on this precise offset, the IPE module performs offset and rotation processing on the downsampled image according to the warp matrix, ensuring that the FOV area of ​​the telephoto camera is located at the center of the downsampled image. Thus, when the IPE module crops the offset and rotated image using the center cropping box calculated by the SAT algorithm module, it obtains a cropped image aligned with the FOV area of ​​the telephoto camera, ensuring a smooth transition in the preview image when the display camera switches from the wide-angle camera to the telephoto camera.

[0194] In S417, the IPE module in the ISP chip upsamples the IPE-cropped image and sends the upsampled image as a preview image for display.

[0195] For example, the IPE module calculates the upsampling ratio based on the size of the IPE cropped image and the preset size of the image to be displayed, and then upsamples the IPE cropped image according to the upsampling ratio to obtain an upsampled image. This upsampled image can then be displayed as a preview image.

[0196] For example, the IPE module sends the upsampled image (i.e., the preview image) to the Camera HAL via the ISP driver. The Camera HAL then sends this preview image as feedback corresponding to the preview frame request to the camera service, which in turn sends the preview image to the camera application for display. For details not explained here, please refer to existing technologies; they will not be elaborated further here.

[0197] Additionally, it should be noted that the image processing workflow for the camera sensor may also include format conversion, noise removal, white balance, etc. For any parts of the workflow not explained in detail here, please refer to existing technologies.

[0198] In this scenario, when the user adjusts the zoom level of the electronic device to 3.5x, the display camera switches from a wide-angle camera to a telephoto camera. At this point, the electronic device displays a preview image generated from the image captured by the telephoto camera sensor. Because the FOV center of the wide-angle camera was previously aligned with the FOV center of the telephoto camera, the field of view of the two preview images displayed by the electronic device (the first frame is the preview image generated from the image captured by the wide-angle camera sensor, and the second frame is the preview image generated from the image captured by the telephoto camera sensor) does not change significantly when the display camera switches, resulting in a smooth transition in the preview image.

[0199] It should be noted that when the user adjusts the zoom level of the electronic device to 3.5x, the display camera of the electronic device becomes a telephoto camera. Since there are no other cameras in the electronic device with a focal length greater than that of the telephoto camera, the FOV center of the telephoto camera no longer needs to be aligned.

[0200] The above process is explained using the example of both the wide-angle and telephoto camera sensors operating at current. In some implementations, the multi-camera decision module only controls both camera sensors to operate at current when the display camera might switch. When the difference between the current zoom level and the zoom level at which the display camera switches is significant, the multi-camera decision module only controls one camera sensor to operate at current. For example, if the current zoom level is 3.3x, the multi-camera decision module controls the telephoto camera sensor to operate at current; if the current zoom level is 1.2x, the multi-camera decision module does not control the telephoto camera sensor to operate at current, and only the wide-angle camera sensor acquires images.

[0201] During user zooming, to ensure consistency in the preview image changes, when the telephoto camera sensor is not in operation (zoomin scenario), the electronic device also aligns the FOV center of the wide-angle camera with that of the telephoto camera. The difference lies in that, since the telephoto camera sensor is not in operation, the SAT algorithm module in Camera HAL can calculate the total FOV center offset between the wide-angle and telephoto cameras using calibration data (e.g., camera spacing, intrinsic and extrinsic matrices). Because the FOV area of ​​the wide-angle camera has already been partially offset during the IFE module processing stage, the SAT algorithm module can calculate the required offset of the wide-angle camera's FOV area during the IPE module processing stage based on the total FOV center offset and the offset already achieved during the IFE module processing stage. Then, it can generate the corresponding warp matrix based on the required offset during the IPE module processing stage. Thus, during the IPE module processing stage, the IPE module continues to process the downsampled image set based on the calculation results of the SAT algorithm module, so that the FOV region of the wide-angle camera continues to shift in the image, obtaining the FOV region to be displayed. Other processes are similar to those described above and will not be repeated here.

[0202] The situation is similar when switching from an ultra-wide-angle camera to a wide-angle camera in a zoom-in scenario. In this case, the FOV center of the ultra-wide-angle camera needs to be aligned with the FOV center of the wide-angle camera. This involves a rough alignment during the IFE module processing stage of the ISP chip, followed by a precise alignment during the IPE module processing stage. For details not explained in sufficient detail here, please refer to the previous section on switching from a wide-angle camera to a telephoto camera; further explanation is unnecessary here.

[0203] After the mobile phone adopts the image alignment method between multiple cameras provided in this embodiment, Figure 9 An example is shown illustrating the changes in the preview screen when the mobile phone switches between camera displays. Among them, Figure 9 (1) is the preview screen before switching the display camera. Figure 9 (2) is the preview screen after switching the camera. (Comparison) Figure 9 As can be seen from (1) and (2), when the display camera of the electronic device is switched, the field of view of the preview image changes very little, that is, the object being photographed shifts very little in the preview frame, and the user will hardly perceive that the preview image is jumping.

[0204] Figure 10 An example is shown illustrating how a portion of the preview screen changes when the phone switches between camera feeds. Specifically, Figure 10(1) is the preview screen before switching the display camera. Figure 10 (2) is the preview screen after switching the camera. (Comparison) Figure 10 As can be seen from (1) and (2), when the display camera of the electronic device switches, not only does the change in the field of view of the preview image be very small, but the clarity of the preview image is also very high. Under the premise of the same preview image switching effect, the image alignment method between multiple cameras provided in this embodiment can make the preview image clearer.

[0205] Similar to the zoom-out scene, in the zoom-out scene, the FOV center of the ultra-wide-angle camera is aligned with the FOV center of the wide-angle camera in stages (coarse alignment in the IFE module processing stage and precise alignment in the IPE module processing stage), and the FOV center of the wide-angle camera is aligned with the FOV center of the telephoto camera (coarse alignment in the IFE module processing stage and precise alignment in the IPE module processing stage).

[0206] In zoom-out scenarios, when the FOV center of the ultra-wide-angle camera is aligned with the FOV center of the wide-angle camera, the wide-angle camera becomes the display camera. After the ultra-wide-angle camera sensor starts transmitting power, the IFE module of the ISP chip crops the image (not the display image) captured by the ultra-wide-angle camera sensor based on the non-center cropping bounding box, resulting in the IFE cropped image. Subsequently, the IPE module of the ISP chip continues to offset the FOV region in the IFE cropped image based on the warp matrix calculated by the SAT algorithm module. In this way, when the display camera of the electronic device switches from the wide-angle camera to the ultra-wide-angle camera, since the FOV center of the ultra-wide-angle camera is already aligned with that of the wide-angle camera, the preview image can transition smoothly, avoiding abrupt image jumps.

[0207] Similarly, in zoom-out scenarios, when the FOV center of the wide-angle camera is aligned with the FOV center of the telephoto camera, the telephoto camera becomes the display camera. After the wide-angle camera sensor starts transmitting power, the IFE module of the ISP chip crops the image captured by the wide-angle camera sensor (the non-display image) according to the non-center cropping box, obtaining the IFE cropped image. Subsequently, the IPE module of the ISP chip continues to offset the FOV region in the IFE cropped image according to the warp matrix calculated by the SAT algorithm module. In this way, when the display camera of the electronic device switches from the telephoto camera to the wide-angle camera, since the FOV center of the wide-angle camera is aligned with the telephoto camera, the preview image can transition smoothly, avoiding abrupt image jumps.

[0208] In summary, the image alignment method between multiple cameras provided in this embodiment achieves alignment between the FOV areas of the multiple cameras in stages. Specifically, coarse alignment is achieved during the IFE module processing stage of the ISP chip, while precise alignment is achieved during the IPE module processing stage. This method ensures a smooth transition in the preview image when switching between multiple cameras without reducing the clarity of the preview image.

[0209] This embodiment also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the image alignment method between multiple cameras in the above embodiment.

[0210] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement the image alignment method between multiple cameras in the above embodiment.

[0211] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory. The memory is used to store computer execution instructions. When the apparatus is running, the processor can execute the computer execution instructions stored in the memory to cause the chip to execute the image alignment method between multiple cameras in the above-described method embodiments.

[0212] In this embodiment, the electronic devices (such as mobile phones), computer storage media, computer program products, or chips are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0213] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0214] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0215] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for image alignment between multiple cameras, characterized in that, Applied in an electronic device, the electronic device includes at least two cameras, and the electronic device also includes an image signal processor (ISP) chip, the ISP chip including a first image processing module and a second image processing module, comprising: In response to the first operation, the current zoom ratio of the electronic device is increased to a first value, and the first image captured by the first camera is obtained; Based on the first value and prior information, a first cropping box is determined in the first image; wherein the first cropping box is not centered with the first image and is offset in the first image towards the field of view (FOV) direction of the second camera; the focal length of the first camera is smaller than that of the second camera. The first image processing module is controlled to crop the first image according to the first cropping frame to obtain the second image; Calculate the first warp matrix and the second cropping box corresponding to the second image; The second image processing module is controlled to process the second image according to the first warp matrix to obtain a third image, and to perform a cropping operation on the third image according to the second cropping box to obtain a fourth image; the fourth image is used to generate a preview image for display.

2. The method according to claim 1, characterized in that, Based on the first value and prior information, a first cropping box is determined in the first image, including: Based on the FOV area of ​​the first camera corresponding to the first value, a third cropping frame is determined in the first image; wherein the third cropping frame is aligned with the center of the first image; Based on the first value and prior information, calculate the target offset of the third cropping frame; The third cropping frame is offset according to the target offset of the third cropping frame to obtain the first cropping frame.

3. The method according to claim 2, characterized in that, Based on the first value and prior information, the target offset of the third cropping frame is calculated, including: Based on the prior information, calculate the total FOV center offset between the first camera and the second camera; The total offset of the FOV center is allocated based on the first value to obtain the target offset of the third cropping frame.

4. The method according to any one of claims 1-3, characterized in that, Also includes: In response to the first operation, the fifth image captured by the second camera is acquired; Calculating the first warp matrix and the second cropping box corresponding to the second image includes: Based on the first value and the fifth image, the FOV area of ​​the second camera is determined in the fifth image; Based on the FOV region of the second camera in the fifth image and the second image, spatial transformation and alignment processing is performed to obtain a first warp matrix and a second cropping box corresponding to the second image.

5. The method according to any one of claims 1-3, characterized in that, The second camera is not activated; Calculating the first warp matrix and the second cropping box corresponding to the second image includes: Based on the calibration information of the first camera and the second camera, calculate the total FOV center offset between the first camera and the second camera; Based on the total offset of the FOV center and the target offset corresponding to the first cropping frame, calculate the offset to be made of the FOV region of the first camera in the second image; Based on the offset to be calculated, the first warp matrix and the second cropping box corresponding to the second image are calculated.

6. The method according to any one of claims 1-3, characterized in that, Also includes: In response to the first operation, the current zoom level of the electronic device is increased to a second value, and the display camera of the electronic device is switched from the first camera to the second camera.

7. The method according to any one of claims 1-3, characterized in that, Also includes: In response to the second operation, the current zoom level of the electronic device is reduced to a third value, and a sixth image captured by the first camera and a seventh image captured by the second camera are acquired; the seventh image is used to generate a preview image to be displayed. Based on the third value and prior information, a fourth cropping box is determined in the sixth image; wherein the fourth cropping box is not centered in the sixth image and is offset in the FOV direction of the second camera in the sixth image; The first image processing module is controlled to crop the sixth image according to the fourth cropping frame to obtain the eighth image; Based on the third value and the seventh image, the FOV area of ​​the second camera is determined in the seventh image; Based on the FOV region of the second camera in the seventh image and the eighth image, spatial transformation and alignment processing is performed to obtain the second warp matrix and the fifth cropping box corresponding to the eighth image; The second image processing module is controlled to process the eighth image according to the second warp matrix to obtain the ninth image, and to perform a cropping operation on the ninth image according to the fifth cropping box to obtain the tenth image.

8. The method according to claim 7, characterized in that, Also includes: In response to the second operation, the current zoom level of the electronic device is reduced to a fourth value, and the display camera of the electronic device is switched from the second camera to the first camera. The tenth image is used to generate a preview image for display.

9. The method according to claim 7, characterized in that, Based on the third value and prior information, a fourth cropping box is determined in the sixth image, including: Based on the FOV area of ​​the first camera corresponding to the third value, a sixth cropping frame is determined in the sixth image; wherein the sixth cropping frame is aligned with the center of the sixth image; Based on the third value and prior information, calculate the target offset of the sixth cropping frame; The sixth cropping frame is offset according to the target offset of the sixth cropping frame to obtain the fourth cropping frame.

10. The method according to claim 1, characterized in that, The prior information includes: The extrinsic parameter matrix between the first camera and the second camera, and the intrinsic parameter matrix between the first camera and the second camera.

11. The method according to claim 1, characterized in that, The first camera is an ultra-wide-angle camera, and the second camera is a wide-angle camera; Alternatively, the first camera may be a wide-angle camera, and the second camera may be a telephoto camera.

12. An electronic device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored on the memory, and when the computer programs are executed by the one or more processors, cause the electronic device to perform the image alignment method between multiple cameras as claimed in any one of claims 1-11.

13. A chip system, characterized in that, When applied in an electronic device, the chip system includes instructions and at least one processor, the at least one processor executing the instructions to cause the electronic device to perform an image alignment method between multiple cameras as described in any one of claims 1-11.

14. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is run on an electronic device, it causes the electronic device to perform the image alignment method between multiple cameras as described in any one of claims 1-11.

Images