Image alignment method, electronic equipment, chip system and storage medium

Abstract

The invention provides an image alignment method among multiple cameras, electronic equipment, a chip system and a storage medium. In the method, alignment of multi-camera FOV areas of the electronic equipment is realized in two stages, rough alignment processing is firstly carried out in an IFE module processing stage, and then fine alignment processing is carried out in an IPE module processing stage. Thus, in a shooting preview scene, alignment of the FOV areas of the multiple cameras is not limited by the size of a reserved margin area when the IFE module cuts the image when the sending and displaying cameras are switched, it can be ensured that preview images can be smoothly transited when the cameras are switched, the definition of the preview images of the electronic equipment cannot be affected, and the image quality of the electronic equipment is improved. Therefore, the use experience of the user is improved.

Description

Technical Field

[0001] This application relates to the technical field of intelligent terminals, and in particular, to an image alignment method, an electronic device, a chip system, and a storage medium. Background Art

[0002] With the development of electronic devices such as mobile phones and tablet computers, the camera function is becoming more and more important to users. Taking mobile phones as an example, in order to provide users with a better camera experience, multiple cameras with different focal lengths are usually set on electronic devices to photograph objects at different distances.

[0003] When photographing objects at different distances, users can switch the camera displayed by the electronic device through a zoom operation to improve the shooting effect of the electronic device. Among them, when the camera displayed by the electronic device switches, the smoother the preview screen switches, the smaller the jump of the preview screen will be. However, in order to ensure that the preview screen can be smoothly switched when the camera displayed by the electronic device switches, it may affect the clarity of the preview screen, thus affecting the user's visual experience. Summary of the Invention

[0004] This application provides an image alignment method, an electronic device, a chip system, and a storage medium. In this method, when the camera displayed switches, the alignment of the FOV (Field of View) regions of multiple cameras is no longer limited by the size of the margin region reserved when the IFE module crops the image. It can not only ensure that the preview screen can smoothly transition when the camera switches, but also will not affect the clarity of the preview screen of the electronic device.

[0005] In a first aspect, an embodiment of this application provides an image alignment method between multiple cameras. This method is applied to an electronic device. The electronic device includes at least two cameras, and the electronic device further includes an Image Signal Processor (ISP) chip, and the ISP chip includes a first image processing module and a second image processing module. The method includes: the electronic device increases the current zoom magnification value of the electronic device to a first value in response to a first operation, and acquires a first image collected by a first camera; the electronic device determines a first cropping frame in the first image according to the first value and prior information; wherein, the first cropping frame is not centered with the first image and offsets in the direction of the Field of View (FOV) of a second camera in the first image; the focal length of the first camera is less than that of the second camera; the electronic device controls the first image processing module to crop the first image according to the first cropping frame to obtain a second image; the electronic device calculates a first warp matrix and a second cropping frame corresponding to the second image; the electronic device controls the second image processing module to process the second image according to the first warp matrix to obtain a third image, and performs a cropping operation on the third image according to the second cropping frame to obtain a fourth image; the fourth image is used to generate a preview image for display.

[0006] Among them, the first image processing module refers to the IFE module in the ISP chip mentioned below, and the second image processing module refers to the IPE module in the ISP chip mentioned below.

[0007] Exemplarily, the first operation can be a zoomin operation achieved by clicking or sliding a zoom control, or a zoomin operation achieved by sliding two fingers away from each other. Among them, the first value belongs to the display zoom magnification range of the first camera.

[0008] Among them, in the image area cropped according to the first crop box, the size of the edge (margin) area is small.

[0009] In this embodiment, the electronic device calculates a warp matrix corresponding to the second image, so that after the second image is processed based on the warp matrix, the FOV area of the first camera can be aligned with the FOV area of the second camera and is located in the central area of the image. Among them, the second crop box is a central crop box and is centrally aligned with the third image after being offset and rotated based on the warp matrix.

[0010] In this embodiment, the alignment of the FOV area of the first camera and the FOV area of the second camera is achieved in two stages. First, a rough alignment process is performed during the IFE module processing stage, and then a fine alignment process is performed during the IPE module processing stage. In this way, in the shooting preview scenario, the alignment of the FOV areas of multiple cameras when the display camera is switched is no longer limited by the size of the margin area reserved when the IFE module crops the image, which can not only ensure that the preview screen can smoothly transition when the camera is switched, but also will not affect the clarity of the preview screen of the electronic device, thereby improving the user experience.

[0011] According to the first aspect, the electronic device determines a first crop box in the first image according to the first value and prior information, including: the electronic device determines a third crop box in the first image according to the FOV area of the first camera corresponding to the first value; among them, the third crop box is centrally aligned with the first image; the electronic device calculates the target offset of the third crop box according to the first value and prior information; the electronic device performs an offset process on the third crop box according to the target offset of the third crop box to obtain the first crop box.

[0012] In this way, during the IFE module processing stage, the IFE module performs non-central cropping on the first image to reduce the size of the margin area in the cropped image.

[0013] According to the first aspect, or any implementation of the above first aspect, the electronic device calculates the target offset of the third cropping frame based on the first value and the prior information, including: the electronic device calculates the total FOV center offset between the first camera and the second camera according to the prior information; the electronic device allocates the total FOV center offset according to the first value to obtain the second offset of the second cropping frame.

[0014] Exemplarily, the rough FOV center offset between the first camera and the second camera is offset_12, the zoom magnification range sent by camera 1 is [m x, n x), and the current zoom magnification value (i.e., the first value) is p x, where m < p < n. Then the second offset of the second cropping frame is: Offset = [(p - m) / (n - m)] * offset_12.

[0015] According to the first aspect, or any implementation of the above first aspect, the method further includes: the electronic device obtains the fifth image collected by the second camera in response to the first operation. Wherein, the electronic device calculates the first warp matrix and the second cropping frame corresponding to the second image, including: the electronic device determines the FOV area of the second camera in the fifth image according to the first value and the fifth image; the electronic device performs spatial transformation alignment processing according to the FOV area of the second camera in the fifth image and the second image to obtain the first warp matrix and the second cropping frame corresponding to the second image.

[0016] In this way, in the case where both the first camera and the second camera are streaming, the electronic device can calculate the fine FOV area offset required to be implemented in the IPE processing stage based on the image spatial transformation alignment processing.

[0017] According to the first aspect, or any implementation of the above first aspect, the second camera is not started. The electronic device calculates the first warp matrix and the second cropping frame corresponding to the second image, including: the electronic device calculates the total FOV center offset between the first camera and the second camera according to the calibration information of the first camera and the second camera; the electronic device calculates the offset to be offset of the FOV area of the first camera in the second image according to the total FOV center offset and the target offset corresponding to the first cropping frame; the electronic device calculates the first warp matrix and the second cropping frame corresponding to the second image according to the offset to be offset.

[0018] That is, subtracting the offset corresponding to the first cropping frame from the total FOV center offset is the offset to be offset of the FOV area of the first camera in the second image. Among them, the offset corresponding to the first cropping frame is the target offset of the third cropping frame mentioned above.

[0019] In this way, in the case where only the first camera is activated, the electronic device can use the calibration data of the first camera and the second camera to calculate the total offset of the FOV center between the first camera and the second camera, and then calculate the fine FOV region offset required to be achieved in the IPE processing stage.

[0020] According to the first aspect, or any implementation manner of the above first aspect, the method further includes: the electronic device increases the current zoom magnification value of the electronic device to a second value in response to a first operation, and switches the display camera of the electronic device from the first camera to the second camera.

[0021] After the display camera of the electronic device is switched from the first camera to the second camera, the image captured by the second camera is used to generate a preview image. Since the FOV center of the first camera has been aligned with the FOV center of the second camera before the display camera is switched, when the display camera is switched from the first camera to the second camera, the field of view angles of the two consecutive preview images displayed on the electronic device (the previous frame is the preview image generated from the image captured by the first camera sensor, and the subsequent frame is the preview image generated from the image captured by the second camera sensor) do not change significantly, and the preview screen smoothly transitions.

[0022] According to the first aspect, or any implementation manner of the above first aspect, the method further includes: the electronic device reduces the current zoom magnification of the electronic device to a third value in response to a second operation, obtains a sixth image captured by the first camera, and a seventh image captured by the second camera; the seventh image is used to generate a preview image for display; the electronic device determines a fourth cropping frame in the sixth image according to the third value and prior information; wherein, the fourth cropping frame is not centered with the sixth image and is offset in the FOV direction of the second camera in the sixth image; the electronic device controls the first image processing module to crop the sixth image according to the fourth cropping frame to obtain an eighth image; the electronic device determines the FOV region of the second camera in the seventh image according to the third value and the seventh image; the electronic device performs a spatial transformation alignment process according to the FOV region of the second camera in the seventh image and the eighth image to obtain a second warp matrix corresponding to the eighth image and a fifth cropping frame; the electronic device controls the second image processing module to process the eighth image according to the second warp matrix to obtain a ninth image, and performs a cropping operation on the ninth image according to the fifth cropping frame to obtain a tenth image.

[0023] Among them, the fifth cropping frame is centered with the ninth image.

[0024] Exemplarily, the second operation may be a zoom out operation achieved by clicking or sliding a zoom control, or a zoom out operation achieved by sliding two fingers away from each other. Among them, the third value belongs to the transmission display zoom magnification range of the second camera.

[0025] In the case where both the first camera and the second camera are streaming, although the second camera is the transmission display camera, the FOV area of the first camera is still aligned with the FOV area of the second camera in two stages. Among them, the electronic device can calculate the fine FOV area offset required for the IPE processing stage based on the image space transformation alignment processing. In this way, in the shooting preview scenario, the alignment of the multi-camera FOV areas during the switching of the transmission display camera is no longer limited by the size of the margin area reserved when the IFE module crops the image, which can not only ensure the smooth transition of the preview screen during the camera switch, but also will not affect the clarity of the preview screen of the electronic device, thereby improving the user experience.

[0026] According to the first aspect, or any one of the implementation manners of the above first aspect, the method further includes: the electronic device reduces the current zoom magnification of the electronic device to a fourth value in response to the second operation, switches the transmission display camera of the electronic device from the second camera to the first camera, and the tenth image is used to generate the preview image for transmission display.

[0027] After switching the transmission display camera of the electronic device from the second camera to the first camera, the image captured by the first camera is used to generate the preview image. Since the FOV center of the first camera has been aligned with the FOV center of the second camera before the transmission display camera is switched, when the transmission display camera is switched from the second camera to the first camera, the field of view angles of the two consecutive preview images displayed on the electronic device (the previous frame is the preview image generated from the image captured by the second camera sensor, and the subsequent frame is the preview image generated from the image captured by the first camera sensor) will not change much, and the preview screen smoothly transitions.

[0028] According to the first aspect, or any one of the implementation manners of the above first aspect, the electronic device determines a fourth crop frame in the sixth image according to the third value and the prior information, including: the electronic device determines a sixth crop frame in the sixth image according to the FOV area of the first camera corresponding to the third value; among them, the sixth crop frame is aligned with the center of the sixth image; the electronic device calculates the target offset of the sixth crop frame according to the third value and the prior information; the electronic device performs an offset process on the sixth crop frame according to the target offset of the sixth crop frame to obtain the fourth crop frame.

[0029] According to the first aspect, or any one of the implementation manners of the above first aspect, the prior information includes:

[0030] The extrinsic matrix between the first camera and the second camera, and the intrinsic matrices of the first camera and the second camera.

[0031] According to the first aspect, or any implementation of the above first aspect, the first camera is an ultra-wide-angle camera and the second camera is a wide-angle camera; or, the first camera is a wide-angle camera and the second camera is a telephoto camera.

[0032] In a second aspect, an embodiment of the present application provides an electronic device. The electronic device includes: one or more processors; a memory; and one or more computer programs, where one or more computer programs are stored on the memory, and when the computer programs are executed by the one or more processors, the electronic device is caused to execute the image alignment method between multiple cameras according to the first aspect and any one of the first aspects.

[0033] The second aspect and any implementation of the second aspect respectively correspond to the first aspect and any implementation of the first aspect. The technical effects corresponding to the second aspect and any implementation of the second aspect can be seen in the technical effects corresponding to the first aspect and any implementation of the first aspect above, and will not be elaborated here.

[0034] In a third aspect, an embodiment of the present application provides a computer-readable storage medium. The computer-readable storage medium includes a computer program, and when the computer program runs on an electronic device, the electronic device is caused to execute the image alignment method between multiple cameras according to the first aspect and any one of the first aspects.

[0035] The third aspect and any implementation of the third aspect respectively correspond to the first aspect and any implementation of the first aspect. The technical effects corresponding to the third aspect and any implementation of the third aspect can be seen in the technical effects corresponding to the first aspect and any implementation of the first aspect above, and will not be elaborated here.

[0036] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, and when the computer program is run, the computer is caused to execute the image alignment method between multiple cameras as described in the first aspect or any one of the first aspects.

[0037] The fourth aspect and any implementation of the fourth aspect respectively correspond to the first aspect and any implementation of the first aspect. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be seen in the technical effects corresponding to the first aspect and any implementation of the first aspect above, and will not be elaborated here.

[0038] Fifth aspect, the present application provides a chip, which includes a processing circuit and transceiver pins. Among them, the transceiver pins and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the image alignment method between multiple cameras as described in the first aspect or any one of the first aspect to control the receiving pins to receive signals and control the sending pins to send signals.

[0039] The fifth aspect and any implementation manner of the fifth aspect respectively correspond to the first aspect and any implementation manner of the first aspect. For the technical effects corresponding to the fifth aspect and any implementation manner of the fifth aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect above, which will not be elaborated here.

[0040] Sixth aspect, the present application provides a chip system, which is applied to an electronic device. The chip system includes instructions and at least one processor, and the at least one processor runs the instructions to enable the electronic device to execute the image alignment method between multiple cameras as described in the first aspect or any one of the first aspect.

[0041] The sixth aspect and any implementation manner of the sixth aspect respectively correspond to the first aspect and any implementation manner of the first aspect. For the technical effects corresponding to the sixth aspect and any implementation manner of the sixth aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect above, which will not be elaborated here. Description of the Drawings

[0042] Figure 1a Schematic diagram of the focal length of the display camera switching of the electronic device shown exemplarily;

[0043] Figure 1b Schematic diagram of the zoom scene of the electronic device shown exemplarily;

[0044] Figure 2 Situation of the preview screen change when the display camera of the electronic device is switched shown exemplarily;

[0045] Figure 3 Processing flow of the electronic device processing the images collected by the camera to generate a preview image shown exemplarily;

[0046] Figure 4a Schematic diagram of the IPE module in the ISP chip performing an offset cutting operation shown exemplarily;

[0047] Figure 4b Schematic diagram of the selection of the central cutting frame shown exemplarily;

[0048] Figure 4cIt is an exemplary comparison schematic diagram showing the influence of the margin area size on the preview image clarity;

[0049] Figure 5 It is a schematic diagram of the hardware structure of an exemplary electronic device;

[0050] Figure 6 It is a schematic diagram of the software structure of an exemplary electronic device;

[0051] Figure 7 It is an exemplary schematic diagram of module interaction;

[0052] Figure 8 It is an exemplary schematic diagram showing the cropping of the IFE module in the ISP chip based on a non-centered cropping frame;

[0053] Figure 9 It is an exemplary situation of the preview screen change when the electronic device switches the sending display camera;

[0054] Figure 10 It is an exemplary situation of the preview screen change when the electronic device switches the sending display camera. Detailed implementation manners

[0055] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0056] In this text, the term "and / or" only describes the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone.

[0057] The terms "first" and "second" etc. in the description and claims of the embodiments of the present application are used to distinguish different objects, rather than to describe the specific order of the objects. For example, the first target object and the second target object etc. are used to distinguish different target objects, rather than to describe the specific order of the target objects.

[0058] In the embodiments of the present application, words such as "exemplary" or "for example" are used to give examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or having more advantages than other embodiments or design solutions. Exactly, using words such as "exemplary" or "for example" aims to present relevant concepts in a specific way.

[0059] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality" refers to two or more. For example, a plurality of processing units refers to two or more processing units; a plurality of systems refers to two or more systems.

[0060] To meet the shooting needs of users, setting cameras with different focal lengths in the same electronic device has become an essential configuration of the electronic device. Among them, the user can switch the camera displayed by the electronic device (hereinafter referred to as the displayed camera) through a zoom operation to improve the shooting effect of the electronic device.

[0061] In a possible implementation manner, an ultra-wide-angle camera, a wide-angle camera, and a telephoto camera are provided in the electronic device. Among them, the FOV of the ultra-wide-angle camera is greater than the FOV of the wide-angle camera, and the FOV of the wide-angle camera is greater than the FOV of the telephoto camera. The electronic device selects one of the cameras as the displayed camera according to the zoom ratio (or zoom factor, ZoomRatio value, etc.) currently set by the user. Refer to Figure 1a , for example, if the user adjusts the zoom ratio to the zoom ratio range [0.6x, 1.0x), the electronic device selects the ultra-wide-angle camera as the displayed camera; if the user adjusts the zoom ratio to the zoom ratio range [1.0x, 3.5x), the electronic device selects the wide-angle camera as the displayed camera; if the user adjusts the zoom ratio to be greater than or equal to 3.5x (for example, the zoom ratio range [3.5x, 5x]), the electronic device selects the telephoto camera as the displayed camera. The values of the above zoom ratio ranges are only for illustrative purposes, and this embodiment does not limit this.

[0062] That is, in the zoom-in scenario, the user's zoom operation can switch the displayed camera of the electronic device from the ultra-wide-angle camera to the wide-angle camera, or from the wide-angle camera to the telephoto camera to magnify the scene (that is, push the scene closer). In the zoom-out scenario, the user's zoom operation can switch the displayed camera of the electronic device from the telephoto camera to the wide-angle camera, or from the wide-angle camera to the ultra-wide-angle camera to reduce the scene (that is, pull the scene farther away).

[0063] It should be noted that in order to make the switching of the displayed camera of the electronic device relatively smooth, the electronic device usually starts to collect images in advance for the target camera (that is, the switched displayed camera) (abbreviated as starting the target camera). Continue to refer to Figure 1a, in the zoom-in scenario, when the zoom ratio is close to 1.0x, for example, when the zoom ratio is 0.8x, the electronic device will activate the wide-angle camera, that is, the electronic device controls the image sensor (sensor) of the wide-angle camera to expose and capture an image (referred to as taking a picture). When the zoom ratio is close to 3.5x, for example, when the zoom ratio is 3.2x, the electronic device will activate the telephoto camera, that is, the electronic device controls the sensor of the telephoto camera to take a picture. Similarly, in the zoom-out scenario, when the zoom ratio is close to 3.5x, for example, when the zoom ratio is 3.7x, the electronic device will activate the wide-angle camera, that is, the electronic device controls the sensor of the wide-angle camera to take a picture. When the zoom ratio is close to 1.0x, for example, when the zoom ratio is 1.2x, the electronic device will activate the ultra-wide-angle camera, that is, the electronic device controls the sensor of the ultra-wide-angle camera to take a picture.

[0064] The above-mentioned electronic device can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), or a dedicated camera (such as a single-lens reflex camera, a compact camera), etc. The present application does not impose any restrictions on the specific type of the above-mentioned electronic device.

[0065] The following takes a mobile phone as an example of the electronic device for relevant explanation.

[0066] Figure 1b An exemplary application scenario is shown. The mobile phone detects a user operation (such as a touch / click operation) on the camera application icon. In response to this operation, it can display, for example, Figure 1b the shooting interface 10 shown in (1) of. The shooting interface 10 can be the user interface of the default shooting mode of the camera application program. The user can take a picture on this interface. The camera application program is an image shooting application program on the electronic device. The present application does not limit the name of this application program. That is to say, the user can open the shooting interface 10 of the camera application program by clicking the camera application icon. It can be understood that the default camera in the default shooting mode is not limited to the rear camera, and the default camera can also be the front camera.

[0067] As Figure 1bAs shown in (1), the shooting interface 10 may include a parameter adjustment area 101, a preview area 102, a camera mode option area 103, a gallery quick control 105, a camera flip control 104, a shutter control 106, and a zoom control 107. Among them, each control in the parameter adjustment area 101 is used for adjusting corresponding shooting parameters, including but not limited to: a flash setting control, an AI recognition switch setting control, a color standard setting control, and more detailed camera setting controls. The preview area 102 can be used to display a preview image, which is an image captured by the mobile phone in real time through the sending camera. The mobile phone can refresh the display content in the preview area 102 in real time, so that the user can view the image captured by the sending camera in time. One or more shooting mode options may be displayed in the camera mode option 103. The one or more shooting mode options may include but not limited to: an aperture mode option, a night scene mode option, a portrait mode option, a photo shooting mode option, a video recording mode option, a professional option mode, and more options. It can be understood that the one or more shooting mode options may be presented as text information on the interface, such as "aperture", "night scene", "portrait", "photo shooting", "video recording", "professional", "more", and may also be presented as icons or other forms of interactive elements (IE), which is not limited in this application.

[0068] Among them, the zoom control 107 is used to trigger the mobile phone to adjust the zoom ratio, so as to adjust the FOV of the mobile phone shooting preview interface 102. The user can adjust the current zoom ratio of the mobile phone by clicking on the zoom control 107 or sliding the zoom control 107, so as to magnify or reduce the scenery in the shooting preview interface. Optionally, the user can also perform a two-finger relative sliding operation or a two-finger away sliding operation on the preview area 102 to adjust the current zoom ratio of the mobile phone, so as to reduce or magnify the scenery in the shooting preview interface.

[0069] Continue to refer to Figure 1b As shown in (1), taking the zoom operation of the user performing a two-finger away sliding in the preview area 102 as an example. In response to the user performing a two-finger away sliding operation in the preview area 102, the mobile phone increases the zoom ratio and magnifies the scenery in the shooting preview interface, which can be referred to Figure 1b as shown in (2).

[0070] Among them, when the user's zoom operation keeps the zoom ratio of the mobile phone within the zoom ratio range of the same camera, the user's zoom operation will not trigger the electronic device to switch the displayed camera. If the user's zoom operation adjusts the zoom ratio of the mobile phone from the displayed zoom ratio range of one camera to the displayed zoom ratio range of another camera, the user's zoom operation will trigger the electronic device to switch the displayed camera. For example, when the user's zoom operation adjusts the zoom ratio of the mobile phone from 1.0x to 3.5x (which can be referred to Figure 1b as shown), 1.0x belongs to the displayed zoom ratio range of the wide-angle camera, and 3.5x belongs to the displayed zoom ratio range of the telephoto camera, then the user's zoom operation will trigger the electronic device to switch the displayed camera from the wide-angle camera to the telephoto camera.

[0071] However, when the displayed camera of the electronic device switches, that is, when the image sensor for the preview images of two consecutive frames switches, due to the differences in the position settings of different cameras in the electronic device, the shooting preview screen of the electronic device will jump. Figure 2 Exemplarily shows the situation where the shooting preview screen jumps when the displayed camera of the mobile phone switches. Among them, Figure 2 in (1) is the shooting preview screen before the displayed camera switches, Figure 2 in (2) is the shooting preview screen after the displayed camera switches. By comparing Figure 2 in (1) and (2), it can be seen that when the displayed camera of the electronic device switches, the field of view angle of the preview image changes, that is, the shooting object shifts in the preview frame, resulting in a relatively large jump in the shooting preview screen that can be visually perceived by the user. Obviously, the switching of the shooting preview screen is rather abrupt, and the closer the shooting distance, the more abrupt the screen switching, and the more obvious the jump in the screen visually perceived by the user.

[0072] In order to improve the user's visual experience and make the switching of the shooting preview screen less abrupt, the electronic device usually aligns the center point of the field of view angle during the multi-camera switch, that is, performs smooth processing on the switching of the shooting preview screen. Among them, the electronic device can achieve smooth processing of the switching of the shooting preview screen according to the FOV offset based on the Spatial Alignment Transform (SAT). However, the FOV offset processing based on SAT needs to rely on the edge (margin) area of the image, and only when the margin area is large enough can the electronic device complete the FOV offset processing based on SAT.

[0073] The following takes the example of the display camera of an electronic device about to switch from a wide-angle camera to a telephoto camera in the zoom-in scenario for explanation. Assume that at the current zoom ratio, both the wide-angle camera sensor and the telephoto camera sensor of the electronic device output images, and the wide-angle camera is the display camera.

[0074] As Figure 3 shown, the processing flow of the ISP (Image Signal Processor) chip in the electronic device for the image output by the wide-angle camera sensor can be roughly divided into the processing stage of the IFE (Image Singnal Processing FrontEnd) module and the processing stage of the IPE (Image Singnal Processing Post End) module.

[0075] The SAT module calculates a central crop (or cut) frame corresponding to the image collected by the wide-angle camera sensor according to the current zoom ratio, and sends the central crop frame to the IFE module. Among them, the central crop frame can be understood as a crop frame aligned with the center of the image collected by the wide-angle camera sensor. Refer to Figure 3 , in the processing stage of the IFE module, the IFE module crops the image collected by the wide-angle camera sensor according to the central crop frame to obtain an IFE cropped image. In the IFE cropped image, in addition to including the image area corresponding to the wide-angle camera display FOV, there is also an edge area (for example, the margin width from the upper boundary of the image area corresponding to the wide-angle camera display FOV to the upper boundary of the IFE cropped image is Margin1, and the margin width from the left boundary of the image area corresponding to the wide-angle camera display FOV to the left boundary of the IFE cropped image is Margin2). Among them, this edge area can prevent black areas (i.e., image-free areas) from appearing due to warp processing of the image in the processing stage of the IPE module. After cropping to obtain the IFE cropped image, the IFE module also performs downsampling processing on the IFE cropped image to obtain a downsampled image, so that the image size of the downsampled image conforms to the preset size of the display.

[0076] The SAT module can also determine the telephoto camera display FOV area in the image collected by the telephoto camera sensor according to the current zoom ratio, perform spatial alignment transformation processing on the telephoto camera display FOV image area and the wide-angle camera display FOV image area, calculate a warp matrix corresponding to the downsampled image, and a crop frame aligned with the image center, and send the warp matrix and the crop frame to the IPE module. Continue to refer to Figure 3, during the IPE module processing stage, the IPE module processes the downsampled image output by the IFE module, such as offsetting and rotating it according to the warp matrix, to obtain the offset and rotated image (i.e., the downsampled image). In the offset and rotated downsampled image, the FOV area sent by the telephoto camera (i.e., the sending area aligned with the center of the FOV sent by the telephoto camera) is located in the central area of the image. Subsequently, the IPE module can crop the offset and rotated image according to the central crop box output by the SAT module to obtain the IPE cropped image. Among them, the crop box output by the SAT module is centered with the offset and rotated image. After the IPE module completes the cropping operation, it can upsample the IPE cropped image to obtain an upsampled image that conforms to the preset size (i.e., the sending size), that is, obtain the sending preview image collected by the wide-angle camera.

[0077] It should be noted that the SAT module may include an ROI (Region of Interest) Translator module and a SAT algorithm module. Among them, the ROI translation module can be used to calculate the central crop box corresponding to the image collected by the sensor according to the current zoom ratio, and the SAT algorithm module can be used to calculate the FOV center offset amount between the cameras based on the SAT algorithm. The division of the functional modules of the SAT module in the ISP chip in this embodiment is not limited.

[0078] It should be noted that if only the sensor of the wide-angle camera outputs an image and the sensor of the telephoto camera does not start streaming at the current zoom ratio, the SAT module can calculate the warp matrix and the crop box according to the calibration data of the telephoto camera and the wide-angle camera (such as the distance between the cameras, the internal parameter matrix, the external parameter matrix, etc.).

[0079] Continue to refer to Figure 4a , the IPE module performs operations such as offsetting and rotating the downsampled image according to the warp matrix calculated by the SAT module to make the FOV area sent by the telephoto camera located in the center of the image. Furthermore, the IPE module can crop the FOV area sent by the telephoto camera in the downsampled image according to the central crop box calculated by the SAT module to obtain the IPE cropped image. It can be seen from this that during the IPE module processing stage, the offset operation of the FOV area to be sent depends on the margin area in the IFE cropped image. In the scenario of close-up shooting, the FOV center offset amount between multiple cameras is generally very large. In this way, if we want to ensure that the FOV centers can be aligned and the preview screen can be smoothly transitioned when switching between multiple cameras, we need to increase the margin area in the IFE cropped image so that the IPE module can successfully complete the offset and crop operations and avoid the existence of black edge areas (i.e., image-free areas) in the IPE cropped image.

[0080] Continue to refer to Figure 4b As shown, in the image collected by the sensor of the wide-angle camera, the display FOV area of the wide-angle camera is the image area that actually needs to be displayed and previewed. The display FOV area of the telephoto camera is the image area that actually needs to be displayed and previewed after the display camera is switched to the telephoto camera. In order to make the picture transition smoothly when the camera is switched, the electronic device needs to perform spatial transformation alignment processing on the display FOV image area of the wide-angle camera and the display FOV image area of the telephoto camera. In this way, the IFE cropped image must increase the margin area to ensure that the IFE cropped image can cover both the display FOV image area of the wide-angle camera and the display FOV image area of the telephoto camera. In the scenario of close-range shooting, the FOV center offset between the wide-angle camera and the telephoto camera is relatively large, which will inevitably lead to a relatively large margin area in the IFE cropped image, that is Figure 4b the sizes of Margin1 and Margin2 in

[0081] However, the increase in the margin area in the IFE cropped image will inevitably reduce the clarity of the preview image of the electronic device. The following explains the reason why the size of the margin area in the IFE cropped image affects the clarity of the preview image.

[0082] Figure 4c An exemplary comparison diagram showing the influence of the margin area size on the clarity of the preview image is shown. Refer to Figure 4cAs shown, the IFE cropped image 201 is an image with a larger margin area cropped by the IFE module based on the central cropping frame, and the IFE cropped image 202 is an image with a smaller margin area cropped by the IFE module based on the central cropping frame. Exemplarily, Margin_1 is greater than Margin_3, and Margin_2 is greater than Margin_4. The IFE module downsamples the IFE cropped image 201 to a preset size (i.e., the display image size) to obtain the downsampled image 202. The IFE module downsamples the IFE cropped image 301 to a preset size to obtain the downsampled image 302. Among them, the image sizes of the downsampled image 202 and the downsampled image 302 are the same. Obviously, since the size of the IFE cropped image 201 is larger than that of the IFE cropped image 301, the downsampling ratio of the IFE cropped image 201 is greater than that of the IFE cropped image 301. By comparing the downsampled image 202 and the downsampled image 302, it can be seen that the proportion of the display FOV area of the wide-angle camera in the downsampled image 202 is larger, and the proportion of the display FOV area of the wide-angle camera in the downsampled image 302 is smaller. The IPE module performs offset rotation processing on the downsampled image 202 according to the warp matrix and the cropping frame calculated by the SAT module, and then crops it to obtain the IPE cropped image 203 corresponding to the display FOV area of the telephoto camera. The IPE module performs offset rotation processing on the downsampled image 302 according to the warp matrix and the cropping frame calculated by the SAT module, and then crops it to obtain the IPE cropped image 303 corresponding to the display FOV area of the telephoto camera. By comparison, the image size of the IPE cropped image 303 is larger than that of the IPE cropped image 203. Subsequently, the IPE module can perform upsampling processing on the IPE cropped image 203 to obtain the upsampled image 204, which is also the display image 1 of the wide-angle camera. Similarly, the IPE module can perform upsampling processing on the IPE cropped image 303 to obtain the upsampled image 304, which is also the display image 2 of the wide-angle camera. Since the sizes of the upsampled image 204 and the upsampled image 304 are the same, the upsampling ratio corresponding to the IPE cropped image 203 must be greater than the upsampling ratio corresponding to the IPE cropped image 303. Among them, the larger the upsampling ratio, the lower the clarity of the obtained display image. That is, the image clarity of the display image 1 of the wide-angle camera is lower than that of the display image 2 of the wide-angle camera.

[0083] An embodiment of the present application provides an image alignment method between multiple cameras. In this method, the electronic device splits the offset of the display FOV area during the IPE module processing stage into two parts. First, the IFE module uses a non-centered cropping frame with a rough offset to complete the cropping operation when cropping the image collected by the sensor. Then, the IPE module finely offsets the display FOV area before cropping the downsampled image to complete the alignment of the display FOV area images between multiple cameras.

[0084] In this way, in the shooting preview scenario, the alignment of the display FOV area images during camera switching is no longer restricted by the size of the margin area in the image cropped by the IFE, that is, the smooth transition of the preview screen during camera switching is no longer restricted by the size of the margin area in the image cropped by the IFE. This image alignment method can not only ensure a smooth transition of the preview screen during camera switching, but also has no impact on the clarity of the preview screen of the electronic device, thereby improving the user experience.

[0085] Figure 5 The schematic structural diagram of the electronic device 100 is shown. It should be understood that Figure 5 The shown electronic device 100 is only an example of an electronic device, and the electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. Figure 5 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0086] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0087] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP) (or ISP chip), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0088] Among them, the controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0089] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0090] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0091] The USB interface 130 is an interface that complies with the USB standard specification. Specifically, it can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as AR devices, etc.

[0092] It can be understood that the interface connection relationships between the modules illustrated in the embodiments of the present application are only illustrative descriptions and do not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0093] The charging management module 140 is used to receive a charging input from a charger. Among them, the charger can be a wireless charger or a wired charger. As Figure 2 shown, in some embodiments of wired charging, the charging management module 140 can receive the charging input of the wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 can receive the wireless charging input through the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141.

[0094] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives the inputs of the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as the battery capacity, the number of battery cycles, and the battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be disposed in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0095] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0096] Antenna 1 and Antenna 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, Antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0097] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc. applied to the electronic device 100. The mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves through Antenna 1, filter, amplify, and perform other processing on the received electromagnetic waves, and then transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves through Antenna 1 and radiate it out. In some embodiments, at least some functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 can be disposed in the same device.

[0098] The wireless communication module 160 can provide solutions for wireless communications including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves through Antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signals to be transmitted from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through Antenna 2 and radiate them out.

[0099] In some embodiments, Antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and Antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technologies.

[0100] The electronic device 100 implements the display function through the GPU, the display screen 194, the application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to execute mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0101] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel.

[0102] The electronic device 100 can implement the shooting function through the ISP chip, the camera 193, the video codec, the GPU, the display screen 194, the application processor, etc.

[0103] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light passes through the lens and is transmitted to the camera photosensitive element. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also optimize the noise, brightness, and skin color of the image through algorithms. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0104] The camera 193 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV, etc. formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0105] In one implementation, the electronic device 100 may include three cameras, namely an ultra-wide-angle camera, a wide-angle camera, and a telephoto camera, and an image sensor (sensor) is separately provided in each camera. Exemplarily, the display zoom magnification range of the ultra-wide-angle camera is [0.6, 1.0), the display zoom magnification range of the wide-angle camera is [1.0, 3.5), and the display zoom magnification range of the telephoto camera is [3.5, 5].

[0106] In some embodiments, the camera can be a TOF (Time of Flight) camera (or lens). The TOF camera is used to collect TOF data. In some implementation manners, the TOF camera can include a TOF sensor, a TOF sensor controller, a TOF light source, and a TOF light source controller.

[0107] In some embodiments, the TOF light source controller is controlled by the TOF sensor controller to control the TOF light source. Under the control of the TOF light source controller, the TOF light source emits infrared (IR) light. The TOF sensor is used to sense the light reflected by the infrared light on an object (such as a human face, etc.) to collect TOF data. The TOF sensor controller and the TOF light source controller can communicate with the processor 110. Among them, the processor 110 can also be used to generate a TOF image according to the TOF data collected by the TOF camera, including an infrared image and a depth image.

[0108] The digital signal processor is used to process digital signals. In addition to being able to process digital image signals, it can also process other digital signals. For example, when the electronic device 100 is selecting a frequency, the digital signal processor is used to perform Fourier transform on the frequency energy, etc.

[0109] The video codec is used to compress or decompress digital video. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0110] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external memory card.

[0111] The internal memory 121 can be used to store computer-executable program code, and the executable program code includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.). In addition, the internal memory 121 can include a high-speed random access memory and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0112] The electronic device 100 can implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, and an application processor, etc. For example, music playback, recording, etc.

[0113] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some functional modules of the audio module 170 can be disposed in the processor 110.

[0114] The pressure sensor is used to sense a pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor can be disposed on the display screen 194. There are many types of pressure sensors, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, etc. The capacitive pressure sensor can include at least two parallel plates having a conductive material. When a force acts on the pressure sensor, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor. In some embodiments, touch operations with the same touch position but different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than a first pressure threshold acts on the short message application icon, the instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

[0115] The acceleration sensor can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0116] The touch sensor, also known as the "touch panel". The touch sensor can be disposed on the display screen 194, and the touch sensor and the display screen 194 form a touch screen, also known as the "touch screen". The touch sensor is used to detect touch operations acting thereon or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor can also be disposed on the surface of the electronic device 100, at a different position from the display screen 194.

[0117] The button 190 includes a power-on button, a volume button, etc. The button 190 can be a mechanical button. It can also be a touch button. The electronic device 100 can receive button inputs and generate key signal inputs related to the user settings and function control of the electronic device 100.

[0118] The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts and can also be used for touch vibration feedback.

[0119] The indicator 192 can be an indicator light and can be used to indicate the charging state, power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0120] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the embodiments of this application, taking the Android system with a layered architecture as an example, the software structure of the electronic device 100 is exemplarily described.

[0121] Figure 6 It is the software structure block diagram of the electronic device 100 in the embodiments of this application.

[0122] The layered architecture of the electronic device 100 divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom are the application layer, the application framework layer, the hardware abstraction layer (HAL), and the kernel (Kernel) layer (also known as the driver layer).

[0123] It can be understood that Figure 6For example, between the application framework layer and the HAL layer, there may also be included layers such as the Android runtime and the system libraries. Among them, the Android Runtime includes core libraries and a virtual machine, and is responsible for the scheduling and management of the Android system. The system libraries can include multiple functional modules. For example: the surface manager, Media Libraries, 3D graphics processing libraries (such as OpenGL ES), 2D graphics engines (such as SGL), etc.

[0124] The application layer can include a series of application packages. Such as Figure 6 As shown, the application packages can include application programs such as camera applications, galleries, applications with camera functions, etc. The application packages can also include applications such as calls, calendars, maps, navigation, music, videos, text messages, etc.

[0125] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the application programs in the application layer. The application framework layer includes some predefined functions.

[0126] Such as Figure 6 As shown, the application framework layer can include a camera service, which can be called by the camera application to implement functions related to shooting. In addition, the application framework layer can also include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc.

[0127] Among them, the window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0128] The content provider is used to store and obtain data, and make these data accessible to application programs. The data can include videos, images, audio, dialed and answered calls, browsing history and bookmarks, phone books, etc.

[0129] The view system includes visible controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build application programs. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon can include a view for displaying text and a view for displaying pictures.

[0130] The phone manager is used to provide the communication function of the electronic device 100. For example, the management of call states (including answering, hanging up, etc.).

[0131] The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.

[0132] The notification manager enables the application to display notification information in the status bar. It can be used to convey notification-type messages, which can automatically disappear after a short stay without user interaction. For example, notification information is used to inform that the download is complete, message reminders, etc. Notification information can also be notifications that appear in the system top status bar in the form of charts or scrollbar text, such as notifications of background-running applications, and can also be notifications that appear in the form of a dialog window on the screen. Notification information can also be, for example, text information prompted in the status bar, emitted prompt sounds, vibrations of the electronic device, flashing of the indicator light, etc.

[0133] It should be noted that the camera application can also call the content provider, resource manager, notification manager, window manager, view system, etc. according to actual business needs, and the embodiments of this application do not make any restrictions on this.

[0134] The kernel layer is the layer between hardware and software. The kernel layer at least includes a camera driver and an ISP driver. The camera driver can be used to drive the hardware module with a shooting function, such as the image sensor (Camera sensor) in the camera. In other words, the above camera driver is responsible for data interaction with the camera. The ISP driver can be used to drive the ISP chip, and specifically can be used to send control instructions to the ISP chip, or transmit image data, etc. For example, the ISP driver can control the IFE module and IPE module in the ISP chip to perform relevant image processing, etc. Of course, the kernel layer can also include a display driver, an audio driver, a sensor driver, etc., and the embodiments of this application do not make any restrictions on this.

[0135] In addition, the HAL layer can encapsulate the driver programs in the kernel layer and provide an interface for the application framework layer to call, shielding the implementation details of the lower-level hardware. As Figure 6 shown, the above HAL layer can include Camera HAL.

[0136] Camera HAL is the Camera core software framework, and the Camera HAL can include an interface module, a Sensornode, an ROI Translator, a SAT algorithm module, a multi-camera decision module, etc.

[0137] Among them, the Sensor node and the interface module are components in the transmission pipeline of image data and control instructions in Camera HAL, and different components also correspond to different functions. For example, the Sensor node can be a control node for the camera sensor, and this Sensor node can control the camera sensor through the camera driver. Another example is that the interface module can be a software interface for the application framework layer, used for data interaction with the application framework layer. Exemplarily, the interface module can also perform data interaction with other modules in Camera HAL (such as the multi-camera decision-making module, Sensor node, ROI translator, SAT algorithm module).

[0138] The multi-camera decision-making module can determine the camera sensor for display and image output (or called starting the stream) according to the application scenario, such as the sensor of the front camera or the sensor of the rear camera, or the sensor of the ultra-wide-angle camera, wide-angle camera, and telephoto camera in the rear camera. Among them, the camera application can transmit information such as the camera mode and zoom parameter selected by the user to the camera service in the application framework layer, and then the camera service transmits it to the multi-camera decision-making module through the interface module in the HAL layer.

[0139] Among them, the multi-camera decision-making module can also estimate the target camera to be switched according to the user's zoom operation, and turn on the camera sensor of the target camera in advance through the camera driver. In addition, the multi-camera decision-making module can also set the camera for display among multiple cameras with the camera sensor turned on.

[0140] In the embodiment of this application, the ROI translator can be used to calculate the IFE cropping frame (abbreviated as non-centered cropping frame) that is not centered with the image collected by the sensor. Specifically, the ROI translator can calculate the IFE cropping frame centered with the image collected by the sensor (abbreviated as centered cropping frame) according to the zoom ratio, calculate the rough offset between multiple cameras according to the prior information, and calculate the offset corresponding to the centered cropping frame according to the zoom ratio, and then apply this offset to the centered cropping frame to obtain the non-centered cropping frame.

[0141] In the embodiment of this application, the SAT algorithm module can be used to perform spatial alignment on the image data according to the zoom ratio and the preset SAT algorithm, and calculate the warp matrix and the cropping frame for use by the IPE module in the ISP chip.

[0142] It can be understood that Figure 6The layers in the shown software structure and the components included in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, or combine certain components, or split certain components, or have different component arrangements, which are not limited in the present application.

[0143] It can be understood that, in order to implement the image alignment method between multiple cameras in the embodiments of the present application, the electronic device includes corresponding hardware and / or software modules for performing various functions. Combining the algorithm steps of each example described in the embodiments disclosed in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described function for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of the present application.

[0144] Taking the zoomin scenario as an example below, the image alignment method between multiple cameras provided in the embodiments of the present application will be explained. In this zoomin scenario, the display camera of the electronic device is a wide-angle camera, and the sensor of the telephoto camera has been activated to start collecting images.

[0145] As Figure 7 shown is the interaction schematic diagram of each module. Referring to Figure 7 , the process of the image alignment method between multiple cameras provided in the embodiments of the present application specifically includes:

[0146] S401, in response to the zoom operation performed by the user in the shooting preview interface, the camera application sends the zoom ratio value to the camera service.

[0147] Among them, the shooting preview interface may be a photo shooting preview interface or a video shooting preview interface, which is not limited in this embodiment. In the zoomin scenario, the zoom operation is an operation to increase the zoom ratio. Exemplarily, the zoom operation may be an operation where the user clicks or slides the zoom control, or an operation of spreading two fingers apart, etc., which is not limited in this embodiment. Correspondingly, in the zoom out scenario, the zoom operation is an operation to decrease the zoom ratio.

[0148] When the user performs a zoom operation, the touch module of the mobile phone generates a touch reporting point operation in response to the user's zoom operation and sends it to the camera application. After receiving the touch reporting point event, the camera application converts the reporting point coordinate data corresponding to the touch reporting point event into a zoom ratio (ZoomRatio) value and sends it to the camera service in the application framework layer.

[0149] S402, the camera service sends a preview frame request to the interface module in the Cameral HAL.

[0150] The camera service writes the received zoom ratio value into the preview frame request (or preview image frame request), and sends the preview frame request to the interface module in the Cameral HAL.

[0151] S403, the interface module passes the preview frame request to the sensor node, and the sensor node sends an image output instruction to the camera driver based on the preview frame request.

[0152] Exemplarily, the image output instruction (or image acquisition instruction) may include, but is not limited to, the camera sensor identifier for exposure image output. In this scenario, the image output instruction is used to instruct the wide-angle camera sensor and the telephoto camera sensor to output images simultaneously. Among them, the image collected by the wide-angle camera sensor is the image that needs to be sent for display.

[0153] It should be noted that due to the rotation buffer mechanism between the application framework (Framework) layer and the HAL layer in the Android architecture camera system. When the user turns on the camera application, the camera service will first continuously send multiple (for example, 8) preview frame requests to the Cameral HAL, and then send a preview image frame request to the Cameral HAL after receiving a preview image frame feedback from the Cameral HAL each time. In the Cameral HAL, the Sensor node will sequentially control the camera sensor to perform exposure image output operations based on these multiple preview frame requests. That is to say, for any preview frame request, after all the preview frame requests before it have been processed by the Sensor node for exposure image output, the Sensor node will control the camera sensor to perform exposure image output operations for this preview frame request.

[0154] S404, the camera driver drives the wide-angle camera sensor to output an image by exposure, and obtains the image collected by the wide-angle camera sensor.

[0155] S405, the camera driver drives the telephoto camera sensor to output an image by exposure, and obtains the image collected by the telephoto camera sensor.

[0156] This embodiment does not limit the sequence of S404 and S405.

[0157] S406, the camera driver sends the image collected by the wide-angle camera sensor to the IFE module in the ISP chip through the ISP driver.

[0158] S407, the camera driver sends the image captured by the telephoto camera sensor to the SAT algorithm module in the Cameral HAL.

[0159] This embodiment does not limit the sequence of S406 and S407.

[0160] S408, the interface module in the Cameral HAL sends the zoom ratio value to the ROI translator.

[0161] S409, the interface module in the Cameral HAL sends the zoom ratio value to the SAT algorithm module.

[0162] This embodiment does not limit the sequence of S408 and S409. At the same time, this embodiment also does not limit the sequence of S403 with S408 and S409.

[0163] S410, the ROI translator in the Cameral HAL calculates the central cropping frame according to the current zoom ratio value.

[0164] Among them, the central cropping frame is centered with the image captured by the wide-angle camera sensor.

[0165] In this embodiment, the central cropping frame is determined based on the display FOV of the wide-angle camera and does not need to include the display FOV of the wide-angle camera. That is to say, when cropping the image captured by the wide-angle camera sensor according to the central cropping frame, the margin area is relatively small.

[0166] Exemplarily, as Figure 8 shown in (1) below, the ROI translator in the Cameral HAL calculates the central cropping frame corresponding to the display FOV of the wide-angle camera according to the current zoom ratio value. In the image area corresponding to this central cropping frame, the margin area is relatively small, that is, the sizes of Margin1 and Margin2 are small.

[0167] S411, the ROI translator in the Cameral HAL calculates the rough FOV center offset according to the prior information and calculates the offset of this central cropping frame according to the current zoom ratio value.

[0168] In the zoom-in scenario, assuming that the user's zoom operation drives the display camera of the electronic device to switch from Camera 1 to Camera 2, the ROI translator in the Cameral HAL needs to calculate the rough FOV center offset between Camera 1 and Camera 2 based on prior information. Subsequently, the ROI translator calculates the offset of the central cropping frame according to the current zoom magnification value and the rough FOV center offset between Camera 1 and Camera 2.

[0169] Assume that the rough FOV center offset between Camera 1 and Camera 2 is offset_12, the display zoom magnification range of Camera 1 is [m x, n x), and the current zoom magnification value is p x, where m < p < n. Then the offset of the central cropping frame is: Offset = [(p - m) / (n - m)] * offset_12.

[0170] In the zoom-in scenario described in this process, since the user's zoom operation drives the display camera of the electronic device to switch from the wide-angle camera to the telephoto camera, the ROI translator in the Cameral HAL needs to calculate the rough FOV center offset between the wide-angle camera and the telephoto camera based on prior information. Subsequently, the ROI translator can calculate the offset of the central cropping frame according to the current zoom magnification value and the rough FOV center offset between the wide-angle camera and the telephoto camera.

[0171] Exemplarily, the rough FOV center offset between the wide-angle camera and the telephoto camera is offset_wt, the display zoom magnification range of the wide-angle camera is [1.0x, 3.5x), and the current zoom magnification value is 3.3x. Then the offset of the central cropping frame is: Offset = [(3.3 - 1.0) / (3.5 - 1.0)] * offset_wt.

[0172] It should be noted that the rough FOV center offset between the cameras is a two-dimensional vector, so the offset of the central cropping frame is a two-dimensional vector.

[0173] As an alternative implementation, the ROI translator can calculate the rough FOV center offset between Camera 1 and Camera 2 based on the calibration data of Camera 1 and Camera 2. Among them, the focal length of Camera 1 is less than that of Camera 2. For example, the ROI translator can calculate the rough FOV center offset between Camera 1 and Camera 2 according to the extrinsic matrix [R, t] between Camera 1 and Camera 2, the intrinsic matrix K1 of Camera 1, and the intrinsic matrix K2 of Camera 2. Among them, R is the rotation matrix and t is the translation vector.

[0174] Assume that a point p2 is selected in the image captured by camera 2, and the point p2 corresponds to a three-dimensional point P in the world coordinate system. The projection point of the three-dimensional point P in the image captured by camera 1 is p1. Then, the rough FOV center offset between camera 1 and camera 2 is:

[0175] offset_12 = p1 - p2.

[0176] where p1 = K1 * [R, t] * Z * K2 -1 * p2, and Z is the shooting distance, which can be obtained through external TOF or AF (Automatic Focus) data.

[0177] where the internal parameter matrix of the camera cx and cy are the coordinates of the origin of the image coordinate system in the pixel coordinate system, fx = f / dx, fy = f / dy, f is the focal length of the camera, dx is the image length of a single pixel in the x direction, and dy is the image length of a single pixel in the x direction.

[0178] S412. The ROI translator in Cameral HAL offsets the central cropping frame according to the offset of the central cropping frame to obtain a non-central cropping frame, and sends the non-central cropping frame to the IFE module in the ISP chip.

[0179] Exemplarily, as Figure 8 shown in (2) below, applying the offset of the central cropping frame to the central cropping frame can obtain the corresponding non-central cropping frame. Among them, "non-central" in the non-central cropping frame means that the cropping frame is not centrally aligned with the image collected by the wide-angle camera sensor.

[0180] Exemplarily, the ROI translator in Cameral HAL can send the non-central cropping frame to the IFE module in the ISP chip through the ISP driver.

[0181] S413. The IFE module in the ISP chip crops the image collected by the wide-angle camera sensor according to the non-central cropping frame to obtain an IFE cropped image, and sends the IFE cropped image to the SAT algorithm module in Cameral HAL.

[0182] Continue to refer to Figure 8 shown in (3) below. The IFE module crops the image output by the wide-angle sensor according to the non-central cropping frame to obtain an IFE cropped image.

[0183] Exemplarily, the IFE module can send the IFE cropped image to the SAT algorithm module in Cameral HAL through the ISP driver.

[0184] In this way, the IFE module crops the image collected by the wide-angle camera sensor based on the non-centered cropping frame, achieving a preliminary alignment between the center point of the FOV of the wide-angle camera and the center point of the FOV of the telephoto camera. Moreover, in the IFE cropped image obtained by the IFE module, the margin area is relatively small, and the clarity of the preview image sent for display will not be lost.

[0185] S414, the IFE module in the ISP chip downsamples the IFE cropped image and sends the downsampled image to the IPE module.

[0186] Exemplarily, the IFE module calculates the downsampling ratio according to the size of the IFE cropped image and the preset size of the image sent for display, and downsamples the IFE cropped image according to this downsampling ratio to obtain the downsampled image.

[0187] S415, the SAT algorithm module in Cameral HAL calculates the warp matrix and the centered cropping frame according to the IFE cropped image, the current zoom magnification value, and the image collected by the telephoto camera sensor, and sends the warp matrix and the centered cropping frame to the IPE module in the ISP chip.

[0188] The SAT algorithm module can determine the FOV region image of the telephoto camera in the image collected by the telephoto camera sensor according to the current zoom magnification value, and based on the preset SAT algorithm, perform a spatial transformation alignment process on the IFE cropped image and the FOV region image of the telephoto camera to obtain the FOV center offset corresponding to the IFE cropped image, and then generate the warp matrix corresponding to the downsampled image and the centered cropping frame corresponding to the downsampled image according to this FOV center offset. Among them, the area covered by the centered cropping frame is the FOV region sent for display.

[0189] Among them, regarding the calculation method of the warp matrix and the centered cropping frame corresponding to the downsampled IFE cropped image, reference can be made to the existing technology and will not be elaborated here.

[0190] Exemplarily, the SAT algorithm module can send the warp matrix and the centered cropping frame corresponding to the downsampled IFE cropped image to the IPE module in the ISP chip through the ISP driver.

[0191] This embodiment does not limit the sequence of S414 and S415.

[0192] S416, the IPE module in the ISP chip performs an offset rotation process on the downsampled image according to the warp matrix, and crops the downsampled image after the offset rotation process according to the centered cropping frame to obtain the IPE cropped image.

[0193] Since the SAT algorithm module calculates the accurate offset between the FOV centers of the wide-angle camera and the telephoto camera, and generates a warp matrix based on this accurate offset, the IPE module performs offset and rotation processing on the downsampled image according to the warp matrix, enabling the FOV area of the telephoto camera to be located at the center of the downsampled image. In this way, when the IPE module crops the offset and rotated image according to the center cropping frame calculated by the SAT algorithm module, a cropped image aligned with the FOV area of the telephoto camera can be obtained, ensuring a smooth transition of the preview screen when the display camera switches from the wide-angle camera to the telephoto camera.

[0194] S417, the IPE module in the ISP chip upsamples the IPE cropped image and sends the upsampled image as the preview image for display.

[0195] Exemplarily, the IPE module calculates the upsampling ratio according to the size of the IPE cropped image and the preset size of the display image, and performs upsampling processing on the IPE cropped image according to this upsampling ratio to obtain the upsampled image. Subsequently, this upsampled image can be sent for display as the preview image.

[0196] Exemplarily, the IPE module sends the upsampled image (i.e., the preview image) to the Cameral HAL through the ISP driver. The Cameral HAL sends this preview image as the feedback corresponding to the preview frame request to the camera service, and then the camera service sends this preview image to the camera application for display. For the parts not explained in detail in this process, reference can be made to the existing technology and will not be elaborated here.

[0197] In addition, it should be noted that the processing flow of the image collected by the camera sensor may also include format conversion, noise removal, white balance, etc. For the parts not explained in detail in this process, reference can be made to the existing technology.

[0198] In this scenario, when the user adjusts the zoom magnification value of the electronic device to 3.5x, the display camera of the electronic device switches from the wide-angle camera to the telephoto camera. At this time, the electronic device displays the preview image generated from the image collected by the telephoto camera sensor. Since before this, the FOV center of the wide-angle camera has been aligned with the FOV center of the telephoto camera, when the display camera switches from the wide-angle camera to the telephoto camera, the field of view angles of the two consecutive preview images displayed on the electronic device (the previous frame is the preview image generated from the image collected by the wide-angle camera sensor, and the subsequent frame is the preview image generated from the image collected by the telephoto camera sensor) do not change significantly, and the preview screen smoothly transitions.

[0199] It should be noted that after the user adjusts the zoom magnification value of the electronic device to 3.5x, the display camera of the electronic device is a telephoto camera, and no other cameras with a focal length greater than that of the telephoto camera are set in the electronic device. Therefore, the FOV center of the telephoto camera does not need to be aligned any further.

[0200] In the above process, the case where both the wide-angle camera sensor and the telephoto camera sensor are powered on is used as an example for explanation. In some implementation manners, the multi-camera decision module controls both camera sensors to be powered on only when the display camera may switch, and when the difference between the current zoom magnification value and the zoom magnification value at which the display camera switches is large, the multi-camera decision module controls only one camera sensor to be powered on. For example, if the current zoom magnification is 3.3x, the multi-camera decision module controls the telephoto camera sensor to be powered on. If the current zoom magnification is 1.2x, the multi-camera decision module does not control the telephoto camera sensor to be powered on, and at this time, only the wide-angle camera sensor captures images.

[0201] During the process of the user zooming, in order to ensure the consistency of the preview screen change, when the telephoto camera sensor is not powered on (zoomin scenario), the electronic device also aligns the FOV center of the wide-angle camera to the FOV center of the telephoto camera. The difference is that since the telephoto camera sensor is not powered on, the SAT algorithm module in the Camera HAL can calculate the total FOV center offset between the wide-angle camera and the telephoto camera based on the calibration data of the wide-angle camera and the telephoto camera (such as the distance between the cameras, the internal parameter matrix, the external parameter matrix, etc.). Since the FOV area of the wide-angle camera has been offset by a part during the IFE module processing stage, the SAT algorithm module can calculate the offset that the FOV area of the wide-angle camera needs to achieve during the IPE module processing stage based on the total FOV center offset and the offset that has occurred during the IFE module processing stage, and then generate a corresponding warp matrix according to the offset that needs to be achieved during the IPE module processing stage. In this way, during the IPE module processing stage, the IPE module continues to process the downsampled image set according to the calculation result of the SAT algorithm module, so that the FOV area of the wide-angle camera continues to be offset in the image to obtain the FOV area to be displayed. Other processes are similar to those described above and will not be elaborated here.

[0202] Regarding the case where the transmission display camera switches from an ultra-wide-angle camera to a wide-angle camera in the zoom-in scenario, it is similar. In this case, it is necessary to align the FOV center of the ultra-wide-angle camera with the FOV center of the wide-angle camera. First, it is a rough alignment during the processing stage of the IFE module of the ISP chip, and then it is a precise alignment during the processing stage of the IPE module of the ISP chip. For the parts not explained in detail here, reference can be made to the case where the transmission display camera switches from a wide-angle camera to a telephoto camera in the previous text, and details will not be elaborated here.

[0203] After the mobile phone adopts the image alignment method between multiple cameras provided in this embodiment, Figure 9 Exemplarily shows the change situation of the shooting preview screen when the transmission display camera of the mobile phone switches. Among them, Figure 9 In (1), it is the shooting preview screen before the transmission display camera switches, Figure 9 In (2), it is the shooting preview screen after the transmission display camera switches. By comparing Figure 9 (1) and (2) in the figure, it can be seen that when the transmission display camera of the electronic device switches, the change in the field of view angle of the preview image is very small, that is, the offset of the shooting object in the preview frame is very small, and users can hardly perceive the jump of the shooting preview screen visually.

[0204] Figure 10 Exemplarily shows the change situation of a partial area of the shooting preview screen when the transmission display camera of the mobile phone switches. Among them, Figure 10 In (1), it is the shooting preview screen before the transmission display camera switches, Figure 10 In (2), it is the shooting preview screen after the transmission display camera switches. By comparing Figure 10 (1) and (2) in the figure, it can be seen that when the transmission display camera of the electronic device switches, not only the change in the field of view angle of the preview image is very small, but also the clarity of the preview screen is very high. On the premise of the same preview screen switching effect, the mobile phone adopting the image alignment method between multiple cameras provided in this embodiment can make the clarity of the preview screen higher.

[0205] Similar to the zoom-in scenario, in the zoom-out scenario, the FOV center of the ultra-wide-angle camera is also aligned with the FOV center of the wide-angle camera in stages (rough alignment in the processing stage of the IFE module and precise alignment in the processing stage of the IPE module), and the FOV center of the wide-angle camera is aligned with the FOV center of the telephoto camera (rough alignment in the processing stage of the IFE module and precise alignment in the processing stage of the IPE module).

[0206] In the zoom out scenario, when aligning the FOV center of the ultra-wide-angle camera with the FOV center of the wide-angle camera, the wide-angle camera is the display camera. After the ultra-wide-angle camera sensor starts to stream, the IFE module of the ISP chip crops the image (non-display image) captured by the ultra-wide-angle camera sensor according to the non-centered cropping frame to obtain the IFE cropped image. Subsequently, the IPE module of the ISP chip continues to offset the FOV area in the IFE cropped image according to the warp matrix calculated by the SAT algorithm module. In this way, when the display camera of the electronic device is switched from the wide-angle camera to the ultra-wide-angle camera, since the FOV center of the ultra-wide-angle camera has been aligned with the wide-angle camera, the preview screen can smoothly transition, and abrupt screen jumps can be avoided.

[0207] Similarly, in the zoom out scenario, when aligning the FOV center of the wide-angle camera with the FOV center of the telephoto camera, the telephoto camera is the display camera. After the wide-angle camera sensor starts to stream, the IFE module of the ISP chip crops the image (non-display image) captured by the wide-angle camera sensor according to the non-centered cropping frame to obtain the IFE cropped image. Subsequently, the IPE module of the ISP chip continues to offset the FOV area in the IFE cropped image according to the warp matrix calculated by the SAT algorithm module. In this way, when the display camera of the electronic device is switched from the telephoto camera to the wide-angle camera, since the FOV center of the wide-angle camera has been aligned with the telephoto camera, the preview screen can smoothly transition, and abrupt screen jumps can be avoided.

[0208] In summary, in the method for image alignment between multiple cameras provided in this embodiment, the alignment between the FOV areas of multiple cameras is achieved in stages. Among them, in the processing stage of the IFE module of the ISP chip, a rough alignment between the FOV areas of multiple cameras is achieved, and in the processing stage of the IPE module of the ISP chip, a precise alignment between the FOV areas of multiple cameras is achieved. In this way, this method ensures that the preview screen can smoothly transition when switching between multiple cameras without reducing the clarity of the preview image.

[0209] This embodiment also provides a computer storage medium, in which computer instructions are stored. When these computer instructions run on an electronic device, the electronic device is made to execute the above-related method steps to implement the method for image alignment between multiple cameras in the above embodiment.

[0210] This embodiment also provides a computer program product. When this computer program product runs on a computer, the computer is made to execute the above-related steps to implement the method for image alignment between multiple cameras in the above embodiment.

[0211] In addition, an embodiment of the present application further provides a device, which may specifically be a chip, a component or a module. The device may include a processor and a memory connected to each other. The memory is used to store computer-executable instructions. When the device runs, the processor may execute the computer-executable instructions stored in the memory, so that the chip executes the image alignment method between multiple cameras in the above method embodiments.

[0212] Among them, the electronic device (such as a mobile phone, etc.), computer storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be elaborated here.

[0213] Through the description of the above embodiments, those skilled in the art can understand that for the convenience and conciseness of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0214] In several embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other may be through some interfaces. The indirect coupling or communication connection of the device or unit may be in electrical, mechanical or other forms.

[0215] As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image alignment method between multiple cameras, characterized in that, it is applied to an electronic device, the electronic device includes at least two cameras, the electronic device further includes an Image Signal Processor (ISP) chip, and the ISP chip includes a first image processing module and a second image processing module, and includes: In response to a first operation, increasing the current zoom magnification value of the electronic device to a first value, and acquiring a first image captured by a first camera; According to the first value and prior information, determining a first cropping frame in the first image; wherein, the first cropping frame is not centered with the first image and is offset in the field of view (FOV) direction of the second camera in the first image; the focal length of the first camera is less than that of the second camera; Controlling the first image processing module to crop the first image according to the first cropping frame to obtain a second image; Calculating a first warp matrix and a second cropping frame corresponding to the second image; Controlling the second image processing module to process the second image according to the first warp matrix to obtain a third image, and performing a cropping operation on the third image according to the second cropping frame to obtain a fourth image; the fourth image is used to generate a preview image for display.

2. The method according to claim 1, characterized in that, determining a first cropping frame in the first image according to the first value and prior information includes: Determining a third cropping frame in the first image according to the FOV region of the first camera corresponding to the first value; wherein, the third cropping frame is centered with the first image; Calculating a target offset of the third cropping frame according to the first value and prior information; Performing an offset process on the third cropping frame according to the target offset of the third cropping frame to obtain the first cropping frame.

3. The method according to claim 2, characterized in that, calculating a target offset of the third cropping frame according to the first value and prior information includes: Calculating the total offset of the FOV centers between the first camera and the second camera according to the prior information; Allocating the total offset of the FOV centers according to the first value to obtain the target offset of the third cropping frame.

4. The method according to any one of claims 1-3, characterized in that, further includes: In response to the first operation, acquiring a fifth image captured by a second camera; Calculating a first warp matrix and a second cropping frame corresponding to the second image includes: Determining the FOV region of the second camera in the fifth image according to the first value and the fifth image; Performing a spatial transformation alignment process according to the FOV region of the second camera in the fifth image and the second image to obtain a first warp matrix and a second cropping frame corresponding to the second image.

5. The method according to any one of claims 1-3, characterized in that, the second camera is not started; Calculating a first warp matrix and a second cropping frame corresponding to the second image includes: Calculate the total FOV center offset between the first camera and the second camera according to the calibration information of the first camera and the second camera; Calculate the offset to be applied to the FOV area of the first camera in the second image according to the total FOV center offset and the target offset corresponding to the first cropping frame; Calculate the first warp matrix and the second cropping frame corresponding to the second image according to the offset to be applied; 6. The method according to any one of claims 1-3, characterized in that, further comprising: In response to the first operation, increasing the current zoom magnification value of the electronic device to a second value, and switching the display camera of the electronic device from the first camera to the second camera.

7. The method according to any one of claims 1-3, characterized in that, further comprising: In response to a second operation, reducing the current zoom magnification of the electronic device to a third value, and obtaining a sixth image captured by the first camera and a seventh image captured by the second camera; the seventh image is used to generate a preview image for display; Determine a fourth cropping frame in the sixth image according to the third value and prior information; wherein, the fourth cropping frame is not centered with the sixth image and is offset in the FOV direction of the second camera in the sixth image; Control the first image processing module to crop the sixth image according to the fourth cropping frame to obtain an eighth image; Determine the FOV area of the second camera in the seventh image according to the third value and the seventh image; Perform spatial transformation alignment processing according to the FOV area of the second camera in the seventh image and the eighth image to obtain a second warp matrix and a fifth cropping frame corresponding to the eighth image; Control the second image processing module to process the eighth image according to the second warp matrix to obtain a ninth image, and perform a cropping operation on the ninth image according to the fifth cropping frame to obtain a tenth image.

8. The method according to claim 7, characterized in that, further comprising: In response to the second operation, reducing the current zoom magnification of the electronic device to a fourth value, and switching the display camera of the electronic device from the second camera to the first camera, and the tenth image is used to generate a preview image for display.

9. The method according to claim 7, characterized in that, Determining a fourth cropping frame in the sixth image according to the third value and prior information includes: Determine a sixth cropping frame in the sixth image according to the FOV area of the first camera corresponding to the third value; wherein, the sixth cropping frame is centered with the sixth image; Calculate the target offset of the sixth cropping frame according to the third value and prior information; Perform an offset process on the sixth cropping frame according to the target offset of the sixth cropping frame to obtain the fourth cropping frame.

10. The method according to claim 1, characterized in that, The prior information includes: The external parameter matrix between the first camera and the second camera, and the internal parameter matrices of the first camera and the second camera.

11. The method according to claim 1, wherein, the first camera is an ultra-wide-angle camera and the second camera is a wide-angle camera; or, the first camera is a wide-angle camera and the second camera is a telephoto camera.

12. An electronic device, wherein, comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored on the memory, and when the computer programs are executed by the one or more processors, the electronic device executes the image alignment method between multiple cameras according to any one of claims 1-11.

13. A chip system, wherein, applied to an electronic device, the chip system includes instructions and at least one processor, and the at least one processor runs the instructions to make the electronic device execute the image alignment method between multiple cameras according to any one of claims 1-11.

14. A computer-readable storage medium, including a computer program, wherein, when the computer program runs on an electronic device, the electronic device is made to execute the image alignment method between multiple cameras according to any one of claims 1-11.

Images