Image processing method, electronic device, chip system and storage medium

Abstract

The application discloses an image processing method, an electronic device, a chip system and a storage medium, relates to the technical field of image processing, and is used for reducing the problem that a preview image displayed by the electronic device shakes when a zoom ratio changes. In the image processing method, when the first zoom ratio is used, the electronic device uses the first camera, and when the zoom ratio changes, the electronic device has a tendency to switch to the second camera or switches to the second camera. The electronic device can compensate for the nonlinear shift of the crop frame in the IFE stage. Specifically, the shift information during the compensation can be determined based on the prior information between the cameras before and after the switching (or before the switching and the imminent switching) and the shift information during the nonlinear shift, and the compensation operation enables the crop frame to finally perform linear shift on the image captured by the first camera.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, electronic device, chip system, and storage medium. Background Technology

[0002] Electronic devices can be equipped with multiple cameras (such as ultra-wide-angle cameras, wide-angle cameras, telephoto cameras, etc.) and can switch between different cameras based on the currently set zoom level to capture objects at different distances. During this process, the electronic device can also control the camera's image sensor to expose the content captured by the camera, achieving the purpose of outputting an image. Before sending the sensor image to the display for preview, the electronic device can perform a series of image processing steps to ensure a better display effect for the preview image.

[0003] One step in the image processing described above is to crop the sensor output image according to the zoom ratio of the electronic device. For example, the center cropping frame is shifted on the sensor output image to obtain a preview image that meets the zoom ratio. However, the shift of the center cropping frame on the sensor output image will cause the final displayed preview image to jitter, thus affecting the user experience. Summary of the Invention

[0004] This application provides an image processing method, an electronic device, a chip system, and a storage medium to reduce the problem of jitter in the preview image displayed by the electronic device when the zoom magnification changes.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, an image processing method is provided, which is applied to an electronic device. The electronic device includes at least two cameras with different focal lengths, and the electronic device uses the first camera when at a first zoom level, and has a tendency to switch to the second camera when the zoom level changes, or switches to the second camera.

[0007] In the above method, the electronic device can display a first preview image corresponding to a first image at a first zoom level, wherein the first image is captured by a first camera. The electronic device can also determine a first cropping frame aligned with the center of the first image based on a second zoom level during the zoom level change process. Then, the electronic device acquires first offset information, during the image front-end processing (IFE) stage at the second zoom level, indicating a non-linear shift of the first cropping frame on the first image. Based on the first offset information and prior information between the first and second cameras, it determines second offset information of the first cropping frame on the first image during the image back-end processing (IPE) stage. Finally, the electronic device performs a compensation operation on the non-linear shift of the first cropping frame on the first image based on the second offset information. This compensation operation causes the first cropping frame to shift linearly on the first image under the influence of the first and second offset information. Afterward, the electronic device displays a second preview image corresponding to the image within the first cropping frame.

[0008] In this process, the image within the first cropping frame changes along with the linear offset of the first cropping frame onto the first image as the zoom level changes.

[0009] In the above method, the electronic device can perform a non-linear offset operation on the cropping frame in the IFE stage and then compensate for this non-linear offset in the IPE stage, so that the cropping frame can finally perform a linear offset on the first image after both the IFE and IPE stages. It can be understood that the second zoom ratio can be one zoom ratio in the zoom ratio change process. The electronic device can perform the above compensation operation for multiple zoom ratios in the zoom ratio change process. The image in the first cropping frame changes with the offset of the first cropping frame in the first image. Therefore, during the zoom ratio change process, the electronic device can continuously display preview images corresponding to different zoom ratios, thus presenting a preview image that follows the zoom ratio change. Furthermore, during the zoom ratio change process, the first cropping frame can achieve continuous linear offset on the first image, so the preview image displayed by the electronic device is also smooth and stable, reducing jitter and improving the user experience.

[0010] In another possible implementation of the first aspect, the electronic device determines the target offset information of the first cropping box on the first image by linearly offsetting it based on the distance between the first camera and the second camera in the prior information. Then, based on the target offset information and the first offset information, it determines the second offset information corresponding to the first cropping box in the IPE stage.

[0011] In this implementation, the target offset information can represent the predicted information under ideal conditions, that is, the offset condition or offset amount that the first cropping box needs to satisfy when linearly offsetting the first image. The first offset information can represent the actual offset amount of the first cropping box on the first image during the IFE stage, that is, the offset amount when performing nonlinear offsetting. Therefore, in order for the first cropping box to satisfy the linear offset condition, the electronic device can determine what kind of offset the first cropping box needs to undergo to satisfy the linear offset, given that it already has the first offset information; that is, it determines the second offset information.

[0012] In another possible implementation of the first aspect, as the zoom ratio changes, both the first offset information and the second offset information exhibit a step-like change. It is understood that due to the specific nature of the first image format, the first offset information is typically a multiple of 2 (representing 2 pixels), thus the change in the first offset information will exhibit a step-like change. Correspondingly, to ensure that the final offset of the first cropping box satisfies linearity, the change in the second offset information should also exhibit a step-like change. This ensures that after superimposing the second offset information on the first offset information (or using the second offset information to compensate for non-linear offset), the total offset of the first cropping box exhibits a linear change, i.e., achieving linear offset of the first cropping box.

[0013] In another possible implementation of the first aspect, the electronic device determines the first warp matrix corresponding to the first cropping frame based on the second offset information, and then performs an offset operation on the nonlinear offset of the first cropping frame on the first image based on the first warp matrix.

[0014] In this implementation, the electronic device can use the warp matrix to offset the first cropping box again during the compensation operation. After the two offset operations are superimposed, the first cropping box finally achieves a linear offset.

[0015] In another possible implementation of the first aspect, the electronic device acquires a second image captured by the electronic device at a second zoom level, wherein the second image is captured by either the first camera or the second camera. Then, the electronic device performs spatial transformation processing based on the second image and the image within the first cropping frame to obtain a second warp matrix corresponding to the first cropping frame, and determines a first warp matrix corresponding to the first cropping frame based on second offset information and the second warp matrix.

[0016] In this implementation, before the compensation operation, the electronic device can first determine the first warp matrix based on the second offset information. Specifically, the electronic device can either superimpose the second offset information onto the second warp matrix, or perform an offset operation on the second warp matrix based on the second offset information, thereby obtaining the first warp matrix.

[0017] In one possible implementation of the first aspect, the prior information between the first camera and the second camera includes one or more of the following: the distance between the first camera and the second camera, the extrinsic parameter matrix between the first camera and the second camera, and the intrinsic parameter matrix between the first camera and the second camera.

[0018] In another possible implementation of the first aspect, when both the first zoom ratio and the second zoom ratio are within a preset zoom range, the electronic device determines the deviation information between the first camera and the second camera based on prior information, and allocates the deviation information according to the second zoom ratio to obtain third offset information of the offset of the first cropping frame on the first image. The electronic device then determines, based on the third offset information and the format type of the first image, first offset information of the non-linear offset of the first cropping frame on the first image during the IFE stage.

[0019] The aforementioned deviation information is used to indicate the positional and / or angular deviations between the cameras, and the preset zoom range is the zoom magnification range corresponding to the focal length of the first camera. In this implementation, the influence of the deviation information between the first image format type and the cameras on the first cropping frame offset can be considered, thereby more accurately determining the first offset information for the non-linear offset of the first cropping frame.

[0020] In another possible implementation of the first aspect, when the second zoom ratio exceeds a preset zoom range, the electronic device determines the deviation information between the first camera and the second camera based on prior information, and determines the third offset information of the first cropping frame's offset on the first image based on the deviation information. The electronic device then determines the first offset information of the first cropping frame's non-linear offset on the first image during the IFE stage based on the third offset information and the format type of the first image.

[0021] In this implementation, if the zoom ratio changes beyond the preset zoom range, it can be understood as a significant zoom ratio change, thus having a substantial impact on the first cropping frame. In this case, the electronic device does not need to redistribute the deviation information between the cameras; instead, it can directly determine the third offset information based on the deviation information. Combined with the influence of the first image format type on the offset, the electronic device can more accurately determine the first offset information for the non-linear offset of the first cropping frame.

[0022] In another possible implementation of the first aspect, the format type of the aforementioned first image includes a Bayer array image, where each pixel in the Bayer array image is a red pixel R, a green pixel G, or a blue pixel B. Furthermore, the image processing method provided in this application can perform processing operations on Bayer array images, thereby reducing the impact of the Bayer array image on the non-linear offset of the cropping frame and improving the stability and smoothness of the preview image display.

[0023] In another possible implementation of the first aspect, the first camera is an ultra-wide-angle camera and the second camera is a wide-angle camera; or, the first camera is a wide-angle camera and the second camera is a telephoto camera or an ultra-wide-angle camera; or, the first camera is a telephoto camera and the second camera is a wide-angle camera.

[0024] It is understandable that the aforementioned zoom ratio changes can include either increasing or decreasing the zoom ratio. When the zoom ratio increases, the electronic device may tend to switch from an ultra-wide-angle camera to a wide-angle camera, or directly switch to a wide-angle camera; or, the electronic device may tend to switch from a wide-angle camera to a telephoto camera, or directly switch to a telephoto camera. When the zoom ratio decreases, the electronic device may tend to switch from a telephoto camera to a wide-angle camera, or directly switch to a wide-angle camera; or, the electronic device may have an area where it switches from a wide-angle camera to an ultra-wide-angle camera, or directly switch to an ultra-wide-angle camera.

[0025] In this implementation method, the electronic device can implement the above image processing method in different scenarios of zoom magnification or reduction, so as to present users with a smoother and more stable preview image in different scenarios.

[0026] Secondly, another electronic device is provided. This electronic device includes at least two cameras with different focal lengths, and, at a first zoom level, the electronic device uses the first camera; when the zoom level changes, the electronic device has a tendency to switch to the second camera, or switches to the second camera. The electronic device also includes: a memory and one or more processors; the memory is coupled to the processors, and the processors are connected to at least two cameras; the memory is used to store computer programs or instructions, and the processors are used to execute the computer programs or instructions, which, when executed by the processors, cause the electronic device to perform the image processing method as described in the first aspect and any implementation thereof.

[0027] Thirdly, a chip system is provided, including an image signal processor (ISP) chip, the ISP chip being used to perform the image processing method as described in the first aspect and any implementation thereof.

[0028] Fourthly, a computer-readable storage medium is provided, including computer instructions, the computer-readable storage medium storing computer instructions or programs that, when executed on a computer, cause the image processing method as described in the first aspect and any implementation thereof to be performed.

[0029] The beneficial effects that the electronic device described in the second aspect, the chip system described in the third aspect, and the computer-readable storage medium described in the fourth aspect can achieve can be referred to the beneficial effects that can be achieved in the first aspect and any of its implementations. Attached Figure Description

[0030] Figure 1 A schematic diagram of a preview page provided for an embodiment of this application;

[0031] Figure 2 A schematic diagram illustrating an image processing procedure provided in an embodiment of this application;

[0032] Figure 3 A schematic diagram of a Bayer array image provided for an embodiment of this application;

[0033] Figure 4 A schematic diagram illustrating the center clipping frame offset provided in an embodiment of this application;

[0034] Figure 5 Schematic diagram of the structure of the electronic device provided in the embodiments of this application Figure 1 ;

[0035] Figure 6 Schematic diagram of the structure of the electronic device provided in the embodiments of this application Figure 2 ;

[0036] Figure 7 A flowchart illustrating the image processing method provided in the embodiments of this application. Figure 1 ;

[0037] Figure 8 A schematic diagram illustrating the zoom range provided in the embodiments of this application;

[0038] Figure 9 This is a schematic diagram of another preview page provided for an embodiment of this application. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0040] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0041] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0042] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0043] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0044] Currently, some electronic devices (such as mobile phones) can be equipped with cameras (e.g., ultra-wide-angle cameras, wide-angle cameras, telephoto cameras, etc.), allowing users to photograph different objects. Before shooting, the user can open the corresponding camera application, and the electronic device will display a preview page for the camera. For example, see [link to mobile phone example]. Figure 1 As shown in (a), the preview page 101 displayed on the mobile phone may include a real-time preview image 102. When the zoom ratio (or zoom ratio, ZoomRatio value, etc.) of the mobile phone changes, the preview image 102 will also change accordingly.

[0045] Furthermore, the preview page 101 may also include other controls to facilitate user control of the camera, such as shooting controls 103, zoom controls 104, etc. Users can adjust the zoom controls 104 to control the zoom ratio of the phone, and the phone can also switch between different cameras to take pictures according to the changes in zoom ratio, thereby presenting a preview image 102 with changing field of view on the preview page 101.

[0046] For example, refer to Figure 1As shown in (a) and (b), if the user adjusts the zoom ratio to the range [0.6x, 1.0x) and the phone selects the ultra-wide-angle camera to take a picture, the field of view of the preview image becomes larger, meaning the objects in the image become smaller; if the user adjusts the zoom ratio to the range [1.0x, 3.5x) and the phone selects the wide-angle camera to take a picture, the field of view of the preview image becomes smaller, and the objects in the image become larger; if the user adjusts the zoom ratio to a value greater than or equal to 3.5x (e.g., the zoom ratio range [3.5x, 5x]) and the phone selects the telephoto camera to take a picture, the field of view of the preview image becomes smaller, and the objects in the image become larger.

[0047] Among them, the field of view of the ultra-wide-angle camera is larger than that of the wide-angle camera, and the field of view of the wide-angle camera is larger than that of the telephoto camera.

[0048] It is understandable that the user's operation of adjusting the zoom control 104 can be seen as an operation of zooming in or out of the preview image 102, or as an operation of wanting to view the zoomed-in or zoomed-out preview image 102.

[0049] During shooting, the electronic device can also control the camera sensor to expose the content captured by the camera, achieving the purpose of outputting an image. Afterwards, the electronic device can use an image signal processor (ISP) to perform a series of processes on the sensor image, ultimately displaying a preview image on the screen. Furthermore, the ISP processing can include two stages: image signal processing front end (IFE) and image signal processing post end (IPE).

[0050] For example, see Figure 2 As shown, in the IFE stage, the zoom level of the electronic device can be obtained, and the sensor output image can be cropped according to the zoom level. Specifically, the central cropping box can be offset on the sensor output image according to the zoom level, such as by enlarging or shrinking the cropping box, thereby obtaining the image content within the central cropping box, i.e., the IFE cropped image. The central cropping box can be determined by a region of interest translator (ROITranslator) based on the current zoom level of the electronic device.

[0051] Next, in the IPE stage, the warp matrix determined by the spatial alignment transform (SAT) module is used to offset and rotate the center cropping box on the aforementioned IFE cropped image, thereby enhancing the quality and performance of the IFE cropped image to obtain the IPE cropped image. Then, the IPE cropped image is sent to the display panel to show a preview image on the screen.

[0052] Specifically, the SAT module can perform spatial alignment transformation on the sensor images before and after the camera switching based on the current zoom level of the electronic device or the camera switching situation, to obtain a warp matrix. Alternatively, in some possible application scenarios, the SAT module can include a ROI Translator, thereby determining the aforementioned center clipping box.

[0053] In the above process, in addition to the IFE stage, the IPE stage can also perform offset operations on the center cropping box. Ideally, the offsets in both the IFE and IPE stages are linear, resulting in a smooth final preview image.

[0054] In practice, the image output from a sensor is a specific format, such as a Bayer array image. A Bayer array image can include several 2x2 arrays, represented as an NxN array, where each 2x2 array contains red (R) pixels, green (G) pixels, and blue (B) pixels. Figure 3 As shown in (a), the array arrangement can be represented as follows: the first row includes B and G, the second row includes G and R, or, for example... Figure 3 As shown in (b), the array arrangement can also be represented as the first row including G and B, and the second row including G and R.

[0055] Furthermore, the central clipping frame must include at least one 2x2 array, represented as an MxM array (M≤N), and each 2x2 array must be identical to the smallest array in the sensor output image, for example, ... Figure 3 The array shown in (a) is, or is Figure 3 The array shown in (b) is shown in the diagram.

[0056] In the aforementioned IFE stage, because the center clipping box must meet the array requirements, its offset on the sensor output image must be a multiple of 2 (representing 2 pixels). However, this offset is not linear but non-linear. Consequently, this non-linear offset of the "center" clipping box will result in the clipping box not being centered on the sensor output image. For an example, see... Figure 4 As shown, the solid line 402 represents the linear offset of the center clipping box under ideal conditions, while the dashed line 401 represents the non-linear offset of the non-center clipping box under actual conditions.

[0057] The non-linear offset of the cropping frame mentioned above can cause the preview image to flicker when displayed after the camera switches, thus affecting the user experience.

[0058] Based on the above, this application provides an image processing method. An electronic device executing this method may include at least two cameras with different focal lengths. When at a first zoom level, the electronic device uses the first camera; when the zoom level changes, the electronic device tends to switch to the second camera, or switches to the second camera. The electronic device can display a first preview image corresponding to a first image at the first zoom level, wherein the first image is captured by the first camera. The electronic device can determine a first cropping frame aligned with the center of the first image based on a second zoom level during the zoom level change process. Then, the electronic device acquires first offset information, during the image front-end processing (IFE) stage at the second zoom level, where the first cropping frame undergoes a non-linear offset on the first image. Then, based on the first offset information and prior information between the first and second cameras, it determines second offset information of the first cropping frame on the first image during the image back-end processing (IPE) stage. Finally, the electronic device performs a compensation operation on the non-linear offset of the first cropping frame on the first image based on the second offset information. The compensation operation is used to make the first cropping frame linearly offset on the first image under the influence of the first and second offset information. Then, the electronic device displays a second preview image corresponding to the image in the first cropping frame.

[0059] In this process, the image within the first cropping frame changes along with the linear offset of the first cropping frame onto the first image as the zoom level changes.

[0060] In the above method, when the zoom level of the electronic device changes, the electronic device can switch to use the corresponding camera, or there is a tendency to switch to another camera. Furthermore, after the cropping box undergoes a non-linear offset in the IFE stage, the electronic device can compensate for this non-linear offset in the IPE stage. Specifically, the offset information during compensation can be determined based on prior information between the cameras before and after the switch (or between the cameras before and about to be switched) and the offset information during the non-linear offset. The compensation operation allows the cropping box to ultimately undergo a linear offset on the first image after the IFE and IPE stages.

[0061] It is understandable that the aforementioned second zoom level can be one zoom level in the zoom level change process. The electronic device can perform the above compensation operation for multiple zoom levels in the zoom level change process. The image in the first cropping frame changes with the offset of the first cropping frame in the first image. Therefore, during the zoom level change process, the electronic device can continuously display preview images corresponding to different zoom levels, thus presenting a preview image that follows the zoom level change. Furthermore, during the zoom level change process, the first cropping frame can achieve continuous linear offset on the first image, so the preview image displayed by the electronic device is also smooth and stable, reducing jitter and thus improving the user experience.

[0062] For example, the aforementioned electronic device's switching of cameras based on zoom ratio, or its tendency to switch cameras, can be expressed as follows: When using a wide-angle camera, if the change in zoom ratio does not exceed the zoom ratio range corresponding to the wide-angle camera's focal length, the electronic device can still use the wide-angle camera. In this case, if the zoom ratio increases, the electronic device tends to switch to a telephoto camera; if the zoom ratio decreases, the electronic device tends to switch to an ultra-wide-angle camera. Alternatively, when using a wide-angle camera, if the decrease in zoom ratio exceeds the zoom ratio range corresponding to the wide-angle camera's focal length, the electronic device switches to an ultra-wide-angle camera; if the increase in zoom ratio exceeds the zoom ratio range corresponding to the wide-angle camera's focal length, the electronic device switches to a telephoto camera. Therefore, the electronic device can implement the aforementioned image processing method in different scenarios of zoom ratio magnification or reduction. Furthermore, the scenarios for displaying preview images involve taking photos or recording videos with the camera, thus providing users with smoother and more stable preview images in different scenarios.

[0063] The image processing method provided in this application can be applied to electronic devices equipped with at least two cameras. Exemplarily, the electronic device can be a mobile phone, tablet computer, digital camera, surveillance equipment, virtual reality (VR) device, augmented reality (AR) device, wearable device, etc. This application does not impose any special limitations on the specific form of the electronic device; it is merely an illustrative example.

[0064] In some embodiments, the electronic devices in this application can all adopt Figure 5 The structure shown. (As illustrated) Figure 5 As shown, the electronic device may include an IFE module 501, an IPE module 502, an ROI module 503, a SAT module 504, and a display module 505.

[0065] The ROI module 503 can acquire the zoom level of the electronic device and the image captured by the camera at the current zoom level. For example, before the zoom level changes, the electronic device is at a first zoom level and using the first camera. At this time, the ROI module 503 can acquire the first zoom level and the first image captured by the first camera. After the zoom level changes, the electronic device is currently at a second zoom level. If the first camera is still being used, the ROI module 503 acquires the second zoom level and the second image captured by the first camera. If the second camera is being used, the ROI module 503 acquires the second zoom level and the second image captured by the second camera.

[0066] The ROI module 503 can also determine the first cropping box on the first image acquired before the change based on the second zoom level during the change process. It can be understood that the first cropping box determined by the ROI module 503 this time is the first cropping box in the initial state, and the first cropping box is aligned with the center of the first image, or in other words, the first cropping box is located at the center of the first image.

[0067] Subsequently, the ROI module 503 can also obtain the first offset information of the first cropping box in the IFE stage, which is non-linearly offset on the first image, at the second zoom level.

[0068] When determining the first offset information, the ROI module 503 can also combine the changed zoom ratio, the format type of the first image before the zoom ratio change, and the prior information of the camera to determine the first offset information.

[0069] For example, the ROI module 503 acquires prior information and, based on this prior information, determines the deviation information (or coarse offset) between the first camera used before the zoom ratio change and the second camera that has been switched or is about to be switched. Then, based on the deviation information and the second zoom ratio, the ROI module 503 determines the third offset information for the first cropping box's offset on the first image. Finally, based on the third offset information and the format type of the first image, the ROI module 503 determines the first offset information for the non-linear offset of the first cropping box on the first image during the IFE stage.

[0070] The aforementioned prior information includes the distance between the cameras and the camera calibration information. The calibration information includes the extrinsic parameter matrix between the cameras and the intrinsic parameter matrix of the cameras. Taking the first and second cameras mentioned above as examples, the prior information may include one or more of the following: the distance between the first and second cameras, the extrinsic parameter matrix between the first and second cameras, and the intrinsic parameter matrix of the first and second cameras.

[0071] The aforementioned deviation information is used to indicate the positional and / or angular deviations between the cameras. Taking the first and second cameras as examples, the deviation information may include the positional and / or angular deviations between the first and second cameras. For example, the ROI module 503 can allocate the deviation information according to the second zoom level, and adjust the first cropping frame on the first image according to the allocated deviation information. That is, the first cropping frame can undergo a first offset operation on the first image to obtain the offset information of the first cropping frame (i.e., the third offset information).

[0072] The format type of the first image can be determined based on the characteristics of the image output from the camera sensor. For example, the format type of the first image can be a Bayer array image, that is, each pixel in the image includes only one color pixel, such as R, G, or B. Furthermore, the smallest unit array in the image is a 2x2 array, and each 2x2 array includes pixels of R, G, and B. The specific arrangement and number of pixels vary depending on the parameters or configuration of the camera sensor. For example, it could be... Figure 3 The array shown in (a) can also be Figure 3 The array shown in (b) is not specifically limited in this embodiment.

[0073] Because of the special format of the first image, the first cropping box must be offset by a multiple of 2 (representing 2 pixels) during the IFE stage, resulting in a non-linear offset of the first cropping box. The ROI module 503 can further determine the first offset information of the first cropping box during the non-linear offset stage, based on the previously determined third offset information and the characteristics of the first image format.

[0074] The purpose of the ROI module 503 is to determine the first offset information required by the IFE module 501 before the IFE module 501 processes the data. The offset operation of the first clipping box is specifically implemented by the IFE module 501 (i.e., the IFE module 501 implements the processing of the IFE stage). Furthermore, as can be seen from the aforementioned embodiments, when the IFE module 501 processes the data, the first clipping box undergoes a non-linear offset.

[0075] Then, the IFE module 501 sends the first cropping frame and the image within the first cropping frame to the SAT module 504.

[0076] In addition, the ROI module 503 can also send the determined first offset information to the SAT module 504.

[0077] The SAT module 504 can determine the compensation offset information for compensating the offset of the first cropping frame in the IFE stage (or the processing stage of the IFE module 501) based on the offset information of the first cropping frame processed by the IFE module 501 and the prior information between the cameras.

[0078] For example, the SAT module 504 determines the linear offset target information of the first cropping box on the first image based on the distance between the first camera and the second camera in the prior information. Then, the SAT module 504 determines the second offset information (i.e., the aforementioned compensation offset information) based on the target offset information and the first offset information.

[0079] The second offset information can be determined by subtracting the first offset information from the target offset information. In other words, the sum of the first and second offset information (or the superposition of the first and second offset information) can be represented as the target offset information. For example, the offset information in this embodiment can be represented as a specific offset amount, and the offset amount itself can be positive or negative.

[0080] Then, the SAT module 504 sends the determined second offset information to the IPE module 502.

[0081] Furthermore, the SAT module 504 can also determine the first warp matrix corresponding to the first clipping frame based on the second offset information, and send the first warp matrix to the IPE module 502 in the form of the second offset information.

[0082] The SAT module 504 can acquire the second image after the change ratio, and perform spatial transformation processing based on the second image and the image in the first cropping frame to obtain the second warp matrix corresponding to the first cropping frame. Then, the second warp matrix is ​​processed according to the second offset information to obtain the first warp matrix corresponding to the first cropping frame.

[0083] The IPE module 502 can perform a compensation operation on the nonlinear offset of the first cropping box on the first image (i.e., the nonlinear offset of the first image in the aforementioned IFE stage) based on the received second offset information. This allows the first cropping box to undergo a linear offset on the first image.

[0084] Furthermore, the IPE module 502 can use the first warp matrix sent by the SAT module 504 to perform a compensation operation on the first clipping frame.

[0085] Then, the IPE module 502 sends the image in the first cropping frame after the compensation operation to the display module 505, which displays the image in the first cropping frame, thereby displaying a preview image on the electronic device.

[0086] It is understandable that the display module 505 can display the preview image at each zoom level during the zoom ratio change process. That is, it can display the first preview image corresponding to the first image captured by the first camera at the first zoom level, and it can also display the second preview image at the second zoom level.

[0087] Understandably, during the IPE stage (or the stage where IPE module 502 processes the image), the image in the first cropping frame changes as the first cropping frame shifts on the first image, so that the preview image finally displayed by display module 505 also changes, thus enabling the preview image to change with the zoom level of the electronic device.

[0088] Furthermore, since the non-linear offset of the first cropping frame is compensated during the IPE stage, the compensated first cropping frame can achieve linear offset. Therefore, the offset of the image in the first cropping frame is also linear, resulting in a smoother display effect in the final generated preview image, reducing jitter and improving the user experience.

[0089] In other embodiments, the electronic device in this application may also employ... Figure 6The hardware structure is shown. (As shown in the image.) Figure 6 As shown, the electronic device includes a processor 601, a transceiver 602, and a communication line 603.

[0090] Furthermore, the electronic device may also include a memory 604. The processor 601, memory 604, and transceiver 602 can be connected via a communication line 603.

[0091] The processor 601 can be a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 601 can also be other devices with processing capabilities, such as circuits, devices, or software modules, without limitation.

[0092] Transceiver 602 is used to communicate with other devices or other communication networks. These other communication networks can be Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc. Transceiver 602 can be a module, circuit, transceiver, or any device capable of enabling communication.

[0093] Communication line 603 is used to transmit information between the various components of an electronic device. Communication line 603 can provide wireless communication solutions including 2G / 3G / 4G / 5G, wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technology.

[0094] Memory 604 is used to store instructions. These instructions can be computer programs.

[0095] The memory 604 can be a read-only memory (ROM) or other type of static storage device that can store static information and / or instructions; it can also be a random access memory (RAM) or other type of dynamic storage device that can store information and / or instructions; it can also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, etc., without limitation.

[0096] It should be noted that the memory 604 can exist independently of the processor 601, or it can be integrated with the processor 601. The memory 604 can be used to store instructions, program code, or some data, etc. The memory 604 can be located inside or outside the electronic device, without limitation. The processor 601 is used to execute the instructions stored in the memory 604 to implement the image processing method provided in the embodiments of this application.

[0097] In one example, processor 601 may include one or more CPUs, such as processor 601 including CPU0 and CPU1.

[0098] As an alternative implementation, the electronic device includes multiple processors, for example, in addition to processor 601, it may also include processor 608, which may include CPU0 and CPU1.

[0099] As an optional implementation, the electronic device also includes at least two image acquisition units with different focal lengths, such as camera 605, camera 606, and camera 607, etc., and the focal lengths of each camera are different to meet the shooting requirements of the electronic device at different zoom levels. Furthermore, the processor is connected to each camera.

[0100] As an alternative implementation, the camera, also known as a camera module, includes a camera sensor. Therefore, the electronic device can include the camera sensor, which can capture images, expose them, and output the image, thus obtaining the image captured by the camera. Furthermore, the processor is connected to both the camera sensor and the camera module.

[0101] It is understood that the connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments, the electronic device may also include more or fewer modules than those provided in the above embodiments, and the modules may also employ different connection methods or combinations of multiple connection methods as described in the above embodiments.

[0102] Furthermore, the actions, terms, etc., involved in the various embodiments of this application can be referenced interchangeably without limitation. The data names or parameter names in the data exchanged between various devices in the embodiments of this application are merely examples, and other names may be used in specific implementations without limitation. The methods in the following embodiments can all be implemented in electronic devices with the above-described hardware structure.

[0103] The following uses the application of the above image processing method to a mobile phone as an example to further illustrate the image processing method in the embodiments of this application. In some embodiments, see Figure 7 As shown, the above image processing method may include the following steps S701-S706.

[0104] S701: At the first zoom level, the mobile phone displays the first preview image corresponding to the first image.

[0105] The mobile phone may include at least two cameras with different focal lengths. During zoom adjustments, if the zoom ratio changes significantly beyond the zoom range of the currently used first camera, the phone can switch to using the second camera. If the zoom ratio does not exceed the zoom range of the currently used first camera, the phone can continue using the first camera. The first image can be captured by the first camera.

[0106] Furthermore, the aforementioned camera can be a telephoto camera, a wide-angle camera, an ultra-wide-angle camera, etc. (See also...) Figure 8 As shown, the zoom range corresponding to the focal length of a wide-angle camera can be [1.0x, 3.5x), the zoom range corresponding to the focal length of a telephoto camera can be [3.5x, 5x], and the zoom range corresponding to the focal length of an ultra-wide-angle camera can be [0.5x, 1.0x]. In this embodiment, the zoom ranges of different cameras are merely illustrative and do not constitute a specific limitation on the zoom ranges of different cameras.

[0107] In some possible implementations, if the zoom ratio change does not exceed the zoom range of the current camera, the phone may tend to switch cameras based on the zoom ratio change. For example, if the phone is currently using a wide-angle camera, and the zoom ratio changes from 1.0X to 2.0X, although the current zoom ratio of 2.0X does not exceed the zoom range of the wide-angle camera, the phone tends to switch to the telephoto camera because the zoom ratio tends to increase. As another example, if the phone is currently using a wide-angle camera, and the zoom ratio changes from 3.0X to 1.0X, although the current zoom ratio of 1.0X does not exceed the zoom range of the wide-angle camera, the phone tends to switch to the ultra-wide-angle camera because the zoom ratio tends to decrease.

[0108] It is understood that the change in zoom magnification in the embodiments of this application is a continuous process. The method in the embodiments of this application can process a certain zoom magnification during the continuous change of zoom magnification and achieve the purpose of displaying the corresponding preview image.

[0109] For example, if a mobile phone continuously changes from a first zoom level to a third zoom level, then the processing can focus on the second zoom level during the change. The second zoom level can be any zoom level within the aforementioned continuous change process.

[0110] In some possible implementations, the phone can respond to the user's zoom operation by changing the zoom magnification. This zoom operation could include the user manipulating zoom controls on the preview page displayed on the phone, or the user inputting a preset gesture on the preview page.

[0111] For example, see Figure 9 As shown in (a), the phone's preview page 901 includes a zoom control 902, which the user can move to control the phone to change the zoom level. Alternatively, see [link to image]. Figure 9 As shown in (b), users can input zoom-in or zoom-out gestures on the preview page 901 to control the phone to change the zoom level.

[0112] In some possible implementations, the camera is also called a camera module, which includes a lens, a camera sensor, etc. The lens receives light from outside the phone, which is then received by the camera sensor. The camera sensor performs photoelectric conversion and other processing to obtain an image (also known as sensor output image). Therefore, the aforementioned acquisition of the first image by the first camera can also be seen as the phone acquiring the camera sensor output image, or the phone acquiring the first image through the camera sensor.

[0113] S702, the mobile phone determines a first cropping frame aligned with the center of the first image based on the second zoom ratio during the zoom ratio change process.

[0114] It can be understood that the first cropping frame determined at this point can also be called the center cropping frame. For example, the determined center cropping frame includes not only specific location information but also information such as resolution. Furthermore, the position and resolution of the center cropping frame can be determined based on the current zoom level. For instance, if the current zoom level is the second zoom level during a zoom cycle, the center cropping frame can be determined based on the second zoom level.

[0115] The first cropping frame determined in step S702 can be regarded as the first cropping frame in the initial state, and the first cropping frame is aligned with the center of the first image.

[0116] The first cropping frame determined in this embodiment is used to crop the image captured by the camera, thereby sending the image in the first cropping frame to the display, that is, using the image in the first cropping frame to generate a preview image displayed on the mobile phone.

[0117] It's understandable that as the zoom level of the phone changes, the first cropping frame constantly shifts relative to the first image, and the image within the first cropping frame also changes, thus causing the preview image displayed on the phone to change continuously. Furthermore, the shift of the first cropping frame here refers to its shift relative to its initial state during the zoom level change.

[0118] Referring to the foregoing embodiments, the offset of the first cropping frame on the first image includes two stages: the offset in the IFE stage and the offset in the IPE stage. For example, step S703 can determine the offset in the IFE stage.

[0119] S703, The mobile phone acquires the first offset information of the first cropping box being non-linearly offset on the first image during the IFE stage at the second zoom level.

[0120] In some possible implementations, when determining the first offset information, the mobile phone can further determine it based on the change in zoom ratio. The change in zoom ratio can include: the zoom ratio changing within the zoom ratio range corresponding to the focal length of the currently used first camera, or the change in zoom ratio exceeding the zoom ratio range corresponding to the focal length of the currently used first camera.

[0121] For example, the zoom ratio can vary within the zoom ratio range corresponding to the focal length of the currently used first camera, which may include both the first zoom ratio and the second zoom ratio being within a preset zoom range. Specifically, if the first camera is a telephoto camera, the preset zoom range is the zoom ratio range corresponding to a telephoto camera; if the first camera is a wide-angle camera, the preset zoom range is the zoom ratio range corresponding to a wide-angle camera; and if the first camera is an ultra-wide-angle camera, the preset zoom range is the zoom ratio range corresponding to an ultra-wide-angle camera.

[0122] When both the first and second zoom levels are within a preset zoom range, the phone can determine the deviation information between the first and second cameras based on prior information. Then, based on the deviation information and the second zoom level, it determines a third offset information for the first cropping frame's shift on the first image. That is, the phone assigns the deviation information to the first cropping frame according to the second zoom level; the offset information assigned to the first cropping frame is the third offset information. Afterward, based on the third offset information and the format type of the first image, the phone determines a first offset information for the non-linear shift of the first cropping frame on the first image.

[0123] In other words, the offset of the first cropping frame during the IFE stage is affected not only by the first image format type but also by the deviation information between the cameras. This embodiment takes both of these effects into account, thereby more accurately determining the first offset information for the non-linear offset of the first cropping frame.

[0124] As another example, a change in zoom magnification exceeding the zoom magnification range corresponding to the focal length of the currently used first camera may include a first zoom magnification within a preset zoom range, while a second zoom magnification exceeds the preset zoom range.

[0125] When the second zoom level exceeds the preset zoom range, the phone can determine the deviation information between the first and second cameras based on prior information. Then, based on the deviation information, it determines the third offset information for the first cropping frame's offset on the first image, for example, using the deviation information as the third offset information. Afterward, based on the third offset information and the format type of the first image, the phone determines the first offset information for the non-linear offset of the first cropping frame on the first image.

[0126] It is evident that if the zoom ratio change exceeds the zoom ratio range of the currently used first camera, it can be understood as a significant zoom ratio change, thus having a substantial impact on the first cropping frame. In this case, the phone does not need to redistribute the deviation information between cameras; instead, it can directly determine the third offset information based on the deviation information. Combined with the influence of the first image format type on the offset, the phone can more accurately determine the first offset information for the non-linear offset of the first cropping frame.

[0127] In some possible implementations, the format type of the first image is determined by the configuration or parameters of the camera sensor. For example, the format type of the first image can be a Bayer array image. Furthermore, the image processing method in this embodiment can perform processing operations on the Bayer array image, thereby reducing the impact of the Bayer array image on the non-linear offset of the cropping frame and improving the stability and smoothness of the preview image display.

[0128] In some possible implementations, the prior information between the first camera and the second camera may include one or more of the following: the distance between the first camera and the second camera, the extrinsic parameter matrix between the first camera and the second camera, and the intrinsic parameter matrix between the first camera and the second camera.

[0129] For example, step S704 can determine the offset in the IPE stage.

[0130] S704. The mobile phone determines the second offset information of the first cropping frame on the first image during the IPE stage based on the first offset information and the prior information between the first camera and the second camera.

[0131] In some possible implementations, the mobile phone can determine the linear offset target information of the first cropping box on the first image based on the distance between the first and second cameras in the prior information. Then, based on the target offset information and the first offset information, the second offset information corresponding to the first cropping box is determined in the IPE stage.

[0132] For example, the target offset information mentioned above can represent the prediction information under ideal conditions, that is, the offset condition or offset amount that the first cropping box needs to satisfy when linearly offsetting the first image. The first offset information can represent the actual offset amount of the first cropping box on the first image, that is, the offset amount when performing nonlinear offsetting.

[0133] Therefore, in order for the first cropping frame to meet the condition of linear offset, the mobile phone can determine what kind of offset is needed for the first cropping frame to meet the linear offset given that it already has the first offset information, that is, determine the second offset information.

[0134] For example, due to the special nature of the first image format, the first offset information is usually a multiple of 2 (representing 2 pixels). As a result, the change of the first offset information will present a step-like change. Correspondingly, in order to make the final offset of the first cropping box satisfy the linear offset, the change of the second offset information should also present a step-like change. This ensures that after superimposing the second offset information on the first offset information (or using the second offset information to compensate for the non-linear offset), the total offset of the first cropping box presents a linear change, that is, to achieve the linear offset of the first cropping box.

[0135] It is understandable that target offset information can also be represented as target offset information at each zoom level, and the target offset information changes linearly with the zoom level.

[0136] Furthermore, when the offset information is a specific offset amount, the phone can subtract the first offset information from the target offset information to obtain the second offset information.

[0137] In the above implementation, the mobile phone can determine the second offset information needed to make the first cropping box linearly offset after the first cropping box has been non-linearly offset. Then, the first cropping box is offset again (or compensated) according to the second offset information in subsequent steps (such as step S704), so that the first cropping box after the IEF stage and IPE stage finally achieves linear offset.

[0138] S705: The mobile phone performs a compensation operation on the non-linear offset of the first cropping box on the first image based on the second offset information.

[0139] The compensation operation is used to linearly offset the first cropping box onto the first image under the influence of the first and second offset information. In other words, after the first cropping box undergoes a non-linear offset onto the first image based on the first offset information, the phone then uses the second offset information to offset the first cropping box onto the first image. The two offsets are superimposed, resulting in a final linear offset for the first cropping box, thus compensating for the non-linear offset in the previous stage.

[0140] In some possible implementations, during the compensation operation described above, the mobile phone can use the warp matrix corresponding to the second offset information to offset the first cropping box.

[0141] For example, the mobile phone can determine the first warp matrix corresponding to the first cropping box based on the second offset information. Then, based on the first warp matrix, an offset operation is performed on the non-linear offset of the first cropping box on the first image.

[0142] Furthermore, when determining the first warp matrix, the mobile phone can acquire a second image captured at the second zoom level. This second image can be captured by either the first or the second camera; this embodiment does not impose any specific limitations on this.

[0143] Next, the phone performs spatial transformation processing on the second image and the image within the first cropping frame to obtain the second warp matrix corresponding to the first cropping frame. Specifically, the phone can perform spatial transformation processing on the image within the first cropping frame based on the second image to obtain the second warp matrix.

[0144] Finally, the phone determines the first warp matrix corresponding to the first cropping frame based on the second offset information and the second warp matrix. In other words, the phone can either overlay the second offset information onto the second warp matrix, or perform an offset operation on the second warp matrix based on the second offset information, to obtain the first warp matrix.

[0145] In the above implementation, the mobile phone can use the warp matrix to offset the first cropping box again during the compensation operation. After the two offset operations are superimposed, the first cropping box finally achieves a linear offset.

[0146] S706: The phone displays the second preview image corresponding to the image in the first cropping frame.

[0147] Understandably, steps S701-S706 described above can determine the corresponding first cropping frame for a given zoom level during the zoom ratio change process, and perform a compensation operation on the first cropping frame. After the compensation operation, the phone then acquires the image within the first cropping frame and displays that image, thereby showing the preview image corresponding to the current zoom ratio.

[0148] In this embodiment, the mobile phone can perform the aforementioned compensation operation for each zoom level or preset zoom level during the zoom ratio change process. As the zoom ratio changes, the image within the first cropping frame changes along with the offset of the first cropping frame onto the first image. Therefore, the mobile phone can continuously display preview images corresponding to different zoom levels, thus presenting a preview image that changes with the zoom ratio. Furthermore, as the zoom ratio changes, the first cropping frame can achieve continuous linear offset on the first image, resulting in a smooth and stable preview image displayed by the mobile phone, reducing jitter and improving the user experience.

[0149] In the above embodiments, a mobile phone is used as an example to describe the image processing method in the embodiments of this application. In other embodiments, when the electronic device is a tablet computer, digital camera, monitoring equipment, VR device, AR device, or wearable device, the electronic device can also perform the method steps of S701-S706 as described above, which will not be repeated in this application.

[0150] This application also provides a chip system applied to the aforementioned electronic device. The chip system includes at least one processor and at least one interface circuit. The processor and the interface circuit are interconnected via lines. For example, the interface circuit can be used to receive signals from other devices (e.g., memory). Or, for example, the interface circuit can be used to send signals to other devices (e.g., the processor). Exemplarily, the interface circuit can read instructions stored in memory and send those instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the various method steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application does not specifically limit this.

[0151] In other embodiments of this application, another chip system is provided, which may include an ISP chip that can perform the various method steps in the above embodiments.

[0152] This application also provides a computer-readable storage medium including computer instructions that, when executed on the electronic device, cause the electronic device to perform the various method steps described in the above embodiments.

[0153] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various method steps in the above embodiments. For example, the computer may be the aforementioned electronic device.

[0154] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0156] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0158] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0159] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized by, The method is applied to an electronic device, the electronic device comprising at least two cameras with different focal lengths, and at a first zoom ratio, the electronic device uses a first camera, and after a change in the zoom ratio, the electronic device has a tendency to switch to a second camera or switches to the second camera; the method comprises: displaying a first preview image corresponding to a first image at the first zoom ratio; the first image is captured by the first camera; determining a first crop frame aligned with a center of the first image based on a second zoom ratio in a change in the zoom ratio; obtaining first offset information of a non-linear offset of the first crop frame on the first image in an image front-end processing (IFE) stage at the second zoom ratio; determining second offset information of the first crop frame on the first image in an image post-end processing (IPE) stage according to the first offset information and prior information between the first camera and the second camera; performing a compensation operation on the non-linear offset of the first crop frame on the first image based on the second offset information; the compensation operation is used to make the first crop frame perform a linear offset on the first image under the action of the first offset information and the second offset information; displaying a second preview image corresponding to an image in the first crop frame; the image in the first crop frame changes following the linear offset of the first crop frame on the first image in the process of the change in the zoom ratio.

2. The method of claim 1, wherein, The determination of the second offset information of the first crop frame on the first image in the IPE stage according to the first offset information and the prior information between the first camera and the second camera comprises: determining target offset information of the linear offset of the first crop frame on the first image according to a distance between the first camera and the second camera in the prior information; determining the second offset information of the first crop frame in the IPE stage according to the target offset information and the first offset information.

3. The method of claim 1, wherein, In the process of the change in the zoom ratio, the first offset information presents a step change, and the second offset information presents a step change.

4. The method of claim 1, wherein, The compensation operation on the non-linear offset of the first crop frame on the first image based on the second offset information comprises: determining a first warp matrix corresponding to the first crop frame according to the second offset information; performing an offset operation on the non-linear offset of the first crop frame on the first image based on the first warp matrix.

5. The method of claim 4, wherein, The determination of the first warp matrix corresponding to the first crop frame according to the second offset information comprises: obtaining a second image captured by the electronic device at the second zoom ratio; the second image is captured by the first camera or by the second camera; performing a spatial change processing based on the second image and the image in the first crop frame to obtain a second warp matrix corresponding to the first crop frame; and displaying a second preview image corresponding to an image in the first crop frame; the image in the first crop frame changes following the linear offset of the first crop frame on the first image in the process of the change in the zoom ratio. According to the second offset information and the second warp matrix, a first warp matrix corresponding to the first clipping frame is determined.

6. The method according to any one of claims 1 to 5, characterized in that, The prior information between the first camera and the second camera includes one or more of a distance between the first camera and the second camera, an extrinsic matrix between the first camera and the second camera, and an intrinsic matrix of the first camera and the second camera.

7. The method according to any one of claims 1 to 5, characterized in that, The first offset information includes: When the first zoom ratio and the second zoom ratio are both in a preset zoom range, According to the prior information, deviation information between the first camera and the second camera is determined; the deviation information is used to represent a positional deviation and / or an angular deviation between the cameras; the preset zoom range is a zoom ratio range corresponding to a focal length range of the first camera; The deviation information is distributed according to the second zoom ratio to obtain third offset information of the offset of the first clipping frame on the first image; According to the third offset information and a format type of the first image, the first offset information of the nonlinear offset of the first clipping frame on the first image in the IFE stage is determined.

8. The method according to any one of claims 1 to 5, characterized in that, The first offset information includes: When the second zoom ratio is out of the preset zoom range, According to the prior information, deviation information between the first camera and the second camera is determined; the deviation information is used to represent a positional deviation and / or an angular deviation between the cameras; the preset zoom range is a zoom ratio range corresponding to a focal length range of the first camera; According to the deviation information, third offset information of the offset of the first clipping frame on the first image is determined; According to the third offset information and a format type of the first image, the first offset information of the nonlinear offset of the first clipping frame on the first image in the IFE stage is determined.

9. The method of claim 7, wherein, The format type of the first image includes a Bayer array image, and each pixel point in the Bayer array image is a red pixel R, a green pixel G, or a blue pixel B.

10. The method according to any one of claims 1 to 5, characterized in that, The first camera is an ultra-wide-angle camera, and the second camera is a wide-angle camera; or the first camera is a wide-angle camera, and the second camera is a long-focus camera or an ultra-wide-angle camera; or the first camera is a long-focus camera, and the second camera is a wide-angle camera.

11. An electronic device, comprising: The electronic device comprises at least two cameras with different focal lengths, and when the zoom ratio is the first zoom ratio, the electronic device uses the first camera; when the zoom ratio changes, the electronic device has a tendency to switch to the second camera or switches to the second camera; further comprising a memory and one or more processors; the memory is coupled with the processor, and the processor is connected with the at least two cameras; the memory is used to store computer programs or instructions; and the processor is used to run the computer programs or instructions, so that the electronic device executes the image processing method according to any one of claims 1-10 when the computer programs or instructions are executed by the processor.

12. A chip system, characterized by The electronic device comprises an image signal processor (ISP) chip, and the ISP chip is used to execute the image processing method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises computer instructions, and the computer readable storage medium stores the computer instructions or programs; when the computer instructions or programs are run on the computer, the image processing method according to any one of claims 1-10 is executed.

Images