Image processing method, electronic equipment, chip system and storage medium

Abstract

The invention discloses an image processing method, electronic equipment, a chip system and a storage medium, relates to the technical field of image processing, and is used for reducing the problem that a preview image displayed by the electronic equipment shakes when a zoom ratio is changed. In the image processing method, the electronic equipment uses the first camera at the first zoom magnification, and after the zoom magnification is changed, the electronic equipment has a trend of switching to the second camera or switching to the second camera. The electronic equipment can perform compensation operation on the nonlinear offset of the cutting frame in the I FE stage, specifically, offset information during compensation can be determined based on prior information before and after switching (or before switching and to be switched) and offset information during nonlinear offset, and the offset information during nonlinear offset can be determined based on prior information between cameras before and after switching (or before switching and to be switched). And the compensation operation can enable the cutting frame to finally perform linear offset on the image acquired by the first camera.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of image processing, and in particular, to an image processing method, an electronic device, a chip system, and a storage medium. Background Art

[0002] Multiple cameras (such as an ultra-wide-angle camera, a wide-angle camera, a telephoto camera, etc.) can be set on an electronic device, and different cameras are switched according to the currently set zoom ratio, etc. to photograph objects at different distances. During this process, the electronic device can also control the image sensor (sensor) of the camera to expose the content photographed by the camera to achieve the purpose of generating an image. Before sending the sensor image for display to present a preview image, the electronic device can perform a series of image processing on the sensor image, so as to ensure a better display effect of the preview image.

[0003] One link in the above image processing is to crop the sensor image according to the zoom ratio of the electronic device. For example, the center crop frame is controlled to shift on the sensor image, so as to obtain a preview image that meets the zoom ratio. However, the shift of the center crop frame on the sensor image will cause the finally displayed preview image to jitter, thus affecting the user experience. Summary of the Invention

[0004] The embodiments of the present application provide an image processing method, an electronic device, a chip system, and a storage medium, which are used to reduce the problem that the preview image displayed by the electronic device jitters when the zoom ratio changes.

[0005] To achieve the above object, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an image processing method is provided. This method is applied to an electronic device. The electronic device includes at least two cameras with different focal lengths. Moreover, when at a first zoom ratio, the electronic device uses a first camera. When the zoom ratio changes, the electronic device has a tendency to switch to a second camera, or switches to a second camera.

[0007] In the above method, the electronic device can display a first preview image corresponding to a first image at a first zoom ratio, where the first image is captured by a first camera. Also, the electronic device can determine a first cropping frame aligned with the center of the first image based on a second zoom ratio during the process of changing the zoom ratio. Then, the electronic device obtains first offset information of non-linear offset of the first cropping frame on the first image during the IFE stage at the second zoom ratio, and then determines second offset information of the first cropping frame on the first image during the IPE stage based on the first offset information and prior information between the first camera and the second camera. Finally, the electronic device performs a compensation operation on the non-linear offset of the first cropping frame on the first image based on the second offset information. The compensation operation is used to make the first cropping frame linearly offset on the first image under the action of the first offset information and the second offset information. Then, the electronic device displays a second preview image corresponding to the image in the first cropping frame.

[0008] Among them, during the process of changing the zoom ratio, the image in the first cropping frame changes following the linear offset of the first cropping frame on the first image.

[0009] In the above method, after the cropping frame undergoes non-linear offset during the IFE stage, the electronic device can perform a compensation operation on the non-linear offset during the IPE stage, so that the cropping frame can finally linearly offset on the first image after the IFE stage and the IPE stage. It can be understood that the above second zoom ratio can be a zoom ratio during the process of changing the zoom ratio, and the electronic device can perform the above compensation operation for multiple zoom ratios during the process of changing the zoom ratio. The image in the first cropping frame changes following the offset of the first cropping frame on the first image. Then, during the process of changing the zoom ratio, the electronic device can continuously display preview images corresponding to different zoom ratios, thereby presenting preview images that follow the change of the zoom ratio. Moreover, during the process of changing the zoom ratio, the first cropping frame can achieve continuous linear offset on the first image, so the preview images displayed by the electronic device are also smooth and stable, and the jitter situation is reduced, thereby improving the user experience.

[0010] In another possible implementation manner of the first aspect, the electronic device determines target offset information of linear offset of the first cropping frame on the first image according to the distance between the first camera and the second camera in the prior information, and then determines second offset information corresponding to the first cropping frame during the IPE stage based on the target offset information and the first offset information.

[0011] In this implementation manner, the target offset information may represent the predicted information in the ideal case, that is, the offset condition or offset amount that the first cropping frame needs to satisfy when linearly offsetting on the first image. The first offset information may represent the actual offset amount of the first cropping frame on the first image during the IFE stage, that is, the offset amount during non-linear offset. Then, in order to enable the first cropping frame to meet the conditions of linear offset, the electronic device may determine how much further offset is required for the first cropping frame to meet the linear offset on the premise that it already has the first offset information, that is, determine the second offset information.

[0012] In another possible implementation manner of the first aspect, during the process of the zoom ratio changing, the first offset information changes stepwise, and the second offset information changes stepwise. It can be understood that due to the particularity of the first image format, the first offset information is usually a multiple of 2 (indicating 2 pixels), and thus the change of the first offset information will show a stepwise change form. Correspondingly, in order to make the final offset of the first cropping frame meet the linear offset, the change of the second offset information should also show a stepwise change form, so as to ensure that after the second offset information is superimposed on the first offset information (or called compensating for the non-linear offset using the second offset information), the total offset amount of the first cropping frame changes linearly, that is, the linear offset of the first cropping frame is achieved.

[0013] In another possible implementation manner of the first aspect, the electronic device determines a first warp matrix corresponding to the first cropping frame according to the second offset information, and then, based on the first warp matrix, performs an offset operation on the non-linear offset of the first cropping frame on the first image.

[0014] In this implementation manner, the electronic device may specifically use the warp matrix to offset the first cropping frame again during the compensation operation, so that after the two offset operations are superimposed, the first cropping frame finally achieves linear offset.

[0015] In another possible implementation manner of the first aspect, the electronic device acquires a second image collected by the electronic device at the second zoom ratio, where the second image is collected by the first camera or the second camera. Then, the electronic device performs a spatial transformation process based on the second image and the image in the first cropping frame to obtain a second warp matrix corresponding to the first cropping frame, and determines the first warp matrix corresponding to the first cropping frame according to the second offset information and the second warp matrix.

[0016] In this implementation manner, before the compensation operation, the electronic device may first determine a first warp matrix according to the second offset information. Specifically, the electronic device may superimpose the second offset information on the basis of the second warp matrix, or perform an offset operation on the second warp matrix based on the second offset information to obtain the first warp matrix.

[0017] In a possible implementation manner of the first aspect, the prior information between the first camera and the second camera includes one or more of the following: the distance between the first camera and the second camera, the external parameter matrix between the first camera and the second camera, and the internal parameter matrices of the first camera and the second camera.

[0018] In another possible implementation manner of the first aspect, when both the first zoom ratio and the second zoom ratio are within a preset zoom range, the electronic device determines the deviation information between the first camera and the second camera according to the prior information, and distributes the deviation information according to the second zoom ratio to obtain the third offset information of the offset of the first cropping frame on the first image. The electronic device then determines, according to the third offset information and the format type of the first image, the first offset information of the non-linear offset of the first cropping frame on the first image in the IFE stage.

[0019] The above deviation information is used to represent the position deviation and / or angle deviation between the cameras, and the preset zoom range is the zoom ratio range corresponding to the focal length of the first camera. In this implementation manner, the influence of the first image format type and the deviation information between the cameras on the offset of the first cropping frame can be considered, so as to more accurately determine the first offset information of the non-linear offset of the first cropping frame.

[0020] In another possible implementation manner of the first aspect, when the second zoom ratio exceeds the preset zoom range, the electronic device determines the deviation information between the first camera and the second camera according to the prior information, and determines the third offset information of the offset of the first cropping frame on the first image according to the deviation information. The electronic device then determines, according to the third offset information and the format type of the first image, the first offset information of the non-linear offset of the first cropping frame on the first image in the IFE stage.

[0021] In this implementation manner, if the change in the zoom ratio exceeds the preset zoom range, it can be understood that the change in the zoom ratio is relatively large, so the influence on the first cropping frame is also relatively large. In this case, the electronic device does not need to distribute the deviation information between the cameras anymore, but can directly determine the third offset information according to the deviation information. Combining with the influence of the first image format type on the offset, the electronic device can more accurately determine the first offset information of the non-linear offset of the first cropping frame.

[0022] In another possible implementation of the first aspect, the format type of the above-mentioned first image includes a Bayer array image, and each pixel point in the Bayer array image is a red pixel R, a green pixel G, or a blue pixel B. Furthermore, the image processing method provided in this application can perform processing operations on the Bayer array image, thereby reducing the influence of the Bayer array image on the non-linear offset of the cropping frame and improving the stability and smoothness of the preview image display.

[0023] In another possible implementation of the first aspect, the first camera is an ultra-wide-angle camera, and the second camera is a wide-angle camera; or, the first camera is a wide-angle camera, and the second camera is a telephoto camera or an ultra-wide-angle camera; or, the first camera is a telephoto camera, and the second camera is a wide-angle camera.

[0024] It can be understood that the above-mentioned change in zoom ratio may include zoom ratio magnification or zoom ratio reduction. When the zoom ratio is magnified, the electronic device may have a tendency to switch from an ultra-wide-angle camera to a wide-angle camera, or directly switch to a wide-angle camera; or, the electronic device may have a tendency to switch from a wide-angle camera to a telephoto camera, or directly switch to a telephoto camera. When the zoom ratio is reduced, the electronic device may have a tendency to switch from a telephoto camera to a wide-angle camera, or directly switch to a wide-angle camera; or, the electronic device has a region to switch from a wide-angle camera to an ultra-wide-angle camera, or directly switch to an ultra-wide-angle camera.

[0025] In the above-mentioned implementation, in different scenarios of zoom ratio magnification or reduction, the electronic device can implement the above-mentioned image processing method, so that in different scenarios, a more smooth and stable preview image can be presented to the user.

[0026] In a second aspect, another electronic device is provided. The electronic device includes at least two cameras with different focal lengths, and when at a first zoom ratio, the electronic device uses the first camera, and when the zoom ratio changes, the electronic device has a tendency to switch to the second camera, or switches to the second camera. The electronic device further includes: a memory and one or more processors; the memory is coupled to the processor, and the processor is connected to at least two cameras; the memory is used to store computer programs or instructions, and the processor is used to run the computer programs or instructions. When the computer programs or instructions are executed by the processor, the electronic device is caused to execute the image processing method as described in the first aspect and any of its implementations.

[0027] In a third aspect, a chip system is provided, including an Image Signal Processor (ISP) chip, and the ISP chip is used to execute the image processing method as described in the first aspect and any of its implementations.

[0028] Fourthly, a computer-readable storage medium is provided, including computer instructions. The computer-readable storage medium stores computer instructions or programs. When the computer instructions or programs are run on a computer, the image processing method described in the first aspect and any of its implementation manners is executed.

[0029] The beneficial effects that can be achieved by the electronic device described in the second aspect, the chip system described in the third aspect, and the computer-readable storage medium described in the fourth aspect can refer to the beneficial effects that can be achieved by the first aspect and any of its implementation manners described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of a preview page provided by an embodiment of the present application;

[0031] Figure 2 A schematic diagram of an image processing process provided by an embodiment of the present application;

[0032] Figure 3 A schematic diagram of a Bayer array image provided by an embodiment of the present application;

[0033] Figure 4 A schematic diagram of the offset of the central cropping frame provided by an embodiment of the present application;

[0034] Figure 5 A structural schematic diagram of an electronic device provided by an embodiment of the present application Figure 1 ;

[0035] Figure 6 A structural schematic diagram of an electronic device provided by an embodiment of the present application Figure 2 ;

[0036] Figure 7 A flowchart of an image processing method provided by an embodiment of the present application Figure 1 ;

[0037] Figure 8 A schematic diagram of the zoom ratio range provided by an embodiment of the present application;

[0038] Figure 9 Another schematic diagram of a preview page provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art without making creative efforts based on the embodiments in the present application belong to the scope of protection of the present application.

[0040] As used herein, the term "and / or" is merely a description of the relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone.

[0041] In the description of the embodiments of the present application, terms such as "first" and "second" in the specification and claims are used to distinguish different objects, rather than to describe a specific order of the objects. For example, the first target object and the second target object are used to distinguish different target objects, rather than to describe a specific order of the target objects.

[0042] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0043] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" refers to two or more. For example, a plurality of processing units refers to two or more processing units; a plurality of systems refers to two or more systems.

[0044] Currently, some electronic devices (such as mobile phones, etc.) can be equipped with cameras (such as ultra-wide-angle cameras, wide-angle cameras, telephoto cameras, etc.), so that users can use the electronic devices to photograph different objects. Before taking a photo, the user can open the application corresponding to the camera, and the electronic device presents a preview page corresponding to the camera to the user. Taking a mobile phone as an example, as shown in (a) of Figure 1 the preview page 101 presented on the mobile phone may include a preview image 102 that is displayed in real time. When the zoom ratio (or called zoom factor, ZoomRatio value, etc.) of the mobile phone changes, the preview image 102 will also change accordingly.

[0045] In addition, the preview page 101 may also include other controls that are convenient for the user to control the camera, such as a shooting control 103, a zoom control 104, etc. The user can adjust the zoom control 104 to control the change of the zoom ratio of the mobile phone, and the mobile phone can also switch different cameras for shooting according to the change of the zoom ratio, so as to present a preview image 102 with a changing field of view on the preview page 101.

[0046] Exemplarily, referring to Figure 1As shown in (a) and (b) therein, if the user adjusts the zoom ratio to the range [0.6x, 1.0x), the mobile phone selects the ultra-wide-angle camera for shooting. Correspondingly, the field of view of the preview image becomes larger, that is, the objects in the image become smaller. If the user adjusts the zoom ratio to the range [1.0x, 3.5x), the mobile phone selects the wide-angle camera for shooting. Correspondingly, the field of view of the preview image becomes smaller, and the objects in the image become larger. If the user adjusts the zoom ratio to be greater than or equal to 3.5x (for example, the range [3.5x, 5x]), the mobile phone selects the telephoto camera for shooting. Correspondingly, the field of view of the preview image becomes even smaller, and the objects in the image become even larger.

[0047] Among them, the field of view of the ultra-wide-angle camera is greater than that of the wide-angle camera, and the field of view of the wide-angle camera is greater than that of the telephoto camera.

[0048] It can be understood that the operation of the user to adjust the zoom control 104 can be regarded as an operation of enlarging or reducing the preview image 102, or regarded as an operation of wanting to view the enlarged or reduced preview image 102.

[0049] During shooting, the electronic device can also control the camera sensor to expose the content captured by the camera to achieve the purpose of obtaining an image. After that, the electronic device can use an image signal processor (ISP) to perform a series of processes on the image output by the sensor, so as to finally display the preview image on the screen. And the processing process of the ISP can include two stages: image signal processing front end (IFE) and image signal processing post end (IPE).

[0050] Exemplarily, as shown in Figure 2 In the IFE stage, the zoom ratio of the electronic device can be obtained, and the image output by the sensor can be cropped according to the zoom ratio. Specifically, the center crop frame can be offset on the image output by the sensor according to the zoom ratio, such as enlarging or reducing the crop frame, so as to obtain the image content in the center crop frame, that is, the IFE cropped image. Among them, the center crop frame can be determined by the region of interest translator (ROITranslator) according to the current zoom ratio of the electronic device.

[0051] After that, in the IPE stage, the warped matrix determined by the spatial alignment transform (SAT) module is used to offset, rotate, etc. the central cropping frame on the above-mentioned IFE cropped image again, so as to perform enhancement operations on the quality and performance of the IFE cropped image and obtain the IPE cropped image. After that, the IPE cropped image is sent for display, so as to display a preview image on the screen.

[0052] Among them, the above SAT module can perform spatial alignment transformation processing on the sensor output images before and after switching according to the change of the current zoom ratio of the electronic device or according to the camera switching situation to obtain the warp matrix. Or, in some possible application scenarios, the SAT module can include a ROI Translator, so that the SAT module can also determine the above central cropping frame.

[0053] In the above process, in addition to the IFE stage being able to perform an offset operation on the central cropping frame, the IPE stage can also perform an offset operation on the central cropping frame. Ideally, the offsets in the IFE stage and the IPE stage are both linear offsets, so that the finally displayed preview image is stable.

[0054] In actual situations, the sensor output image obtained through the sensor is an image in a specific format, such as a Bayer array image, etc. Among them, the Bayer array image can include several 2*2 arrays and is represented as an N*N array, and each 2*2 array includes a red pixel (R), a green pixel (G), and a blue pixel (B). For example Figure 3 as shown in (a) of, the array arrangement can be represented as the first row including B, G, and the second row including G, R, or, for another example Figure 3 as shown in (b) of, the array arrangement can also be represented as the first row including G, B, and the second row including G, R.

[0055] And, the central cropping frame needs to include at least one 2*2 array and is represented as an M*M array (M≤N), and each 2*2 array needs to be the same as the smallest array in the sensor output image, for example Figure 3 the array shown in (a) of, or Figure 3 the array shown in (b) of.

[0056] In the above IFE stage, since the central cropping frame needs to meet the above array requirements, when the central cropping frame is offset on the sensor output image, the offset needs to be a multiple of 2 (representing 2 pixels). However, such an offset is not a linear offset but a non-linear offset. Correspondingly, the non-linear offset of the "central" cropping frame also causes the cropping frame not to be at the center of the sensor output image. Exemplarily, see Figure 4 as shown, where the dashed line 401 represents the linear offset of the central cropping frame under ideal conditions, and the solid line 402 represents the non-linear offset of the non-central cropping frame under actual conditions.

[0057] The non-linear offset of the above cropping frame will cause the preview image finally sent for display to jitter when the camera is switched, thus affecting the user experience.

[0058] Based on the above, the embodiments of the present application provide an image processing method. The electronic device executing this method may include at least two cameras with different focal lengths. And when at the first zoom ratio, the electronic device uses the first camera. When the zoom ratio changes, the electronic device has a tendency to switch to the second camera or switches to the second camera. The electronic device can display a first preview image corresponding to a first image at the first zoom ratio, where the first image is captured by the first camera. And the electronic device can determine a first cropping frame aligned with the center of the first image based on a second zoom ratio during the change of the zoom ratio. Then, the electronic device obtains first offset information of the non-linear offset of the first cropping frame on the first image during the image front-end processing IFE stage at the second zoom ratio, and then determines second offset information of the first cropping frame on the first image during the image back-end processing IPE stage according to the first offset information and the prior information between the first camera and the second camera. Finally, the electronic device performs a compensation operation on the non-linear offset of the first cropping frame on the first image based on the second offset information. Wherein, the compensation operation is used to make the first cropping frame linearly offset on the first image under the action of the first offset information and the second offset information. Then, the electronic device displays a second preview image corresponding to the image in the first cropping frame.

[0059] Wherein, the image in the first cropping frame changes following the linear offset of the first cropping frame on the first image during the change of the zoom ratio.

[0060] In the above method, when the zoom ratio of the electronic device changes, the electronic device can switch to use the corresponding camera or has a tendency to switch to another camera for use. Moreover, the electronic device can also perform a compensation operation on the non-linear offset in the IPE stage after the cropping frame undergoes a non-linear offset in the IFE stage. Specifically, the offset information during compensation can be determined based on the prior information between the cameras before and after the switch (or before the switch and the camera about to be switched) and the offset information during the non-linear offset. The compensation operation enables the cropping frame to finally perform a linear offset on the first image after the IFE stage and the IPE stage.

[0061] It can be understood that the above second zoom ratio can be a zoom ratio during the process of zoom ratio change. The electronic device can perform the above compensation operation for multiple zoom ratios during the process of zoom ratio change. The image in the first cropping frame changes following the offset of the first cropping frame on the first image. Then, during the process of zoom ratio change, the electronic device can continuously display preview images corresponding to different zoom ratios, thereby presenting preview images that follow the zoom ratio change. Moreover, during the process of zoom ratio change, the first cropping frame can achieve continuous linear offset on the first image, so the preview images displayed by the electronic device are also smooth and stable, with less jitter, thus improving the user experience.

[0062] Exemplarily, the electronic device switching cameras or having a tendency to switch cameras according to the zoom ratio can be expressed as: when currently using a wide-angle camera, if the change in the zoom ratio does not exceed the zoom ratio range corresponding to the focal length of the wide-angle camera, the electronic device can still use the wide-angle camera. In this case, if the zoom ratio increases, the electronic device has a tendency to switch to a telephoto camera; if the zoom ratio decreases, the electronic device has a tendency to switch to an ultra-wide-angle camera. Or, when currently using a wide-angle camera, if the zoom ratio decreases and exceeds the zoom ratio range corresponding to the focal length of the wide-angle camera, the electronic device switches to an ultra-wide-angle camera; if the zoom ratio increases and exceeds the zoom ratio range corresponding to the focal length of the wide-angle camera, the electronic device switches to a telephoto camera. Thus, in different scenarios of zoom ratio magnification or reduction, the electronic device can implement the above image processing method. Moreover, the above scenarios of displaying preview images involve scenarios such as taking pictures or recording videos using the camera, so that in different scenarios, a smoother and more stable preview image can be presented to the user.

[0063] The above-mentioned image processing method provided by the embodiments of the present application can be applied to an electronic device equipped with at least two cameras. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a digital camera, a monitoring device, a virtual reality (VR) device, an augmented reality (AR) device, a wearable device, etc. In the embodiments of the present application, there is no special limitation on the specific form of the electronic device, and it is only an exemplary illustration here.

[0064] In some embodiments, the electronic devices in the embodiments of the present application can all adopt Figure 5 the structure shown. As Figure 5 shown, the electronic device can include an IFE module 501, an IPE module 502, an ROI module 503, a SAT module 504, and a display module 505.

[0065] Among them, the ROI module 503 can obtain the zoom ratio of the electronic device and the image captured by the camera at the current zoom ratio. For example, before the zoom ratio changes, the electronic device is at the first zoom ratio and uses the first camera. At this time, the ROI module 503 can obtain the first zoom ratio and the first image captured by the first camera; after the zoom ratio changes, the electronic device is currently at the second zoom ratio. If the first camera is still used, the ROI module 503 obtains the second zoom ratio and the second image captured by the first camera. If the second camera is used, the ROI module 503 obtains the second zoom ratio and the second image captured by the second camera.

[0066] The ROI module 503 can also determine a first cropping frame on the first image captured before the change based on the second zoom ratio during the change process. It can be understood that the first cropping frame determined by the ROI module 503 this time is the first cropping frame in the initial state, and the first cropping frame is aligned with the center of the first image, or rather, the first cropping frame is at the center position of the first image.

[0067] After that, the ROI module 503 can also obtain the first offset information of the above-mentioned first cropping frame during the IFE stage and non-linearly offset on the first image at the second zoom ratio.

[0068] Among them, when determining the first offset information, the ROI module 503 can also combine the changed zoom ratio, the format type of the first image before the zoom ratio change, the prior information of the camera, etc. to determine the first offset information.

[0069] For example, the ROI module 503 obtains prior information and determines deviation information (or a rough offset) between a first camera used before the zoom ratio change and a second camera that has been switched or is about to be switched according to the prior information. Then, the ROI module 503 determines third offset information of the offset of the first cropping frame on the first image according to the deviation information and the second zoom ratio. Finally, the ROI module 503 determines first offset information of the non-linear offset of the first cropping frame on the first image in the IFE stage according to the third offset information and the format type of the first image.

[0070] The above prior information includes the distance between cameras and the calibration information of the cameras. The calibration information includes the external parameter matrix between cameras, the internal parameter matrix of the cameras, etc. Taking the above first camera and second camera as an example, the prior information may include one or more of the distance between the first camera and the second camera, the external parameter matrix between the first camera and the second camera, and the internal parameter matrix of the first camera and the second camera.

[0071] The above deviation information is used to represent the position deviation and / or angle deviation between cameras. Taking the above first camera and second camera as an example, the deviation information may include the position deviation and / or angle deviation between the first camera and the second camera. Exemplarily, the ROI module 503 may allocate the deviation information according to the second zoom ratio and adjust the first cropping frame on the first image according to the allocated deviation information, that is, the first cropping frame may perform a first offset operation on the first image, so as to obtain the offset information of the first cropping frame (i.e., the third offset information).

[0072] The above format type of the first image may be determined according to the characteristics of the image output by the camera sensor. For example, the format type of the first image may be a Bayer array image, that is, each pixel point in the image only includes a pixel of one color, such as R, G, or B. And, the smallest unit array in the image is a 2*2 array, and each 2*2 array includes pixels such as R, G, and B. The specific arrangement and quantity of the pixels are different according to the parameters or configurations of the camera sensor. For example, it may be the array shown in (a) in Figure 3 or the array shown in (b) in Figure 3 This application embodiment does not make specific limitations on this.

[0073] Since the format type of the first image is special, when the first cropping frame is offset on the first image in the IFE stage, the offset amount needs to be a multiple of 2 (indicating 2 pixels), resulting in a non-linear offset of the first cropping frame on the first image. The ROI module 503 can, on the basis of the third offset information determined above, further combine the characteristics of the first image format type to continue to determine the first offset information when the first cropping frame is non-linearly offset in the IFE stage.

[0074] The purpose of the above processing by the ROI module 503 is to determine the first offset information required during the processing of the IFE module 501 before the IFE module 501 processes. The offset operation of the first cropping frame is specifically implemented by the IFE module 501 (i.e., the IFE module 501 implements the processing process in the IEF stage). And, as can be seen from the foregoing embodiments, when the IFE module 501 processes, the first cropping frame is non-linearly offset.

[0075] After that, the IFE module 501 sends the first cropping frame and the image in the first cropping frame, etc. to the SAT module 504.

[0076] In addition, the ROI module 503 can also send the determined first offset information to the SAT module 504.

[0077] The SAT module 504 can determine the compensation offset information for compensating the offset of the first cropping frame in the IFE stage (or the processing stage of the IFE module 501) according to the offset information of the first cropping frame during the processing of the IFE module 501, the prior information between cameras, etc.

[0078] For example, the SAT module 504 determines the target offset information for linearly offsetting the first cropping frame on the first image according to the distance between the first camera and the second camera in the prior information. After that, the SAT module 504 determines the second offset information (i.e., the above compensation offset information) according to the target offset information and the first offset information.

[0079] Among them, the second offset information can be determined by subtracting the first offset information from the target offset information. That is to say, the sum of the first offset information and the second offset information (or the superposition of the first offset information and the second offset information) can be expressed as the target offset information. Exemplarily, the offset information in the embodiments of the present application can be expressed as a specific offset amount (offset), and the offset amount itself has positive and negative values.

[0080] After that, the SAT module 504 sends the determined second offset information to the IPE module 502.

[0081] Further, the SAT module 504 can also determine a first warp matrix corresponding to the first cropping frame according to the second offset information, and send the first warp matrix to the IPE module 502 in the form of the second offset information.

[0082] Among them, the SAT module 504 can obtain the second image after the change magnification changes, and perform spatial transformation processing based on the second image and the image in the first cropping frame to obtain the second warp matrix corresponding to the first cropping frame. Then, the second warp matrix is processed according to the second offset information to obtain the first warp matrix corresponding to the first cropping frame.

[0083] The IPE module 502 can perform a compensation operation on the non-linear offset of the first cropping frame on the first image (i.e., the non-linear offset of the first image in the foregoing IFE stage) based on the received second offset information. Thus, the first cropping frame performs a linear offset on the first image.

[0084] Further, the IPE module 502 can use the first warp matrix sent by the SAT module 504 to perform a compensation operation on the first cropping frame.

[0085] After that, the IPE module 502 sends the image in the first cropping frame after the compensation operation to the display module 505, and the display module 505 displays the image in the first cropping frame, so as to display a preview image on the electronic device.

[0086] It can be understood that the display module 505 can display the preview image at each zoom magnification during the zoom magnification change process, that is, it can display the first preview image corresponding to the first image captured by the first camera at the first zoom magnification, or it can also display the second preview image at the second zoom magnification.

[0087] It can be understood that in the IPE stage (or the stage processed by the IPE module 502), the image in the first cropping frame changes following the offset of the first cropping frame on the first image. Therefore, the preview image finally displayed by the display module 505 also changes, so as to realize that the preview image changes following the zoom magnification of the electronic device.

[0088] Moreover, since the non-linear offset of the first cropping frame is compensated in the IPE stage, the first cropping frame after compensation can achieve a linear offset. Therefore, the offset of the image in the first cropping frame is also linear. Furthermore, the finally generated preview image will present a relatively smooth display effect, and the occurrence of jitter is also reduced, improving the user experience.

[0089] In some other embodiments, the electronic device in the embodiments of the present application can also adopt Figure 6The hardware structure shown. As Figure 6 shown, the electronic device includes a processor 601, a transceiver 602, and a communication line 603.

[0090] Furthermore, the electronic device may further include a memory 604. Among them, the processor 601, the memory 604, and the transceiver 602 may be connected through the communication line 603.

[0091] Among them, the processor 601 is a central processing unit (CPU), a general-purpose processor, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 601 may also be other devices with processing functions, such as circuits, devices, or software modules, without limitation.

[0092] The transceiver 602 is used to communicate with other devices or other communication networks. The other communication network may be an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The transceiver 602 may be a module, a circuit, a transceiver, or any device capable of implementing communication.

[0093] The communication line 603 is used to transmit information between the components included in the electronic device. The communication line 603 may provide wireless communication solutions including 2G / 3G / 4G / 5G, wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.

[0094] The memory 604 is used to store instructions. Among them, the instructions may be computer programs.

[0095] Among them, the memory 604 can be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions, or a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions. It can also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, without limitation.

[0096] It should be noted that the memory 604 can exist independently of the processor 601 or be integrated with the processor 601. The memory 604 can be used to store instructions, program codes, or some data, etc. The memory 604 can be located inside the electronic device or outside the electronic device, without limitation. The processor 601 is used to execute the instructions stored in the memory 604 to implement the image processing method provided by the embodiments of the present application.

[0097] In one example, the processor 601 can include one or more CPUs. For example, the processor 601 includes CPU0 and CPU1.

[0098] As an alternative implementation, the electronic device includes multiple processors. For example, in addition to the processor 601, it can also include a processor 608, and the processor 608 can include CPU0 and CPU1.

[0099] As an alternative implementation, the electronic device further includes at least two image acquirers with different focal lengths, such as cameras 605, 606, 607, etc. Moreover, the focal lengths of the respective cameras are different, so as to meet the shooting requirements of the electronic device at different zoom ratios. And the processor is respectively connected to the cameras.

[0100] As another alternative implementation, the camera is also called a camera module, and the camera module includes a camera sensor. Thus, the electronic device can include a camera sensor. The electronic device can collect an image through the camera sensor and output an image after exposure to the image, so that the electronic device obtains the image collected by the camera. And the processor is respectively connected to the camera sensor.

[0101] It can be understood that the connection relationships between the modules illustrated in the embodiments of the present application are only illustrative descriptions and do not constitute a structural limitation on the electronic device. In other embodiments, the electronic device may also include more or fewer modules than those provided in the above embodiments, and different connection manners as described in the above embodiments, or a combination of multiple connection manners, may also be adopted between the various modules.

[0102] In addition, actions, terms, etc. involved between the embodiments of the present application can be referred to each other without limitation. The data names or parameter names in the data, etc. for the interaction between various devices in the embodiments of the present application are only examples, and other names may also be adopted in specific implementations without limitation. The methods in the following embodiments can all be implemented in an electronic device having the above hardware structure.

[0103] Taking the application of the above image processing method to a mobile phone as an example, the image processing method in the embodiments of the present application will be further described. In some embodiments, referring to Figure 7 as shown, the above image processing method may include the following steps S701 - S706.

[0104] S701. The mobile phone displays a first preview image corresponding to a first image at a first zoom ratio.

[0105] Among them, the mobile phone may include at least two cameras with different focal lengths. During the process of the change of the zoom ratio of the mobile phone, if the change of the zoom ratio is too large and exceeds the zoom ratio range corresponding to the focal length of the currently used first camera, then the mobile phone can switch from the current first camera to the second camera for use; if the zoom ratio does not exceed the zoom ratio range corresponding to the focal length of the currently used first camera, then the mobile phone can still use the current first camera. The above first image may be captured by the first camera.

[0106] And, the above cameras may be telephoto cameras, wide - angle cameras, ultra - wide - angle cameras, etc. Among them, referring to Figure 8 as shown, the zoom ratio range corresponding to the focal length of the wide - angle camera may be [1.0x, 3.5x), the zoom ratio range corresponding to the focal length of the telephoto camera may be [3.5x, 5x], and the zoom ratio range corresponding to the focal length of the ultra - wide - angle camera may be [0.5x, 1.0x). In the embodiments of the present application, the zoom ratio ranges of different cameras are only illustrative descriptions and do not serve as specific limitations on the zoom ratio ranges of different cameras.

[0107] In some possible implementation manners, during the process of the change in the zoom ratio, if the change in the zoom ratio of the mobile phone does not exceed the zoom ratio range corresponding to the current camera, the mobile phone may also have a tendency to switch cameras according to the change in the zoom ratio. For example, when the mobile phone is currently using the wide-angle camera, during the process of the change in the zoom ratio, if the zoom ratio changes from 1.0X to 2.0X, although the current zoom ratio of 2.0X does not exceed the zoom ratio of the wide-angle camera, since the zoom ratio has a tendency to increase, the mobile phone has a tendency to switch to the telephoto camera. For another example, when the mobile phone is currently using the wide-angle camera, during the process of the change in the zoom ratio, if the zoom ratio changes from 3.0X to 1.0X, although the current zoom ratio of 1.0X does not exceed the zoom ratio of the wide-angle camera, since the zoom ratio has a tendency to decrease, the mobile phone has a tendency to switch to the ultra-wide-angle camera.

[0108] It can be understood that the change in the zoom ratio in the embodiments of the present application is a continuous change process. The method in the embodiments of the present application can process a certain zoom ratio during the continuous change process of the zoom ratio and achieve the purpose of displaying the corresponding preview image.

[0109] For example, if the mobile phone continuously changes from the first zoom ratio to the third zoom ratio, then the second zoom ratio during the change process can be processed. Among them, the second zoom ratio can be any zoom ratio in the above continuous change process.

[0110] In some possible implementation manners, the mobile phone can respond to the user's zoom operation to change the zoom ratio. Among them, the user's zoom operation may include the user operating the zoom control on the preview page displayed on the mobile phone, or the user inputting a preset gesture on the preview page displayed on the mobile phone, etc.

[0111] Exemplarily, as shown in (a) of Figure 9 , the preview page 901 of the mobile phone includes a zoom control 902, and the user can move the zoom control 902 to control the mobile phone to change the zoom ratio. Or, as shown in (b) of Figure 9 , the user can input a gesture of zooming in or out on the preview page 901 to control the mobile phone to change the zoom ratio.

[0112] In some possible implementation manners, the camera is also called a camera module, and the camera module includes a lens, a camera sensor, etc. Among them, the lens receives the light outside the mobile phone and is received by the camera sensor. The camera sensor obtains an image (also called the sensor output image) after photoelectric conversion and other processes. Therefore, the above-mentioned mobile phone obtaining the first image collected by the first camera can also be regarded as the mobile phone obtaining the sensor output image, or the mobile phone obtaining the first image through the camera sensor.

[0113] S702. The mobile phone determines a first cropping frame aligned with the center of the first image based on the second zoom ratio during the process of zoom ratio change.

[0114] Herein, it can be understood that the first cropping frame determined at this time can also be referred to as the center cropping frame. Exemplarily, the determined center cropping frame includes not only specific position information but also information such as resolution. Moreover, the position, resolution, etc. of the center cropping frame can be determined according to the current zoom ratio. For example, during the process of the zoom ratio of the mobile phone changing, if the current zoom ratio is the second zoom ratio, the center cropping frame can be determined based on the second zoom ratio.

[0115] The first cropping frame determined in this step S702 can be regarded as the first cropping frame in the initial state, and the first cropping frame is aligned with the center of the first image.

[0116] The above-mentioned first cropping frame determined in the embodiments of the present application is used to crop the image collected by the camera, so as to send the image in the first cropping frame for display, that is, to generate a preview image displayed on the mobile phone by using the image in the first cropping frame.

[0117] It can be understood that during the process of the zoom ratio of the mobile phone changing, due to the continuous change of the zoom ratio, the first cropping frame also continuously shifts on the first image, the image in the first cropping frame also continuously changes, and thus the preview image displayed on the mobile phone also continuously changes. Moreover, the shift of the first cropping frame here refers to the shift of the first cropping frame relative to the first cropping frame in the initial state during the process of zoom ratio change.

[0118] Referring to the content of the foregoing embodiments, the shift of the first cropping frame on the first image includes two stages, that is, the shift in the IFE stage and the shift in the IPE stage. Exemplarily, the following step S703 can determine the shift in the IFE stage.

[0119] S703. The mobile phone obtains first offset information of the non-linear offset of the first cropping frame on the first image in the IFE stage at the second zoom ratio.

[0120] In some possible implementation manners, when the mobile phone determines the first offset information, it can further be determined according to the change situation of the zoom ratio. Among them, the change situation of the zoom ratio can include: the zoom ratio changes within the zoom ratio range corresponding to the focal length of the first camera currently in use, the change of the zoom ratio exceeds the zoom ratio range corresponding to the focal length of the first camera currently in use, etc.

[0121] Exemplarily, the zoom ratio varies within the zoom ratio range corresponding to the focal length of the first camera currently in use, which may include that both the first zoom ratio and the second zoom ratio are within a preset zoom range. Among them, if the first camera is a telephoto camera, the preset zoom range is the zoom ratio range corresponding to the telephoto camera; if the first camera is a wide-angle camera, the preset zoom range is the zoom ratio range corresponding to the wide-angle camera; if the first camera is an ultra-wide-angle camera, the preset zoom range is the zoom ratio range corresponding to the ultra-wide-angle camera.

[0122] When both the first zoom ratio and the second zoom ratio are within the preset zoom range, the mobile phone can determine the deviation information between the first camera and the second camera according to prior information. Then, according to the deviation information and the second zoom ratio, the mobile phone determines the third offset information of the offset of the first crop box on the first image, that is, the mobile phone distributes the deviation information to the first crop box according to the second zoom ratio, where the offset information distributed to the first crop box is the third offset information. After that, the mobile phone determines the first offset information of the non-linear offset of the first crop box on the first image according to the third offset information and the format type of the first image.

[0123] That is to say, the offset of the first crop box in the IFE stage is not only affected by the format type of the first image, but also affected by the deviation information between the cameras. In the embodiments of the present application, these two aspects of influence can be considered, so as to more accurately determine the first offset information of the non-linear offset of the first crop box.

[0124] Exemplarily again, the change in the zoom ratio exceeds the zoom ratio range corresponding to the focal length of the first camera currently in use, which may include that the first zoom ratio is within the preset zoom range while the second zoom ratio exceeds the preset zoom range.

[0125] When the second zoom ratio exceeds the preset zoom range, the mobile phone can determine the deviation information between the first camera and the second camera according to prior information. Then, according to the deviation information, the mobile phone determines the third offset information of the offset of the first crop box on the first image, for example, taking the deviation information as the third offset information. After that, the mobile phone determines the first offset information of the non-linear offset of the first crop box on the first image according to the third offset information and the format type of the first image.

[0126] It can be seen that if the change in the zoom ratio exceeds the zoom ratio range corresponding to the currently used first camera, it can be understood that the change in the zoom ratio is relatively large. Therefore, the impact on the first cropping frame is also relatively large. In this case, the mobile phone does not need to allocate the deviation information between the cameras again, but can directly determine the third offset information according to the deviation information. Combining the influence of the first image format type on the offset, the mobile phone can more accurately determine the first offset information for the non-linear offset of the first cropping frame.

[0127] In some possible implementation manners, the format type of the above first image is determined by the configuration or parameters of the camera sensor. For example, the format type of the first image may be a Bayer array image. Furthermore, the image processing method in the embodiments of the present application can perform processing operations on the Bayer array image, thereby reducing the influence of the Bayer array image on the non-linear offset of the cropping frame and improving the stability and smoothness of the preview image display.

[0128] In some possible implementation manners, the prior information between the above first camera and the second camera may include one or more of the following: the distance between the first camera and the second camera, the external parameter matrix between the first camera and the second camera, and the internal parameter matrices of the first camera and the second camera.

[0129] Exemplarily, the following step S704 can determine the offset in the IPE stage.

[0130] S704. The mobile phone determines, according to the first offset information and the prior information between the first camera and the second camera, the second offset information of the first cropping frame on the first image in the IPE stage.

[0131] In some possible implementation manners, the mobile phone can determine the target offset information for the linear offset of the first cropping frame on the first image according to the distance between the first camera and the second camera in the prior information. Then, according to the target offset information and the first offset information, the second offset information corresponding to the first cropping frame in the IPE stage is determined.

[0132] Exemplarily, the above target offset information may represent the predicted information in the ideal case, that is, the offset condition or offset amount that needs to be satisfied when the first cropping frame performs a linear offset on the first image. The first offset information may represent the actual offset amount of the first cropping frame on the first image, that is, the offset amount during the non-linear offset.

[0133] Then, in order to enable the first cropping frame to meet the conditions of the linear offset, the mobile phone can determine how much additional offset is required for the first cropping frame to meet the linear offset on the premise that the first cropping frame already has the first offset information, that is, determine the second offset information.

[0134] Exemplarily, due to the particularity of the first image format, the first offset information is usually a multiple of 2 (indicating 2 pixels). Thus, the change of the first offset information will present a stepped change form. Correspondingly, in order to make the final offset of the first cropping frame satisfy linear offset, the change of the second offset information should also present a stepped change form, so as to ensure that after the second offset information is superimposed on the first offset information (or it can be said that the non-linear offset is compensated by using the second offset information), the total offset of the first cropping frame presents a linear change, that is, the linear offset of the first cropping frame is realized.

[0135] It can be understood that the target offset information can also be expressed as the target offset information at each zoom ratio, and moreover, the target offset information changes linearly with the change of the zoom ratio.

[0136] Moreover, further, when the offset information is a specific offset value (offset), the mobile phone can subtract the first offset information from the target offset information to obtain the second offset information.

[0137] In the above implementation manner, the mobile phone can, on the basis that the first cropping frame has undergone non-linear offset, further determine the second offset information required to make the first cropping frame perform linear offset, so that subsequently (such as in step S704), the first cropping frame is offset again (or it can be called a compensation operation) according to the second offset information, so that the first cropping frame finally realizes linear offset after passing through the IEF stage and the IPE stage.

[0138] S705. The mobile phone, based on the second offset information, performs a compensation operation on the non-linear offset of the first cropping frame on the first image.

[0139] Among them, the compensation operation is used to make the first cropping frame perform linear offset on the first image under the action of the first offset information and the second offset information. That is to say, after the first cropping frame performs non-linear offset on the first image according to the first offset information, the mobile phone then uses the second offset information to offset the first cropping frame on the first image, and the offsets of the two times are superimposed, so that the final offset of the first cropping frame is a linear offset, thus realizing the compensation for the non-linear offset in the previous stage.

[0140] In some possible implementation manners, during the above compensation operation, the mobile phone can use the warp matrix corresponding to the second offset information to realize the offset operation of the first cropping frame.

[0141] Exemplarily, the mobile phone can determine the first warp matrix corresponding to the first cropping frame according to the second offset information. Then, based on the first warp matrix, the offset operation is performed on the non-linear offset of the first cropping frame on the first image.

[0142] Further, when determining the first warp matrix, the mobile phone can obtain a second image captured at a second zoom ratio. The second image can be captured by the first camera or the second camera, and the embodiments of the present application do not make specific limitations in this regard.

[0143] After that, the mobile phone performs a spatial transformation process on the basis of the second image and the image in the first cropping frame to obtain a second warp matrix corresponding to the first cropping frame. Specifically, the mobile phone can perform a spatial transformation process on the image in the first cropping frame based on the second image to obtain the second warp matrix.

[0144] Finally, the mobile phone determines the first warp matrix corresponding to the first cropping frame according to the second offset information and the second warp matrix. That is to say, the mobile phone can superimpose the second offset information on the basis of the second warp matrix, or perform an offset operation on the second warp matrix based on the second offset information to obtain the first warp matrix.

[0145] In the above implementation, during the compensation operation, the mobile phone can specifically use the warp matrix to offset the first cropping frame again, so that after the two offset operations are superimposed, the first cropping frame finally realizes a linear offset.

[0146] S706. The mobile phone displays a second preview image corresponding to the image in the first cropping frame.

[0147] It can be understood that the above steps S701-S706 can be used to determine a corresponding first cropping frame for a zoom ratio during the zoom ratio change process and perform a compensation operation on the first cropping frame. After the compensation operation, the mobile phone obtains the image in the first cropping frame and displays the image, so as to display the preview image corresponding to the current zoom ratio.

[0148] In the embodiments of the present application, the mobile phone can perform the above compensation operation for each zoom ratio or preset zoom ratio during the zoom ratio change process. During the zoom ratio change process, the image in the first cropping frame changes following the offset of the first cropping frame in the first image. Then, during the zoom ratio change process, the mobile phone can continuously display preview images corresponding to different zoom ratios, so as to present a preview image that follows the zoom ratio change. Moreover, during the zoom ratio change process, the first cropping frame can realize continuous linear offset on the first image, so the preview image displayed by the mobile phone is also smooth and stable, and the jitter situation is reduced, thereby improving the user experience.

[0149] In the above embodiments, the electronic device is taken as a mobile phone as an example to describe the image processing method in the embodiments of the present application. In some other embodiments, when the electronic device is a tablet computer, a digital camera, a monitoring device, a VR device, an AR device, or a wearable device, the electronic device can also execute the method steps such as the aforementioned S701-S706, which will not be elaborated herein in the present application.

[0150] The embodiments of the present application further provide a chip system, which is applied to the above-mentioned electronic device. The chip system includes at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected through a line. For example, the interface circuit can be used to receive signals from other devices (such as a memory). For another example, the interface circuit can be used to send signals to other devices (such as a processor). Exemplarily, the interface circuit can read the instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can execute each of the method steps in the above embodiments. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiments of the present application.

[0151] In some other embodiments of the present application, another chip system is further provided, which can include an ISP chip, and the ISP chip can execute each of the method steps in the above embodiments.

[0152] The embodiments of the present application further provide a computer-readable storage medium, which includes computer instructions. When the computer instructions run on the above-mentioned electronic device, the electronic device can execute each of the method steps in the above embodiments.

[0153] The embodiments of the present application further provide a computer program product. When the computer program product runs on a computer, the computer can execute each of the method steps in the above embodiments. For example, the computer can be the above-mentioned electronic device.

[0154] Through the description of the above implementation manners, those skilled in the art can clearly understand that for the convenience and conciseness of description, only the above division of each functional module is used as an example for illustration. In practical applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0155] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections between each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0156] The units described as separate components may or may not be physically separated. The components displayed as units can be one physical unit or multiple physical units, that is, they can be located in one place, or they can be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0157] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0158] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read only memory (ROM), random access memory (RAM), magnetic disks or optical discs and other various media that can store program codes.

[0159] The above content is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that, Applied to an electronic device, the electronic device includes at least two cameras with different focal lengths, and when at the first zoom ratio, the electronic device uses the first camera. After the zoom ratio changes, the electronic device has a tendency to switch to the second camera or switches to the second camera; the method includes: At the first zoom ratio, display a first preview image corresponding to the first image; the first image is captured by the first camera; Based on a second zoom ratio during the change of the zoom ratio, determine a first cropping frame aligned with the center of the first image; Obtain first offset information of non-linear offset of the first cropping frame on the first image during the image front-end processing IFE stage at the second zoom ratio; According to the first offset information and the prior information between the first camera and the second camera, determine second offset information of the first cropping frame on the first image during the image back-end processing IPE stage; Based on the second offset information, perform a compensation operation on the non-linear offset of the first cropping frame on the first image; the compensation operation is used to make the first cropping frame linearly offset on the first image under the action of the first offset information and the second offset information; Display a second preview image corresponding to the image in the first cropping frame; the image in the first cropping frame changes following the linear offset of the first cropping frame on the first image during the change of the zoom ratio.

2. The method according to claim 1, characterized in that, The step of determining the second offset information of the first cropping frame on the first image during the image back-end processing IPE stage according to the first offset information and the prior information between the first camera and the second camera includes: According to the distance between the first camera and the second camera in the prior information, determine the target offset information of linear offset of the first cropping frame on the first image; According to the target offset information and the first offset information, determine the second offset information corresponding to the first cropping frame during the IPE stage.

3. The method according to claim 2 or 3, characterized in that, During the change of the zoom ratio, the first offset information shows a stepped change, and the second offset information shows a stepped change.

4. The method according to any one of claims 1 to 3, characterized in that, The step of performing a compensation operation on the non-linear offset of the first cropping frame on the first image based on the second offset information includes: According to the second offset information, determine a first warp matrix corresponding to the first cropping frame; Based on the first warp matrix, perform an offset operation on the non-linear offset of the first cropping frame on the first image.

5. The method according to claim 4, wherein The step of determining a first warp matrix corresponding to the first cropping frame according to the second offset information includes: Obtain a second image captured by the electronic device at the second zoom ratio; the second image is captured by the first camera or the second camera; Based on the second image and the image in the first cropping frame, perform a spatial transformation process to obtain a second warp matrix corresponding to the first cropping frame; Determine the first warp matrix corresponding to the first cropping frame according to the second offset information and the second warp matrix.

6. The method according to any one of claims 1-5, characterized in that, The prior information between the first camera and the second camera includes one or more of the following: the distance between the first camera and the second camera, the external parameter matrix between the first camera and the second camera, and the internal parameter matrices of the first camera and the second camera.

7. The method according to any one of claims 1-6, characterized in that, The obtaining of the first offset information of the first cropping frame performing non-linear offset on the first image in the IFE stage at the second zoom ratio includes: When both the first zoom ratio and the second zoom ratio are within a preset zoom range, Determine the deviation information between the first camera and the second camera according to the prior information; the deviation information is used to represent the position deviation and / or angle deviation existing between the cameras; the preset zoom range is the zoom ratio range corresponding to the focal length of the first camera; Allocate the deviation information according to the second zoom ratio to obtain the third offset information of the offset of the first cropping frame on the first image; Determine the first offset information of the first cropping frame performing non-linear offset on the first image in the IFE stage according to the third offset information and the format type of the first image.

8. The method according to any one of claims 1 to 6, characterized in that The obtaining of the first offset information of the first cropping frame performing non-linear offset on the first image in the IFE stage at the second zoom ratio includes: When the second zoom ratio exceeds the preset zoom range, Determine the deviation information between the first camera and the second camera according to the prior information; the deviation information is used to represent the position deviation and / or angle deviation existing between the cameras; the preset zoom range is the zoom ratio range corresponding to the focal length of the first camera; Determine the third offset information of the offset of the first cropping frame on the first image according to the deviation information; Determine the first offset information of the first cropping frame performing non-linear offset on the first image in the IFE stage according to the third offset information and the format type of the first image.

9. The method according to claim 7 or 8, characterized in that, The format type of the first image includes a Bayer array image, and each pixel point in the Bayer array image is a red pixel R, a green pixel G, or a blue pixel B.

10. The method according to any one of claims 1-9, characterized in that, The first camera is an ultra-wide-angle camera, and the second camera is a wide-angle camera; or, the first camera is a wide-angle camera, and the second camera is a telephoto camera or an ultra-wide-angle camera; or, the first camera is a telephoto camera, and the second camera is a wide-angle camera.

11. An electronic device, characterized in that, It includes at least two cameras with different focal lengths. When at the first zoom ratio, the electronic device uses the first camera. After the zoom ratio changes, the electronic device has a tendency to switch to the second camera or switches to the second camera. It also includes: a memory and one or more processors; the memory is coupled to the processor, and the processor is connected to the at least two cameras; the memory is used to store computer programs or instructions; the processor is used to run the computer programs or instructions. When the computer programs or instructions are executed by the processor, the electronic device executes the image processing method described in any one of claims 1-10.

12. A chip system, characterized in that, It includes an Image Signal Processor (ISP) chip, and the ISP chip is used to execute the image processing method described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, It includes computer instructions. The computer-readable storage medium stores computer instructions or programs. When the computer instructions or programs run on a computer, the image processing method described in any one of claims 1-10 is executed.

Images