Image processing method and related apparatus thereof

Abstract

The application relates to the field of image processing, and provides an image processing method and related equipment thereof, which is applied to an electronic device including a first camera and a second camera. The method comprises the following steps: starting a camera application program; displaying a first image, which is obtained by collecting an image by the first camera; receiving a first zooming operation; displaying and saving a second image, which is obtained by correcting an image collected by the first camera by using an offset between the image collected by the first camera and an image collected by the second camera, wherein the offset is obtained by performing direction filtering on a feature point pair matched by feature points of the image collected by the first camera and feature points of the image collected by the second camera. The application can realize smooth zooming and smooth switching of the camera by performing direction filtering.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to an image processing method and a related device thereof. BACKGROUND

[0002] With the development of the shooting function in electronic devices, camera applications are increasingly widely used in electronic devices. In order to obtain a better shooting experience, multiple cameras are usually provided on the current electronic devices, and the focal lengths of the cameras are different.

[0003] In shooting, in response to the operation of a user, the electronic device can switch the cameras with different focal lengths to perform zoom shooting, and can also process the captured image in combination with the digital zoom mode to meet various high-magnification shooting scenarios. However, in the zooming process, due to the different depths of the shooting objects, such as the influence of the foreground and the background, there is a jump in the imaging content and size between the switching of different cameras, and the cameras cannot be smoothly switched. When digital zooming, only the imaging content at the center of the field of view can be enlarged and cropped, which is very limited.

[0004] Therefore, how to smoothly switch the cameras during shooting has become a problem to be solved. SUMMARY

[0005] The present application provides an image processing method and a related device thereof, which can achieve smooth zooming and smooth switching of cameras by performing direction filtering.

[0006] In a first aspect, an image processing method is provided, applied to an electronic device including a first camera and a second camera, and the method includes: starting a camera application; displaying a first image, the first image being obtained by the first camera capturing an image; receiving a first zooming operation; displaying and saving a second image, the second image being obtained by correcting the image captured by the first camera using an offset between the image captured by the first camera and the image captured by the second camera, the offset being obtained by performing direction filtering on a feature point pair matched by a feature point of the image captured by the first camera and a feature point of the image captured by the second camera.

[0007] In the embodiments of this application, feature point matching is performed on the images captured by the first camera and the images captured by the second camera. Then, directional filtering is applied to the matched feature point pairs to obtain the offset between the two images. This offset is then used to correct the image captured by the first camera, and the corrected second image is displayed. By performing directional filtering during the offset calculation process, this application can filter a large number of feature point pairs, retaining those with consistent characteristics (e.g., consistent direction or slope). This makes the feature point pairs used for offset calculation more stable, resulting in a more stable FOV for the corrected image, ensuring smooth zooming and smooth switching during camera switching.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: receiving a second zoom operation; displaying and saving a third image, the third image being obtained from an image captured by the second camera.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: acquiring a first initial image captured by the first camera and a second initial image captured by the second camera; performing feature point detection and registration on the first initial image and the second initial image to obtain multiple matching feature point pairs; calculating a vector formed by each pair of feature point pairs; determining the slope of each vector; dividing all feature point pairs into multiple point sets according to the slope of each vector; determining the offset based on multiple feature point pairs in the point set that includes the largest number of feature point pairs; and correcting the first initial image based on the offset to obtain the second image.

[0010] In the embodiments of this application, feature point detection and registration are performed on the images captured by the first camera and the images captured by the second camera. Then, directional filtering is applied to the matched feature point pairs to obtain the offset between the two images. This offset is then used to correct the image captured by the first camera, and the corrected second image is then displayed. By performing directional filtering during the offset calculation process, this application can filter a large number of feature point pairs, retaining those with consistent characteristics (e.g., consistent direction or slope). This makes the feature point pairs used for offset calculation more stable, resulting in a more stable FOV for the corrected image, ensuring smooth zooming and smooth switching during camera switching.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes, before calculating the vector formed by each pair of said feature points, removing feature point pairs that are mismatched.

[0012] In the embodiments of this application, after testing, the number of correctly registered feature points is greater than the number of incorrectly registered feature points in each scenario. Therefore, in order to improve processing efficiency and the effect of subsequent processing, the incorrectly registered feature points can be removed, and only the correctly registered feature point pairs can be retained.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, based on the slope of each vector, all feature point pairs are divided into multiple point sets, including: sorting the slopes corresponding to all feature point pairs and setting an index; calculating the difference between the slope corresponding to the current index feature point pair and the average slope corresponding to all previous index feature point pairs according to the index order; if the difference is too large, the current index feature point pair is recorded as a feature point pair with a large difference, and the previous set of feature point pairs recorded as having a large difference, up to the last set of feature point pairs recorded as having a large difference, and multiple feature point pairs between the two sets of feature point pairs are divided into a point set.

[0014] In this embodiment, feature point detection and registration are performed on two frames of images captured by the first and second cameras. Then, the slope of the vector formed by each pair of feature points is used to divide all feature point pairs into multiple point sets. The offset is then determined according to the point set containing the most feature point pairs. For two frames of images, the point set containing the most feature point pairs has a more consistent feature point pair representation, making the determined offset more representative. For the two image streams acquired by the first and second cameras, the point set containing the most feature point pairs is relatively stable across multiple consecutive frames. Therefore, when offsetting the two image streams, the offset amplitude and direction are more consistent, ensuring that the FOV of the image remains consistent during zooming and camera switching, thus guaranteeing a smooth transition.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: responding to the first zoom operation; determining that the zoom ratio of the electronic device satisfies the zoom ratio range of the first camera foreground display and the second camera for background operation.

[0016] It should be understood that "foreground display" refers to the image captured by the camera being used for display; "background operation" refers to the image captured by the camera being used for image processing, not for display.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: responding to the second zoom operation; determining that the zoom ratio of the electronic device satisfies the zoom ratio range of the first camera running in the background or being turned off, while the second camera is displayed in the foreground.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, the first zoom operation or the second zoom operation includes: a two-finger reverse swipe operation, a swipe operation on the displayed zoom control, a voice operation, or an air gesture operation.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the first camera includes a wide-angle camera, and / or the second camera includes a telephoto camera or an ultra-wide-angle camera.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, the first zoom operation and the second zoom operation are consecutive zoom operations.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, during the first zoom operation, when the difference between the current zoom ratio and the target zoom ratio is equal to or less than a preset value, the offset remains unchanged and is equal to a first offset; wherein, the target zoom ratio is used to indicate the zoom ratio at which the display is switched from the first camera to the second camera; and the first offset is used to indicate the last determined offset when the difference between the current zoom ratio and the target zoom ratio is greater than the preset value.

[0022] In this embodiment of the application, the offset is stabilized at the zoom level corresponding to the near camera switching point (e.g., if the offset remains unchanged, the last offset determined previously is used). Through the above method, the offset change can be guaranteed to be linear and stable, thereby making the image change smoothly and avoiding abnormalities such as image jitter and jumps.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: determining the parameter value corresponding to the current frame by using the offset between the image captured by the first camera and the image captured by the second camera, and all previous offsets; determining the difference between the parameter value corresponding to the current frame and the parameter value corresponding to the previous frame; if it is greater than a threshold, then using the offset corresponding to the previous frame as the offset corresponding to the current frame.

[0024] In this embodiment, at the zoom level corresponding to the near camera switching point, each time an offset is determined, a parameter value is calculated using this offset and all previous offsets. Abnormal offsets are filtered out based on the magnitude of the parameter value, while stable offsets are retained. This method ensures that the offset change is linear and stable, resulting in smooth image changes and avoiding image jitter, jumps, or other anomalies.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: if the difference between three consecutive frames is greater than the threshold, then clear all offsets and re-determine.

[0026] In conjunction with the first aspect, in some implementations of the first aspect, the parameter values ​​include the mean and / or variance of the offset.

[0027] In the embodiments of this application, the mean and variance included in the parameter values ​​can be used to represent the stability of all offsets.

[0028] In a second aspect, an electronic device is provided, comprising: one or more processors, a memory, a first camera, and a second camera; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, and the one or more processors calling the computer instructions to cause the electronic device to execute:

[0029] Open the camera application; display the first image, which is the image captured by the first camera; receive the first zoom operation; display and save the second image, which is the image captured by the first camera after correction using the offset between the image captured by the first camera and the image captured by the second camera. The offset is obtained by performing directional filtering on the feature point pairs that match the feature points of the image captured by the first camera and the feature points of the image captured by the second camera.

[0030] It should be understood that the extensions, limitations, explanations and descriptions of the relevant content in the first aspect above also apply to the same content in the second aspect.

[0031] Thirdly, a chip system is provided, the chip system being applied to an electronic device, the chip system including one or more processors, the processors being configured to invoke computer instructions to cause the electronic device to perform any of the image processing methods in the first aspect.

[0032] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing computer program code, which, when executed by an electronic device, causes the electronic device to perform any of the image processing methods in the first aspect.

[0033] Fifthly, a computer program product is provided, the computer program product comprising: computer program code, which, when executed by an electronic device, causes the electronic device to perform any of the image processing methods in the first aspect. Attached Figure Description

[0034] Figure 1This is a schematic diagram of a hardware system for an electronic device applicable to this application;

[0035] Figure 2 This is a schematic diagram of a software system applicable to an electronic device of this application;

[0036] Figure 3 This is a schematic diagram of the arrangement of multiple cameras on an electronic device according to an embodiment of this application;

[0037] Figure 4 This is a schematic diagram illustrating the zoom ratio corresponding to different types of cameras provided in this application embodiment;

[0038] Figure 5 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0039] Figure 6 This is a schematic diagram of images captured by the main camera and the telephoto camera provided in the embodiments of this application;

[0040] Figure 7 This is a set of schematic diagrams of image streams involved in existing technologies;

[0041] Figure 8 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0042] Figure 9 This is a schematic flowchart of another image processing method provided in the embodiments of this application;

[0043] Figure 10 This is a schematic flowchart of another image processing method provided in the embodiments of this application;

[0044] Figure 11 This is a set of schematic diagrams of image streams involved in the embodiments of this application;

[0045] Figure 12 This is a schematic diagram illustrating the relationship between zoom ratio and offset in the embodiments of this application;

[0046] Figure 13 It is a set of image streams involved in existing technology;

[0047] Figure 14 This is a schematic flowchart of another image processing method provided in the embodiments of this application;

[0048] Figure 15 This refers to a set of image streams involved in the embodiments of this application;

[0049] Figure 16 This is a schematic flowchart of another image processing method provided in the embodiments of this application;

[0050] Figure 17 This is a schematic diagram illustrating an application scenario for zoom switching provided in an embodiment of this application;

[0051] Figure 18 This is a schematic diagram illustrating an application scenario for zoom switching provided in an embodiment of this application;

[0052] Figure 19 This is a schematic diagram of another electronic device provided in an embodiment of this application;

[0053] Figure 20 This is a schematic diagram of an electronic device applicable to this application. Detailed Implementation

[0054] In the embodiments of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0055] First, some terms used in the embodiments of this application will be explained to facilitate understanding by those skilled in the art.

[0056] 1. Field of View (FOV): In optical instruments, the FOV is the angle between the two edges of the lens, representing the maximum range through which the image of the target object can pass through the lens. The size of the FOV determines the field of view of the optical instrument. The larger the FOV, the wider the field of view, but the lower the optical magnification. In other words, objects beyond this angle will not be captured by the lens. A shorter focal length results in a wider horizontal field of view, and thus a smaller image. The horizontal field of view narrows as the focal length increases, while the size of the object being photographed increases accordingly.

[0057] 2. Registration refers to the matching of geographic coordinates between different images obtained by different imaging methods within the same region. This includes three aspects: geometric correction, projection transformation, and ensuring the images are displayed at the same scale.

[0058] 3. Zoom ratio: Zoom ratio is used to indicate the zoom level of an electronic device when shooting.

[0059] The above is a brief introduction to the terms used in the embodiments of this application, and will not be repeated below.

[0060] The hardware system, software system, and application scenarios of the electronic device provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0061] For example, electronic device 100 may be a mobile phone, smart screen, tablet computer, wearable electronic device, in-vehicle electronic device, augmented reality (AR) device, virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), projector, etc. This application embodiment does not limit the specific type of electronic device 100.

[0062] See Figure 1 The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0063] It should be noted that, Figure 1 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include... Figure 1 The components shown may include more or fewer components, or the electronic device 100 may include... Figure 1 The components shown may be a combination of certain components, or the electronic device 100 may include... Figure 1 Sub-components of some of the components shown. Figure 1 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0064] Processor 110 may include one or more processing units. For example, processor 110 may include at least one of the following processing units: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and neural network processing unit (NPU). These different processing units may be independent devices or integrated devices.

[0065] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0066] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0067] For example, the processor 110 can be used to execute the image processing method of the embodiments of this application; for example, opening a camera application; displaying a first image, the first image being obtained by capturing an image from a first camera; receiving a first zoom operation; displaying and saving a second image, the second image being obtained by correcting the image captured by the first camera using an offset between the image captured by the first camera and the image captured by the second camera, the offset being obtained by performing directional filtering on feature point pairs that match the feature points of the image captured by the first camera and the feature points of the image captured by the second camera.

[0068] Figure 1 The connection relationships between the modules shown are merely illustrative and do not constitute a limitation on the connection relationships between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of various connection methods described in the above embodiments.

[0069] The wireless communication function of electronic device 100 can be realized through devices such as antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0070] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0071] Electronic device 100 can implement display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0072] Display screen 194 can be used to display images or videos.

[0073] For example, in an embodiment of this application, the display screen 194 may be used to display a second image.

[0074] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0075] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can perform algorithmic optimization of image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0076] The camera 193 (also known as a lens) is used to capture still images or videos. It can be activated via application commands to enable photo-taking, such as capturing images of any scene. The camera may include components such as an imaging lens, filters, and an image sensor. Light emitted or reflected by an object enters the imaging lens, passes through the filter, and is finally focused onto the image sensor. The imaging lens is primarily used to focus and image the light emitted or reflected by all objects within the shooting field of view (also known as the scene to be captured, the target scene, or the scene image the user expects to capture). The filter is primarily used to filter out excess light waves (such as infrared light waves other than visible light). The image sensor can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The image sensor is primarily used to perform photoelectric conversion on the received light signal, converting it into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into standard RGB, YUV, and other image signal formats.

[0077] For example, the gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, y-axis, and z-axis). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the shake of the electronic device 100, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 100 by moving in the opposite direction, thus achieving image stabilization. The gyroscope sensor 180B can also be used in scenarios such as navigation and motion-sensing games.

[0078] In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0079] The camera 193 can be located in front of the electronic device 100 or on the back of the electronic device 100. The specific number and arrangement of the cameras can be set according to the requirements, and this application does not impose any restrictions.

[0080] For example, the electronic device 100 includes a front-facing camera and a rear-facing camera. For instance, either the front-facing camera or the rear-facing camera may include one or more cameras. Taking an electronic device 100 with four rear-facing cameras as an example, when the electronic device 100 activates the four rear-facing cameras to take pictures, it can use the image processing method provided in the embodiments of this application.

[0081] Alternatively, the camera can be mounted on an external accessory of the electronic device 100, which is rotatably connected to the frame of the phone. The angle formed between the external accessory and the display screen 194 of the electronic device 100 can be any angle between 0 and 360 degrees. For example, when the electronic device 100 takes a selfie, the external accessory rotates the camera to face the user. Of course, when the phone has multiple cameras, only some cameras can be mounted on the external accessory, while the remaining cameras are mounted on the main body of the electronic device 100. This application embodiment does not impose any restrictions on this.

[0082] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0083] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, and MPEG 4.

[0084] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, y-axis, and z-axis). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the shake of the electronic device 100, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 100 by moving in the opposite direction, thus achieving image stabilization. The gyroscope sensor 180B can also be used in scenarios such as navigation and motion-sensing games.

[0085] The accelerometer 180E can detect the magnitude of acceleration of the electronic device 100 in various directions (typically the x-axis, y-axis, and z-axis). When the electronic device 100 is stationary, it can detect the magnitude and direction of gravity. The accelerometer 180E can also be used to identify the attitude of the electronic device 100, serving as input parameters for applications such as screen orientation switching and pedometers.

[0086] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some embodiments, such as in a shooting scenario, the electronic device 100 can utilize the distance sensor 180F to measure distance for fast focusing.

[0087] The ambient light sensor 180L is used to sense the brightness of ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0088] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to perform functions such as unlocking, accessing application locks, taking photos, and answering calls.

[0089] Touch sensor 180K, also known as a touch device, can be disposed on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a touch screen. Touch sensor 180K is used to detect touch operations applied to or near it. Touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be disposed on the surface of electronic device 100, and in a different location from display screen 194.

[0090] The hardware system of electronic device 100 has been described in detail above. The software system of electronic device 100 will be introduced below.

[0091] Figure 2 This is a schematic diagram of the software system of the electronic device provided in the embodiments of this application.

[0092] like Figure 2 As shown, the system architecture may include an application layer 210, an application framework layer 220, a hardware abstraction layer 230, a driver layer 240, and a hardware layer 250.

[0093] Application layer 210 may include applications such as camera application, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0094] The application framework layer 220 provides application programming interfaces (APIs) and programming frameworks for applications in the application layer; the application framework layer may include some predefined functions.

[0095] For example, the application framework layer 220 may include a camera access interface; the camera access interface may include camera management and camera devices. Specifically, camera management can be used to provide an access interface for managing cameras; camera devices can be used to provide an interface for accessing cameras.

[0096] The hardware abstraction layer 230 is used to abstract hardware. For example, the hardware abstraction layer can encompass the camera abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer can call camera algorithms.

[0097] For example, the hardware abstraction layer 230 includes a camera hardware abstraction layer and a camera algorithm; the camera algorithm may include software algorithms for image processing.

[0098] For example, the camera algorithm library may include algorithms corresponding to the image processing methods provided in the embodiments of this application.

[0099] For example, the algorithm in a camera algorithm can refer to something that does not depend on specific hardware implementation; for example, code that can typically run on a CPU.

[0100] The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include a camera driver.

[0101] Hardware layer 250 is located at the lowest level of the operating system; such as Figure 2 As shown, hardware layer 250 may include camera 1, camera 2, camera 3, etc. Camera 1, camera 2, and camera 3 may correspond to multiple cameras on an electronic device.

[0102] For ease of understanding, the following description will take a mobile phone with the above-mentioned hardware and software structure as an example, and will first describe in detail the camera on the electronic device 100 to which the method provided in this application embodiment is applicable.

[0103] The electronic device to which the method provided in this application is applicable has at least a plurality of cameras 193, for example, three types of cameras 193; the three types of cameras are a main camera (e.g., a wide-angle camera), an ultra-wide-angle camera, and a telephoto camera; the three cameras can be used to capture the same scene to be captured.

[0104] Optionally, the electronic device 100 may also have other cameras 193. The type of camera 193 and the number of each type of camera 193 can be set as needed, and this application embodiment does not impose any restrictions on this.

[0105] For example, such as Figure 3 The illustration uses an electronic device 100 with three cameras 193 as an example; the arrangement of the three cameras can be as follows: Figure 3 As shown in (a) or as in Figure 3 As shown in (b); for example, the three cameras 193 can be a main camera 1931 (e.g., a wide-angle camera), an ultra-wide-angle camera 1932, and a telephoto camera 1933.

[0106] It should be understood that the above are merely examples of two arrangement methods, and other arrangement methods are also possible; the specific arrangement method can be designed and modified as needed, and the embodiments of this application do not impose any restrictions on this.

[0107] It should be noted that when the three cameras are shooting, the field of view of the main camera 1931 is usually larger than that of the telephoto camera 1933; while the field of view of the ultra-wide-angle camera 1932 is larger than that of the main camera 1931; the field of view of the ultra-wide-angle camera 1932 may overlap with that of the main camera 1931; that is to say, the ultra-wide-angle camera 1932 can capture the scene captured by the main camera 1931 and the surrounding scene.

[0108] It should be understood that the field of view of the telephoto camera 1933 is smaller than that of the main camera 1931, and there may be overlap between the field of view of the main camera 1931 and the telephoto camera 1933; that is, the main camera 1931 can capture the scene captured by the telephoto camera 1933 and its surrounding scene. Similarly, the field of view of the ultra-wide-angle camera 1932 may overlap with that of the telephoto camera 1933; that is, the ultra-wide-angle camera 1932 can capture the scene captured by the telephoto camera 1933 and its surrounding scene.

[0109] Among them, the ultra-wide-angle camera 1932 is suitable for shooting close-ups due to its smaller focusing distance; and, as the name suggests, the ultra-wide-angle camera 1932 is suitable for shooting scenes with a large field of view; the main camera 1931 is more suitable for shooting portraits due to its higher resolution, while the telephoto camera 1933 is more suitable for shooting distant close-ups.

[0110] For example, such as Figure 4 As shown, the zoom ratio of an ultra-wide-angle camera can be less than M times the zoom ratio; the zoom ratio of a wide-angle camera, i.e., the main camera, can be in the range of [M, N); and the zoom ratio of a telephoto camera can be greater than or equal to N times the zoom ratio.

[0111] For example, M can be 1 and N can be 2.5; then the zoom ratio of the ultra-wide-angle camera is less than 1x zoom ratio (1×); the zoom ratio of the wide-angle camera is from 1x zoom ratio to 2.5x zoom ratio (1×~2.5×); the zoom ratio of the telephoto camera is greater than or equal to 2.5x zoom ratio.

[0112] It should be understood that during the shooting process of electronic devices, the greater the zoom ratio, the smaller the corresponding field of view.

[0113] The following is combined Figure 5The application scenarios of the image processing method provided in the embodiments of this application are illustrated with examples.

[0114] The method in this application embodiment can be applied to shooting scenarios, previewing scenarios, recording video scenarios, or video call scenarios, etc.; through the method in this application embodiment, smooth zoom and / or smooth switching between different cameras in electronic devices can be achieved, improving the user's shooting experience and image quality.

[0115] For example, the preview scenarios include, but are not limited to, the following scenarios:

[0116] Photo preview, aperture preview, night scene preview, portrait preview, video preview, or professional preview, etc.

[0117] It should be understood that a preview scene can refer to a scene in which an electronic device captures an image before the button indicating that the camera has been clicked, in a certain shooting mode.

[0118] In one example, such as Figure 5 As shown in (a), after the electronic device enters the camera application, it can activate the default shooting mode; in the shooting mode, the electronic device can enter the default shooting mode, which can refer to a shooting mode in which the wide-angle camera is used as the main camera and the zoom ratio is a single zoom ratio (1×), and the electronic device displays the image captured by the main camera; as Figure 5 As shown in (b), in response to user operation, the electronic device can switch to display the image captured by the telephoto camera when the zoom ratio meets the zoom range corresponding to the telephoto camera.

[0119] For example, the image processing method in this application embodiment can also be applied to video call scenarios, wherein the video call scenario may include, but is not limited to, the following scenarios:

[0120] Video calls, video conferencing applications, long and short video applications, live video applications, online video courses, intelligent portrait camera movement applications, video recording by system camera function, video surveillance, or portrait shooting scenarios such as intelligent doorbells, etc.

[0121] It should be understood that the above are illustrative examples of application scenarios and do not limit the application scenarios of this application in any way.

[0122] Currently, during shooting, electronic devices can zoom by switching between cameras with different focal lengths in response to user input. They can also combine digital zoom with image processing to meet various high-magnification shooting scenarios. To achieve smooth zooming, existing technologies typically employ the SAT algorithm (which corrects each pair of simultaneously acquired images from the two cameras used for zoom switching to achieve consistent field of view).

[0123] For example, such as Figure 6 As shown, when switching from the main camera to the telephoto camera, both the main camera and the telephoto camera capture images before the switch, and the image captured by the main camera is displayed. After the switch, both the main camera and the telephoto camera continue to capture images, but the image captured by the telephoto camera is displayed. During this zoom process, to maintain smooth zooming, the field of view (FOV) of each pair of images captured by the main camera and the telephoto camera must remain consistent. To this end, as the zoom magnification increases, the image captured by the telephoto camera is typically used as the reference. Correction parameters are calculated to transform the image captured by the main camera into the image captured by the telephoto camera, and then applied to the image captured by the main camera, thus ensuring that the FOV of the corrected image is consistent with that of the image captured by the telephoto camera. These correction parameters generally include a rotation amount representing the rotation relationship and an offset amount representing the offset relationship. The rotation amount can be obtained using existing related technologies, which will not be discussed here. In existing technologies, the offset amount can be calculated based on the differences between the feature points after registering the images captured by the main camera and the telephoto camera.

[0124] However, this calculation process cannot guarantee the accuracy of registration. For example, in scenes with repetitive textures or weak textures, incorrectly registered feature points may occur. Incorrect feature point registration will lead to errors in the calculation of feature point differences, i.e., offsets, resulting in incorrect correction parameters. Consequently, the FOV of the image corrected using these parameters will not match that of the image captured by the telephoto camera. Repetitive texture scenes refer to images with repetitive patterns or designs, such as floral patterns or tree patterns. Texture scenes refer to texture features with relatively small feature values, such as wall cracks or brick patterns.

[0125] Furthermore, even if all feature points are correctly registered, the offsets of feature points at different depths will differ, and a single, uniform offset cannot represent the offset of all feature points in the image. For example, if the depth of the subject differs—such as the depth of the foreground and background—then the feature points extracted from the foreground and background will correspond to different depths. Since there is a perspective relationship between the foreground and background in the image, for the same physical distance, the offset of feature points in the foreground will be larger than that in the background. Therefore, when performing correction, an offset determined in a fixed way cannot be used as the offset of all feature points in the image.

[0126] However, the center-first and foreground-first processing strategies provided in the existing technology result in the collection of feature points with smaller or closer depths after the depth is calculated to calculate the offset. When there are not enough feature points with closer depths extracted from the foreground during zooming, feature points with farther depths in the background are added to calculate the offset, which leads to abnormal image effects after correction.

[0127] For example, Figure 7 This refers to a set of image streams involved in existing technologies. For example... Figure 7 As shown in (a) to (f), assuming the acquired image only contains foreground and background, during zooming, there is a significant difference between the foreground and background images from the two cameras. If the foreground is to be aligned, the background cannot be aligned; similarly, if the background is to be aligned, the foreground cannot be aligned. Since only one depth can be guaranteed at a time, the algorithm needs to continuously determine whether to align the foreground or the background depth, resulting in repeated jumping between foreground and background in the zooming process, making it impossible to smoothly switch between cameras.

[0128] In view of this, embodiments of this application provide an image processing method and related equipment. In embodiments of this application, feature point detection and registration are performed on images captured by a first camera and images captured by a second camera. Then, directional filtering is applied to the matched feature point pairs to obtain the offset between the two images. The image captured by the first camera is then corrected based on this offset, and the corrected second image is displayed. By performing directional filtering during the offset calculation process, this application can filter a large number of feature point pairs, retaining feature point pairs with consistent characteristics (such as consistent direction or slope). This makes the feature point pairs used to calculate the offset more stable, thereby making the determined corrected image FOV more stable, ensuring smooth zooming and smooth switching during camera switching.

[0129] Optionally, the directional filtering process may include using the slope of the vector formed by each pair of feature points to divide all feature point pairs into multiple point sets, and then determining the offset according to the point set containing the most feature point pairs.

[0130] For two frames of images, the set of points with the most feature point pairs exhibits more consistent feature point pair representations, making the determined offset more representative. For the two image streams acquired by the first and second cameras, the set of points with the most feature point pairs across consecutive frames is relatively stable. This ensures that the offset amplitude and direction are consistent when performing offset correction on the two image streams, thereby maintaining a consistent field of view (FOV) during zooming and camera switching, guaranteeing a smooth transition.

[0131] The following is combined Figure 8 A schematic flowchart illustrating the image processing method provided in the embodiments of this application is described in detail.

[0132] Figure 8 This is a schematic flowchart of an image processing method provided in an embodiment of this application. The method can be... Figure 1 The electronic device shown executes the method 300, which includes steps S310 to S340. Steps S310 to S340 are described in detail below.

[0133] It should be understood that in the embodiments of this application, the electronic device includes a first camera and a second camera; wherein the first camera and the second camera are cameras of different types.

[0134] S310, launch the camera application.

[0135] For example, a user can instruct an electronic device to launch the camera application by clicking the icon of the "Camera" application.

[0136] For example, when an electronic device is locked, a user can instruct the device to launch the camera application by swiping right on the screen. Alternatively, if the device is locked and the lock screen includes a camera application icon, the user can instruct the device to launch the camera application by tapping the icon. Or, if the device is running another application with permission to access the camera application, the user can instruct the device to launch the camera application by tapping the corresponding control. For instance, if the device is running an instant messaging application, the user can instruct the device to launch the camera application by selecting a control that enables camera functionality.

[0137] It should be understood that the above is an example of how to open a camera application; the camera application can also be opened by voice commands or other methods; this application does not limit this in any way.

[0138] It should also be understood that launching the camera application can mean running the camera application.

[0139] S320. Display the first image, which is obtained by capturing an image from the first camera.

[0140] For example, the first camera can be Figure 3 The wide-angle camera shown can be a main camera, or it can be any other camera with a wider field of view than a telephoto camera.

[0141] For example, the first image can be an RGB image captured by the main camera, or the first image can be an RGB image processed by a series of camera algorithms from a Raw image captured by the main camera.

[0142] For example, Figure 17 The preview image shown in (a) is the first image captured and displayed by the first camera. For example, Figure 18 The video image shown in (a) is the first image captured and displayed by the first camera.

[0143] Optionally, when displaying the first image captured by the first camera, the depth of the subject can also be captured.

[0144] For example, before displaying the first image, the first camera can also perform parallax detection on the content in the first image to determine the depth of the different shooting objects included in the first image.

[0145] For example, such as Figure 18 As shown in (a), when the first camera sends the first image to the display, it can perform parallax detection to determine the depth corresponding to different shooting objects. This depth information can be displayed on the first image or not, and this application does not limit it.

[0146] It should be understood that the first image may include one or more subjects, and when displayed, each subject identified based on AI detection can have a detection box displayed simultaneously.

[0147] It should also be understood that the type of the subject can be preset as needed. For example, if the type is set to a face, the target object can be used to indicate a face identified in the first image. The type of subject can also include plants, animals, etc. This application embodiment does not impose any limitations in this regard.

[0148] S330, received the first zoom operation.

[0149] Optionally, receiving the first zoom operation may include receiving a first zoom operation for the first image.

[0150] It should be understood that zooming on the first image can be a user instruction, or it can be an instruction for zooming automatically triggered by the electronic device based on AI detection. The target object is one of the subjects included in the first image.

[0151] For example, when the zoom operation indicates the operation command triggered by the user, the zoom operation may include a two-finger reverse swipe operation, a click operation, a voice operation, an air gesture operation, etc., for the first image. This application embodiment does not impose any limitations on this.

[0152] For example, such as Figure 18 (a) and Figure 18 As shown in (b), when the electronic device displays one or more objects to be photographed and a zoom control, the zoom operation may include a click operation on the target object in the first image and a swipe operation on the zoom control; or, the zoom operation may also include a click operation on the target object in the first image and a two-finger reverse swipe operation. It should be understood that when the zoom operation includes two operations, the user must first perform the click operation, and then perform the swipe operation on the zoom control or the two-finger reverse swipe operation on the target object. The zoom operation may also be other operations, or it may include three or more sub-operations; this application embodiment does not impose any limitations on this.

[0153] S340, Display and save the second image.

[0154] The second image is obtained by correcting the offset between the image captured by the first camera and the image captured by the second camera. The offset is obtained by performing directional filtering on the feature point pairs that match the feature points of the image captured by the first camera and the feature points of the image captured by the second camera.

[0155] For example, the first image can be an RGB image, and the corresponding second image can be an RGB image.

[0156] It should be understood that, in response to the first zoom operation, feature point detection and registration are performed on the images captured by the first camera and the images captured by the second camera. Since the first camera and the second camera capture the same scene, the feature points extracted from the two frames have a large number of feature points indicating the same target. Therefore, multiple pairs of matching feature points can be obtained. By performing directional filtering on the matching feature point pairs, the offset between the images captured by the two cameras can be obtained. Based on this offset, the image captured by the first camera is corrected to obtain the second image.

[0157] For example, the first zoom operation is either an operation that increases the zoom magnification, or an operation that decreases the zoom magnification. When the zoom magnification increases, the size of the subject included in the second image is larger than the size of the subject included in the first image. When the zoom magnification decreases, the size of the subject included in the second image is smaller than the size of the subject included in the first image.

[0158] Optionally, the target object in the second image is centered within the second image.

[0159] Optionally, the second image may include all or part of the target object. For example, such as Figure 18 As shown, assuming the target object is the fourth photographed object, after magnification, the second image can include the entire fourth photographed object, or the second image can also include partial content of the fourth photographed object, such as the face of the fourth photographed object; when magnified further, the second image can also include a part of the face of the fourth photographed object.

[0160] Optionally, to ensure smooth magnification of the target object displayed by the electronic device during zooming, the above-mentioned S340 can be executed multiple times, with the target object in each acquired second image being slightly larger than the target object in the previous acquired image. That is, in response to the zoom operation, the electronic device corrects the image acquired by the first camera using the offset between the image acquired by the second camera, acquires and displays multiple second images, all of which include the target object, and the target object in these multiple second images gradually increases in size from its original size. The number of iterations and the magnitude of size change can be set as needed, and this application does not limit this. The original size refers to the size of the target object in the first image.

[0161] It should be understood that S340 can be executed simultaneously with S330, or S340 can be executed after S330. This application embodiment does not impose any restrictions on this.

[0162] For example, a second image can be displayed when in a preview scene. These preview shooting modes include, but are not limited to, shooting modes such as night scene preview mode, video preview mode, photo preview mode, and portrait preview mode.

[0163] For example, when in a recording scene, a second image can be displayed and saved.

[0164] Optionally, the method further includes:

[0165] Responding to the first zoom operation;

[0166] Determine the zoom ratio of the electronic device to meet the zoom ratio range of the first camera for foreground display and the second camera for background operation.

[0167] For example, in response to the first zoom operation, when the zoom ratio has reached the minimum zoom ratio of the second camera but not the maximum zoom ratio of the first camera (e.g., the telephoto camera is at least 2x and the main camera is at most 5x), the second image acquired by the first camera continues to be displayed. However, this second image is obtained by correcting the offset between the image acquired by the first camera and the image acquired by the second camera. The offset is obtained by performing directional filtering on the feature point pairs that match the feature points of the image acquired by the first camera and the feature points of the image acquired by the second camera.

[0168] For example, the first zoom operation is switching from 1x to 4x. The first camera is always in the foreground display, but when zooming to 2x, the second camera starts running in the background. At this time, the method provided in the embodiments of this application can be triggered to calculate the offset, correct the image captured by the first camera, and generate a second image for display and saving.

[0169] This application provides an image processing method and related equipment. In this embodiment, feature point matching is performed on images captured by a first camera and images captured by a second camera. Then, directional filtering is applied to the matched feature point pairs to obtain the offset between the two images. This offset is then used to correct the image captured by the first camera, and the corrected second image is displayed. By performing directional filtering during the offset calculation process, this application can filter a large number of feature point pairs, retaining those with consistent characteristics (e.g., consistent direction or slope). This makes the feature point pairs used for offset calculation more stable, resulting in a more stable FOV for the corrected image, ensuring smooth zooming and smooth switching during camera switching.

[0170] For example, Figure 11 This refers to a set of image streams involved in this application. For example...Figure 11 (a), (b), (c), (d), (e) Figure 11 As shown in (f), assuming that the acquired image only has foreground and background, during the zoom process, since there are more feature points with greater depth in the background and the features are more consistent, the offset is mainly calculated based on the feature points in the background. In this way, the corrected image can be smoothly enlarged with the background as the reference, which in turn can make the FOV of the image stream smoothly enlarged and the camera can be smoothly switched during the zoom process.

[0171] Figure 9 An exemplary flowchart of an image processing method for another electronic device 100 is shown.

[0172] Taking the first camera as the main camera and the second camera as a telephoto camera as an example, the method 400 includes S401 to S408; S401 to S408 are described in detail below.

[0173] S401. Acquire the first initial image captured by the main camera.

[0174] For example, the image captured by the main camera can be a Raw image captured by the wide-angle camera, or a YUV image.

[0175] S402, Acquire the second initial image captured by the telephoto camera.

[0176] For example, the image captured by the telephoto camera can be a Raw image or a YUV image captured by the telephoto camera.

[0177] S403. Perform image feature point detection and registration on the first initial image and the second initial image to obtain multiple pairs of feature points.

[0178] Optionally, image feature point detection can employ any existing image feature point detection algorithm, and this application does not impose any limitations on it.

[0179] For example, image feature point detection is performed on the first initial image and the second initial image respectively, extracting M feature points from the first initial image and N feature points from the second initial image; then, the M feature points and N feature points are registered. M and N are both integers greater than 0.

[0180] It should be understood that each feature point pair includes one feature point belonging to the first initial image and one feature point belonging to the second initial image; these two feature points are used to indicate the same target. The registered feature points have a one-to-one correspondence.

[0181] Optionally, after S403, the method may further include: filtering correctly registered feature point pairs.

[0182] Specifically, correctly registered feature point pairs can be selected by eliminating incorrectly registered feature point pairs.

[0183] After testing, it was found that the number of correctly registered feature points exceeded the number of incorrectly registered feature points in each scenario. Therefore, in order to improve processing efficiency and the effect of subsequent processing, the incorrectly registered feature points can be removed, and only the correctly registered feature point pairs can be retained.

[0184] S404. Calculate the vector formed by each pair of feature points in all feature point pairs.

[0185] This vector is used to indicate the direction and length between pairs of feature points.

[0186] S405. Determine the slope of each vector.

[0187] It should be understood that the slope is the direction of the vector. When the slopes of multiple vectors are the same, it means that the offsets of the corresponding feature point pairs are the same, which means that the depths of the corresponding feature point pairs are the same.

[0188] S406. Based on the slope of each vector, divide all feature point pairs into multiple point sets, each point set may contain one or more feature point pairs.

[0189] For example, different slope thresholds can be set, or based on the difference between adjacent slopes, or based on the slope, all feature point pairs can be divided into multiple point sets using methods such as clustering.

[0190] For example, given 10 feature point pairs, the slopes corresponding to the 10 determined vectors are 0.81, 0.81, 0.81, 0.82, 0.84, 0.84, 0.85, 0.85, 0.85, and 0.88, respectively. These 10 slopes can be divided into three point sets based on the difference between adjacent slopes. The first point set corresponds to slopes of 0.81, 0.81, 0.81, and 0.82; the second point set corresponds to slopes of 0.84, 0.84, 0.85, 0.85, and 0.85; and the third point set corresponds to a slope of 0.88.

[0191] Optionally, the above S406 may include the following S4061 to S4063, which will be described in detail below.

[0192] S4061. Sort all feature points by their corresponding slopes and assign them a sequence number.

[0193] Here, for the convenience of subsequent calculations, all feature point pairs can be sorted in descending or ascending order of slope, and a sequence number can be assigned to each feature point pair.

[0194] S4062. Calculate the slope corresponding to the feature point pair of the current sequence number and the difference between the slope and the average slope corresponding to all previous feature point pairs, according to the sequence number order.

[0195] It should be understood that when the index is 1, the average slope has not been calculated before. Therefore, when calculating, we can start from the slope corresponding to the feature point pair of the second index. The average slope corresponding to the feature point pairs of all previous indexes is the slope corresponding to the feature point pair of the first index.

[0196] S4063. If the difference is too large, the feature point pair with the current sequence number is recorded as the feature point pair with too large a difference, and the previous set of feature point pairs recorded as the feature point pair with too large a difference, the previous set of feature point pairs recorded as the feature point pair with too large a difference, and multiple feature point pairs between the two sets of feature point pairs are divided into a point set.

[0197] Optionally, a difference threshold can be set. If the determined difference is greater than or equal to the difference threshold, it means that the difference is too large and S4063 needs to be executed. If the determined difference is less than the difference threshold, it means that the difference is not large and S4062 and S4063 can be repeated for the next set of feature point pairs.

[0198] For example, assuming the difference threshold is 0.02, if the slopes of the 10 determined vectors are sorted from smallest to largest, the 10 slopes are 0.801, 0.801, 0.801, 0.802, 0.804, 0.804, 0.805, 0.805, 0.805, and 0.808. Following the sequence, the slope 0.801 corresponding to feature point pair number 2 is determined to be 0.801, and the difference between this slope and the slope 0.801 corresponding to feature point pair number 1 is 0. If there is no difference, proceed to the next step. The slope 0.801 corresponding to feature point pair number 3 is found to be 0 different from the average slope 0.801 of the two feature point pairs numbered 1 and 2, so the calculation continues. The slope 0.802 corresponding to feature point pair number 4 is found to be 0.01 different from the average slope 0.801 of the three feature point pairs numbered 1 to 3. This difference is less than the difference threshold of 0.02, indicating a small difference, so the calculation continues. The slope 0.804 of feature point pair number 5 is found to be 0.0275 different from the average slope 0.80125 of the four feature point pairs numbered 1 to 4. This difference is greater than the difference threshold of 0.02, indicating that feature point pair number 5 is inconsistent with the previous vector features. Therefore, the four feature point pairs numbered 1 to 4 can be divided into a point set, which includes these four feature point pairs.

[0199] S407. Determine the offset based on the feature point pairs in the point set that contain the most feature point pairs.

[0200] S408. Based on the offset, the first initial image is corrected to obtain the corrected image.

[0201] Optionally, when correcting the first initial image, the rotation amount determined by the relevant method and the offset amount determined by the method provided in the embodiments of this application can be combined to correct the first initial image together to obtain the corrected image.

[0202] In this embodiment, feature point detection and registration are performed on two frames of images captured by the first and second cameras. Then, the slope of the vector formed by each pair of feature points is used to divide all feature point pairs into multiple point sets. The offset is then determined according to the point set containing the most feature point pairs. For two frames of images, the point set containing the most feature point pairs has a more consistent feature point pair representation, making the determined offset more representative. For the two image streams acquired by the first and second cameras, the point set containing the most feature point pairs is relatively stable across multiple consecutive frames. Therefore, when offsetting the two image streams, the offset amplitude and direction are more consistent, ensuring that the FOV of the image remains consistent during zooming and camera switching, thus guaranteeing a smooth transition.

[0203] In this embodiment, the offset at different zoom ratios is obtained by allocating a total offset, typically in the form of an exponential curve. The unit of offset is pixels.

[0204] For example, such as Figure 12 As shown in (a), taking a zoom ratio of 2.5 when switching from the first camera to the second camera for display as an example, the closer to the camera switching point, the larger the offset is allocated for a smaller zoom ratio range. For example, when zooming to 1.1x, the total offset may be 100, and the offset allocated for that zoom ratio may be 10; when zooming to 1.2x, the total offset may be 105, and the offset allocated for that zoom ratio may be 10.5; when zooming to 2.45x, the total offset may be 100, and the offset allocated for that zoom ratio may be 96; when zooming to 2.48x, the total offset may be 105, and the offset allocated for that zoom ratio may be 101.

[0205] As can be seen, a smaller offset was allocated to the zoom range of 1.1x to 1.2x, which is far from the switching point of 2.5x; a 0.1x zoom range has an offset difference of 0.5 pixels. However, a larger offset was allocated to the zoom range of 2.45x to 2.48x, which is close to the switching point of 2.5x; a 0.03x zoom range has an offset difference of 5 pixels.

[0206] It should be understood that the total offset is calculated from two images corresponding to the two cameras. Taking the first camera as the main camera and the second camera as the telephoto lens, with a zoom ratio of [1x, 2.5x] as an example, the specific steps are as follows: crop the main camera image at any zoom ratio to align the FOV size with the telephoto image (the FOV of the main camera image will change during the zoom process, but the FOV size of the telephoto image remains at 2.5x). Then, register the two images with the same FOV size (but with rotational offset) to obtain the offset between the two images. This can be understood as the total offset (the FOV offset corresponding to 2.5x. Different zoom ratios must be aligned at this scale for comparison. The offsets mentioned below are all at this scale). Theoretically, it also needs to be mapped to the current FOV size of the main camera.

[0207] Assuming the camera is fixed, the total offset at different zoom levels is also fixed, let's say 100, calculated as described above. Based on the allocation logic (for simplicity, we assume it's linear, but it's actually a curve), the offset allocated to 1x is 0, and the offset allocated to 2.5x is 100. Therefore, the offset allocated to 1.1x is approximately 100*(1.1-1) / (2.5-1) ≈ 6.667, the offset allocated to 1.2x is approximately 13.333, and the offset allocated to 2x is approximately 66.667. The calculation process for other zoom levels is the same and will not be described further here.

[0208] When the camera is not fixed, the images at each zoom level will differ using the above calculation method. This difference comes from various factors, mainly considering the differences in imaging caused by camera movement. Consequently, the selection of feature points, registration, and the resulting offset will also differ. However, since these differences are minor, the main image subject theoretically changes very little. Therefore, the total offset is on the same order of magnitude as the offset when the camera is fixed. Thus, at different zoom levels, there may be total offsets of 105, 98, 103, etc. Therefore, theoretically, the entire process involves glitches and fluctuations, rather than appearing suddenly.

[0209] Based on this, the allocation is performed. The offset corresponding to 1.1x may be 7 (total offset is 105), 6.53 (total offset is 98), or 6.86 (total offset is 103); the offset corresponding to 2.4x may be 98, 91.42, 93.52, etc. As can be seen from the allocation logic, when the zoom ratio is small, the difference in total offset will be reduced (105 and 98 differ by 7 pixels, but when allocated to 1.1x, the difference is only 0.5 pixels, which is indistinguishable to the naked eye). However, when the zoom ratio is large, this difference will be obvious, thus causing visually visible jitter.

[0210] Another reason for the visually visible jitter is that the total offset theoretically needs to be mapped to the current FOV size of the main camera. The FOV change from 1x to 1.1x is 1.1 times (magnified by 1.1 times), while the FOV change from 2.4x to 2.5x is 2.5 / 2.4 = 1.042 times (magnified by 1.042 times). Therefore, the magnification from 1x to 1.1x is more obvious visually, and the offset will be weakened. The magnification is not obvious in the process from 2.4x to 2.5x, so the change in offset is easier to see.

[0211] For example, factors such as external environmental influences or the vibration of the electronic device itself (e.g., hand tremors caused by the user's breathing when taking a photo) can cause the image content captured by the camera to shift. Therefore, the offset allocated to each frame may fluctuate. Figure 12 As shown in (b), glitches / fluctuations will occur throughout the zoom process. This will manifest on the display as an anomaly where there is no noticeable shaking (mainly imperceptible to the naked eye) when the zoom point is far from the camera, but noticeable shaking of the displayed content occurs when the zoom point is close to the camera.

[0212] In addition, during zooming, for example, when there is a moving target in the scene (such as a running dog), when the first and second cameras are out of focus during zooming, or when the two cameras shake inconsistently, the offset value may also be abnormal, which will cause one or two frames of the image to show abnormal shaking effect during the display process.

[0213] For example, Figure 13 This refers to a set of image streams involved in existing technologies. For example... Figure 13 During the zoom process in (a), (b) to (d), due to one of the reasons mentioned above, an abnormal offset occurred, resulting in the following: Figure 13 The image shown in (c) exhibits abnormal phenomena such as jitter and jumps.

[0214] In response, this application also provides an image processing method. In this application embodiment, the offset is stabilized at the zoom level corresponding to the near camera switching point (e.g., if the offset remains unchanged, the last determined offset is used); and / or, each time an offset is determined, a parameter value is calculated using this offset and all previous offsets. Abnormal offsets are filtered out based on the magnitude of the parameter value, while stable offsets are retained. Through the above method, the offset change can be ensured to be linear and stable, thereby enabling smooth image changes and avoiding abnormalities such as image jitter and jumps.

[0215] Figure 14A schematic flowchart illustrating another image processing method of an electronic device 100 is provided. The method 500 includes steps S501 to S506; steps S501 to S506 are described in detail below.

[0216] S501, Launch the camera application.

[0217] S502, Display the first image.

[0218] S503, received the first zoom operation.

[0219] The contents of S501 to S503 are consistent with those described in S310 to S330 above, and can be referred to the above introduction, so they will not be repeated here.

[0220] S504. During the first zoom operation, when the difference between the current zoom ratio and the target zoom ratio is equal to or less than a preset value, the offset remains unchanged and is equal to the first offset.

[0221] The target zoom ratio is used to indicate the zoom ratio when switching from displaying the first camera to displaying the second camera; the first offset is used to indicate the last offset determined when the difference between the current zoom ratio and the target zoom ratio is greater than a preset value.

[0222] For example, if the target zoom ratio is 2.5x and the preset value is 0.1x, during the first zoom operation from 1x to 2.5x, when zooming from 1x to 2.4x, since the difference between 2.4x and 2.5x equals the preset value, no further offset calculation or allocation is needed. Instead, the first offset, which is the last or most recent offset determined, is used, allowing the offset for the current frame to continue using the previously calculated offset. Subsequently, when zooming from 2.4x to 2.5x, the first offset continues to be used.

[0223] It should be understood that the offset occurs. Figure 12 The jagged or fluctuating images shown typically occur near the zoom level at the switching point between the two cameras. In actual use, it is rare to stay at this focal length for an extended period. This is because, on the one hand, to solve the jagged image problem, and on the other hand, to save computing resources, the above method can be used to apply the offset determined in the last frame near the editing zoom level at the switching point between the two cameras. Under the allocation logic, the change of the offset can be guaranteed to be linear, thereby solving the image jitter problem.

[0224] S505. Combine the offset between the image captured by the first camera and the image captured by the second camera with all previous offsets to determine the parameter value corresponding to the current frame.

[0225] Optionally, the parameter values ​​include the mean and / or variance of the offset. Of course, the parameter values ​​may also include other types of numerical values, which are not limited in this embodiment.

[0226] S506. Determine the difference between the parameter value corresponding to the current frame and the parameter value corresponding to the previous frame; if it is less than the threshold, update the offset, that is, make the determined new offset the offset corresponding to the current frame. If it is greater than the threshold, proceed to S507; if the difference between the current frame and the parameter value corresponding to the previous frame is greater than the threshold, proceed to S508.

[0227] In the embodiments of this application, the threshold comparison method after statistical analysis of mean and variance adopts the 3sigma theory, which can be referred to in related technologies and will not be elaborated here.

[0228] S507. Use the offset corresponding to the previous frame as the offset corresponding to the current frame.

[0229] S508. Clear all offsets and re-determine.

[0230] For example, when the second frame is the current frame, the offset between the second frame image captured by the first camera and the second frame image captured by the second camera, such as P2, is used, along with all previous offsets (such as the offset P1 between the first frame image captured by the first camera and the second frame image captured by the second camera). The variance of P1 and P2 is calculated as a1, and the mean is b1. If the difference is greater than a threshold, the offset corresponding to the first frame is used as the offset corresponding to the second frame. If the differences determined by the above method for the subsequent third and fourth frames are all greater than the threshold, all offsets are cleared, and the calculation is repeated.

[0231] When the third frame is the current frame, the offset between the third frame image captured by the first camera and the third frame image captured by the second camera (e.g., P3) is used, along with all previous offsets (e.g., the offset between the first frame image captured by the first camera and the first frame image captured by the second camera, and the offset between the second frame image captured by the first camera and the second frame image captured by the second camera, P2). The variance of P1, P2, and P3 is calculated as a2, and the mean is b2, which is then used as the parameter value corresponding to the third frame. Then, the difference between the variance a2 and the variance a1 corresponding to the second frame is determined, and / or the difference between the mean b2 and the mean b1 corresponding to the second frame is determined. If the difference is greater than a threshold, the offset corresponding to the second frame is used as the offset corresponding to the third frame. If the differences determined in the subsequent fourth and fifth frames using the above method are all greater than the threshold, all offsets are cleared, and the calculation is repeated.

[0232] It should be understood that the mean and variance included in the parameter values ​​can be used to represent the stability of all offsets. It should also be understood that by calculating the difference between the parameter values ​​corresponding to the current frame and those corresponding to the previous frame, when the difference is large (i.e., the offset is unstable), the offset corresponding to the previous frame can be used as the offset corresponding to the current frame to avoid large offset differences, thereby preventing jumps in the image.

[0233] In this embodiment, the offset is stabilized at the zoom level corresponding to the near camera switching point (e.g., if the offset remains unchanged, the last determined offset is used); and / or, each time an offset is determined, a parameter value is calculated using this offset and all previous offsets. Abnormal offsets are filtered out based on the magnitude of the parameter value, while stable offsets are retained. This method ensures that the offset change is linear and stable, resulting in smooth image changes and avoiding image jitter, jumps, or other anomalies.

[0234] For example, Figure 15 This refers to a set of image streams involved in the embodiments of this application. For example... Figure 15 During the zoom process in (a), (b), (c) to (d), by combining the image processing method provided in this application, abnormal offsets can be avoided, thereby making the image change smoothly and eliminating abnormal phenomena such as image jitter and jumps.

[0235] Figure 16 An exemplary flowchart of an image processing method for another electronic device 100 is shown. The method 600 includes steps S601 to S609; steps S601 to S609 are described in detail below.

[0236] S601. Acquire the image captured by the main camera.

[0237] For example, the image captured by the main camera can be a raw image captured by the wide-angle camera; or, the image captured by the main camera can be a raw image captured by the ultra-wide-angle camera.

[0238] S602, First front-end processing.

[0239] Optionally, the first front-end processing may include algorithms for converting the Raw image captured by the main camera into a YUV image; this application does not limit the algorithms in any way.

[0240] For example, the first front-end processing may refer to the image processing algorithm performed in the ISP to convert a Raw image to a YUV image. The first front-end processing may also include subsequent operations such as cropping and magnification based on the zoom ratio in response to zoom operations.

[0241] S603: Acquire images captured by the telephoto camera.

[0242] For example, the image captured by the telephoto camera can be a raw image captured by the telephoto camera.

[0243] Optionally, S604 and S601 can be executed simultaneously, or S604 and S601 can be executed sequentially.

[0244] S604, Second Front-End Processing.

[0245] Optionally, the second front-end processing may include algorithms for converting Raw images captured by a telephoto camera into YUV images; this application does not impose any limitations on the algorithms.

[0246] For example, the second front-end processing may refer to the image processing algorithm performed in the ISP to convert the Raw image to a YUV image. The second front-end processing may also include subsequent operations such as cropping and magnification based on the zoom ratio in response to zoom operations.

[0247] After S605 is executed, the processed image undergoes a first backend processing; the implementation of the first backend processing can be found in the relevant description of S603.

[0248] For example, when the zoom ratio is between 1x and 2.5x, only S601 is executed; when zooming to 2.5x to 5x, both S601 and S603 are executed simultaneously; when zooming to 5x and above, only S603 is executed.

[0249] S605, First backend processing.

[0250] For example, the first backend processing includes, but is not limited to: brightening processing, noise reduction processing, saturation adjustment processing, cropping processing, or deformation processing.

[0251] S606, Smoothing (or Alignment).

[0252] For example, smoothing is performed on the image stream captured by the main camera and the image stream captured by the telephoto camera; this can be understood as the input data for smoothing being two image streams (e.g., the image stream captured by the main camera and the image stream captured by the telephoto camera), and the smoothed image stream being one image stream. This smoothing process is the one provided in the embodiments of this application. Figure 8 , Figure 9 , Figure 10 and / or Figure 14 The image processing method shown.

[0253] For example, when the zoom magnification increases from 2x to 2.5x, the smoothed image stream is based on the image stream captured by the telephoto camera, and the image stream of the main camera is processed using the image processing method provided in the embodiments of this application.

[0254] For example, when the zoom magnification decreases from 5x to 2.5x, the smoothed image stream is based on the image stream captured by the main camera, and the image stream of the telephoto camera is processed using the image processing method provided in the embodiments of this application.

[0255] For example, when the zoom magnification increases and zooms from 2.4x to 2.5x, it is possible to utilize Figure 14 The image processing method shown filters and modifies the offset corresponding to each frame of the image.

[0256] In the embodiments of this application, smoothing processing can avoid obvious jumps between foreground and background in the image when switching cameras.

[0257] S607, Second Backend Processing.

[0258] For example, the second back-end processing includes stabilization processing; for instance, image frames in the image stream can be stabilized according to jitter parameters to obtain a processed image.

[0259] S608. Display the processed image.

[0260] For example, when the zoom magnification increases and the zoom ratio is between 1x and 2x, the processed image is the image corresponding to the image captured by the main camera; when zooming to 2x to 5x, the processed image is the image captured by the main camera, which has been corrected and processed based on the image captured by the telephoto camera; when zooming to 5x and above, the processed image is the image corresponding to the image captured by the telephoto camera.

[0261] S609, Zoom operation detected.

[0262] The zoom operation can be referred to in the description in S230 above.

[0263] Optionally, the electronic device may display an image captured by the main camera, or an image captured by the telephoto camera, and the electronic device may detect user interaction with the displayed image.

[0264] It should be understood that the above examples are provided to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments of this application to the specific values ​​or scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples, and such modifications or changes also fall within the scope of the embodiments of this application.

[0265] For example, Figure 17 This is a schematic diagram of an application scenario provided by an embodiment of this application.

[0266] like Figure 17 As shown in (a), in response to a user's operation on the camera application, the electronic device can display a preview interface 1301, which includes a preview window and shooting controls. The preview image 1302 displayed in the preview window can include a first subject, a second subject, and a third subject. Here, when the camera application is opened, the zoom level is 1x by default. Figure 17 As shown in (b), suppose a user wants to zoom in on the third subject located in the lower left corner of preview image 1302, the user can perform a two-finger reverse swipe on the screen; in response to this action, for example, as Figure 17 As shown in (c), the zoom ratio can be increased to 10x, the imaging size of the third subject increases from small to large, and the third subject can respond to the movement of the user's finger and be located in the middle of the preview image 1303.

[0267] It can adapt to the increase in zoom magnification and camera switching. Figure 8 , Figure 9 , Figure 10 or Figure 14 The image processing methods described herein.

[0268] It should be understood that the above describes a preview scenario in photo shooting mode. This application embodiment can also be applied to preview scenarios in other shooting modes such as video recording. A preview scenario can refer to the scenario where the electronic device captures an image before clicking the shooting control in a certain shooting mode.

[0269] For example, Figure 18 This is a schematic diagram illustrating another application scenario provided by an embodiment of this application.

[0270] Once the camera app is opened, in response to the user's click on the recording controls, the electronic device can begin recording video and display the video recording interface 1401, such as... Figure 18 As shown in (a), the video recording interface 1401 may include a recording window, a pause control, an end control, and a zoom control. The video image 1402 displayed in the recording window may include a first subject, a second subject, a third subject, and a fourth subject. Here, the zoom ratio is set to 1x by default when video recording begins.

[0271] If the electronic device supports and enables AI detection and parallax detection functions, it can detect image content and depth when capturing video image 1402, and display multiple detection boxes when displaying video image 1402. Each detection box is used to indicate the position information of a captured object in video image 1402. Figure 18 As shown in (a), four detection boxes are displayed in video image 1402, which respectively indicate the position information of the first subject to the fourth subject.

[0272] Optionally, if the AI ​​detection and parallax detection functions are only used for depth detection of faces, the electronic device may display only two detection boxes when displaying video image 1402. These two detection boxes indicate the face of the second subject and the face of the fourth subject, respectively.

[0273] like Figure 18 As shown in (a), during recording, assuming the user wants to zoom in on the face of the fourth subject located on the left side of video image 1402, the user can click on the detection box corresponding to the fourth subject on the screen. Figure 18 As shown in (b), in response to a user's click operation, the detection box of the fourth subject included in the video image 1403 can be selected, for example, the selected detection box can have a different color and / or style compared to other detection boxes. Then, the electronic device can receive a user's sliding operation on the zoom control, such as... Figure 18 As shown in (c), in response to a sliding operation on the zoom control (assuming a sliding to 8x), the electronic device can display a video image 1404 in which the face of the fourth subject included in the video image 1404 is larger than the face of the fourth subject included in the video image 1402, and the face of the fourth subject can be located in the center of the video image 1404 in response to the user's finger movement.

[0274] It should be noted that if the electronic device detects only one subject or face in the image, the user does not need to click on the detection box corresponding to the subject or face. The electronic device will automatically select it. Then, in response to the user's sliding operation on the zoom control, the electronic device will zoom in and center the subject or face.

[0275] It can adapt during zoom ratio changes and camera switching. Figure 8 , Figure 9 , Figure 10 or Figure 14 The image processing methods described herein.

[0276] It should be understood that the above is a video recording scenario, and the embodiments of this application can also be applied to scenarios such as video calls.

[0277] For example, video call scenarios may include, but are not limited to, the following: video calls, video conferencing applications, long and short video applications, live video applications, online video courses, portrait intelligent camera movement applications, video recording by system camera recording function, video surveillance, or shooting scenarios such as smart doorbell cameras.

[0278] The above text combined Figures 1 to 18 The image processing method provided in the embodiments of this application has been described in detail; the following will be combined with Figure 19 and Figure 20 The apparatus embodiments of this application are described in detail below. It should be understood that the apparatus in the embodiments of this application can perform the various methods described in the foregoing embodiments of this application, that is, the specific working processes of the various products described below can be referred to the corresponding processes in the foregoing method embodiments.

[0279] Figure 19 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 1500 includes a processing module 1510 and a display module 1520; the electronic device 1500 also includes a first camera and a second camera.

[0280] The processing module 1510 is used to: open the camera application; the display module 1520 is used to: display a first image, which is obtained by the first camera; receive a first zoom operation; display and save a second image, which is obtained by correcting the image obtained by the first camera using the offset between the image obtained by the first camera and the image obtained by the second camera. The offset is obtained by performing directional filtering on the feature point pairs that match the feature points of the image obtained by the first camera and the feature points of the image obtained by the second camera.

[0281] Optionally, as an embodiment, the display module 1520 is further configured to: receive a second zoom operation; display and save a third image, the third image being obtained from an image captured by a second camera.

[0282] It should be noted that the aforementioned electronic device 1500 is embodied in the form of a functional unit. The term "module" here can be implemented in software and / or hardware, without specific limitations.

[0283] For example, a "module" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application-specific integrated circuit (ASIC), electronic circuitry, a processor (e.g., a shared processor, a proprietary processor, or a group processor) and memory for executing one or more software or firmware programs, integrated logic circuitry, and / or other suitable components that support the described functions.

[0284] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0285] Figure 20 A schematic diagram of the structure of an electronic device provided in this application is shown. Figure 20 The dashed lines indicate that the unit or module is optional. The electronic device 1600 can be used to implement the methods described in the above method embodiments.

[0286] Electronic device 1600 includes one or more processors 1601, which can support the implementation of the image processing method in the method embodiments of electronic device 1600. Processor 1601 can be a general-purpose processor or a special-purpose processor. For example, processor 1601 can be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, such as discrete gate, transistor logic devices, or discrete hardware components.

[0287] The processor 1601 can be used to control the electronic device 1600, execute software programs, and process data from the software programs. The electronic device 1600 may also include a communication unit 1605 for inputting (receiving) and outputting (transmitting) signals.

[0288] For example, electronic device 1600 may be a chip, communication unit 1605 may be the input and / or output circuit of the chip, or communication unit 1605 may be the communication interface of the chip, and the chip may be a component of terminal device or other electronic device.

[0289] For example, electronic device 1600 can be a terminal device, communication unit 1605 can be the transceiver of the terminal device, or communication unit 1605 can be the transceiver circuit of the terminal device.

[0290] The electronic device 1600 may include one or more memories 1602, which store a program 1604. The program 1604 can be executed by the processor 1601 to generate instructions 1603, causing the processor 1601 to execute the method described in the above method embodiments according to the instructions 1603.

[0291] Optionally, the memory 1602 may also store data. Optionally, the processor 1601 may also read the data stored in the memory 1602, which may be stored at the same memory address as the program 1604, or the data may be stored at a different memory address than the program 1604.

[0292] The processor 1601 and memory 1602 can be configured separately or integrated together, for example, integrated on the system on chip (SOC) of the terminal device.

[0293] For example, the memory 1602 can be used to store the related program 1604 of the image processing method provided in the embodiments of this application, and the processor 1601 can be used to call the related program 1604 of the image processing method stored in the memory 1602 during video processing to execute the image processing method of the embodiments of this application; for example, opening the camera application; opening the camera application; displaying a first image, the first image being obtained by the first camera capturing an image; receiving a first zoom operation; displaying and saving a second image, the second image being obtained by correcting the image captured by the first camera using the offset between the image captured by the first camera and the image captured by the second camera, the offset being obtained by performing directional filtering processing on the feature point pairs that match the feature points of the image captured by the first camera and the feature points of the image captured by the second camera.

[0294] This application also provides a computer program product that, when executed by processor 1601, implements the methods described in any of the method embodiments of this application.

[0295] The computer program product can be stored in memory 1602, for example, program 1604. Program 1604 is finally converted into an executable object file that can be executed by processor 1601 after processing such as preprocessing, compilation, assembly and linking.

[0296] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer, implements the methods described in any of the method embodiments of this application. The computer program may be a high-level language program or an executable object program.

[0297] The computer-readable storage medium is, for example, memory 1602. Memory 1602 can be volatile memory or non-volatile memory, or memory 1602 can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0298] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process and technical effects of the above-described apparatus and equipment can be referred to the corresponding processes and technical effects in the foregoing method embodiments, and will not be repeated here.

[0299] In the several embodiments provided in this application, the systems, apparatuses, and methods disclosed can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not performed. The apparatus embodiments described above are merely illustrative; the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components can be combined or integrated into another system. Furthermore, the coupling between units or components can be direct coupling or indirect coupling, including electrical, mechanical, or other forms of connection.

[0300] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0301] Furthermore, the terms "system" and "network" are often used interchangeably in this paper. The term "and / or" in this paper merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " in this paper generally indicates that the preceding and following related objects have an "or" relationship.

[0302] In summary, the above description is merely a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image processing method, characterized in that, Applied to an electronic device, the electronic device including a first camera and a second camera, the method includes: Open the camera application; The first image is displayed, which is obtained by the first camera capturing the image. Received the first zoom operation; Display and save a second image, which is obtained by correcting the image captured by the first camera using the offset between the image captured by the second camera and the image captured by the second camera. The offset is obtained by performing directional filtering on feature point pairs that match the feature points of the first camera image and the feature points of the second camera image. The directional filtering process refers to dividing all feature point pairs into multiple point sets based on the slope of the vector formed by each set of feature point pairs, and selecting the feature point pairs that include the point set with the largest number of feature point pairs. The feature point pairs in the point set that include the largest number of feature point pairs have the same direction or the same slope. The offset is determined based on the feature point pairs in the point set that include the point set with the largest number of feature point pairs.

2. The image processing method according to claim 1, characterized in that, The method further includes: Received second zoom operation; Display and save a third image, which is obtained from an image captured by the second camera.

3. The image processing method according to claim 1 or 2, characterized in that, The method for determining the slope of the vector formed by each set of feature point pairs includes: Acquire images captured by the first camera and images captured by the second camera; Feature point detection and registration are performed on the images captured by the first camera and the images captured by the second camera to obtain multiple sets of matching feature point pairs; Calculate the vector formed by each pair of feature points; Determine the slope of each of the vectors.

4. The image processing method according to claim 3, characterized in that, Before calculating the vector formed by each pair of feature points, the method further includes: removing feature point pairs with mismatches.

5. The image processing method according to claim 3 or 4, characterized in that, The method of dividing all feature point pairs into multiple point sets includes: Sort all feature points according to their corresponding slopes and assign them a sequence number; Calculate the slope corresponding to the feature point pair with the current index in the order of the index, and the difference between the slope and the average slope corresponding to all previous feature point pairs. If the difference is too large, the feature point pair with the current sequence number is recorded as a feature point pair with too large a difference. The previous set of feature point pairs recorded as a feature point pair with too large a difference, the previous set of feature point pairs recorded as a feature point pair with too large a difference, and multiple feature point pairs between the two sets of feature point pairs are divided into a point set.

6. The image processing method according to any one of claims 1 to 5, characterized in that, The method further includes: responding to the first zoom operation; The zoom ratio of the electronic device is determined to be within the zoom ratio range that allows the first camera to display data in the foreground and the second camera to operate in the background.

7. The image processing method according to claim 2, characterized in that, The method further includes: responding to the second zoom operation; The zoom ratio of the electronic device is determined to be within the range that the first camera is running in the background or off, while the second camera is displayed in the foreground.

8. The image processing method according to claim 2 or 7, characterized in that, The first zoom operation or the second zoom operation includes: a two-finger reverse swipe operation, a swipe operation on the displayed zoom control, a voice operation, or an air gesture operation.

9. The image processing method according to any one of claims 1 to 8, characterized in that, The first camera includes a wide-angle camera, and / or the second camera includes a telephoto camera or an ultra-wide-angle camera.

10. The image processing method according to claim 2, characterized in that, The first zoom operation and the second zoom operation are consecutive zoom operations.

11. The image processing method according to claim 1, characterized in that, During the first zoom operation, when the difference between the current zoom ratio and the target zoom ratio is equal to or less than a preset value, the offset remains unchanged and is equal to the first offset. Wherein, the target zoom ratio is used to indicate the zoom ratio when switching from the first camera to the second camera for foreground display; the first offset is used to indicate the last offset determined when the difference between the current zoom ratio and the target zoom ratio is greater than the preset value.

12. The image processing method according to claim 1 or 11, characterized in that, The method further includes: The parameter value corresponding to the current frame is determined by using the offset between the image captured by the first camera and the image captured by the second camera, and by comparing it with all previous offsets. Determine the difference between the parameter value corresponding to the current frame and the parameter value corresponding to the previous frame; If the value is greater than the threshold, the offset corresponding to the previous frame is used as the offset corresponding to the current frame.

13. The image processing method according to claim 12, characterized in that, The method further includes: If the difference between three consecutive frames is greater than the threshold, then clear all offsets and redetermine.

14. The image processing method according to claim 12 or 13, characterized in that, The parameter values ​​include the mean and / or variance of the offset.

15. An electronic device, characterized in that, The electronic device includes: one or more processors, a memory, a first camera and a second camera; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to perform the image processing method as described in any one of claims 1 to 14.

16. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the processors being configured to invoke computer instructions to cause the electronic device to perform the image processing method as described in any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the image processing method according to any one of claims 1 to 14.

Images