Image processing method and related equipment thereof

Abstract

The invention relates to the field of image processing, and provides an image processing method and related equipment thereof, the image processing method is applied to electronic equipment, the electronic equipment comprises a first camera and a second camera, and the method comprises the following steps: starting a camera application program; displaying a first image, wherein the first image is obtained by collecting an image by a first camera; receiving a first zooming operation; a second image is displayed and stored, and the second image is obtained after the image collected by the first camera and the image collected by the second camera are corrected by using the offset amount between the image collected by the first camera and the image collected by the second camera; the offset is obtained by performing direction filtering processing on a feature point pair matched with the feature point of the image acquired by the first camera and the feature point of the image acquired by the second camera. According to the invention, through direction filtering, smooth zooming and smooth switching of the camera can be realized.

Description

Technical Field

[0001] This application relates to the field of image processing, and particularly to an image processing method and related devices thereof. Background Art

[0002] With the development of the shooting function in electronic devices, camera applications are becoming more and more widely used in electronic devices. To obtain a better shooting experience, multiple cameras are usually set on current electronic devices, and the focal lengths corresponding to each camera are different.

[0003] During shooting, in response to the user's operation, the electronic device can perform zoom shooting by switching cameras with different focal lengths, and can also process the captured image by combining digital zoom to meet various high-magnification shooting scenarios. However, during the zooming process, due to the different depths of the shooting objects, such as the influence of the foreground and background, there are jumps in the imaging content and size during the switching of different cameras, and the cameras cannot be smoothly switched; while digital zoom can only magnify and crop the imaging content in the center part of the field of view angle, which is very limited.

[0004] Therefore, during the shooting process, how to smoothly switch cameras has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides an image processing method and related devices thereof, which can achieve smooth zooming and smooth switching of cameras by performing direction filtering.

[0006] In a first aspect, an image processing method is provided, which is applied to an electronic device. The electronic device includes a first camera and a second camera. The method includes: starting a camera application; displaying a first image, where the first image is obtained by the first camera capturing an image; receiving a first zoom operation; displaying and saving a second image, where the second image is obtained by correcting the image captured by the first camera using an offset between the image captured by the first camera and the image captured by the second camera, and the offset is obtained by performing direction filtering on the feature point pairs that match the feature points of the image captured by the first camera and the feature points of the image captured by the second camera.

[0007] In an embodiment of the present application, by performing feature point matching on the images collected by the first camera and the second camera, and then performing direction filtering processing on the matched feature point pairs, the offset between the two images can be obtained; then, in combination with this offset, the image collected by the first camera is corrected, and after correction, the obtained second image is displayed. By performing direction filtering processing during the calculation of the offset, the present application can screen a large number of feature point pairs, and retain as many feature point pairs with the same manifestation form (such as the same direction or the same slope) as possible, so that the feature point pairs used for calculating the offset are relatively stable, and further make the determined FOV of the corrected image relatively stable, ensuring smooth zooming and smooth switching during camera switching.

[0008] Combined with the first aspect, in some implementation manners of the first aspect, the method further includes: receiving a second zoom operation; displaying and saving a third image, where the third image is obtained by collecting an image with the second camera.

[0009] Combined with the first aspect, in some implementation manners of the first aspect, the method further includes: obtaining a first initial image collected by the first camera and a second initial image collected by the second camera; performing feature point detection and registration on the first initial image and the second initial image to obtain multiple groups of matched feature point pairs; calculating the vectors formed by each pair of the feature point pairs; determining the slope of each vector; dividing all the feature point pairs into multiple point sets according to the slope of each vector; determining the offset according to multiple groups of feature point pairs in the point set with the largest number of included feature point pairs; and correcting the first initial image in combination with the offset to obtain the second image.

[0010] In an embodiment of the present application, by performing feature point detection and registration on the images collected by the first camera and the second camera, and then performing direction filtering processing on the matched feature point pairs, the offset between the two images can be obtained; then, in combination with this offset, the image collected by the first camera is corrected, and after correction, the obtained second image is displayed. By performing direction filtering processing during the calculation of the offset, the present application can screen a large number of feature point pairs, and retain as many feature point pairs with the same manifestation form (such as the same direction or the same slope) as possible, so that the feature point pairs used for calculating the offset are relatively stable, and further make the determined FOV of the corrected image relatively stable, ensuring smooth zooming and smooth switching during camera switching.

[0011] Combined with the first aspect, in some implementation manners of the first aspect, before calculating the vectors formed by each pair of the feature point pairs, the method further includes: removing the feature point pairs with incorrect matching.

[0012] In the embodiments of the present application, through testing, the number of correctly registered feature points in each scenario is more than that of incorrectly registered feature points. Therefore, in order to improve the processing efficiency and the effect of subsequent processing, the incorrectly registered feature points can be removed, and only the correctly registered feature point pairs are retained.

[0013] In combination with the first aspect, in some implementation manners of the first aspect, according to the slope of each of the vectors, all the feature point pairs are divided into multiple point sets, including: sorting the slopes corresponding to all the feature point pairs and setting serial numbers; calculating, in the order of the serial numbers, the difference between the slope corresponding to the feature point pair of the current serial number and the average slope corresponding to the feature point pairs of all the previous serial numbers; if the difference is too large, the feature point pair of the current serial number is recorded as a feature point pair with too large a difference, and the previous group of feature point pairs of the feature point pair recorded as having too large a difference this time, up to the feature point pair recorded as having too large a difference last time, and multiple feature point pairs between the two groups of feature point pairs are divided into one point set.

[0014] In the embodiments of the present application, by performing feature point detection and registration on two frames of images collected by the first camera and the second camera, and then using the slope of the vector formed by each group of feature point pairs to divide all the feature point pairs into multiple point sets, and then determining the offset according to the point set with the largest number of feature point pairs. For two frames of images, the feature point pairs included in the point set with the largest number of feature point pairs have a more consistent manifestation form, and the determined offset is more representative. For the two image streams obtained by the first camera and the second camera, the point set with the largest number of feature point pairs between consecutive multiple frames of images is relatively stable. Therefore, when the two image streams are corrected for offset, the offset amplitude and direction are relatively consistent, and further, during the zooming process and the camera switching process, the FOV of the image can be kept consistent to ensure a smooth transition.

[0015] In combination with the first aspect, in some implementation manners of the first aspect, the method further includes: in response to the first zoom operation; determining that the zoom magnification of the electronic device satisfies the zoom magnification range in which the first camera is for foreground display and the second camera is for background operation.

[0016] It should be understood that foreground display means that the image collected by this camera is used for display; background operation means that the image collected by this camera is used for image processing and not for display.

[0017] In combination with the first aspect, in some implementation manners of the first aspect, the method further includes: in response to the second zoom operation; determining that the zoom magnification of the electronic device satisfies the zoom magnification range in which the first camera is for background operation or turned off, and the second camera is for foreground display.

[0018] In combination with the first aspect, in some implementations of the first aspect, the first zoom operation or the second zoom operation includes: a two-finger reverse sliding operation, a sliding operation on a displayed zoom control, a voice operation, or an air gesture operation.

[0019] In combination with the first aspect, in some implementations of the first aspect, the first camera includes a wide-angle camera, and / or the second camera includes a telephoto camera or an ultra-wide-angle camera.

[0020] In combination with the first aspect, in some implementations of the first aspect, the first zoom operation and the second zoom operation are continuous zoom operations.

[0021] In combination with the first aspect, in some implementations of the first aspect, during the first zoom operation, when the difference between the current zoom ratio and the target zoom ratio is equal to or less than a preset value, the offset remains unchanged and is equal to the first offset; wherein, the target zoom ratio is used to indicate the zoom ratio at which the foreground display is switched from the first camera to the second camera; the first offset is used to indicate the offset determined last time when the difference between the current zoom ratio and the target zoom ratio is greater than the preset value.

[0022] In the embodiments of the present application, by stabilizing the offset at the zoom ratio corresponding to the adjacent camera switching point (such as keeping the offset unchanged and using the offset determined last time before), through the above method, it can be ensured that the change of the offset is linear and stable, so that the image can change smoothly, avoiding abnormalities such as image jitter and jump.

[0023] In combination with the first aspect, in some implementations of the first aspect, the method further includes: determining a parameter value corresponding to the current frame by using the offset between the image collected by the first camera and the image collected by the second camera and all previous offsets; determining the difference between the parameter value corresponding to the current frame and the parameter value corresponding to the previous frame; if it is greater than the threshold, using the offset corresponding to the previous frame as the offset corresponding to the current frame.

[0024] In the embodiments of the present application, at the zoom ratio corresponding to the adjacent camera switching point, when the offset is determined each time, the parameter value is calculated by using the offset and all previous offsets, and the abnormal offset is screened out by the size of the parameter value, and the stable offset is retained. Through the above method, it can be ensured that the change of the offset is linear and stable, so that the image can change smoothly, avoiding abnormalities such as image jitter and jump.

[0025] In combination with the first aspect, in some implementations of the first aspect, the method further includes: if the differences corresponding to three consecutive frames are all greater than the threshold, clear all offsets and re-determine them.

[0026] In combination with the first aspect, in some implementations of the first aspect, the parameter value includes the mean and / or variance of the offsets.

[0027] In the embodiments of the present application, the mean and variance included in the parameter value can be used to represent the stability of all offsets.

[0028] In a second aspect, an electronic device is provided. The electronic device includes: one or more processors, a memory, a first camera, and a second camera; the memory is coupled to the one or more processors, and the memory is used to store computer program code. The computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to perform:

[0029] Open the camera application; display a first image, where the first image is obtained by the first camera capturing an image; receive a first zoom operation; display and save a second image, where the second image is obtained by correcting the image captured by the first camera using the offset between the image captured by the first camera and the image captured by the second camera, and the offset is obtained by performing a direction filtering process on the feature point pairs that match the feature points of the image captured by the first camera and the feature points of the image captured by the second camera.

[0030] It should be understood that the expansions, limitations, explanations, and descriptions of the relevant content in the above first aspect also apply to the same content in the second aspect.

[0031] In a third aspect, a chip system is provided. The chip system is applied to an electronic device. The chip system includes one or more processors, and the processors are used to call computer instructions to cause the electronic device to perform any one of the image processing methods in the first aspect.

[0032] In a fourth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores computer program code. When the computer program code is run by an electronic device, it causes the electronic device to perform any one of the image processing methods in the first aspect.

[0033] In a fifth aspect, a computer program product is provided. The computer program product includes: computer program code. When the computer program code is run by an electronic device, it causes the electronic device to perform any one of the image processing methods in the first aspect. Description of the Drawings

[0034] Figure 1It is a schematic diagram of the hardware system of an electronic device applicable to this application;

[0035] Figure 2 It is a schematic diagram of the software system of an electronic device applicable to this application;

[0036] Figure 3 It is a schematic diagram of the arrangement of multiple cameras on an electronic device provided by an embodiment of this application;

[0037] Figure 4 It is a schematic diagram of the zoom ratio corresponding to different types of cameras provided by an embodiment of this application;

[0038] Figure 5 It is a schematic diagram of an application scenario provided by an embodiment of this application;

[0039] Figure 6 It is a schematic diagram of the images collected by the main camera and the telephoto camera provided by an embodiment of this application;

[0040] Figure 7 It is a set of image stream diagrams related to the prior art;

[0041] Figure 8 It is a schematic flowchart of an image processing method provided by an embodiment of this application;

[0042] Figure 9 It is a schematic flowchart of another image processing method provided by an embodiment of this application;

[0043] Figure 10 It is a schematic flowchart of yet another image processing method provided by an embodiment of this application;

[0044] Figure 11 It is a set of image stream diagrams related to an embodiment of this application;

[0045] Figure 12 It is a schematic diagram of the relationship between the zoom ratio and the offset involved in an embodiment of this application;

[0046] Figure 13 It is a set of image streams related to the prior art;

[0047] Figure 14 It is a schematic flowchart of yet another image processing method provided by an embodiment of this application;

[0048] Figure 15 It is a set of image streams related to an embodiment of this application;

[0049] Figure 16 It is a schematic flowchart of yet another image processing method provided by an embodiment of this application;

[0050] Figure 17 is a schematic diagram of an application scenario of zoom switching provided by an embodiment of the present application;

[0051] Figure 18 is a schematic diagram of an application scenario of zoom switching provided by an embodiment of the present application;

[0052] Figure 19 is a schematic diagram of another electronic device provided in an embodiment of the present application;

[0053] Figure 20 is a schematic diagram of an electronic device suitable for the present application. DETAILED DESCRIPTION

[0054] In the embodiments of the present application, the following terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present embodiment, unless otherwise specified, "plurality" means two or more.

[0055] First, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0056] 1. Field of view (FOV), in optical instruments, is the angle formed by the two edges of the maximum range of the image of the target object that can pass through the lens, with the lens of the optical instrument as the vertex. The size of the field of view angle determines the field of view of the optical instrument. The larger the field of view angle, the larger the field of view and the smaller the optical magnification, that is, the target object will not be captured by the lens if it exceeds this angle. The shorter the focal length, the wider the horizontal field of view, and the smaller the image. The horizontal field of view becomes narrower as the focal length increases, and the object being photographed increases accordingly.

[0057] 2. Registration refers to the matching of geographic coordinates of different images obtained by different imaging methods in the same area, including geometric correction, projection transformation and the same scale.

[0058] 3. Zoom ratio: The zoom ratio is used to indicate the zoom size of an electronic device when shooting.

[0059] The above is a brief introduction to the terms involved in the embodiments of the present application, which will not be repeated below.

[0060] The hardware system, software system and application scenarios of the electronic device provided in the embodiments of the present application are described below in conjunction with the accompanying drawings.

[0061] Exemplarily, the electronic device 100 may be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, a vehicle-mounted electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, etc. The embodiments of the present application do not impose any restrictions on the specific type of the electronic device 100.

[0062] Referring to Figure 1 , the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0063] It should be noted that Figure 1 the structure shown does not constitute a specific limitation on the electronic device 100. In some other embodiments of the present application, the electronic device 100 may include more or fewer components than Figure 1 the components shown, or, the electronic device 100 may include a combination of some of the components shown Figure 1 , or, the electronic device 100 may include sub-components of some of the components shown Figure 1 . Figure 1 The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0064] The processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). Among them, different processing units may be independent devices or integrated devices.

[0065] The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0066] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may store the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0067] Exemplarily, the processor 110 may be used to execute the image processing method of the embodiments of the present application; for example, to start a camera application; to display a first image, which is obtained by the first camera capturing an image; to receive a first zoom operation; to display and save a second image, which is obtained by correcting the image captured by the first camera using the offset between the image captured by the first camera and the image captured by the second camera, and the offset is obtained by performing a direction filtering process on the feature point pairs that match the feature points of the image captured by the first camera and the feature points of the image captured by the second camera.

[0068] Figure 1 The connection relationships shown between the various modules are only illustrative and do not constitute a limitation on the connection relationships between the modules of the electronic device 100. Optionally, the various modules of the electronic device 100 may also adopt a combination of the various connection methods in the above embodiments.

[0069] The wireless communication function of the electronic device 100 may be implemented by devices such as antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor.

[0070] Antenna 1 and Antenna 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, Antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0071] The electronic device 100 can implement a display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0072] The display screen 194 can be used to display images or videos.

[0073] Exemplarily, in the embodiments of the present application, the display screen 194 can be used to display a second image.

[0074] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor, etc.

[0075] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light passes through the lens and is transmitted to the camera photosensitive element. The optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can perform algorithm optimization on the noise, brightness, and color of the image. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0076] The camera 193 (which can also be referred to as a lens) is used to capture still images or videos. It can be triggered to turn on through application instructions to implement the photographing function, such as taking images of any scene. The camera may include components such as an imaging lens, a filter, and an image sensor. The light emitted or reflected by an object enters the imaging lens, passes through the filter, and finally converges on the image sensor. The imaging lens is mainly used to converge and form an image of the light emitted or reflected by all objects in the photographing perspective (which can also be referred to as the scene to be photographed, the target scene, or can be understood as the scene image that the user expects to photograph); the filter is mainly used to filter out the redundant light waves in the light (such as light waves other than visible light, such as infrared); the image sensor can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The image sensor is mainly used to perform photoelectric conversion on the received optical signal, convert it into an electrical signal, and then transmit the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV, etc. formats.

[0077] Exemplarily, the gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, the y-axis, and the z-axis) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake during photographing. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of jitter of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and makes the lens offset the jitter of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used in scenarios such as navigation and motion-sensing games.

[0078] In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0079] Among them, the camera 193 can be located in the front of the electronic device 100 or in the back of the electronic device 100. The specific number and arrangement of the cameras can be set according to requirements, and the present application does not make any restrictions.

[0080] Exemplarily, the electronic device 100 includes a front camera and a rear camera. For example, either the front camera or the rear camera can include 1 or more cameras. Taking the electronic device 100 having 4 rear cameras as an example, in this way, when the electronic device 100 starts 4 rear cameras for photographing, the image processing method provided by the embodiments of the present application can be used.

[0081] Alternatively, the camera is disposed on an external accessory of the electronic device 100. The external accessory is rotatably connected to the frame of the mobile phone, and the angle formed between the external accessory and the display screen 194 of the electronic device 100 is any angle between 0 and 360 degrees. For example, when the electronic device 100 takes a self-portrait, the external accessory drives the camera to rotate to a position facing the user. Of course, when the mobile phone has multiple cameras, only some of the cameras can be disposed on the external accessory, and the remaining cameras can be disposed on the body of the electronic device 100. The embodiments of the present application do not impose any restrictions on this.

[0082] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0083] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0084] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, y-axis, and z-axis) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of shake of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and makes the lens offset the shake of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used in scenarios such as navigation and motion sensing games.

[0085] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally the x-axis, y-axis, and z-axis). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. The acceleration sensor 180E can also be used to identify the posture of the electronic device 100 and serve as an input parameter for application programs such as horizontal and vertical screen switching and pedometer.

[0086] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance through infrared or laser. In some embodiments, for example, in a shooting scenario, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0087] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance during photography. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touch.

[0088] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement functions such as unlocking, accessing the application lock, taking pictures, and answering incoming calls.

[0089] The touch sensor 180K, also known as a touch control device. The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, which is also called a touch control screen. The touch sensor 180K is used to detect touch operations acting on it or in its vicinity. The touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100 and at a different position from the display screen 194.

[0090] The hardware system of the electronic device 100 has been described in detail above. Next, the software system of the electronic device 100 will be introduced.

[0091] Figure 2 It is a schematic diagram of the software system of the electronic device provided by the embodiments of the present application.

[0092] As Figure 2 shown, the system architecture may include an application layer 210, an application framework layer 220, a hardware abstraction layer 230, a driver layer 240, and a hardware layer 250.

[0093] The application layer 210 may include application programs such as a camera application, a gallery, a calendar, a call, a map, a navigation, a WLAN, a Bluetooth, music, a video, and a short message.

[0094] The application framework layer 220 provides application programming interfaces (APIs) and programming frameworks for the application programs in the application layer; the application framework layer may include some predefined functions.

[0095] For example, the application framework layer 220 may include a camera access interface; the camera access interface may include camera management and a camera device. Among them, the camera management can be used to provide an access interface for managing the camera; the camera device can be used to provide an access interface for the camera.

[0096] The hardware abstraction layer 230 is used to abstract the hardware. For example, the hardware abstraction layer may include a camera abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer may call camera algorithms.

[0097] For example, the hardware abstraction layer 230 includes a camera hardware abstraction layer and camera algorithms; the camera algorithms may include software algorithms for image processing.

[0098] Exemplarily, the camera algorithm library may include algorithms corresponding to the image processing method provided in the embodiments of the present application.

[0099] Exemplarily, the algorithms in the camera algorithms may refer to those that do not depend on specific hardware for implementation; for example, code that can generally run on a CPU, etc.

[0100] The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include a camera driver.

[0101] The hardware layer 250 is located at the bottom layer of the operating system; as Figure 2 shown, the hardware layer 250 may include camera 1, camera 2, camera 3, etc. Among them, camera 1, camera 2, and camera 3 may correspond to multiple cameras on the electronic device.

[0102] For the sake of easy understanding, taking the electronic device 100 as a mobile phone with the above software and hardware structure as an example, the cameras on the electronic device 100 to which the method provided in the embodiments of the present application is applicable will be described in detail first.

[0103] The electronic device to which the method provided in the embodiments of the present application is applicable has at least multiple cameras 193. For example, it has 3 types of cameras 193; the 3 types of cameras are respectively a main camera (for example, a wide-angle camera), an ultra-wide-angle camera, and a telephoto camera; the 3 types of cameras can be used to shoot the same scene to be shot.

[0104] Optionally, the electronic device 100 may also have other cameras 193, and the types of the cameras 193 and the number of each type of camera 193 can be set according to needs, and the embodiments of the present application do not impose any restrictions on this.

[0105] Exemplarily, as Figure 3 shown, taking the electronic device 100 that can have 3 cameras 193 as an example for illustration; the arrangement of the 3 cameras can be as shown in (a) in Figure 3 , or as shown in (b) in Figure 3 ; for example, the 3 cameras 193 can be a main camera 1931 (for example, a wide-angle camera), an ultra-wide-angle camera 1932, and a telephoto camera 1933.

[0106] It should be understood that the above are only examples of two arrangements, and there can also be other arrangements; the specific arrangement can be designed and changed according to needs, and the embodiments of the present application do not impose any restrictions on this.

[0107] It should be noted that during the shooting of the above three cameras, generally, the field of view angle range corresponding to the main camera 1931 is larger than the field of view angle range corresponding to the telephoto camera 1933; and the field of view angle range corresponding to the ultra-wide-angle camera 1932 is larger than the field of view angle range corresponding to the main camera 1931; there may be an overlap between the field of view angles of the ultra-wide-angle camera 1932 and the main camera 1931; that is to say, the ultra-wide-angle camera 1932 can capture the scene content captured by the main camera 1931 and the surrounding scene content.

[0108] It should be understood that the field of view angle range corresponding to the telephoto camera 1933 is smaller than the field of view angle range corresponding to the main camera 1931, and there may be an overlap between the field of view angles of the main camera 1931 and the telephoto camera 1933; that is to say, the main camera 1931 can capture the scene content captured by the telephoto camera 1933 and the surrounding scene content. There may be an overlap between the field of view angles of the ultra-wide-angle camera 1932 and the telephoto camera 1933; that is to say, the ultra-wide-angle camera 1932 can capture the scene content captured by the telephoto camera 1933 and the surrounding scene content.

[0109] Among them, due to the small focusing distance, the ultra-wide-angle camera 1932 is suitable for shooting close-ups; and, as the name implies, the ultra-wide-angle camera 1932 is suitable for shooting scenes with a relatively large field of view angle; the main camera 1931, due to its high clarity, is more suitable for shooting portraits, while the telephoto camera 1933 is more suitable for shooting long-distance close-ups.

[0110] Exemplarily, as Figure 4 shown, the zoom ratio of the ultra-wide-angle camera can be less than the M-fold zoom ratio; the zoom ratio range of the wide-angle camera, i.e., the main camera, can be [M, N); the zoom ratio of the telephoto camera can be greater than or equal to the N-fold zoom ratio.

[0111] For example, M can be 1 and N can be 2.5; then the zoom ratio of the ultra-wide-angle camera is less than 1-fold zoom ratio (1×); the zoom ratio range of the wide-angle camera is from 1-fold zoom ratio to 2.5-fold zoom ratio [1×~2.5×); the zoom ratio of the telephoto camera is greater than or equal to 2.5-fold zoom ratio.

[0112] It should be understood that during the shooting process of the electronic device, the larger the zoom ratio, the smaller the corresponding field of view angle.

[0113] Next, in combination with Figure 5An example of the application scenario of the image processing method provided in the embodiments of this application is given.

[0114] The method in the embodiments of this application can be applied to scenarios such as taking pictures, previewing, recording videos, or video calling; through the method in the embodiments of this application, smooth zooming and / or smooth switching between different cameras in an electronic device can be achieved, improving the user's shooting experience and image quality.

[0115] Exemplarily, the preview scenario includes but is not limited to the following scenarios:

[0116] Photo preview, aperture preview, night scene preview, portrait preview, video recording preview, or professional preview, etc.

[0117] It should be understood that the preview scenario may refer to the scenario where an electronic device captures images before clicking the button indicating shooting in a certain shooting mode.

[0118] In one example, as shown in (a) of Figure 5 , after the electronic device enters the camera application, the default photo-taking mode can be enabled; in the photo-taking mode, the electronic device can enter the default shooting mode, which may refer to the shooting mode with the wide-angle camera as the main camera and the zoom ratio as the single zoom ratio (1×), and the electronic device displays the image captured by the main camera; as shown in (b) of Figure 5 , in response to the user's operation, when the zoom ratio meets the zoom range corresponding to the telephoto camera, the electronic device can switch to display the image captured by the telephoto camera.

[0119] Exemplarily, the image processing method in the embodiments of this application can also be applied to the video calling scenario, where the video calling scenario may include but is not limited to the following scenarios:

[0120] Video calling, video conferencing applications, long and short video applications, video live streaming applications, video online course applications, portrait intelligent panning application scenarios, recording videos using the system camera video recording function, video surveillance, or portrait shooting scenarios such as intelligent doorbells, etc.

[0121] It should be understood that the above is an example of the application scenario and does not limit the application scenario of this application in any way.

[0122] Currently, during shooting, in response to the user's operation, the electronic device can perform zoom shooting by switching cameras with different focal lengths, and can also process the captured image in combination with digital zoom to meet various high-magnification shooting scenarios. During the zooming process, in order to achieve smooth zooming, the SAT algorithm is usually adopted in the prior art (in the image streams collected by the two cameras used for zoom switching, each pair of simultaneously collected images is corrected to achieve the consistency of the FOV).

[0123] Exemplarily, as Figure 6 shown, when zooming from the main camera to the telephoto camera, before the switch, both the main camera and the telephoto camera collect images, and the image collected by the main camera is displayed. After the switch, both the main camera and the telephoto camera continue to collect images, but the image collected by the telephoto camera is displayed. During this zooming process, in order to keep the zoom smooth, it is necessary to make the FOV of each pair of images collected by the main camera and the telephoto camera consistent. For this, during the process of increasing the zoom ratio, usually taking the image collected by the telephoto camera as the reference, the correction parameters for transforming the image collected by the main camera to the image collected by the telephoto camera are calculated, and then the correction parameters are applied to the image collected by the main camera, so that the FOV of the corrected image is consistent with that of the image collected by the telephoto camera. Among them, the correction parameters generally include the rotation amount representing the rotation relationship and the offset amount representing the offset relationship. The rotation amount can be obtained through the existing related technologies and will not be introduced here. In the prior art, the offset amount can be calculated based on the difference of feature points after feature point registration of the image collected by the main camera and the image collected by the telephoto camera.

[0124] However, this calculation process cannot guarantee the accuracy of registration. For example, in the case of a repetitive texture scene, or a weak texture scene, etc., there may be misregistered feature points. The misregistration of feature points will lead to the error in calculating the difference of feature points, that is, the offset amount, resulting in incorrect correction parameters, and further the FOV of the image corrected by the correction parameters cannot be consistent with that of the image collected by the telephoto camera. The repetitive texture scene refers to the image content with repetitive styles, patterns, etc., such as flower patterns, tree patterns, etc. The weak texture scene refers to the texture features with relatively small eigenvalues, such as wall cracks, brick patterns, etc.

[0125] In addition, even if all feature points are correctly registered, the offsets of feature points corresponding to different depths are different, and a unified offset cannot be used to represent the offsets of all feature points in the image. For example, if the depths of the photographed objects are different, such as the depths of the foreground and the background, the depths corresponding to the feature points extracted in the foreground and the background are different. Then, since the foreground and the background in the image have a perspective relationship of being larger near and smaller far away, the offset of the feature points in the foreground for the same actual physical distance is larger, and the offset of the feature points in the background is smaller. Therefore, when performing correction, the offset determined in a fixed way cannot be used as the offset of all feature points in the image.

[0126] However, the center-priority and foreground-priority processing strategies provided in the prior art result in that, after the depth is calculated, feature points with a smaller depth or a closer depth are collected to calculate the offset; when the feature points with a closer depth extracted from the foreground are insufficient during the zoom process, feature points with a farther depth in the background are added to calculate the offset, resulting in abnormal image effects after correction.

[0127] For example, Figure 7 is a set of image streams involved in the prior art. Figure 7 As shown in (a) to (f) in the figure, assuming that the captured image only has the foreground and the background, during the zoom process, there is a large difference between the foreground and the background of the images of the two cameras. If you want to align the foreground, the background will not be aligned; similarly, if you want to align the background, the foreground will not be aligned. Since only one depth can be guaranteed each time, the algorithm needs to continuously determine which depth of field to align, the foreground or the background, which causes the foreground and the background to jump repeatedly during the zoom process, and the camera cannot be switched smoothly.

[0128] In view of this, the embodiment of the present application provides an image processing method and related equipment; in the embodiment of the present application, the offset between the two images can be obtained by performing feature point detection and registration on the image captured by the first camera and the image captured by the second camera, and then performing directional filtering on the matched feature point pairs; the image captured by the first camera is corrected in combination with the offset, and the obtained second image is displayed after correction. The present application can screen a large number of feature point pairs by performing directional filtering in the process of calculating the offset, and retain feature point pairs with consistent expressions (such as consistent directions or consistent slopes) as much as possible, so that the feature point pairs used to calculate the offset can be relatively stable, and then the determined corrected image FOV can be relatively stable, ensuring smooth zooming and smooth switching during camera switching.

[0129] Optionally, the direction filtering process may include dividing all pairs of feature points into multiple point sets by using the slopes of the vectors formed by each pair of feature points, and then determining the offset according to the point set with the largest number of pairs of feature points.

[0130] For two frames of images, the pairs of feature points included in the point set with the largest number of pairs of feature points have a more consistent manifestation form, and the determined offset is more representative. For the two image streams obtained by the first camera and the second camera, the point set with the largest number of pairs of feature points among consecutive multiple frames of images is relatively stable. Therefore, when the two image streams are corrected for offset, the offset amplitude and direction are relatively consistent. Furthermore, during the zooming process and the camera switching process, the FOV of the image can be kept consistent to ensure a smooth transition.

[0131] The following combines Figure 8 to describe in detail the schematic flowchart of the image processing method provided by the embodiments of the present application.

[0132] Figure 8 is the schematic flowchart of the image processing method provided by the embodiments of the present application. This method can be executed by Figure 1 the electronic device shown; this method 300 includes steps S310 to S340, and the steps S310 to S340 will be described in detail below.

[0133] It should be understood that in the embodiments of the present application, the electronic device includes a first camera and a second camera; wherein, the first camera and the second camera are cameras of different types.

[0134] S310. Start the camera application.

[0135] Exemplarily, the user can click the icon of the "Camera" application to instruct the electronic device to start the camera application.

[0136] Exemplarily, when the electronic device is in the locked screen state, the user can use a gesture of swiping right on the display screen of the electronic device to instruct the electronic device to start the camera application. Or, when the electronic device is in the locked screen state and the locked screen interface includes the icon of the camera application, the user clicks the icon of the camera application to instruct the electronic device to start the camera application. Or, when the electronic device is running other applications and the application has the permission to call the camera application; the user can click the corresponding control to instruct the electronic device to start the camera application. For example, when the electronic device is running an instant messaging application, the user can select the control of the camera function to instruct the electronic device to start the camera application, etc.

[0137] It should be understood that the above is an illustrative description of the operation of opening the camera application; the camera application can also be opened by voice instructions or instructions of other operations for the electronic device; the present application does not make any limitation thereto.

[0138] It should also be understood that starting the camera application may refer to running the camera application.

[0139] S320. Display a first image, where the first image is obtained by the first camera collecting an image.

[0140] Exemplarily, the first camera may be Figure 3 the wide-angle camera or the main camera shown in the figure, or may also be other cameras with a larger field of view angle range than the telephoto camera.

[0141] Exemplarily, the first image may be an RGB image collected by the main camera, or rather, the first image may be an RGB image obtained by processing the Raw image collected by the main camera through a series of camera algorithms.

[0142] For example, Figure 17 the preview image shown in (a) in the figure is the first image collected and sent for display by the first camera. Another example, Figure 18 the video image shown in (a) in the figure is the first image collected and sent for display by the first camera.

[0143] Optionally, when displaying the first image collected by the first camera, the depth corresponding to the shooting object may be collected together.

[0144] Exemplarily, before displaying the first image, the first camera may also perform parallax detection on the content in the first image to determine the depths corresponding to different shooting objects included in the first image.

[0145] For example, as shown in (a) in Figure 18 the figure, when the first camera sends the first image for display, it may perform parallax detection to determine the depths corresponding to different shooting objects, and the depth information may or may not be displayed on the first image. The present application does not make any limitation thereto.

[0146] It should be understood that the first image may include one or more shooting objects, and when displayed, a detection frame may be displayed for each shooting object determined based on AI detection.

[0147] It should also be understood that the type of the shooting object can be preset as needed. For example, when the set type is a human face, the target object can be used to indicate a human face recognized in the first image. The type of the shooting object may also include plants, animals, etc. The embodiments of the present application do not make any limitation thereto.

[0148] S330. A first zoom operation is received.

[0149] Optionally, receiving the first zoom operation may include: receiving a first zoom operation for a first image.

[0150] It should be understood that the zoom operation for the first image may indicate a user's operation instruction, or may also indicate a zoom operation instruction automatically triggered by the electronic device based on AI detection. The target object is one of the shooting objects included in the first image.

[0151] Exemplarily, when the zoom operation indicates an operation instruction triggered by the user, the zoom operation may include a two-finger reverse sliding operation, a click operation, a voice operation, an air gesture operation, etc. for the first image. The embodiments of the present application do not impose any restrictions on this.

[0152] Exemplarily, as shown in Figure 18 (a) in and Figure 18 (b) in, when the electronic device displays one or more shooting objects and zoom controls, the zoom operation may include a click operation on the target object in the first image and a sliding operation on the zoom control; or, the zoom operation may also include a click operation on the target object in the first image and a two-finger reverse sliding operation. It should be understood that when the zoom operation includes two operations, when executing, the user needs to first execute the click operation, and then execute the sliding operation on the zoom control or the two-finger reverse sliding operation on the target object. The zoom operation may also be other operations, or the zoom operation may also include three or more sub-operations. The embodiments of the present application do not impose any restrictions on this.

[0153] S340. Display and save a second image.

[0154] The second image is obtained by correcting the image collected by the first camera using the offset between the image collected by the second camera. The offset is obtained by performing a direction filtering process on the feature point pairs that match the feature points of the image collected by the first camera and the feature points of the image collected by the second camera.

[0155] Exemplarily, the first image may be an RGB image, and the corresponding second image is an RGB image.

[0156] It should be understood that in response to the first zoom operation, feature point detection and registration are performed on the images captured by the first camera and the second camera; since the first camera and the second camera capture the same shooting scene, a large number of feature points indicating the same target exist in the two frames of images extracted, so multiple pairs of matching feature point pairs can be obtained; by performing direction filtering processing on the matching feature point pairs, the offset of the images captured by the two cameras can be obtained, and based on this offset, the image captured by the first camera is corrected to obtain a second image.

[0157] For example, the first zoom operation is an operation with an increasing zoom ratio, or the first zoom operation is an operation with a decreasing zoom ratio. When it is an operation with an increasing zoom ratio, the size of the shooting object included in the second image is larger than the size of the shooting object included in the first image. When it is an operation with a decreasing zoom ratio, the size of the shooting object included in the second image is smaller than the size of the shooting object included in the first image.

[0158] Optionally, the target object in the second image is centered in the second image.

[0159] Optionally, the second image includes all or partial content of the target object. For example, as Figure 18 shown, assuming the target object is the fourth shooting object, after magnification, the second image may include the entire fourth shooting object, or the second image may also include partial content of the fourth shooting object, such as the face of the fourth shooting object; when further magnified, the second image may also include partial content of the face of the fourth shooting object.

[0160] Optionally, in order to smoothly magnify the target object displayed by the electronic device during the zoom process, the above S340 can be executed multiple times in a loop, and each time the target object in the obtained second image is a little larger than the target object obtained last time. That is, in response to the zoom operation, the electronic device corrects the image captured by the first camera using the offset between the images captured by the second camera, obtains and displays multiple second images, all of which include the target object, and the target object in these multiple second images gradually becomes larger from the original size. The number of loop executions and the size change range can be set as needed, and the present application does not limit this. Among them, the original size indicates the size of the target object in the first image.

[0161] It should be understood that S340 can be executed while S330 is being executed, or S340 can be executed after S330 is executed. The embodiments of the present application do not impose any restrictions on this.

[0162] Exemplarily, when in the preview scenario, a second image can be displayed. Among them, the preview shooting modes include but are not limited to: for example, night scene preview mode, video recording preview mode, photo shooting preview mode, portrait preview mode, and other shooting modes.

[0163] Exemplarily, when in the video recording scenario, the second image can be displayed and saved.

[0164] Optionally, the method further includes:

[0165] In response to a first zoom operation;

[0166] Determine that the zoom ratio of the electronic device satisfies the zoom multiple range for the front-end display of the first camera and the background operation of the second camera.

[0167] Exemplarily, in response to a first zoom operation, when the zoom ratio has reached the minimum zoom ratio of the second camera but has not reached the maximum zoom ratio of the first camera (for example, the telephoto camera has a minimum of 2x and the main camera has a maximum of 5x), at this time, continue to display the second image obtained by the first camera, but this second image is obtained after correcting the offset between the image collected by the first camera and the image collected by the second camera. The offset is obtained by performing direction filtering processing on the feature point pairs that match the feature points of the image collected by the first camera and the feature points of the image collected by the second camera.

[0168] For example, the first zoom operation is to switch from 1x to 4x. The first camera has been displaying in the foreground, but when zooming to 2x, the second camera starts to run in the background. At this time, the method provided in the embodiments of the present application can be triggered to generate a second image for display and saving by calculating the offset and correcting the image collected by the first camera.

[0169] The embodiments of the present application provide an image processing method and related devices; in the embodiments of the present application, by performing feature point matching on the images collected by the first camera and the second camera, and then performing direction filtering processing on the matched feature point pairs, the offset between the two images can be obtained; then, in combination with this offset, the image collected by the first camera is corrected, and the corrected second image is displayed. In the present application, by performing direction filtering processing during the calculation of the offset, a large number of feature point pairs can be screened, and the feature point pairs with the same manifestation form (such as the same direction or the same slope) are retained as much as possible, so that the feature point pairs used for calculating the offset are relatively stable, and further the FOV of the corrected image determined is relatively stable, ensuring smooth zooming and smooth switching during the camera switching process.

[0170] Exemplarily, Figure 11 is a set of image streams related to the present application. Such asFigure 11 as shown in (a), (b), (c), (d), (e) to Figure 11 (f) thereof, assuming that the captured image only has a foreground and a background, during the zooming process, since there are more feature points with a greater depth in the background and the features are relatively consistent, therefore, during the zooming process, the offset is mainly calculated based on the feature points in the background. In this way, the corrected image can be smoothly enlarged with the background as the reference all the time, and further, the FOV of the image stream can be smoothly enlarged, and the camera can be smoothly switched during the zooming process.

[0171] Figure 9 An exemplary schematic flowchart of an image processing method for another electronic device 100 is shown.

[0172] Taking the first camera as the main camera and the second camera as the telephoto camera as an example, the method 400 includes S401 to S408; S401 to S408 will be described in detail below.

[0173] S401. Obtain a first initial image captured by the main camera.

[0174] Exemplarily, the image captured by the main camera can be a Raw image or a YUV image captured by a wide-angle camera.

[0175] S402. Obtain a second initial image captured by the telephoto camera.

[0176] Exemplarily, the image captured by the telephoto camera can be a Raw image or a YUV image captured by the telephoto camera.

[0177] S403. Perform image feature point detection and registration on the first initial image and the second initial image to obtain multiple pairs of feature point pairs.

[0178] Optionally, any existing image feature point detection algorithm can be used for image feature point detection, and the present application does not make any limitation thereto.

[0179] Exemplarily, image feature point detection is respectively performed on the first initial image and the second initial image, M feature points in the first initial image and N feature points in the second initial image are extracted; then, the M feature points and the N feature points are registered. Both M and N are integers greater than 0.

[0180] It should be understood that each pair of feature points includes a feature point belonging to the first initial image and a feature point belonging to the second initial image, and the two feature points are used to indicate the same target. The registered feature points have a one-to-one correspondence.

[0181] Optionally, after S403, the method may further include: screening the correctly registered feature point pairs.

[0182] Specifically, the correctly registered feature point pairs can be screened out by removing the misregistered feature point pairs.

[0183] After testing, the number of correctly registered feature points in each scenario is more than that of misregistered feature points. Therefore, in order to improve the processing efficiency and the effect of subsequent processing, the misregistered feature points can be removed, and only the correctly registered feature point pairs are retained.

[0184] S404. Calculate the vectors formed by each pair of feature points among all the feature point pairs.

[0185] This vector is used to indicate the direction and length between the feature point pairs.

[0186] S405. Determine the slope of each vector.

[0187] It should be understood that the slope is the direction of the vector. When the slopes of multiple vectors are the same, it means that the offsets of the corresponding feature point pairs are the same, that is, it means that the depths of the corresponding feature point pairs are the same.

[0188] S406. Divide all the feature point pairs into multiple point sets according to the slope of each vector. Each point set can contain one or more pairs of feature point pairs.

[0189] Exemplarily, different slope thresholds can be set, or based on the difference magnitude between adjacent slopes, or all the feature point pairs can also be divided into multiple point sets according to the slope by using methods such as clustering.

[0190] For example, there are 10 feature point pairs, and the slopes corresponding to the determined 10 vectors are 0.81, 0.81, 0.81, 0.82, 0.84, 0.84, 0.85, 0.85, 0.85, 0.88 respectively. According to the difference magnitude between adjacent slopes, these 10 slopes can be divided into three point sets. Among them, the slopes corresponding to the first point set include 0.81, 0.81, 0.81, 0.82; the slopes corresponding to the second point set include 0.84, 0.84, 0.85, 0.85, 0.85; the slope corresponding to the third point set includes 0.88.

[0191] Optionally, the above S406 may include the following S4061 to S4063, which are introduced separately below for S4061 to S4063.

[0192] S4061. Sort the slopes corresponding to all the feature point pairs and set serial numbers.

[0193] Here, only for the convenience of subsequent calculations, all pairs of feature points can be sorted in descending or ascending order of slope, and serial numbers are set for each pair of feature points in order.

[0194] S4062. Calculate the difference between the slope corresponding to the pair of feature points with the current serial number and the average slope corresponding to all pairs of feature points with previous serial numbers, in the order of the serial numbers.

[0195] It should be understood that when the serial number is 1, the average slope has not been calculated before. Therefore, when calculating, the calculation can start from the slope corresponding to the pair of feature points with the 2nd serial number, and the average slope corresponding to all pairs of feature points with previous serial numbers is the slope corresponding to the pair of feature points with the 1st serial number.

[0196] S4063. If the difference is too large, the pair of feature points with the current serial number is recorded as the pair of feature points with too large a difference, and the pair of feature points before the pair of feature points recorded as having too large a difference this time, up to the pair of feature points recorded as having too large a difference last time, and multiple pairs of feature points between the two pairs of feature points are divided into a point set.

[0197] Optionally, a difference threshold can be set. If the determined difference is greater than or equal to the difference threshold, it means the difference is too large and S4063 needs to be executed; if the determined difference is less than the difference threshold, it means the difference is not large and the S4062 and S4063 can be repeatedly executed for the next pair of feature points.

[0198] Exemplarily, assume that the difference threshold is 0.02. If the slopes corresponding to the 10 determined vectors are sorted in ascending order of slope, the 10 slopes are 0.801, 0.801, 0.801, 0.802, 0.804, 0.804, 0.805, 0.805, 0.805, 0.808 respectively. According to the serial number order, the difference between the slope 0.801 corresponding to the feature point pair with serial number 2 and the slope 0.801 corresponding to the feature point pair with serial number 1 is 0, which means there is no difference, so continue to calculate the next one; the difference between the slope 0.801 corresponding to the feature point pair with serial number 3 and the average slope 0.801 of the two groups of feature point pairs with serial numbers 1 and 2 is still 0, which means there is no difference, and continue to calculate the next one; the difference between the slope 0.802 corresponding to the feature point pair with serial number 4 and the average slope 0.801 of the three groups of feature point pairs with serial numbers 1 to 3 is 0.01. This difference is less than the difference threshold 0.02, and the difference is not significant, so continue to calculate the next one. The difference between the slope 0.804 of the feature point pair with serial number 5 and the average slope 0.80125 of the four groups of feature point pairs with serial numbers 1 to 4 is 0.0275. This difference is greater than the difference threshold 0.02, indicating that the feature point pair with serial number 5 is inconsistent with the previous vector features. Therefore, the four groups of feature point pairs with serial numbers 1 to 4 can be divided into a point set, and this point set includes these 4 groups of feature point pairs.

[0199] S407. Determine the offset according to the feature point pairs in the point set with the largest number of feature point pairs.

[0200] S408. Combine this offset to correct the first initial image to obtain the corrected image.

[0201] Optionally, when correcting the first initial image, the rotation amount determined by the relevant method and the offset determined by the method provided in the embodiments of the present application can be combined to correct the first initial image together to obtain the corrected image.

[0202] In the embodiments of the present application, by performing feature point detection and registration on two frames of images collected by a first camera and a second camera, then, using the slopes of the vectors formed by each pair of feature points, all pairs of feature points are divided into multiple point sets, and then the offset is determined according to the point set including the largest number of pairs of feature points. For two frames of images, the pairs of feature points included in the point set with the largest number of pairs of feature points have a more consistent manifestation form, and the determined offset is more representative. For two image streams obtained by the first camera and the second camera, the point set including the largest number of pairs of feature points among consecutive multiple frames of images is relatively stable. Therefore, when the two image streams are corrected for offset, the offset amplitude and direction are relatively consistent. Furthermore, during the zooming process and the camera switching process, the FOV of the image can be kept consistent, ensuring a smooth transition.

[0203] In the embodiments of the present application, the offsets in different zoom magnification ranges are obtained by allocating the total offset. For example, the offset is usually allocated in the form of an exponential curve. The unit of the offset is pixels.

[0204] Exemplarily, as Figure 12 shown in (a) of, taking the zoom magnification of switching from the first camera to the second camera for display as 2.5 as an example, the closer to the camera switching point, the larger the offset amplitude allocated for a smaller range of zoom magnification. For example, when zooming to 1.1x, the total offset may be 100, and the offset allocated for this zoom magnification may correspond to 10; when zooming to 1.2x, the total offset may be 105, and the offset allocated for this zoom magnification may be 10.5; when zooming to 2.45x, the total offset may be 100, and the offset allocated for this zoom magnification may be 96; when zooming to 2.48x, the total offset may be 105, and the offset allocated for this zoom magnification may be 101.

[0205] It can be seen from this that for the zoom magnification range of 1.1x to 1.2x, which is far from the switching point of 2.5x, a smaller offset is allocated, and there is an offset difference of 0.5 for a zoom magnification amplitude of 0.1x; while for the zoom magnification range of 2.45x to 2.48x, which is close to the switching point of 2.5x, a larger offset is allocated, and there is an offset difference of 5 pixels for a zoom magnification amplitude of 0.03x.

[0206] It should be understood that the total offset is calculated from two images corresponding to two cameras. Taking the first camera as the main camera and the second camera as the telephoto camera, and the zoom ratio in the range of [1x, 2.5x] as an example, the specific steps are as follows: Crop the main camera image at any zoom ratio so that the FOV size is aligned with the telephoto image (the FOV of the main camera image will change during the zoom process, but the FOV size of the telephoto image is always the one corresponding to 2.5x). Then, register the two images with the same FOV size (but with rotational offset), and the offset between the two images can be obtained. This can be understood as the total offset (the offset corresponding to the FOV of 2.5x. Different zoom ratios need to be aligned at this scale for comparison. The offsets mentioned below are all at this scale). In theory, it also needs to be mapped to the current FOV size of the main camera.

[0207] Assume that the camera is fixed. Through the above calculation method, the total offset at different zoom ratios is also fixed. Assume it is 100 for all. Then, based on the allocation logic (for the convenience of example, assume it is linear, but actually it is a curve), the offset allocated to 1x is 0, the offset allocated to 2.5x is 100. Then, the offset allocated to 1.1x is 100*(1.1 - 1) / (2.5 - 1) ≈ 6.667, the offset allocated to 1.2x is approximately 13.333, and the offset allocated to 2x is approximately 66.667. The calculation processes for other zoom ratios are the same and will not be introduced here.

[0208] When the camera is not fixed, through the above calculation method, the images corresponding to each zoom ratio will be different. This difference comes from various situations. Here, we mainly consider the differences in imaging caused by camera movement. Then, the selection of feature points, registration, and the resulting offset results will also be different accordingly. However, since these differences are small, theoretically, the main body of the image changes little. Therefore, the total offset is of the same order of magnitude as the offset when the camera is fixed. Then, there may be total offsets such as 105, 98, 103, etc. at different zoom ratios. So, theoretically, there are burrs and fluctuations throughout the process, rather than sudden occurrences.

[0209] Based on this, for the allocation, the offset corresponding to 1.1x may be 7 (total offset is 105), may be 6.53 (total offset is 98), or may be 6.86 (total offset is 103); the offset corresponding to 2.4x may be 98, 91.42, 93.52, etc. It can be seen that due to the allocation logic, when the zoom ratio is small, the difference in the total offset will be reduced (the difference between 105 and 98 is 7 pixels, but when allocated to 1.1x, it only differs by 0.5 pixels, which is indistinguishable to the naked eye). However, when the zoom ratio is large, this difference will be obvious, thus causing visible jitter.

[0210] Another visually visible cause of jitter is that the total offset theoretically needs to be mapped to the current FOV size of the main camera. The change from 1x to 1.1x FOV is 1.1 times (magnified by 1.1 times), while the change from 2.4x to 2.5x FOV is 2.5 / 2.4 = 1.042 times (magnified by 1.042 times). Therefore, the magnification from 1x to 1.1x is more obvious visually, and the offset will be weakened. The magnification from 2.4x to 2.5x is not obvious, so it is easier to see the change in the offset.

[0211] Exemplarily, affected by external environmental factors or the jitter of the electronic device itself (such as hand jitter caused by the user's breathing fluctuations when the user holds the electronic device to take pictures), etc., the image content collected by the camera is offset. Therefore, the offset allocated to each frame of image may have a floating offset, as shown in (b) of Figure 12 , that is, there will be burrs / fluctuations during the entire zoom process. Then, reflected on the display interface, there will be no obvious jitter (mainly indistinguishable to the naked eye) at the switching point far from the camera, and an abnormal situation of obvious jitter of the displayed content will occur at the switching point close to the camera.

[0212] In addition, during the zoom process, for example, when there is a moving target in the scene (such as a running puppy), when the first camera and the second camera are out of focus during the zoom process, and when the jitters of the two cameras are inconsistent, etc., there will also be an abnormal situation of the offset value, which will further cause abnormal jitter effects in one or two frames of images during the display process.

[0213] Exemplarily, Figure 13 is a set of image streams related to the prior art. During the zoom process of (a), (b) to (d) in Figure 13 , due to one of the above reasons, an abnormal offset occurs, and then abnormal phenomena such as image jitter and jump as shown in (c) of Figure 13 will occur.

[0214] In response to this, the embodiment of the present application also provides an image processing method. In the embodiment of the present application, by stabilizing the offset at the zoom ratio corresponding to the camera switching point (such as keeping the offset unchanged and using the last determined offset before), and / or, when determining the offset each time, using the offset and all the previous offsets to calculate a parameter value, and screening out the abnormal offset through the size of the parameter value, and retaining the stable offset. Through the above method, it can be ensured that the change of the offset is linear and stable, so that the image can change smoothly and avoid abnormal situations such as image jitter and jump.

[0215] Figure 14Schematic flowchart exemplarily showing another image processing method for an electronic device 100. The method 500 includes S501 to S506; S501 to S506 will be described in detail below.

[0216] S501. Start the camera application.

[0217] S502. Display the first image.

[0218] S503. Receive the first zoom operation.

[0219] S501 to S503 are the same as those described in the foregoing S310 to S330, and reference can be made to the above introduction, which will not be elaborated here.

[0220] S504. During the first zoom operation, when the difference between the current zoom ratio and the target zoom ratio is equal to or less than a preset value, the offset remains unchanged and is equal to the first offset;

[0221] wherein, the target zoom ratio is used to indicate the zoom ratio for switching from the front camera for foreground display to the second camera for foreground display; the first offset is used to indicate the offset determined last time when the difference between the current zoom ratio and the target zoom ratio is greater than the preset value.

[0222] For example, the target zoom ratio is 2.5x and the preset value is 0.1x; during the first zoom operation from 1x to 2.5x, when zooming from 1x to 2.4x, since the difference between 2.4x and 2.5x is equal to the preset value, at this time, the offset may no longer be calculated or allocated, but the first offset is used. The first offset is the offset determined last time or the most recent time, and the offset corresponding to the current frame continues to use the offset calculated previously. After that, when zooming from 2.4x to 2.5x, the offset continues to use the first offset.

[0223] It should be understood that the situation where the offset shows Figure 12 the burrs or fluctuations shown usually appears near the zoom ratio at the switching point of the two cameras. During use, it rarely stays in this focal range for a long time. Because, on the one hand, to solve the burr problem, and on the other hand, to save computing resources, therefore, the above method can be used to adopt the offset determined by the last frame before near the editing ratio at the switching point of the two cameras. Under the allocation logic, it can ensure that the change of the offset is linear, thereby solving the image jitter problem.

[0224] S505. Combine the offset between the image collected by the first camera and the image collected by the second camera, and all the previous offsets to determine the parameter value corresponding to the current frame.

[0225] Optionally, the parameter value includes the mean and / or variance of the offset. Of course, the parameter value can also include other types of numerical values, which are not limited in the embodiments of the present application.

[0226] S506. Determine the difference between the parameter value corresponding to the current frame and the parameter value corresponding to the previous frame; if it is less than the threshold, update the offset, that is, take the determined new offset as the offset corresponding to the current frame. If it is greater than the threshold, execute S507; if the differences corresponding to three consecutive frames are all greater than the threshold, execute S508.

[0227] In the embodiments of the present application, the threshold comparison method after mean and variance statistics adopts the 3sigma theory. For related technologies, reference can be made, and details are not described here.

[0228] S507. Use the offset corresponding to the previous frame as the offset corresponding to the current frame.

[0229] S508. Clear all offsets and re-determine them.

[0230] For example, when the second frame is the current frame, calculate the variance a1 and mean b1 of P1 and P2, where P1 is the offset between the first frame image collected by the first camera and the second frame image collected by the second camera, and P2 is the offset between the second frame image collected by the first camera and the second frame image collected by the second camera. If the difference is greater than the threshold, use the offset corresponding to the first frame as the offset corresponding to the second frame. If the differences determined for the subsequent third frame and fourth frame are both greater than the threshold according to the above method, clear all offsets and re-calculate.

[0231] When the third frame is the current frame, calculate the variance a2 and mean b2 of P1, P2, and P3, where P3 is the offset between the third frame image collected by the first camera and the third frame image collected by the second camera, and P1 and P2 are the offsets between the first frame image collected by the first camera and the first frame image collected by the second camera, and between the second frame image collected by the first camera and the second frame image collected by the second camera respectively, as the parameter value corresponding to the third frame. Then, determine the difference between the variance a2 and the variance a1 corresponding to the second frame, and / or determine the difference between the mean b2 and the mean b1 corresponding to the second frame. If the difference is greater than the threshold, use the offset corresponding to the second frame as the offset corresponding to the third frame. If the differences determined for the subsequent fourth frame and fifth frame are both greater than the threshold according to the above method, clear all offsets and re-calculate.

[0232] It should be understood that the mean and variance included in the parameter values can be used to represent the stability of all offsets. It should also be understood that by calculating the difference between the parameter value corresponding to the current frame and the parameter value corresponding to the previous frame, when the difference is large, that is, the offset is unstable, the offset corresponding to the previous frame can be used as the offset corresponding to the current frame to avoid large differences in offsets, thereby avoiding jumps in the image.

[0233] In the embodiments of the present application, by stabilizing the offset at the zoom ratio corresponding to the adjacent camera switching point (if the offset remains unchanged, use the last determined offset before); and / or, when determining the offset each time, use the offset and all previous offsets to calculate the parameter value, and screen out abnormal offsets through the magnitude of the parameter value, and retain stable offsets. Through the above method, it can be ensured that the change of the offset is linear and stable, so that the image can change smoothly, avoiding abnormalities such as image jitter and jump.

[0234] Exemplarily, Figure 15 is a set of image streams related to the embodiments of the present application. As Figure 15 In the zooming processes of (a), (b), (c) to (d) as shown, by processing in combination with the image processing method provided in the present application, abnormal offsets can be avoided, so that the image can change smoothly and no longer exhibit abnormalities such as image jitter and jump.

[0235] Figure 16 Exemplarily shown is a schematic flowchart of an image processing method of another electronic device 100. The method 600 includes S601 to S609; S601 to S609 are described in detail below.

[0236] S601. Obtain the image collected by the main camera.

[0237] Exemplarily, the image collected by the main camera can be a Raw image collected by a wide-angle camera; or, the image collected by the main camera can be a Raw image collected by an ultra-wide-angle camera.

[0238] S602. First front-end processing.

[0239] Optionally, the first front-end processing may include algorithms related to converting the Raw image collected by the main camera into a YUV image; the present application does not make any limitations on the algorithms.

[0240] Exemplarily, the first front-end processing may refer to the image processing algorithm for converting a Raw image to a YUV image executed in the ISP. The first front-end processing may further include subsequent operations such as cropping and magnifying based on the zoom ratio in response to a zoom operation.

[0241] S603. Obtain the image captured by the telephoto camera.

[0242] Exemplarily, the image captured by the telephoto camera may be a Raw image captured by the telephoto camera.

[0243] Optionally, S604 and S601 may be executed simultaneously, or S604 and S601 may be executed successively.

[0244] S604. Second front-end processing.

[0245] Optionally, the second front-end processing may include algorithms related to converting the Raw image captured by the telephoto camera into a YUV image; this application does not impose any limitations on the algorithms.

[0246] Exemplarily, the second front-end processing may refer to the image processing algorithm for converting the Raw image to a YUV image executed in the ISP. The second front-end processing may also include subsequent operations such as cropping and magnifying based on the zoom ratio in response to a zoom operation.

[0247] After executing S605, perform first back-end processing on the processed image; among them, the implementation method of the first back-end processing may refer to the relevant description of S603.

[0248] For example, when the zoom ratio is within 1x - 2.5x, only execute S601; when zooming to 2.5x - 5x, start to execute S601 and S603 simultaneously; when zooming to 5x and above, only execute S603.

[0249] S605. First back-end processing.

[0250] Exemplarily, the first back-end processing includes but is not limited to: brightening processing, denoising processing, saturation adjustment processing, cropping processing, or distortion processing.

[0251] S606. Smoothing processing (or called alignment processing).

[0252] Exemplarily, perform smoothing processing on the image stream captured by the main camera and the image stream captured by the telephoto camera; it can be understood that the data input for smoothing processing is two image streams (for example, the image stream captured by the main camera and the image stream captured by the telephoto camera), and the smoothed image stream is one image stream. This smoothing processing is the Figure 8 、 Figure 9 、 Figure 10 and / or Figure 14 image processing method shown.

[0253] Exemplarily, when the zoom magnification increases and zooms from 2x to 2.5x, the smoothed image stream is based on the image stream captured by the telephoto camera, and the image stream of the main camera is processed using the image processing method provided by the embodiments of the present application.

[0254] Exemplarily, when the zoom magnification decreases and zooms from 5x to 2.5x, the smoothed image stream is based on the image stream captured by the main camera, and the image stream of the telephoto camera is processed using the image processing method provided by the embodiments of the present application.

[0255] Exemplarily, when the zoom magnification increases and zooms from 2.4x to 2.5x, the Figure 14 shown image processing method can be used to screen and change the offset corresponding to each frame of the image.

[0256] In the embodiments of the present application, through smoothing processing, it is possible to avoid an obvious jump between the foreground and the background of the image when switching cameras.

[0257] S607, Second backend processing.

[0258] Exemplarily, the second backend processing includes anti-shake processing; for example, the image frames in the image stream can be subjected to anti-shake processing according to the shake parameters to obtain the processed image.

[0259] S608, Display the processed image.

[0260] For example, when the zoom magnification increases and the zoom ratio is in the range of 1x to 2x, the processed image is the processed image corresponding to the image captured by the main camera; when zooming to 2x to 5x, the processed image is: the image captured by the main camera, and the image is corrected and processed based on the image captured by the telephoto camera; when zooming to 5x and above, the processed image is the processed image corresponding to the image captured by the telephoto camera.

[0261] S609, Detect a zoom operation.

[0262] This zoom operation can refer to the description in S230 above.

[0263] Optionally, the electronic device can display the image captured by the main camera, or the image captured by the telephoto camera, and the electronic device detects a user operation on the displayed image.

[0264] It should be understood that the above examples are for helping those skilled in the art to understand the embodiments of the present application, rather than limiting the embodiments of the present application to the specific numerical values or specific scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or changes according to the above examples, and such modifications or changes also fall within the scope of the embodiments of the present application.

[0265] Exemplarily, Figure 17 FIG. is a schematic diagram of an application scenario provided by an embodiment of the present application.

[0266] As Figure 17 shown in (a) of, in response to a user's operation on the camera application, the electronic device may display a preview interface 1301, and the preview interface 1301 includes a preview window and a shooting control. Among them, the preview image 1302 displayed in the preview window may include a first shooting object, a second shooting object, and a third shooting object. Here, when the camera application is opened, the zoom ratio is defaulted to 1x. As Figure 17 shown in (b) of, assuming that the user hopes to zoom in on the third shooting object located in the lower left of the preview image 1302, the user may perform an operation of sliding two fingers in opposite directions on the screen; in response to this operation, for example, as Figure 17 shown in (c) of, the zoom ratio may be increased to 10x, the imaging size corresponding to the third shooting object changes from small to large, and the third shooting object may be located at the middle position of the preview image 1303 in response to the movement of the user's finger.

[0267] During the process of increasing the zoom ratio and switching the camera, it can adapt to Figure 8 , Figure 9 , Figure 10 or Figure 14 the image processing methods introduced.

[0268] It should be understood that the above is the preview scenario in the shooting mode, and the embodiments of the present application may also be applicable to preview scenarios in other shooting modes such as video recording. The preview scenario may refer to the scenario where the electronic device captures images before clicking the shooting control in a certain shooting mode.

[0269] Exemplarily, Figure 18 FIG. is another schematic diagram of an application scenario provided by an embodiment of the present application.

[0270] After the camera application is opened, in response to the user's click operation on the recording control, the electronic device may start recording a video and display a video recording interface 1401, as Figure 18 shown in (a) of, the video recording interface 1401 may include a recording window, a pause control, an end control, and a zoom control. Among them, the video image 1402 displayed in the recording window may include a first shooting object, a second shooting object, a third shooting object, and a fourth shooting object. Here, when the video recording starts, the zoom ratio is defaulted to 1x.

[0271] If the electronic device supports and enables the AI detection and parallax detection functions, the electronic device can detect the image content and depth when collecting the video image 1402, and when displaying the video image 1402, display multiple detection frames, each detection frame being used to indicate the position information of a captured object in the video image 1402. As Figure 18 shown in (a) of

[0272]

[0273] Figure 18 Figure 18 Figure 18

[0274]

[0275] Figure 8 Figure 9 Figure 10 Figure 14

[0276] When the zoom ratio changes and the camera switches, it can adapt to Figure 8 、 Figure 9 、 Figure 10 or Figure 14 the image processing methods introduced.

[0276] It should be understood that the above is a video recording scenario, and the embodiments of the present application can also be applied to scenarios such as video calls.

[0277] Exemplarily, the video call scenario may include but is not limited to the following scenarios: video calls, video conferencing applications, long and short video applications, video live streaming applications, video online course applications, portrait intelligent panning application scenarios, recording videos using the system camera recording function, video monitoring, or shooting scenarios such as intelligent doorbells.

[0278] As described above in conjunction with Figures 1 to 18 The image processing method provided by the embodiments of the present application is described in detail; hereinafter, in conjunction with Figure 19 and Figure 20 The device embodiments of the present application will be described in detail. It should be understood that the devices in the embodiments of the present application can execute various methods of the foregoing embodiments of the present application, that is, the specific working processes of the following various products can refer to the corresponding processes in the foregoing method embodiments.

[0279] Figure 19 FIG. 16 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device 1500 includes a processing module 1510 and a display module 1520; the electronic device 1500 further includes a first camera and a second camera.

[0280] Among them, the processing module 1510 is used to: start the camera application program; the display module 1520 is used to: display a first image, the first image is obtained by the first camera collecting an image; receive a first zoom operation; display and save a second image, the second image is obtained by correcting the image collected by the first camera using the offset between the image collected by the second camera, and the offset is obtained by performing a direction filtering process on the feature point pairs that match the feature points of the image collected by the first camera and the feature points of the image collected by the second camera.

[0281] Optionally, as an embodiment, the display module 1520 is further used to: receive a second zoom operation; display and save a third image, the third image is obtained by the second camera collecting an image.

[0282] It should be noted that the above electronic device 1500 is embodied in the form of functional units. The term "module" here can be implemented in software and / or hardware forms, and no specific limitation is made thereto.

[0283] For example, a "module" may be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group of processors, etc.) for executing one or more software or firmware programs, and a memory, a combined logic circuit, and / or other suitable components that support the described functions.

[0284] Therefore, the units of the various examples described in the embodiments of the present application can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such an implementation should not be considered to exceed the scope of the present application.

[0285] Figure 20 The structural schematic diagram of an electronic device provided by the present application is shown. Figure 20 The dotted line in [the figure] indicates that the unit or the module is optional. The electronic device 1600 can be used to implement the method described in the above method embodiments.

[0286] The electronic device 1600 includes one or more processors 1601, and the one or more processors 1601 can support the electronic device 1600 to implement the image processing method in the method embodiments. The processor 1601 can be a general-purpose processor or a dedicated processor. For example, the processor 1601 can be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gate, transistor logic devices, or discrete hardware components.

[0287] The processor 1601 can be used to control the electronic device 1600, execute software programs, and process the data of the software programs. The electronic device 1600 may further include a communication unit 1605 for implementing signal input (reception) and output (transmission).

[0288] For example, the electronic device 1600 may be a chip, and the communication unit 1605 may be the input and / or output circuit of the chip. Alternatively, the communication unit 1605 may be the communication interface of the chip, and the chip may be a component of a terminal device or other electronic devices.

[0289] As another example, the electronic device 1600 may be a terminal device, and the communication unit 1605 may be the transceiver of the terminal device. Alternatively, the communication unit 1605 may be the transceiver circuit of the terminal device.

[0290] The electronic device 1600 may include one or more memories 1602, on which a program 1604 is stored. The program 1604 can be run by the processor 1601 to generate instructions 1603, so that the processor 1601 executes the methods described in the above method embodiments according to the instructions 1603.

[0291] Optionally, data may also be stored in the memory 1602. Optionally, the processor 1601 may also read the data stored in the memory 1602. The data may be stored at the same storage address as the program 1604, or may be stored at a different storage address from the program 1604.

[0292] The processor 1601 and the memory 1602 may be provided separately or integrated together. For example, they may be integrated on a system on chip (SOC) of a terminal device.

[0293] Exemplarily, the memory 1602 may be used to store the relevant program 1604 of the image processing method provided in the embodiments of the present application. The processor 1601 may be used to call the relevant program 1604 of the image processing method stored in the memory 1602 during video processing to execute the image processing method of the embodiments of the present application; for example, opening a camera application; opening a camera application; displaying a first image, where the first image is obtained by collecting an image with a first camera; receiving a first zoom operation; displaying and saving a second image, where the second image is obtained by correcting the image collected by the first camera using the offset between the image collected by the first camera and the image collected by a second camera, and the offset is obtained by performing a direction filtering process on the feature point pairs that match the feature points of the image collected by the first camera and the feature points of the image collected by the second camera.

[0294] The present application also provides a computer program product, which implements the methods described in any of the method embodiments of the present application when executed by the processor 1601.

[0295] The computer program product can be stored in the memory 1602, such as the program 1604. After processes such as preprocessing, compilation, assembly, and linking, the program 1604 is finally converted into an executable target file that can be executed by the processor 1601.

[0296] This application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, it implements the method described in any one of the method embodiments of this application. The computer program can be a high-level language program or an executable target program.

[0297] The computer-readable storage medium is, for example, the memory 1602. The memory 1602 can be a volatile memory or a non-volatile memory, or the memory 1602 can include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0298] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes and technical effects of the above-described devices and equipment can refer to the corresponding processes and technical effects in the foregoing method embodiments, and will not be elaborated herein.

[0299] In several embodiments provided by the present application, the disclosed systems, devices, and methods can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not executed. The device embodiments described above are merely illustrative. The division of units is only a logical function division. In actual implementation, there may be other division methods. Multiple units or components can be combined or integrated into another system. In addition, the coupling between units or between components can be direct coupling or indirect coupling. The above coupling includes electrical, mechanical, or other forms of connection.

[0300] It should be understood that in various embodiments of the present application, the magnitudes of the sequence numbers of the various processes do not mean the order of execution is prior or subsequent. The order of execution of the various processes should be determined by their functions and internal logics, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0301] In addition, the terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the preceding and following associated objects.

[0302] In summary, the above description is only a preferred embodiment of the technical solution of the present application and is not used to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that, Applied to an electronic device, the electronic device including a first camera and a second camera, the method comprising: Launching a camera application; Displaying a first image, the first image being obtained by the first camera capturing an image; Receiving a first zoom operation; Displaying and saving a second image, the second image being obtained by correcting the image captured by the first camera using an offset between the image captured by the first camera and the image captured by the second camera, the offset being obtained by performing a direction filtering process on the feature point pairs that match between the feature points of the image captured by the first camera and the feature points of the image captured by the second camera.

2. The image processing method according to claim 1, wherein The method further comprises: Receiving a second zoom operation; Displaying and saving a third image, the third image being obtained by the second camera capturing an image.

3. The image processing method according to claim 1 or 2, characterized in that, The method further comprises: Obtaining a first initial image captured by the first camera and a second initial image captured by the second camera; Performing feature point detection and registration on the first initial image and the second initial image to obtain multiple sets of matching feature point pairs; Calculating the vectors formed by each pair of the feature point pairs; Determining the slope of each of the vectors; Dividing all the feature point pairs into multiple point sets according to the slope of each of the vectors; Determining the offset according to the multiple sets of feature point pairs in the point set including the largest number of feature point pairs; Combining the offset to correct the first initial image to obtain the second image.

4. The image processing method according to claim 3, wherein Before calculating the vectors formed by each pair of the feature point pairs, the method further comprises: eliminating the mismatched feature point pairs.

5. The image processing method according to claim 3 or 4, characterized in that, Dividing all the feature point pairs into multiple point sets according to the slope of each of the vectors, including: Sorting the slopes corresponding to all the feature point pairs and setting serial numbers; Calculating the difference between the slope corresponding to the feature point pair of the current serial number and the average slope corresponding to all the previous serial number feature point pairs in the order of the serial numbers; If the difference is too large, the feature point pair of the current serial number is recorded as a feature point pair with too large a difference, and the previous set of feature point pairs up to the last recorded feature point pair with too large a difference, as well as multiple feature point pairs between the two sets of feature point pairs, are divided into one point set.

6. The image processing method according to any one of claims 1 to 5, characterized in that The method further comprises: in response to the first zoom operation; Determining that the zoom magnification of the electronic device satisfies the zoom magnification range for the first camera to be foreground-displayed and the second camera to be background-operated.

7. The image processing method according to claim 2, wherein The method further comprises: in response to the second zoom operation; Determining that the zoom magnification of the electronic device satisfies the zoom magnification range for the first camera to be background-operated or turned off and the second camera to be foreground-displayed.

8. The image processing method according to claim 2 or 7, characterized in that, The first zoom operation or the second zoom operation includes: a two-finger reverse sliding operation, a sliding operation on a displayed zoom control, a voice operation, or an air gesture operation.

9. The image processing method according to any one of claims 1 to 8, characterized in that, The first camera includes a wide-angle camera, and / or, the second camera includes a telephoto camera or an ultra-wide-angle camera.

10. The image processing method according to claim 2, characterized in that The first zoom operation and the second zoom operation are continuous zoom operations.

11. The image processing method according to claim 1, wherein During the first zoom operation, when the difference between the current zoom ratio and the target zoom ratio is equal to or less than a preset value, the offset remains unchanged and is equal to the first offset; wherein, the target zoom ratio is used to indicate the zoom ratio for switching from foreground display by the first camera to foreground display by the second camera; the first offset is used to indicate the offset determined last time when the difference between the current zoom ratio and the target zoom ratio is greater than the preset value.

12. The image processing method according to claim 1 or 11, characterized in that, The method further includes: Determining a parameter value corresponding to the current frame by using the offset between the image collected by the first camera and the image collected by the second camera and all previous offsets; Determining the difference between the parameter value corresponding to the current frame and the parameter value corresponding to the previous frame; If it is greater than the threshold, the offset corresponding to the previous frame is used as the offset corresponding to the current frame.

13. The image processing method according to claim 12, wherein The method further includes: If the differences corresponding to three consecutive frames are all greater than the threshold, clear all offsets and re-determine.

14. The image processing method according to claim 12 or 13, characterized in that, The parameter value includes the mean and / or variance of the offset.

15. An electronic device, characterized in that, The electronic device includes: one or more processors, a memory, a first camera, and a second camera; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to execute the image processing method according to any one of claims 1 to 14.

16. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system includes one or more processors, and the processors are used to call computer instructions to cause the electronic device to execute the image processing method according to any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to execute the image processing method according to any one of claims 1 to 14.

Images