Image processing method and related equipment thereof

Abstract

The invention relates to the field of image processing, and provides an image processing method and related equipment thereof, the image processing method is applied to electronic equipment, the electronic equipment comprises a first camera and a second camera, and the method comprises the following steps: starting a camera application program; displaying a first image, wherein the first image is obtained by collecting an image by a first camera; receiving a zooming operation; displaying and storing a second image, wherein the second image is obtained by performing color brightness alignment processing on the image acquired by the second camera and the last frame of image acquired by the first camera before switching; the weight allocated to the image acquired by the second camera in the color brightness alignment processing comprises the minimum value of the first parameter, the second parameter and the third parameter, and the first parameter corresponds to the current zoom ratio. According to the invention, the weight is allocated in combination with the current zoom ratio in the color brightness alignment processing, so that the smooth transition of the image brightness and color before and after the zoom switching of the camera can be realized.

Description

Technical Field

[0001] This application relates to the field of image processing, and particularly to an image processing method and related devices thereof. Background Art

[0002] With the development of the shooting function in electronic devices, camera applications are becoming more and more widely used in electronic devices. To obtain a better shooting experience, multiple cameras are usually set on current electronic devices, and the focal lengths corresponding to each camera are different.

[0003] During shooting, in response to the user's operation, the electronic device can perform zoom shooting by switching cameras with different focal lengths, and can also process the captured image by combining digital zoom to meet various high-magnification shooting scenarios. However, during the process of switching cameras for zooming, due to the different characteristics of different cameras themselves, the images displayed before and after the camera switch have inconsistent colors and brightness, and even jump, seriously affecting the user experience.

[0004] Therefore, during the zoom switching process, how to avoid brightness and color jumps has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides an image processing method and related devices thereof. By combining the current zoom ratio to allocate weights in color and brightness alignment processing, smooth transitions of image brightness and color before and after switching cameras for zooming can be achieved.

[0006] In a first aspect, an image processing method is provided, which is applied to an electronic device. The electronic device includes a first camera and a second camera. The method includes: starting a camera application; displaying a first image, where the first image is obtained by the first camera capturing an image; receiving a zoom operation; displaying and saving a second image, where the second image is obtained by performing color and brightness alignment processing on the image captured by the second camera and the last frame of the image captured by the first camera before switching; the weight allocated to the image captured by the second camera in the color and brightness alignment processing includes the minimum value among a first parameter, a second parameter, and a third parameter, and the first parameter corresponds to the current zoom ratio.

[0007] Optionally, in an embodiment, allocating a weight to the image captured by the second camera in the color and brightness alignment processing means allocating a weight to a reference frame generated after the first frame of the image captured by the second camera is acted on by a target 3D lut table.

[0008] In the embodiments of the present application, in order to combine the intensity of the color and brightness alignment process with the FOV corresponding to the current image to achieve fast convergence, the intensity of the color and brightness alignment process can be combined with the zoom ratio to construct the relationship between the weight and the current zoom ratio. Thus, during the fast zoom process, the intensity of the color and brightness alignment process can be adaptively adjusted according to the current corresponding FOV, rather than a fixed uniform change, thereby avoiding the problem of color and brightness deviation after fast zoom in the related art.

[0009] In combination with the first aspect, in some implementation manners of the first aspect, the first parameter is determined by the first formula: 1 - (Curzr - Refzr) / ((1 - 1 / i)*Curzr); where Curzr is used to indicate the current zoom ratio, Refzr is used to indicate the zoom ratio corresponding to the last frame of the image captured by the first camera before switching, and i is greater than 0.

[0010] Optionally, as an implementation manner, i can take the value of 2; then the first formula can be expressed as: 1 - (Curzr - Refzr) / (0.5*Curzr).

[0011] In the embodiments of the present application, in order to combine the intensity of the color and brightness alignment process with the FOV corresponding to the current image and make the MCC algorithm achieve fast convergence during fast zoom, and since there is an inverse proportional relationship between the FOV and the zoom ratio, thus, the present application combines the intensity of the color and brightness alignment process with the zoom ratio to construct the relationship between the weight and the current zoom ratio. When the current zoom ratio changes, the first parameter will change, and the weight assigned to the image for color and brightness alignment will also change accordingly. Thus, during the fast zoom process, the weight assigned in the MCC algorithm can be adaptively adjusted according to the FOV, rather than a uniform change.

[0012] In combination with the first aspect, in some implementation manners of the first aspect, the second parameter is determined by the second formula: PreIntensity - 1 / N; where PreIntensity is used to indicate the weight corresponding to the previous frame of the image captured by the second camera, and N is the preset total number of frames.

[0013] In the embodiments of the present application, considering that there may be a scenario of stopping zoom after fast zoom, in order to continue the transition of the color and brightness of the image after stopping zoom, the weight can be determined by combining the weight corresponding to the previous frame of the image captured by the second camera and the preset total number of frames for completing the transition.

[0014] In combination with the first aspect, in certain implementations of the first aspect, the third parameter is determined by the third formula: 1 - CurIdx / N; where CurIdx is used to indicate the Mth frame image captured by the second camera after the switch, N is the preset total number of frames, and M is less than or equal to N.

[0015] In the embodiments of the present application, during normal camera switching, in the order of increasing number of frames of the images captured by the second camera, the weights assigned to the reference frames show a decreasing trend, while the weights assigned to the images captured by the second camera show an increasing trend.

[0016] In combination with the first aspect, in certain implementations of the first aspect, before performing color and brightness alignment processing on the images captured by the second camera and the last frame image captured by the first camera before the switch, the method further includes: traversing each pixel of the first frame image captured by the second camera and the last frame image captured by the first camera before the switch to determine the mapping relationship; based on the mapping relationship, obtaining the target 3D lut table; the color and brightness alignment processing of the images captured by the second camera and the last frame image captured by the first camera before the switch includes: combining the target 3D lut table, transforming the first frame image captured by the second camera after the switch to obtain a reference frame; fusing the Mth frame image captured by the second camera with the reference frame and the assigned weight to obtain the Mth frame target image, and the second image includes the Mth frame target image, where M is an integer greater than or equal to 1.

[0017] In combination with the first aspect, in certain implementations of the first aspect, the assigned weight further includes: the difference between 1 and the minimum value; the weight corresponding to the Mth frame image captured by the second camera is the difference, and the weight corresponding to the reference frame is the minimum value.

[0018] In combination with the first aspect, in certain implementations of the first aspect, before traversing each pixel, the method further includes: preprocessing; where the preprocessing includes at least one of registration, brightness alignment, downsampling, and blurring.

[0019] In the embodiments of the present application, through preprocessing, the content, brightness, and other information of the images captured by the second camera and the last frame image captured by the first camera before the switch can be aligned, and the amount of data to be processed subsequently can be reduced.

[0020] In combination with the first aspect, in certain implementations of the first aspect, the zoom operation includes: a two-finger reverse sliding operation, a sliding operation on the displayed zoom control, a voice operation, or an air gesture operation.

[0021] In connection with the first aspect, in certain implementations of the first aspect, the zoom operation includes one or more back-and-forth sliding operations on the displayed zoom control.

[0022] In connection with the first aspect, in certain implementations of the first aspect, the first camera includes a wide-angle camera, and / or the second camera includes a telephoto camera or an ultra-wide-angle camera.

[0023] In a second aspect, an electronic device is provided, the electronic device includes: one or more processors, a memory, a first camera, and a second camera; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to perform:

[0024] Open the camera application; display a first image, the first image is obtained by the first camera capturing an image; receive a zoom operation; display and save a second image, the second image is obtained by performing color and brightness alignment processing on the image captured by the second camera and the last frame image captured by the first camera before switching; the weights assigned to the image captured by the second camera in the color and brightness alignment processing include the minimum of a first parameter, a second parameter, and a third parameter, and the first parameter corresponds to the current zoom ratio.

[0025] It should be understood that the extensions, limitations, explanations, and descriptions of the relevant content in the above first aspect also apply to the same content in the second aspect.

[0026] In a third aspect, a chip system is provided, the chip system is applied to an electronic device, the chip system includes one or more processors, and the processors are used to call computer instructions to cause the electronic device to perform any one of the image processing methods in the first aspect.

[0027] In a fourth aspect, a computer-readable storage medium is provided, the computer-readable storage medium stores computer program code, and when the computer program code is run by an electronic device, it causes the electronic device to perform any one of the image processing methods in the first aspect.

[0028] In a fifth aspect, a computer program product is provided, the computer program product includes: computer program code, and when the computer program code is run by an electronic device, it causes the electronic device to perform any one of the image processing methods in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic diagram of a hardware system of an electronic device applicable to the present application;

[0030] Figure 2 It is a schematic diagram of a software system applicable to an electronic device of the present application;

[0031] Figure 3 It is a schematic diagram of the arrangement of multiple cameras on an electronic device provided by an embodiment of the present application;

[0032] Figure 4 It is a schematic diagram of the zoom ratio corresponding to different types of cameras provided by an embodiment of the present application;

[0033] Figure 5 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0034] Figure 6 It is a schematic diagram of the images collected before and after the camera switch without processing provided by an embodiment of the present application;

[0035] Figure 7 It is a schematic flowchart of an MCC algorithm provided by an embodiment of the present application;

[0036] Figure 8 It is a schematic diagram of color brightness alignment processing provided by an embodiment of the present application;

[0037] Figure 9 It is a schematic diagram of the images collected before and after the camera switch after processing provided by an embodiment of the present application;

[0038] Figure 10 It is a schematic diagram of two frames of images displayed before zooming and after performing multiple zoom operations quickly back and forth;

[0039] Figure 11 It is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0040] Figure 12 It is a schematic diagram of zoom switching provided by an embodiment of the present application;

[0041] Figure 13 It is another schematic diagram of the images collected before and after the camera switch after processing provided by an embodiment of the present application;

[0042] Figure 14 It is a schematic flowchart of another image processing method provided by an embodiment of the present application;

[0043] Figure 15 It is a schematic diagram of an application scenario of zoom switching provided by an embodiment of the present application;

[0044] Figure 16 It is a schematic diagram of an application scenario of zoom switching provided by an embodiment of the present application;

[0045] Figure 17is a schematic diagram of an electronic device according to an embodiment of the present application;

[0046] Figure 18 is a schematic diagram of an electronic device suitable for the present application. DETAILED DESCRIPTION

[0047] In the embodiments of the present application, the following terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present embodiment, unless otherwise specified, "plurality" means two or more.

[0048] First, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0049] 1. Field of view (FOV), in optical instruments, is the angle formed by the two edges of the maximum range of the image of the target object that can pass through the lens, with the lens of the optical instrument as the vertex. The size of the field of view angle determines the field of view of the optical instrument. The larger the field of view angle, the larger the field of view and the smaller the optical magnification, that is, the target object will not be captured by the lens if it exceeds this angle. The shorter the focal length, the wider the horizontal field of view, and the smaller the image. The horizontal field of view becomes narrower as the focal length increases, and the object being photographed increases accordingly.

[0050] 2. Registration refers to the matching of geographic coordinates of different images obtained by different imaging methods in the same area, including geometric correction, projection transformation and the same scale.

[0051] 3. Zoom ratio: The zoom ratio is used to indicate the zoom size of an electronic device when shooting.

[0052] The above is a brief introduction to the terms involved in the embodiments of the present application, which will not be repeated below.

[0053] The hardware system, software system and application scenarios of the electronic device provided in the embodiments of the present application are described below in conjunction with the accompanying drawings.

[0054] Exemplarily, the electronic device 100 may be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, an in-vehicle electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, etc. The embodiments of the present application do not impose any restrictions on the specific type of the electronic device 100.

[0055] Referring to Figure 1 , the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0056] It should be noted that Figure 1 the structure shown does not constitute a specific limitation on the electronic device 100. In some other embodiments of the present application, the electronic device 100 may include more or fewer components than Figure 1 the components shown, or, the electronic device 100 may include a combination of some of the components Figure 1 shown, or, the electronic device 100 may include sub-components of some of the components Figure 1 shown. Figure 1 The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0057] The processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). Among them, different processing units may be independent devices or integrated devices.

[0058] The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0059] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may store the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0060] Exemplarily, the processor 110 may be used to execute the image processing method of the embodiments of the present application; for example, opening a camera application; displaying a first image, where the first image is obtained by collecting an image with a first camera; receiving a zoom operation; displaying and saving a second image, where the second image is obtained by performing color and brightness alignment processing on the image collected by a second camera and the last frame of the image collected by the first camera before switching; the weights assigned to the image collected by the second camera in the color and brightness alignment processing include the minimum value of a first parameter, a second parameter, and a third parameter, and the first parameter corresponds to the current zoom ratio.

[0061] Figure 1 The connection relationships shown between the modules are only illustrative and do not constitute a limitation on the connection relationships between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of various connection methods in the above embodiments.

[0062] The wireless communication function of the electronic device 100 may be implemented by devices such as an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, a modem processor, and a baseband processor.

[0063] Antenna 1 and Antenna 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0064] The electronic device 100 can implement the display function through the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0065] The display screen 194 can be used to display images or videos.

[0066] Exemplarily, in the embodiments of the present application, the display screen 194 can be used to display a second image.

[0067] The electronic device 100 can implement the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc.

[0068] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light passes through the lens and is transmitted to the camera photosensitive element. The optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can perform algorithm optimization on the noise, brightness, and color of the image. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be provided in the camera 193.

[0069] The camera 193 (which can also be referred to as a lens) is used to capture static images or videos. It can be triggered to turn on through application instructions to achieve the photographing function, such as taking images of any scene. The camera may include components such as an imaging lens, a filter, and an image sensor. The light emitted or reflected by an object enters the imaging lens, passes through the filter, and finally converges on the image sensor. The imaging lens is mainly used to converge and form an image of the light emitted or reflected by all objects in the photographing perspective (which can also be referred to as the scene to be photographed, the target scene, or the scene image that the user expects to photograph); the filter is mainly used to filter out the excess light waves in the light (such as light waves other than visible light, such as infrared); the image sensor can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The image sensor is mainly used to perform photoelectric conversion on the received optical signal, convert it into an electrical signal, and then transmit the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into a standard RGB, YUV, etc. format image signal.

[0070] Exemplarily, the gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, y-axis, and z-axis) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake during photographing. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of jitter of the electronic device 100, calculates the distance that the lens module needs to compensate based on the angle, and makes the lens offset the jitter of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used in scenarios such as navigation and motion sensing games.

[0071] In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0072] Among them, the camera 193 can be located in the front of the electronic device 100 or in the back of the electronic device 100. The specific number and arrangement of the cameras can be set according to requirements, and the present application does not make any restrictions.

[0073] Exemplarily, the electronic device 100 includes a front camera and a rear camera. For example, either the front camera or the rear camera can include one or more cameras. Taking the electronic device 100 having 4 rear cameras as an example, in this way, when the electronic device 100 starts the 4 rear cameras for photographing, the image processing method provided by the embodiments of the present application can be used.

[0074] Alternatively, the camera is disposed on an external accessory of the electronic device 100. The external accessory is rotatably connected to the frame of the mobile phone. The angle formed between the external accessory and the display screen 194 of the electronic device 100 is any angle between 0 and 360 degrees. For example, when the electronic device 100 takes a self-portrait, the external accessory drives the camera to rotate to a position facing the user. Of course, when the mobile phone has multiple cameras, only some of the cameras can be disposed on the external accessory, and the remaining cameras are disposed on the body of the electronic device 100. The embodiments of the present application do not impose any restrictions on this.

[0075] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0076] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0077] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, y-axis, and z-axis) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake during shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the shaking angle of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and makes the lens offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used in scenarios such as navigation and motion-sensing games.

[0078] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in each direction (generally the x-axis, y-axis, and z-axis). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. The acceleration sensor 180E can also be used to identify the posture of the electronic device 100 and serve as input parameters for application programs such as horizontal and vertical screen switching and pedometers.

[0079] The distance sensor 180F is used to measure distances. The electronic device 100 can measure distances through infrared or laser. In some embodiments, for example, in a shooting scenario, the electronic device 100 can use the distance sensor 180F to measure distances to achieve rapid focusing.

[0080] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance during photography. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touch.

[0081] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement functions such as unlocking, accessing the application lock, taking pictures, and answering incoming calls.

[0082] The touch sensor 180K, also known as a touch control device. The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, and the touch screen is also known as a touch control screen. The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100 and at a different position from the display screen 194.

[0083] The hardware system of the electronic device 100 has been described in detail above. Next, the software system of the electronic device 100 will be introduced.

[0084] Figure 2 It is a schematic diagram of the software system of the electronic device provided by the embodiments of the present application.

[0085] As Figure 2 shown, the system architecture may include an application layer 210, an application framework layer 220, a hardware abstraction layer 230, a driver layer 240, and a hardware layer 250.

[0086] The application layer 210 may include application programs such as a camera application, a gallery, a calendar, a call, a map, a navigation, a WLAN, a Bluetooth, music, a video, and a short message.

[0087] The application framework layer 220 provides application programming interfaces (APIs) and programming frameworks for the application programs in the application layer; the application framework layer may include some predefined functions.

[0088] For example, the application framework layer 220 may include a camera access interface; the camera access interface may include camera management and a camera device. Among them, the camera management can be used to provide an access interface for managing the camera; the camera device can be used to provide an access interface for the camera.

[0089] The hardware abstraction layer 230 is used to abstract the hardware. For example, the hardware abstraction layer may include a camera abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer can call camera algorithms.

[0090] For example, the hardware abstraction layer 230 includes a camera hardware abstraction layer and camera algorithms; the camera algorithms may include software algorithms for image processing.

[0091] Exemplarily, the camera algorithm library may include the algorithms corresponding to the image processing method provided in the embodiments of the present application.

[0092] Exemplarily, the algorithms in the camera algorithms may refer to those that do not rely on specific hardware for implementation; for example, code that can generally run on a CPU, etc.

[0093] The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include a camera driver.

[0094] The hardware layer 250 is located at the bottom layer of the operating system; as Figure 2 shown, the hardware layer 250 may include Camera 1, Camera 2, Camera 3, etc. Among them, Camera 1, Camera 2, and Camera 3 may correspond to multiple cameras on the electronic device.

[0095] For ease of understanding, taking the electronic device 100 as a mobile phone with the above software and hardware structure as an example, the cameras on the electronic device 100 to which the method provided in the embodiments of the present application is applicable will be described in detail first.

[0096] The electronic device to which the method provided in the embodiments of the present application is applicable has at least multiple cameras 193. For example, it has 3 types of cameras 193; the 3 types of cameras are respectively a main camera (for example, a wide-angle camera), an ultra-wide-angle camera, and a telephoto camera; the 3 types of cameras can be used to shoot the same scene to be shot.

[0097] Optionally, the electronic device 100 may also have other cameras 193, and the types of the cameras 193 and the number of each type of camera 193 can be set as needed, and the embodiments of the present application do not impose any restrictions on this.

[0098] Exemplarily, as Figure 3 shown, taking the electronic device 100 as an example with 3 cameras 193 for illustration; the arrangement of the 3 cameras can be as shown in (a) in Figure 3 , or as shown in (b) in Figure 3 ; for example, the 3 cameras 193 can be a main camera 1931 (for example, a wide-angle camera), an ultra-wide-angle camera 1932, and a telephoto camera 1933.

[0099] It should be understood that the above are only examples of two arrangement methods, and other arrangement methods are also possible; the specific arrangement method can be designed and changed according to needs, and the embodiments of the present application do not impose any restrictions on this.

[0100] It should be noted that during the shooting of the above three cameras, generally, the field of view angle range corresponding to the main camera 1931 is larger than the field of view angle range corresponding to the telephoto camera 1933; and the field of view angle range corresponding to the ultra-wide-angle camera 1932 is larger than the field of view angle range corresponding to the main camera 1931; there may be an overlap between the field of view angles of the ultra-wide-angle camera 1932 and the main camera 1931; that is to say, the ultra-wide-angle camera 1932 can capture the scene content captured by the main camera 1931 and the surrounding scene content.

[0101] It should be understood that the field of view angle range corresponding to the telephoto camera 1933 is smaller than the field of view angle range corresponding to the main camera 1931, and there may be an overlap between the field of view angles of the main camera 1931 and the telephoto camera 1933; that is to say, the main camera 1931 can capture the scene content captured by the telephoto camera 1933 and the surrounding scene content. There may be an overlap between the field of view angles of the ultra-wide-angle camera 1932 and the telephoto camera 1933; that is to say, the ultra-wide-angle camera 1932 can capture the scene content captured by the telephoto camera 1933 and the surrounding scene content.

[0102] Among them, due to the small focusing distance, the ultra-wide-angle camera 1932 is suitable for shooting close-ups; and, as the name implies, the ultra-wide-angle camera 1932 is suitable for shooting scenes with a relatively large field of view angle; the main camera 1931, due to its high clarity, is more suitable for shooting portraits, while the telephoto camera 1933 is more suitable for shooting distant scenes in close-up.

[0103] Exemplarily, as Figure 4 shown, the zoom ratio of the ultra-wide-angle camera can be less than the M-fold zoom ratio; the zoom ratio range of the wide-angle camera, that is, the main camera, can be [M, N); the zoom ratio of the telephoto camera can be greater than or equal to the N-fold zoom ratio.

[0104] For example, M can be 1 and N can be 2.5; then the zoom ratio of the ultra-wide-angle camera is less than 1-fold zoom ratio (1×); the zoom ratio range of the wide-angle camera is from 1-fold zoom ratio to 2.5-fold zoom ratio [1×~2.5×); the zoom ratio of the telephoto camera is greater than or equal to 2.5-fold zoom ratio.

[0105] It should be understood that during the shooting process of the electronic device, the larger the zoom ratio, the smaller the corresponding field of view angle.

[0106] Next, in combination with Figure 5An example of the application scenario of the image processing method provided in the embodiments of the present application is given.

[0107] The method in the embodiments of the present application can be applied to scenarios such as taking pictures, previewing, recording videos, or video calls; through the method in the embodiments of the present application, smooth zoom and / or smooth switching between different cameras can be achieved in an electronic device, improving the user's shooting experience and image quality.

[0108] Exemplarily, the preview scenario includes but is not limited to the following scenarios:

[0109] Taking picture preview, aperture preview, night scene preview, portrait preview, video recording preview, or professional preview, etc.

[0110] It should be understood that the preview scenario may refer to the scenario where an electronic device captures images before clicking the button indicating shooting in a certain shooting mode.

[0111] In one example, as shown in (a) of Figure 5 , after the electronic device enters the camera application, the default shooting mode can be enabled; in the shooting mode, the electronic device can enter the default shooting mode, which may refer to the shooting mode where the wide-angle camera is used as the main camera and the zoom ratio is the single zoom ratio (1×), and the electronic device displays the image captured by the main camera; as shown in (b) of Figure 5 , in response to the user's operation, when the zoom ratio meets the zoom range corresponding to the telephoto camera, the electronic device can switch to display the image captured by the telephoto camera.

[0112] Exemplarily, the image processing method in the embodiments of the present application can also be applied to the video call scenario, where the video call scenario may include but is not limited to the following scenarios:

[0113] Video calls, video conferencing applications, long and short video applications, video live streaming applications, video online course applications, portrait intelligent panning application scenarios, recording videos using the system camera's video recording function, video monitoring, or portrait shooting scenarios such as intelligent doorbells, etc.

[0114] It should be understood that the above is an example of the application scenario and does not limit the application scenario of the present application in any way.

[0115] Currently, during shooting, in response to a user's operation, an electronic device can perform zoom shooting by switching cameras with different focal lengths, and can also process the captured image in combination with digital zoom to meet various high-magnification shooting scenarios. However, during the process of switching cameras for zooming, due to the different characteristics of different cameras themselves, the colors and brightness of the images sent for display before and after the camera switch are inconsistent, and there may even be jumps in color and brightness, seriously affecting the user experience.

[0116] Exemplarily, Figure 6 shows a schematic diagram of images captured before and after a camera switch without processing. As Figure 6 shown, when zooming from the main camera to the telephoto camera, before the switch, the main camera captures and sends the image for display, and the image sent for display is as shown in Figure 6 (a); after the switch, the telephoto camera captures and sends the image for display, and the image sent for display is as shown in Figure 6 (b). During this switching process, the colors and brightness of the two frames of images displayed before and after the switch are completely different, resulting in problems of jumps in color and brightness.

[0117] To address this problem, the embodiments of the present application provide a multi-camera consistency (Multi-cam consistency, MCC) algorithm for processing images, enabling the image sent for display to smoothly transition from the color and brightness corresponding to the camera before the switch to the color and brightness corresponding to the camera after the switch, ensuring the consistency of image color and brightness when the camera is switched.

[0118] Exemplarily, Figure 7 is a schematic flowchart of an MCC algorithm provided by an embodiment of the present application. As Figure 7 shown, the MCC algorithm 300 may include the following S301 to S306.

[0119] S301. Obtain the image A0 captured by the main camera.

[0120] Exemplarily, the main camera is the camera that sends the image for display before the switch. The image A0 may be an image in the Raw domain captured by the main camera.

[0121] S302. Obtain the image B0 captured by the telephoto camera.

[0122] Exemplarily, the telephoto camera is the camera that sends the image for display after the switch. The image B0 may be an image in the Raw domain captured by the telephoto camera.

[0123] S303. Register and align the brightness of the image A0 and the image B0.

[0124] Exemplarily, the image A0 can be registered with the image B0 with the image B0 as a reference. The registration method can be the registration method provided by the related art, and the embodiments of the present application do not limit this.

[0125] Exemplarily, a statistical method can be used to align the image brightness.

[0126] S304. Downsampling and blurring processing.

[0127] The downsampling and blurring processing methods can be the methods provided by the related art, and the embodiments of the present application do not limit this.

[0128] It should be noted that subsequent processing needs to traverse each pixel of the image to construct a 3D lut table. However, there are differences in the FOV between the two frames of images obtained by the two cameras and they cannot be aligned. Therefore, the two frames of images can be blurred first to blur the pixel-level differences, or it can also be called noise reduction, making the color and brightness smoother and ensuring that the overall color and brightness of the image remain unchanged.

[0129] S305. Traverse each pixel of the processed image A0 and the processed image B0, determine the mapping relationship, and update the mapping relationship on the basis 3D lut table to obtain the target 3D lut table.

[0130] S306. Use the target 3D lut table to perform color and brightness alignment processing in combination with the processed image A0 and the processed image B0.

[0131] Exemplarily, as Figure 8 shown, when the target 3D lut table acts on the image B0, the fully acting image B can be obtained A . It should be understood that it is equivalent to fully applying the mapping relationship between the image A0 and the image B0 to the image B0. Thus, the obtained image B A can be considered to have the same color and brightness as the image A0 and be the closest.

[0132] Then, in order to gradually transition from the color and brightness of the camera before switching to the color and brightness of the camera after switching, the image B A is blended (alpha blending) with each frame of the image stream collected by the telephoto camera after switching, and the weight corresponding to the image B A gradually decreases in order, while the weight corresponding to each frame of the image stream gradually increases in order. Thus, the processed image stream will completely transition to the color and brightness corresponding to the camera after switching when the corresponding weight is 1. It should be understood that the image color and brightness alignment processing is the above-mentioned processing process of using the target 3D lut table to act and perform blending.

[0133] For example, as Figure 8 shown, assume that when switching from the main camera to the telephoto camera at 2.5X, the image captured by the main camera at 2.49X is image A0, and the image captured by the telephoto camera at 2.51X is image B0. Assume that the preset number of transition frames is 30. Then, according to the above method, applying the obtained target 3D lut table to image B0 can obtain image B A ; then, for the first frame image B0 of the telephoto camera, combining image B A , and performing fusion based on weights 0 and 1 to obtain image Bˊ0; for the second frame image B1 captured by the telephoto camera, combining image B A , and performing fusion based on weights 1 / 30 and 29 / 30 to obtain image Bˊ1; for the third frame image B2 captured by the telephoto camera, combining image B A , and performing fusion based on weights 2 / 30 and 28 / 30 to obtain image Bˊ2; and so on. For the image B 29 captured by the telephoto camera, combining image B A , and performing fusion based on weights 1 and 0 to obtain image Bˊ 29 ; based on this, the obtained 30 images from image Bˊ0 to image Bˊ 29 are the 30 transition frames that gradually transition from the color and brightness corresponding to the main camera to the color and brightness corresponding to the telephoto camera.

[0134] In addition, it should be understood that since image Bˊ0 obtained after the camera switch is based on image B A with a weight of 1, it can be considered that image Bˊ0 is equivalent to image B A , and image B A is obtained by conversion according to the target 3D lut table. Therefore, the color and brightness of image Bˊ0 are basically the same as those of image A0. Therefore, through the MCC algorithm provided by the embodiments of the present application, the problem of color and brightness jump in the images before and after zoom switching can be avoided.

[0135] Exemplarily, Figure 9 shows a schematic diagram of the images captured before and after the camera switch after processing. As Figure 9 shown, when zooming from the main camera to the telephoto camera, before the switch, the main camera captures an image and sends it for display, and the displayed image is as Figure 9 shown in (a); and after the switch, the telephoto camera captures an image and sends it for display, and the displayed image is as Figure 9As shown in (b) therein. During this switching process, since the images captured by the telephoto camera are processed in combination with the MCC algorithm introduced above, the color and brightness of the switched image are aligned with those of the image before switching. Therefore, the color and brightness of the images before and after switching are basically the same, and there is no jump in color and brightness.

[0136] Based on Figure 8 As can be seen from the example shown, the MCC algorithm is based on the idea of smooth transition, ensuring that before and after the camera is switched, the color and brightness gradually and smoothly transition from those of the previous camera to the corresponding color and brightness of the current camera. However, in fact, the zoom sliding is not necessarily a uniform and smooth process. If the user performs multiple zoom operations quickly and back and forth, at this time, due to the imaging differences between the two cameras themselves, the center positions of the images captured by the two cameras cannot be guaranteed to be exactly aligned. Therefore, there will be a certain error in the target 3D lut table calculated before and after each switch; then, after multiple back-and-forth zoom operations, the error will continue to accumulate and become larger, resulting in the failure of the MCC algorithm after multiple rapid zoom operations, and the color and brightness of the displayed image are inconsistent with those of the images captured by the two cameras, and there is no corresponding relationship either, that is, the problem of image distortion occurs in the displayed image.

[0137] Exemplarily, Figure 10 are schematic diagrams of two frames of images sent for display before zooming and after multiple rapid back-and-forth zoom operations. As Figure 10 shown, assuming that 2.5X is the switching point, the camera for sending the display can be switched from the main camera to the telephoto camera. If the user quickly zooms back and forth between 1X and 3X multiple times, the corresponding camera will switch from the main camera to the telephoto camera and then from the telephoto camera to the main camera multiple times. Before the zoom switch, the image sent for display may be as Figure 10 shown in (a) therein, the tea box is normally displayed in green, and the central area of the tea box is normally displayed in yellow. However, when multiple rapid zoom switches are performed, the image may be as Figure 10 shown in (b) therein. At this time, the whole image is yellowish and the brightness is dim, and there are certain deviations in the color and brightness of the whole image.

[0138] In view of this, an embodiment of the present application provides an image processing method and related devices; in the embodiment of the present application, in order to combine the action intensity of color and brightness alignment processing with the FOV corresponding to the current image to achieve fast convergence, the action intensity of color and brightness alignment processing can be combined with the zoom ratio to construct the relationship between the weight and the current zoom ratio; thus, during the fast zoom process, the action intensity of color and brightness alignment processing can be adaptively adjusted according to the current corresponding FOV, rather than a fixed uniform change, thereby avoiding the problem of color and brightness deviation after fast zoom in the related art.

[0139] The following Figure 11 will describe in detail the schematic flowchart of the image processing method provided by the embodiment of the present application.

[0140] Figure 11 is the schematic flowchart of the image processing method provided by the embodiment of the present application. This method can be executed by Figure 1 the electronic device shown; this method 400 includes steps S410 to S440, and the following will describe steps S410 to S440 in detail.

[0141] It should be understood that in the embodiment of the present application, the electronic device includes a first camera and a second camera; wherein, the first camera and the second camera are cameras of different types.

[0142] S410. Start the camera application.

[0143] Exemplarily, the user can click the icon of the "Camera" application to instruct the electronic device to start the camera application.

[0144] Exemplarily, when the electronic device is in the locked screen state, the user can swipe right on the display screen of the electronic device to instruct the electronic device to start the camera application. Or, when the electronic device is in the locked screen state and the locked screen interface includes the icon of the camera application, the user clicks the icon of the camera application to instruct the electronic device to start the camera application. Or, when the electronic device is running other applications and the application has the permission to call the camera application; the user can click the corresponding control to instruct the electronic device to start the camera application. For example, when the electronic device is running an instant messaging application, the user can select the control of the camera function to instruct the electronic device to start the camera application, etc.

[0145] It should be understood that the above is an example of the operation of starting the camera application; the electronic device can also be instructed to start the camera application through voice instructions or other operations; the present application makes no limitation thereto.

[0146] It should also be understood that starting the camera application may mean running the camera application.

[0147] S420. Display a first image, which is obtained by the first camera capturing an image.

[0148] Exemplarily, the first camera may be Figure 3 the wide-angle camera, the main camera shown, or may also be other cameras with a larger field of view angle range than the telephoto camera.

[0149] Exemplarily, the first image may be an RGB image captured by the main camera, or rather, the first image may be an RGB image obtained by processing the Raw image captured by the main camera through a series of camera algorithms.

[0150] For example, Figure 15 the preview image shown in (a) of Figure 16 is the first image captured and sent for display by the first camera. Again, for example,

[0151] It should be understood that the first image may include one or more shooting objects, and when displayed, a detection frame may be displayed for each shooting object determined based on AI detection.

[0152] It should also be understood that the type of the shooting object can be preset as needed. For example, when the set type is a human face, the target object can be used to indicate a human face recognized in the first image. The type of the shooting object may also include plants, animals, etc. The embodiments of the present application do not impose any restrictions on this.

[0153] S430. Receive a zoom operation.

[0154] Optionally, receiving a zoom operation may include: receiving a zoom operation for the first image.

[0155] It should be understood that the zoom operation for the first image may indicate a user's operation instruction, or may also indicate a zoom operation instruction automatically triggered by the electronic device based on AI detection. The target object is one of the shooting objects included in the first image.

[0156] Exemplarily, when the zoom operation indicates an operation instruction triggered by the user, the zoom operation may include a two-finger reverse sliding operation, a click operation, a voice operation, an air gesture operation, etc. for the first image. The embodiments of the present application do not impose any restrictions on this.

[0157] Exemplarily, as in Figure 16 (a) and Figure 16As shown in (b) of , when the electronic device displays one or more shooting objects and zoom controls, the zoom operation may include a click operation on the target object in the first image and a swipe operation on the zoom control; alternatively, the zoom operation may also include a click operation on the target object in the first image and a two-finger reverse swipe operation. It should be understood that when the zoom operation includes two operations, when executing, the user needs to first perform the click operation, and then perform the swipe operation on the zoom control or the two-finger reverse swipe operation on the target object. The zoom operation may also be other operations, or the zoom operation may also include three or more sub-operations, and the embodiments of the present application do not impose any restrictions on this.

[0158] Optionally, when the zoom operation indicates a swipe operation on the displayed zoom control, the zoom operation may include one or more back-and-forth swipe operations.

[0159] For example, as Figure 12 shown by line ③ in , the zoom operation refers to multiple back-and-forth swipe operations on the zoom control, so that the zoom ratio slides from 1X to 3.5X, then from 3.5X to 2X, from 2X to 3.4X, and then from 3.4X to 2.3X, etc.

[0160] S440. Display and save the second image.

[0161] The second image is obtained by performing color and brightness alignment processing on the image collected by the second camera and the last frame image collected by the first camera before switching. The weight assigned to the image collected by the second camera in this color and brightness alignment processing includes the minimum value among the first parameter, the second parameter, and the third parameter, and the first parameter corresponds to the current zoom ratio.

[0162] It should be understood that if the current zoom ratio is different, the first parameter will be different, and the minimum value determined among the first parameter, the second parameter, and the third parameter will be different. Therefore, the weight assigned to the image collected by the second camera in the color and brightness alignment processing will be different.

[0163] In the embodiments of the present application, since the image collected by the second camera after switching and the last frame image collected by the first camera before switching are subjected to color and brightness alignment processing, it is possible to make the colors and brightness of the images before and after the camera switch consistent.

[0164] Optionally, before performing the color and brightness alignment processing on the image collected by the second camera and the last frame image collected by the first camera before switching, the method may further include:

[0165] Step 1, preprocess the first frame image collected by the second camera and the last frame image collected by the first camera before switching. The preprocessing includes at least one of registration, brightness alignment, downsampling, and blurring.

[0166] It should be understood that the items and order of preprocessing can be set and adjusted as needed, and the embodiments of the present application do not limit this. In the embodiments of the present application, through preprocessing, information such as the content and brightness of the image collected by the second camera and the last frame image collected by the first camera before switching can be aligned, and the amount of data to be processed subsequently can be reduced.

[0167] Step 2, traverse the two processed frames of images pixel by pixel to determine the mapping relationship.

[0168] Step 3, update the basic 3D lut table based on the mapping relationship to obtain the target 3D lut table.

[0169] The target 3D lut table is used to indicate the color correspondence relationship between the first frame image collected by the second camera and the last frame image collected by the first camera before switching.

[0170] Here, the target 3D lut table is a mapping in the RGB domain.

[0171] It should be noted that since the target 3D lut table is a mapping in the RGB domain, therefore, subsequently combining the target 3D lut table to transform and process the image, the color and brightness can be aligned accordingly.

[0172] Step 4, combine the target 3D lut table to transform the first frame image collected by the second camera after switching to obtain a reference frame.

[0173] For example, combining Figure 8 , the first frame image collected by the second camera is image B0. Combining the target 3D lut table, the first frame image collected by the second camera after switching is transformed, and the obtained reference frame is image B A .

[0174] Step 5, fuse the Mth frame image collected by the second camera and the reference frame by combining the assigned weight to obtain the Mth frame target image. The second image includes the reference frame and the (M - 1)th frame target image, where M is an integer greater than or equal to 2. Among them, the assigned weight is the minimum value among the first parameter, the second parameter, and the third parameter, and the first parameter corresponds to the current zoom ratio.

[0175] Optionally, the first parameter is determined by the first formula: 1 - (Curzr - Refzr) / (0.5 * Curzr).

[0176] Among them, Curzr is used to indicate the current zoom ratio, and Refzr is used to indicate the zoom ratio corresponding to the last frame of the image captured by the first camera before switching.

[0177] Exemplarily, during the fast zoom process, if one of the zooms is from 0.99X to 1.5X (such as switching from an ultra-wide-angle camera to a main camera, with 1X as the switching point), then, to ensure that the weights assigned in the color and brightness alignment process correspond to the zoom ratio and achieve fast convergence, the present application presets to fully transition to the color and brightness of the main camera at twice the zoom ratio corresponding to the last frame of the image captured by the first camera before switching, that is, at a zoom ratio of 1.98X. Thus, by using the change ratio of the FOV, the following formula can be obtained:

[0178] 1 - (1 / Curzr - 1 / Refzr) / (1 / 2Refzr - 1 / Refzr); after substituting the values, we can get: 1 - (1 / 1.5X - 1 / 0.99X) / (1 / 1.98X - 1 / 0.99X); where, 1 / 1.5X - 1 / 0.99X is used to indicate the change amount between the FOV corresponding to the current zoom ratio and the FOV corresponding to the last frame of the image captured by the first camera before switching, and 1 / 1.98X - 1 / 0.99X is used to indicate the change amount between the FOV corresponding to the target zoom ratio at the end of the transition and the FOV corresponding to the last frame of the image captured by the first camera before switching. Combining this example, after organizing the above formula, the first formula for determining the first parameter can be obtained.

[0179] In addition, optionally, the 2 times in the above example is an example. When using i to represent the multiple of the zoom ratio after the transition relative to the zoom ratio corresponding to the last frame of the image captured by the first camera before switching, by using the change ratio of the FOV, the following formula can be obtained:

[0180] 1 - (1 / Curzr - 1 / Refzr) / (1 / iRefzr - 1 / Refzr); after organizing the above formula, we can get 1 - (Curzr - Refzr) / ((1 - 1 / i)*Curzr). Where, i is greater than 0. For example, i can be a decimal, fraction or integer greater than 0.

[0181] In an embodiment of the present application, in order to combine the intensity of the color brightness alignment process with the FOV corresponding to the current image and enable the MCC algorithm to achieve rapid convergence during rapid zooming, and since there is an inverse proportional relationship between the FOV and the zoom ratio, therefore, the present application combines the intensity of the color brightness alignment process with the zoom ratio to construct the relationship between the weight and the current zoom ratio. When the current zoom ratio changes, the first parameter will change, and the weight assigned to the image undergoing color brightness alignment processing will also change accordingly. Thus, during the rapid zooming process, the weight assigned in the MCC algorithm can be adaptively adjusted with the FOV instead of changing uniformly.

[0182] Optionally, the second parameter is determined by the second formula: PreIntensity - 1 / N; where PreIntensity is used to indicate the weight corresponding to the previous frame of the image captured by the second camera, and N is the preset total number of frames.

[0183] In an embodiment of the present application, considering that there may still be a scenario of stopping zooming after rapid zooming, in order to continue to perform smooth transition of the image color and brightness after stopping zooming, the weight can be determined by combining the weight corresponding to the previous frame of the image captured by the second camera and the preset total number of frames, so that the weight shows a linearly decreasing trend.

[0184] It should be noted that since PreIntensity is less than 1 and the weight corresponding to each frame decreases by 1 / N, there is no direct corresponding relationship between N and PreIntensity. Therefore, during the process of the weight decreasing to 0, the number of frames for transition may be less than the preset total number of frames, that is, the actual number of frames for transition is less than the preset total number of frames.

[0185] For example, as Figure 12 shown by line ② in [reference], assuming that the zooming stops after changing from 1X to 3X, then based on the first formula, the first parameter is determined to be 2 / 3 and will no longer change. At this time, in order to ensure that the color and brightness can continue to be smoothly transitioned after the zooming stops, 20 frames can be preset for transition, and taking 2 / 3 as the weight corresponding to the previous frame of the image captured by the second camera as an example to start determining the second parameter. For example: for the 1st frame of the transition, the determined second parameter is 2 / 3 - 1 / 20 = 37 / 60; for the 2nd frame of the transition, the determined second parameter is (2 / 3 - 1 / 20) - 1 / 20 = 34 / 60; for the 3rd frame of the transition, the determined second parameter is 34 / 60 - 1 / 20 = 31 / 60; and so on. For the 13th frame of the transition, the determined second parameter is 1 / 60. At the 14th frame, the transition is completed and the weight changes to 0. It should be understood that during this process, the actual number of frames for transition is less than the preset total number of frames.

[0186] Optionally, the third parameter is determined by a third formula: 1 - CurIdx / N; where CurIdx is used to indicate the Mth frame image captured by the second camera after switching, N is the preset total number of frames, and M is less than or equal to N.

[0187] It should be understood that the logic of the third parameter remains unchanged, that is, when the normal camera switches, in the order of increasing number of frames of the images captured by the second camera, the weight assigned to the reference frame shows a decreasing trend, and the weight assigned to the images captured by the second camera shows an increasing trend. Among them, the reference frame is the image obtained after the first frame image captured by the second camera after switching is processed by the target 3D lut table.

[0188] Exemplarily, if the preset total number of frames is 30 frames, for the first frame image captured by the second camera after switching, the assigned weight is 1 / 30, and the weight assigned to the reference frame is 29 / 30; for the second frame image captured by the second camera after switching, the assigned weight is 2 / 30, and the weight assigned to the reference frame is 28 / 30; and so on in sequence, which will not be elaborated here.

[0189] In the embodiments of the present application, when performing color and brightness alignment processing, the first parameter, the second parameter, and the third parameter can be determined based on the above three formulas, and the minimum value of the first parameter, the second parameter, and the third parameter is used as the weight finally assigned to the reference frame. Then, the weight assigned to the Mth frame image captured by the second camera is the difference between 1 and the minimum value, and the weight assigned to the reference frame is the minimum value. The image and the assigned weight are combined for weighted fusion, so that the fused image can be used as the second image for display.

[0190] Exemplary one, assume that the user slides from 0.9x zoom to 1x and then stops. Then, the first parameter determined by the first formula is always 1, while the second parameter determined by the second formula and the third parameter determined by the third formula will gradually decrease.

[0191] Exemplary two, assume that the user quickly zooms from 0.9x to 2x. Then, the first parameter determined by the first formula will drop rapidly, the second parameter determined by the second formula is larger than the value of the first parameter, and the third parameter determined by the third formula is related to the number of frames.

[0192] Exemplary three, assume that the user quickly zooms from 0.9x to 1.5x and then stops. Then, during the rapid sliding process, refer to the first example. Since the zoom ratio no longer changes, the first parameter determined by the first formula no longer changes, the second parameter determined by the second formula will slowly decrease based on the first parameter, and the third parameter determined by the third formula has been slowly decreasing.

[0193] In consideration of the above three situations, the embodiments of the present application thus construct the usage logic of these three parameters.

[0194] Here, it should also be noted that the reason for selecting the minimum value from the first parameter, the second parameter, and the third parameter is that the MCC algorithm requires the weight to transition from 1 to 0. Assuming the above three formulas are three curves (curves of the weight varying with the zoom magnification, with an overall downward trend), always using the minimum value at the same zoom magnification can ensure that the effect of the 3D lut decreases from large to small.

[0195] Combined with Figure 12 In the example shown, when the zoom operation includes multiple back-and-forth sliding operations on the zoom control, each time the first camera is switched to the second camera and each time the second camera is switched to the first camera, the color and brightness alignment processing steps provided by the embodiments of the present application can be used for processing.

[0196] Optionally, when switching from the second camera to the first camera, the second image is obtained by performing color and brightness alignment processing on the image captured by the first camera and the last frame of the image captured by the second camera before the switch. The weight assigned to the image captured by the first camera in this color and brightness alignment processing includes the minimum value among the first parameter, the second parameter, and the third parameter, and the first parameter corresponds to the current zoom magnification.

[0197] Exemplarily, the first image can be an RGB image, and correspondingly, the second image is an RGB image.

[0198] It should be understood that S440 can be executed while S430 is being executed, or S440 can be executed after S430. The embodiments of the present application do not impose any restrictions on this.

[0199] Exemplarily, when in the preview scene, the second image can be displayed. Among them, the preview shooting modes include but are not limited to: for example, night scene preview mode, video recording preview mode, photo shooting preview mode, portrait preview mode, and other shooting modes.

[0200] Exemplarily, when in the video recording scene, the second image can be displayed and saved.

[0201] The embodiments of the present application provide an image processing method; in the embodiments of the present application, the zoom switch is subdivided into three cases, and the weight logics allocated during color and brightness alignment processing are different in different cases. For example, in the case of normal zoom (such as single or slow zoom switch), according to the current number of frames, the color and brightness corresponding to the camera before the switch can be gradually transitioned to the color and brightness corresponding to the camera after the switch; while in the case of fast zoom, in order to combine the effect intensity of color and brightness alignment processing with the FOV corresponding to the current image to achieve fast convergence, the effect intensity of color and brightness alignment processing can be combined with the zoom ratio to construct the relationship between the weight and the current zoom ratio; thus, during the fast zoom process, the effect intensity of color and brightness alignment processing can be adaptively adjusted according to the current corresponding FOV, rather than a fixed uniform change, thereby avoiding the problem of color and brightness deviation after fast zoom in the related art.

[0202] In addition, in the case of stopping zoom after fast zoom, it can be allocated by combining the weight corresponding to the previous frame and the number of frames of transition, ensuring that the weight between each frame and the previous frame image can be smoothly transitioned after stopping editing.

[0203] Exemplarily, Figure 13 FIG. is a schematic diagram of two frames of images sent for display before zoom and after multiple fast back-and-forth zoom operations. As Figure 13 shown, assuming that 2.5X is the switching point, the camera sent for display can be switched from the main camera to the telephoto camera. If the user quickly zooms back and forth multiple times between 1X and 3X, the corresponding camera will be switched from the main camera to the telephoto camera and then from the telephoto camera to the main camera multiple times. Before the zoom switch, the image sent for display may be as Figure 13 shown in (a) of FIG., the tea box is normally displayed in green, and the central area of the tea box is normally displayed in yellow. After multiple fast zoom switches and processing by combining the weights provided by the embodiments of the present application, the image may be as Figure 13 shown in (b) of FIG., at this time, the color and brightness in the entire image are consistent with the image before the switch, and there is no deviation problem.

[0204] Figure 16 Exemplarily shown is a schematic flowchart of another image processing method of the electronic device 100. The method 600 includes S601 to S609; S601 to S609 will be described in detail below.

[0205] S601. Obtain the image collected by the main camera.

[0206] Exemplarily, the image collected by the main camera can be a Raw image collected by a wide-angle camera; or, the image collected by the main camera can be a Raw image collected by an ultra-wide-angle camera.

[0207] S602, First front - end processing.

[0208] Optionally, the first front - end processing may include algorithms related to converting the Raw image captured by the main camera into a YUV image; this application does not make any limitations on the algorithms.

[0209] Exemplarily, the first front - end processing may refer to the image - processing algorithm in the ISP that converts the Raw image to a YUV image. The first front - end processing may also include subsequent operations such as cropping and zooming based on the zoom ratio in response to a zoom operation.

[0210] S603, Obtain the image captured by the tele - camera.

[0211] Exemplarily, the image captured by the tele - camera may be the Raw image captured by the tele - camera.

[0212] Optionally, S603 and S601 may be executed simultaneously, or S603 and S601 may be executed successively.

[0213] S604, Second front - end processing.

[0214] Optionally, the second front - end processing may include algorithms related to converting the Raw image captured by the tele - camera into a YUV image; this application does not make any limitations on the algorithms.

[0215] Exemplarily, the second front - end processing may refer to the image - processing algorithm in the ISP that converts the Raw image to a YUV image. The second front - end processing may also include subsequent operations such as cropping and zooming based on the zoom ratio in response to a zoom operation.

[0216] After executing S602, perform first back - end processing on the processed image; among them, the implementation method of the first back - end processing can refer to the relevant description of S605.

[0217] S605, First back - end processing.

[0218] Exemplarily, the first back - end processing includes but is not limited to: brightening processing, denoising processing, saturation adjustment processing, cropping processing, or distortion processing.

[0219] S606, Smoothing processing (or called alignment processing).

[0220] Exemplarily, smoothing processing is performed on the image stream collected by the main camera and the image stream collected by the telephoto camera; it can be understood that the data input for smoothing processing is two image streams (for example, the image stream collected by the main camera and the image stream collected by the telephoto camera), and the smoothed image stream is one image stream. This smoothing processing may include the Figure 11 image processing method shown.

[0221] In an embodiment of the present application, through smoothing processing, when switching cameras, obvious jumps in color and brightness of the image can be avoided.

[0222] S607. Second back-end processing.

[0223] Exemplarily, the second back-end processing includes anti-shake processing; for example, the image frames in the image stream can be subjected to anti-shake processing according to the shake parameters to obtain processed images.

[0224] S608. Display the processed image.

[0225] For example, the processed image is: the image obtained by performing color and brightness alignment processing on the image collected by the camera after zoom switching and the last frame image collected by the camera before switching.

[0226] S609. A zoom operation is detected.

[0227] This zoom operation may refer to the description in S430 above.

[0228] Optionally, the electronic device may display the image collected by the main camera, or the image collected by the telephoto camera, and the electronic device detects a user operation on the displayed image.

[0229] It should be understood that the above examples are for helping those skilled in the art to understand the embodiments of the present application, rather than limiting the embodiments of the present application to the specific numerical values or specific scenarios illustrated. Those skilled in the art can clearly make various equivalent modifications or changes based on the above examples, and such modifications or changes also fall within the scope of the embodiments of the present application.

[0230] Exemplarily, Figure 15 is a schematic diagram of an application scenario provided by an embodiment of the present application.

[0231] As Figure 15 shown in (a) of, in response to a user operation on the camera application, the electronic device may display a preview interface 1301, and the preview interface 1301 includes a preview window and shooting controls. Among them, the preview image 1302 displayed in the preview window may include a first shooting object, a second shooting object, and a third shooting object. Here, when the camera application is opened, the zoom ratio is defaulted to 1x. AsFigure 15 As shown in (b) therein, assuming that the user wishes to zoom in on the third subject located in the lower left corner of the preview image 1302, the user can perform an operation of pinching out with two fingers on the screen; in response to this operation, for example, as shown in (c) of Figure 15 , the zoom ratio can be increased to 10x, the imaging size corresponding to the third subject changes from small to large, and the third subject can be located at the middle position of the preview image 1303 in response to the movement of the user's finger. In the embodiments of the present application, the user can perform multiple operations of pinching out with two fingers and pinching in with two fingers back and forth on the screen.

[0232] It should be understood that the above is the preview scenario in the camera mode, and the embodiments of the present application can also be applicable to the preview scenarios in other shooting modes such as video recording. The preview scenario may refer to the scenario where the electronic device captures images before clicking the shooting control in a certain shooting mode.

[0233] Exemplarily, Figure 16 is a schematic diagram of another application scenario provided by the embodiments of the present application.

[0234] After the camera application is opened, in response to the user's click operation on the recording control, the electronic device can start recording a video and display a video recording interface 1401, as shown in (a) of Figure 16 . The video recording interface 1401 may include a recording window, a pause control, an end control, and a zoom control. Among them, the video image 1402 displayed in the recording window may include a first subject, a second subject, a third subject, and a fourth subject. Here, when the video recording starts, the zoom ratio defaults to 1x.

[0235] If the electronic device supports and enables the AI detection function, the electronic device can detect the image content when collecting the video image 1402, and when displaying the video image 1402, display a plurality of detection frames, and each detection frame is used to indicate the position information of a subject in the video image 1402. As shown in (a) of Figure 16 , four detection frames are displayed in the video image 1402, and the four detection frames respectively indicate the position information of the first subject to the fourth subject.

[0236] Optionally, if the AI detection function is only used for face detection, the electronic device can only display two detection frames when displaying the video image 1402, and the two detection frames respectively indicate the faces of the second subject and the fourth subject.

[0237] As shown in Figure 16As shown in (a) in [description], during the recording process, assume that the user hopes to zoom in on the face of the fourth shooting object located on the left side in the video image 1402. The user can click on the detection box corresponding to the fourth shooting object on the screen. As Figure 16 shown in (b) in [description], in response to the user's click operation, the detection box of the fourth shooting object included in the video image 1403 can be in a selected state. For example, compared with other detection boxes, the selected detection box can have a different color and / or style. Then, the electronic device can receive the user's sliding operation on the zoom control. As Figure 16 shown in (c) in [description], in response to the sliding operation on the zoom control (assuming it is slid to 8x), the electronic device can display the video image 1404. The face of the fourth shooting object included in this video image 1404 is larger in size than the face of the fourth shooting object included in the video image 1402, and the face of the fourth shooting object can be located in the middle of the video image 1404 in response to the movement of the user's finger.

[0238] It should be noted that when the electronic device detects the image content and only detects one shooting object or one face, the user does not need to click on the detection box corresponding to the shooting object or face, and the electronic device automatically makes it in a selected state; then, directly in response to the user's sliding operation on the zoom control, it zooms in on and centers the display of the one shooting object or one face. In the embodiments of the present application, the user can perform multiple sliding operations on the zoom control back and forth on the screen.

[0239] It should be understood that the above is a video recording scenario, and the embodiments of the present application can also be applied to scenarios such as video calls.

[0240] Exemplarily, the video call scenario can include but is not limited to the following scenarios: video calls, video conferencing applications, long and short video applications, video live streaming applications, video online course applications, portrait intelligent camera movement applications, recording videos using the system camera recording function, video monitoring, or shooting scenarios such as smart doorbells.

[0241] As described above in combination with Figures 1 to 16 the image processing method provided by the embodiments of the present application is described in detail; below, in combination with Figure 17 and Figure 18 the device embodiments of the present application will be described in detail. It should be understood that the devices in the embodiments of the present application can execute various methods of the foregoing embodiments of the present application, that is, the specific working processes of the following various products can refer to the corresponding processes in the foregoing method embodiments.

[0242] Figure 17It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device 1500 includes a processing module 1510 and a display module 1520; a telephoto camera is also included in the electronic device 1500.

[0243] Among them, the processing module 1510 is configured to: start a camera application program; the display module 1520 is configured to display a first image, which is obtained by a first camera capturing an image; receive a zoom operation; display and save a second image, which is obtained by performing color and brightness alignment processing on an image captured by a second camera and the last frame image captured by the first camera before switching; the weight assigned to the image captured by the second camera in the color and brightness alignment processing includes the minimum value of a first parameter, a second parameter, and a third parameter, and the first parameter corresponds to the current zoom ratio.

[0244] It should be noted that the above electronic device 1500 is embodied in the form of functional units. The term "module" here can be implemented in software and / or hardware forms, and no specific limitation is made thereto.

[0245] For example, the "module" can be a software program, a hardware circuit, or a combination of the two that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group of processors, etc.) for executing one or more software or firmware programs, a memory, a merged logic circuit, and / or other suitable components that support the described functions.

[0246] Therefore, the units of each example described in the embodiments of the present application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0247] Figure 18 A schematic structural diagram of an electronic device provided by the present application is shown. Figure 18 The dotted line in indicates that the unit or the module is optional. The electronic device 1600 can be used to implement the method described in the above method embodiment.

[0248] The electronic device 1600 includes one or more processors 1601, and the one or more processors 1601 can support the electronic device 1600 to implement the image processing method in the method embodiments. The processor 1601 can be a general-purpose processor or a dedicated processor. For example, the processor 1601 can be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

[0249] The processor 1601 can be used to control the electronic device 1600, execute software programs, and process the data of software programs. The electronic device 1600 may further include a communication unit 1605 for implementing signal input (reception) and output (transmission).

[0250] For example, the electronic device 1600 can be a chip, and the communication unit 1605 can be the input and / or output circuit of the chip, or the communication unit 1605 can be the communication interface of the chip, and the chip can be a component of a terminal device or other electronic devices.

[0251] For another example, the electronic device 1600 can be a terminal device, and the communication unit 1605 can be the transceiver of the terminal device, or the communication unit 1605 can be the transceiver circuit of the terminal device.

[0252] The electronic device 1600 may include one or more memories 1602, on which there is a program 1604. The program 1604 can be run by the processor 1601 to generate instructions 1603, so that the processor 1601 executes the method described in the above method embodiments according to the instructions 1603.

[0253] Optionally, data may also be stored in the memory 1602. Optionally, the processor 1601 can also read the data stored in the memory 1602. The data can be stored at the same storage address as the program 1604, or the data can be stored at a different storage address from the program 1604.

[0254] The processor 1601 and the memory 1602 can be set separately or integrated together. For example, they can be integrated on a system on chip (SOC) of a terminal device.

[0255] Exemplarily, the memory 1602 can be used to store the relevant program 1604 of the image processing method provided in the embodiments of the present application. The processor 1601 can be used to call the relevant program 1604 of the image processing method stored in the memory 1602 during video processing to execute the image processing method of the embodiments of the present application; for example, to start the camera application program; to display the first image, where the first image is obtained by the first camera capturing an image; to receive a zoom operation; to display and save the second image, where the second image is obtained by performing color and brightness alignment processing on the image captured by the second camera and the last frame image captured by the first camera before switching; in the color and brightness alignment processing, the weight assigned to the image captured by the second camera includes the minimum value among the first parameter, the second parameter, and the third parameter, and the first parameter corresponds to the current zoom magnification.

[0256] The present application also provides a computer program product, which when executed by the processor 1601 implements the method described in any method embodiment of the present application.

[0257] This computer program product can be stored in the memory 1602, for example, it is the program 1604. The program 1604 undergoes processes such as preprocessing, compilation, assembly, and linking, and is finally converted into an executable target file that can be executed by the processor 1601.

[0258] The present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by the computer, it implements the method described in any method embodiment of the present application. This computer program can be a high-level language program or an executable target program.

[0259] The computer-readable storage medium is, for example, the memory 1602. The memory 1602 may be a volatile memory or a non-volatile memory, or the memory 1602 may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM).

[0260] Those skilled in the art can clearly understand that, for the convenience and conciseness of description, the specific working processes and the resulting technical effects of the above-described devices and apparatuses can refer to the corresponding processes and technical effects in the foregoing method embodiments, and will not be described herein again.

[0261] In several embodiments provided in the present application, the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, some features of the above-described method embodiments can be ignored or not executed. The above-described apparatus embodiments are merely illustrative. The division of units is only a logical function division, and there may be other division methods in actual implementation. Multiple units or components can be combined or integrated into another system. In addition, the coupling between units or the coupling between components can be direct coupling or indirect coupling, and the above coupling includes electrical, mechanical, or other forms of connection.

[0262] It should be understood that in various embodiments of the present application, the sequence numbers of the respective processes do not imply the order of execution, and the order of execution of the respective processes should be determined according to their functions and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0263] In addition, the terms "system" and "network" are often used interchangeably herein. The term "and / or" in this article is merely a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after.

[0264] In summary, the above description is only a preferred embodiment of the technical solution of the present application, and is not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that, Applied to an electronic device, the electronic device includes a first camera and a second camera, and the method includes: Open the camera application; Display a first image, which is obtained by the first camera capturing an image; Receive a zoom operation; Display and save a second image, which is obtained by performing color and brightness alignment processing on the image captured by the second camera and the last frame image captured by the first camera before switching; in the color and brightness alignment processing, the weight assigned to the image captured by the second camera includes the minimum value among a first parameter, a second parameter, and a third parameter, and the first parameter corresponds to the current zoom ratio.

2. The image processing method according to claim 1, characterized in that, The first parameter is determined by the first formula: 1 - (Curzr - Refzr) / ((1 - 1 / i)*Curzr); Wherein, Curzr is used to indicate the current zoom ratio, Refzr is used to indicate the zoom ratio corresponding to the last frame image captured by the first camera before switching, and i > 0.

3. The image processing method according to claim 1 or 2, characterized in that The second parameter is determined by the second formula: PreIntensity - 1 / N; Wherein, PreIntensity is used to indicate the weight corresponding to the previous frame image captured by the second camera, and N is the preset total number of frames.

4. The image processing method according to any one of claims 1 to 3, characterized in that, The third parameter is determined by the third formula: 1 - CurIdx / N; Wherein, CurIdx is used to indicate the Mth frame image captured by the second camera after switching, N is the preset total number of frames, and M ≤ N.

5. The image processing method according to any one of claims 1 to 4, before performing color and brightness alignment processing on the image captured by the second camera and the last frame image captured by the first camera before switching, the method further includes: Traverse the first frame image captured by the second camera and the last frame image captured by the first camera pixel by pixel to determine the mapping relationship; Based on the mapping relationship, obtain a target 3D lut table; The color and brightness alignment processing of the image captured by the second camera and the last frame image captured by the first camera before switching includes: Combined with the target 3D lut table, transform the first frame image captured by the second camera after switching to obtain a reference frame; Fuse the Mth frame image captured by the second camera with the reference frame and combine the assigned weight to obtain the Mth frame target image, and the second image includes the Mth frame target image, where M is an integer greater than or equal to 1.

6. The image processing method according to claim 5, wherein The assigned weight further includes: the difference between 1 and the minimum value; The weight corresponding to the Mth frame image captured by the second camera is the difference value, and the weight corresponding to the reference frame is the minimum value.

7. The image processing method according to claim 5 or 6, characterized in that, Before traversing pixel by pixel, the method further includes: preprocessing; Wherein, the preprocessing includes at least one of registration, brightness alignment, downsampling, and blurring.

8. The image processing method according to any one of claims 1 to 7, characterized in that The zoom operation includes: a two-finger reverse sliding operation, a sliding operation on the displayed zoom control, a voice operation, or an air gesture operation.

9. The image processing method according to claim 8, wherein The zoom operation includes one or more back-and-forth sliding operations on the displayed zoom control.

10. The image processing method according to any one of claims 1 to 9, characterized in that, The first camera includes a wide-angle camera, and / or the second camera includes a telephoto camera or an ultra-wide-angle camera.

11. An electronic device, characterized in that, The electronic device includes: one or more processors, a memory, a first camera, and a second camera; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to execute the image processing method according to any one of claims 1 to 10.

12. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system includes one or more processors, and the processors are used to call computer instructions to cause the electronic device to execute the image processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to execute the image processing method according to any one of claims 1 to 10.

Images