Image processing method and related apparatus thereof

Abstract

The application relates to the field of image processing, and provides an image processing method and related equipment thereof, which is applied to an electronic device including a first camera and a second camera. The method comprises the following steps: starting a camera application program; displaying a first image, which is obtained by collecting an image by the first camera; receiving a zooming operation; displaying and saving a second image, which is obtained by performing color and brightness alignment processing on an image collected by the second camera and a last frame of image collected by the first camera before switching; and the weight assigned to the image collected by the second camera in the color and brightness alignment processing comprises a minimum value in a first parameter, a second parameter and a third parameter, and the first parameter corresponds to a current zooming multiple. In the color and brightness alignment processing, the weight is assigned in combination with the current zooming multiple, so that the smooth transition of image brightness and color before and after switching the camera in zooming can be realized.

Description

Technical Field

[0001] This application relates to the field of image processing, specifically to an image processing method and related equipment. Background Technology

[0002] With the development of shooting functions in electronic devices, camera applications are becoming increasingly widespread. To achieve a better photography experience, current electronic devices typically feature multiple cameras, each with a different focal length.

[0003] During shooting, in response to user input, electronic devices can zoom by switching between cameras with different focal lengths. They can also process the captured images using digital zoom to meet various high-magnification shooting scenarios. However, during the zoom and camera switching process, due to the different characteristics of different cameras, inconsistencies in color and brightness can occur in the displayed images before and after the switch, sometimes even resulting in abrupt changes, severely impacting the user experience.

[0004] Therefore, how to avoid abrupt changes in brightness and color during zoom switching has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides an image processing method and related device, which can achieve a smooth transition of image brightness and color before and after zoom switching by combining the current zoom ratio with the weight allocation in the color brightness alignment processing.

[0006] In a first aspect, an image processing method is provided, applied to an electronic device, the electronic device including a first camera and a second camera, the method including: opening a camera application; displaying a first image, the first image being an image captured by the first camera; receiving a zoom operation; displaying and saving a second image, the second image being obtained by color and brightness alignment processing of the image captured by the second camera and the last frame image captured by the first camera before the switch; the weight assigned to the image captured by the second camera in the color and brightness alignment processing includes the minimum value of a first parameter, a second parameter, and a third parameter, the first parameter corresponding to the current zoom ratio.

[0007] Optionally, in one embodiment, the weighting of the image captured by the second camera in the color brightness alignment process refers to the weighting of the reference frame generated after the first frame image captured by the second camera is processed by the target 3Dlut table.

[0008] In the embodiments of this application, in order to combine the strength of the color brightness alignment processing with the FOV corresponding to the current image to achieve fast convergence, the strength of the color brightness alignment processing can be combined with the zoom ratio to construct the relationship between the weight and the current zoom ratio. Thus, during fast zooming, the strength of the color brightness alignment processing can be adaptively adjusted according to the current FOV, rather than changing uniformly in a fixed manner, thereby avoiding the problem of color and brightness deviation after fast zooming in related technologies.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the first parameter is determined by the first formula: 1-(Curzr-Refzr) / ((1-1 / i)*Curzr); where Curzr is used to indicate the current zoom ratio, Refzr is used to indicate the zoom ratio corresponding to the last frame image captured by the first camera before switching, and i is greater than 0.

[0010] Alternatively, as an implementation, i can take the value 2; then the first formula can be expressed as: 1-(Curzr-Refzr) / (0.5*Curzr).

[0011] In this embodiment, to combine the intensity of color-brightness alignment processing with the FOV of the current image, and to enable the MCC algorithm to converge quickly during rapid zooming, and since there is an inverse relationship between FOV and zoom ratio, this application combines the intensity of color-brightness alignment processing with the zoom ratio to construct a relationship between weights and the current zoom ratio. When the current zoom ratio changes, the first parameter changes, and the weights assigned to the image undergoing color-brightness alignment processing also change accordingly. Therefore, during rapid zooming, the weights assigned in the MCC algorithm adaptively adjust with the FOV, rather than changing uniformly.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the second parameter is determined by the second formula: PreIntensity-1 / N; where PreIntensity is used to indicate the weight corresponding to the previous frame image captured by the second camera, and N is the preset total number of frames.

[0013] In this embodiment of the application, considering that there may be a scenario where zooming stops after rapid zooming, in order to continue the transition of image color and brightness after zooming stops, the weight can be determined by combining the weight of the previous frame image captured by the second camera and the preset total number of frames to complete the transition.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the third parameter is determined by a third formula: 1-CurIdx / N; where CurIdx is used to indicate the Mth frame image captured by the second camera after switching, N is the preset total number of frames, and M is less than or equal to N.

[0015] In this embodiment of the application, during normal camera switching, the weight assigned to the reference frame decreases in the order of increasing image frame number acquired by the second camera, while the weight assigned to the image acquired by the second camera increases.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, before performing color and brightness alignment processing on the image captured by the second camera and the last frame image captured by the first camera before switching, the method further includes: traversing the first frame image captured by the second camera and the last frame image captured by the first camera before switching pixel by pixel to determine a mapping relationship; obtaining a target 3Dlut table based on the mapping relationship; performing color and brightness alignment processing on the image captured by the second camera and the last frame image captured by the first camera before switching includes: transforming the first frame image captured by the second camera after switching in conjunction with the target 3Dlut table to obtain a reference frame; fusing the Mth frame image captured by the second camera with the reference frame and in conjunction with the assigned weights to obtain the Mth frame target image, wherein the second image includes the Mth frame target image, and M is an integer greater than or equal to 1.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, the allocated weights further include: the difference between 1 and the minimum value; the weight corresponding to the Mth frame image captured by the second camera is the difference, and the weight corresponding to the reference frame is the minimum value.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: preprocessing before traversing pixel by pixel; wherein the preprocessing includes at least one of registration, brightness alignment, downsampling, and blurring.

[0019] In this embodiment of the application, by performing preprocessing, the content, brightness and other information of the image captured by the second camera and the last frame image captured by the first camera before switching can be aligned, and the amount of data to be processed in the future can be reduced.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, the zoom operation includes: a two-finger reverse swipe operation, a swipe operation on the displayed zoom control, a voice operation, or an air gesture operation.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, the zoom operation includes one or more back-and-forth sliding operations on the displayed zoom control.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, the first camera includes a wide-angle camera, and / or the second camera includes a telephoto camera or an ultra-wide-angle camera.

[0023] In a second aspect, an electronic device is provided, comprising: one or more processors, a memory, a first camera, and a second camera; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, and the one or more processors calling the computer instructions to cause the electronic device to execute:

[0024] Open the camera application; display the first image, which is the image captured by the first camera; receive the zoom operation; display and save the second image, which is the image captured by the second camera and the last frame image captured by the first camera before switching, after color and brightness alignment processing; the weight assigned to the image captured by the second camera in the color and brightness alignment processing includes the minimum value of the first parameter, the second parameter and the third parameter, and the first parameter corresponds to the current zoom ratio.

[0025] It should be understood that the extensions, limitations, explanations and descriptions of the relevant content in the first aspect above also apply to the same content in the second aspect.

[0026] Thirdly, a chip system is provided, the chip system being applied to an electronic device, the chip system including one or more processors, the processors being configured to invoke computer instructions to cause the electronic device to perform any of the image processing methods in the first aspect.

[0027] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing computer program code, which, when executed by an electronic device, causes the electronic device to perform any of the image processing methods in the first aspect.

[0028] Fifthly, a computer program product is provided, the computer program product comprising: computer program code, which, when executed by an electronic device, causes the electronic device to perform any of the image processing methods in the first aspect. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of a hardware system for an electronic device applicable to this application;

[0030] Figure 2 This is a schematic diagram of a software system applicable to an electronic device of this application;

[0031] Figure 3 This is a schematic diagram of the arrangement of multiple cameras on an electronic device according to an embodiment of this application;

[0032] Figure 4 This is a schematic diagram illustrating the zoom ratio corresponding to different types of cameras provided in this application embodiment;

[0033] Figure 5 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0034] Figure 6 This is a schematic diagram of images captured before and after camera switching without any processing, provided in an embodiment of this application.

[0035] Figure 7 This is a flowchart illustrating an MCC algorithm provided in an embodiment of this application;

[0036] Figure 8 This is a schematic diagram of a color brightness alignment process provided in an embodiment of this application;

[0037] Figure 9 This is a schematic diagram of images captured before and after camera switching, provided in an embodiment of this application;

[0038] Figure 10 This is a schematic diagram of two frames of images displayed before zooming and after multiple rapid back-and-forth zooming operations.

[0039] Figure 11 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0040] Figure 12 This is a schematic diagram of zoom switching provided in an embodiment of this application;

[0041] Figure 13 This is a schematic diagram of another processed image captured before and after camera switching, provided in an embodiment of this application;

[0042] Figure 14 This is a schematic flowchart of another image processing method provided in the embodiments of this application;

[0043] Figure 15 This is a schematic diagram illustrating an application scenario for zoom switching provided in an embodiment of this application;

[0044] Figure 16 This is a schematic diagram illustrating an application scenario for zoom switching provided in an embodiment of this application;

[0045] Figure 17This is a schematic diagram of an electronic device according to an embodiment of this application;

[0046] Figure 18 This is a schematic diagram of an electronic device applicable to this application. Detailed Implementation

[0047] In the embodiments of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0048] First, some terms used in the embodiments of this application will be explained to facilitate understanding by those skilled in the art.

[0049] 1. Field of View (FOV): In optical instruments, the FOV is the angle between the two edges of the lens, representing the maximum range through which the image of the target object can pass through the lens. The size of the FOV determines the field of view of the optical instrument. The larger the FOV, the wider the field of view, but the lower the optical magnification. In other words, objects beyond this angle will not be captured by the lens. A shorter focal length results in a wider horizontal field of view, and thus a smaller image. The horizontal field of view narrows as the focal length increases, while the size of the object being photographed increases accordingly.

[0050] 2. Registration refers to the matching of geographic coordinates between different images obtained by different imaging methods within the same region. This includes three aspects: geometric correction, projection transformation, and ensuring the images are displayed at the same scale.

[0051] 3. Zoom ratio: Zoom ratio is used to indicate the zoom level of an electronic device when shooting.

[0052] The above is a brief introduction to the terms used in the embodiments of this application, and will not be repeated below.

[0053] The hardware system, software system, and application scenarios of the electronic device provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0054] For example, electronic device 100 may be a mobile phone, smart screen, tablet computer, wearable electronic device, in-vehicle electronic device, augmented reality (AR) device, virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), projector, etc. This application embodiment does not limit the specific type of electronic device 100.

[0055] See Figure 1 The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0056] It should be noted that, Figure 1 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include... Figure 1 The components shown may include more or fewer components, or the electronic device 100 may include... Figure 1 The components shown may be a combination of certain components, or the electronic device 100 may include... Figure 1 Sub-components of some of the components shown. Figure 1 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0057] Processor 110 may include one or more processing units. For example, processor 110 may include at least one of the following processing units: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and neural network processing unit (NPU). These different processing units may be independent devices or integrated devices.

[0058] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0059] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0060] For example, the processor 110 can be used to execute the image processing method of the embodiments of this application; for example, opening a camera application; displaying a first image, which is obtained by capturing an image from a first camera; receiving a zoom operation; displaying and saving a second image, which is obtained by color and brightness alignment processing of the image captured by the second camera and the last frame image captured by the first camera before switching; the weight assigned to the image captured by the second camera in the color and brightness alignment processing includes the minimum value of a first parameter, a second parameter and a third parameter, and the first parameter corresponds to the current zoom ratio.

[0061] Figure 1 The connection relationships between the modules shown are merely illustrative and do not constitute a limitation on the connection relationships between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of various connection methods described in the above embodiments.

[0062] The wireless communication function of electronic device 100 can be realized through devices such as antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0063] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0064] Electronic device 100 can implement display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0065] Display screen 194 can be used to display images or videos.

[0066] For example, in an embodiment of this application, the display screen 194 may be used to display a second image.

[0067] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0068] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can perform algorithmic optimization of image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0069] The camera 193 (also known as a lens) is used to capture still images or videos. It can be activated via application commands to enable photo-taking, such as capturing images of any scene. The camera may include components such as an imaging lens, filters, and an image sensor. Light emitted or reflected by an object enters the imaging lens, passes through the filter, and is finally focused onto the image sensor. The imaging lens is primarily used to focus and image the light emitted or reflected by all objects within the shooting field of view (also known as the scene to be captured, the target scene, or the scene image the user expects to capture). The filter is primarily used to filter out excess light waves (such as infrared light waves other than visible light). The image sensor can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The image sensor is primarily used to perform photoelectric conversion on the received light signal, converting it into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into standard RGB, YUV, and other image signal formats.

[0070] For example, the gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, y-axis, and z-axis). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the shake of the electronic device 100, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 100 by moving in the opposite direction, thus achieving image stabilization. The gyroscope sensor 180B can also be used in scenarios such as navigation and motion-sensing games.

[0071] In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0072] The camera 193 can be located in front of the electronic device 100 or on the back of the electronic device 100. The specific number and arrangement of the cameras can be set according to the requirements, and this application does not impose any restrictions.

[0073] For example, the electronic device 100 includes a front-facing camera and a rear-facing camera. For instance, either the front-facing camera or the rear-facing camera may include one or more cameras. Taking an electronic device 100 with four rear-facing cameras as an example, when the electronic device 100 activates the four rear-facing cameras to take pictures, it can use the image processing method provided in the embodiments of this application.

[0074] Alternatively, the camera can be mounted on an external accessory of the electronic device 100, which is rotatably connected to the frame of the phone. The angle formed between the external accessory and the display screen 194 of the electronic device 100 can be any angle between 0 and 360 degrees. For example, when the electronic device 100 takes a selfie, the external accessory rotates the camera to face the user. Of course, when the phone has multiple cameras, only some cameras can be mounted on the external accessory, while the remaining cameras are mounted on the main body of the electronic device 100. This application embodiment does not impose any restrictions on this.

[0075] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0076] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, and MPEG 4.

[0077] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, y-axis, and z-axis). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the shake of the electronic device 100, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 100 by moving in the opposite direction, thus achieving image stabilization. The gyroscope sensor 180B can also be used in scenarios such as navigation and motion-sensing games.

[0078] The accelerometer 180E can detect the magnitude of acceleration of the electronic device 100 in various directions (typically the x-axis, y-axis, and z-axis). When the electronic device 100 is stationary, it can detect the magnitude and direction of gravity. The accelerometer 180E can also be used to identify the attitude of the electronic device 100, serving as input parameters for applications such as screen orientation switching and pedometers.

[0079] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some embodiments, such as in a shooting scenario, the electronic device 100 can utilize the distance sensor 180F to measure distance for fast focusing.

[0080] The ambient light sensor 180L is used to sense the brightness of ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0081] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to perform functions such as unlocking, accessing application locks, taking photos, and answering calls.

[0082] Touch sensor 180K, also known as a touch device, can be disposed on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a touch screen. Touch sensor 180K is used to detect touch operations applied to or near it. Touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be disposed on the surface of electronic device 100, and in a different location from display screen 194.

[0083] The hardware system of electronic device 100 has been described in detail above. The software system of electronic device 100 will be introduced below.

[0084] Figure 2 This is a schematic diagram of the software system of the electronic device provided in the embodiments of this application.

[0085] like Figure 2 As shown, the system architecture may include an application layer 210, an application framework layer 220, a hardware abstraction layer 230, a driver layer 240, and a hardware layer 250.

[0086] Application layer 210 may include applications such as camera application, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0087] The application framework layer 220 provides application programming interfaces (APIs) and programming frameworks for applications in the application layer; the application framework layer may include some predefined functions.

[0088] For example, the application framework layer 220 may include a camera access interface; the camera access interface may include camera management and camera devices. Specifically, camera management can be used to provide an access interface for managing cameras; camera devices can be used to provide an interface for accessing cameras.

[0089] The hardware abstraction layer 230 is used to abstract hardware. For example, the hardware abstraction layer can encompass the camera abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer can call camera algorithms.

[0090] For example, the hardware abstraction layer 230 includes a camera hardware abstraction layer and a camera algorithm; the camera algorithm may include software algorithms for image processing.

[0091] For example, the camera algorithm library may include algorithms corresponding to the image processing methods provided in the embodiments of this application.

[0092] For example, the algorithm in a camera algorithm can refer to something that does not depend on specific hardware implementation; for example, code that can typically run on a CPU.

[0093] The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include a camera driver.

[0094] Hardware layer 250 is located at the lowest level of the operating system; such as Figure 2 As shown, hardware layer 250 may include camera 1, camera 2, camera 3, etc. Camera 1, camera 2, and camera 3 may correspond to multiple cameras on an electronic device.

[0095] For ease of understanding, the following description will take a mobile phone with the above-mentioned hardware and software structure as an example, and will first describe in detail the camera on the electronic device 100 to which the method provided in this application embodiment is applicable.

[0096] The electronic device to which the method provided in this application is applicable has at least a plurality of cameras 193, for example, three types of cameras 193; the three types of cameras are a main camera (e.g., a wide-angle camera), an ultra-wide-angle camera, and a telephoto camera; the three cameras can be used to capture the same scene to be captured.

[0097] Optionally, the electronic device 100 may also have other cameras 193. The type of camera 193 and the number of each type of camera 193 can be set as needed, and this application embodiment does not impose any restrictions on this.

[0098] For example, such as Figure 3 The illustration uses an electronic device 100 with three cameras 193 as an example; the arrangement of the three cameras can be as follows: Figure 3 As shown in (a) or as in Figure 3 As shown in (b); for example, the three cameras 193 can be a main camera 1931 (e.g., a wide-angle camera), an ultra-wide-angle camera 1932, and a telephoto camera 1933.

[0099] It should be understood that the above are merely examples of two arrangement methods, and other arrangement methods are also possible; the specific arrangement method can be designed and modified as needed, and the embodiments of this application do not impose any restrictions on this.

[0100] It should be noted that when the three cameras are shooting, the field of view of the main camera 1931 is usually larger than that of the telephoto camera 1933; while the field of view of the ultra-wide-angle camera 1932 is larger than that of the main camera 1931; the field of view of the ultra-wide-angle camera 1932 may overlap with that of the main camera 1931; that is to say, the ultra-wide-angle camera 1932 can capture the scene captured by the main camera 1931 and the surrounding scene.

[0101] It should be understood that the field of view of the telephoto camera 1933 is smaller than that of the main camera 1931, and there may be overlap between the field of view of the main camera 1931 and the telephoto camera 1933; that is, the main camera 1931 can capture the scene captured by the telephoto camera 1933 and its surrounding scene. Similarly, the field of view of the ultra-wide-angle camera 1932 may overlap with that of the telephoto camera 1933; that is, the ultra-wide-angle camera 1932 can capture the scene captured by the telephoto camera 1933 and its surrounding scene.

[0102] Among them, the ultra-wide-angle camera 1932 is suitable for shooting close-ups due to its smaller focusing distance; and, as the name suggests, the ultra-wide-angle camera 1932 is suitable for shooting scenes with a large field of view; the main camera 1931 is more suitable for shooting portraits due to its higher resolution, while the telephoto camera 1933 is more suitable for shooting distant close-ups.

[0103] For example, such as Figure 4 As shown, the zoom ratio of an ultra-wide-angle camera can be less than M times the zoom ratio; the zoom ratio of a wide-angle camera, i.e., the main camera, can be in the range of [M, N); and the zoom ratio of a telephoto camera can be greater than or equal to N times the zoom ratio.

[0104] For example, M can be 1 and N can be 2.5; then the zoom ratio of the ultra-wide-angle camera is less than 1x zoom ratio (1×); the zoom ratio of the wide-angle camera is from 1x zoom ratio to 2.5x zoom ratio (1×~2.5×); the zoom ratio of the telephoto camera is greater than or equal to 2.5x zoom ratio.

[0105] It should be understood that during the shooting process of electronic devices, the greater the zoom ratio, the smaller the corresponding field of view.

[0106] The following is combined with Figure 5The application scenarios of the image processing method provided in the embodiments of this application are illustrated with examples.

[0107] The method in this application embodiment can be applied to shooting scenarios, previewing scenarios, recording video scenarios, or video call scenarios, etc.; through the method in this application embodiment, smooth zoom and / or smooth switching between different cameras in electronic devices can be achieved, improving the user's shooting experience and image quality.

[0108] For example, the preview scenarios include, but are not limited to, the following scenarios:

[0109] Photo preview, aperture preview, night scene preview, portrait preview, video preview, or professional preview, etc.

[0110] It should be understood that a preview scene can refer to a scene in which an electronic device captures an image before the button indicating that the camera has been clicked, in a certain shooting mode.

[0111] In one example, such as Figure 5 As shown in (a), after the electronic device enters the camera application, it can activate the default shooting mode; in the shooting mode, the electronic device can enter the default shooting mode, which can refer to a shooting mode in which the wide-angle camera is used as the main camera and the zoom ratio is a single zoom ratio (1×), and the electronic device displays the image captured by the main camera; as Figure 5 As shown in (b), in response to user operation, the electronic device can switch to display the image captured by the telephoto camera when the zoom ratio meets the zoom range corresponding to the telephoto camera.

[0112] For example, the image processing method in this application embodiment can also be applied to video call scenarios, wherein the video call scenario may include, but is not limited to, the following scenarios:

[0113] Video calls, video conferencing applications, long and short video applications, live video applications, online video courses, intelligent portrait camera movement applications, video recording by system camera function, video surveillance, or portrait shooting scenarios such as intelligent doorbells, etc.

[0114] It should be understood that the above are illustrative examples of application scenarios and do not limit the application scenarios of this application in any way.

[0115] Currently, during shooting, electronic devices can switch between cameras with different focal lengths to zoom in response to user input. They can also combine digital zoom with image processing to meet various high-magnification shooting scenarios. However, during the zoom and camera switching process, due to the different characteristics of different cameras, inconsistencies in color and brightness can occur in the displayed images before and after the switch, and even abrupt changes in color and brightness may occur, severely impacting the user experience.

[0116] For example, Figure 6 This diagram illustrates images captured before and after a camera switch, without any processing. Figure 6 As shown, when switching from the main camera to the telephoto camera, the main camera captures and displays the image before the switch. The displayed image is as follows: Figure 6 As shown in (a); after switching, the image is captured by the telephoto camera and sent to the display, and the displayed image is as follows. Figure 6 As shown in (b) above. During this switching process, the colors and brightness of the two frames displayed before and after the switch are completely different, resulting in a jump in color and brightness.

[0117] To address this issue, this application provides a Multi-cam consistency (MCC) algorithm for image processing, enabling the displayed image to smoothly transition from the color and brightness corresponding to the previous camera to the color and brightness corresponding to the new camera, thus ensuring the consistency of image color and brightness during camera switching.

[0118] For example, Figure 7 This is a flowchart illustrating an MCC algorithm provided in an embodiment of this application. Figure 7 As shown, the MCC algorithm 300 may include the following S301 to S306.

[0119] S301. Obtain image A0 captured by the main camera.

[0120] For example, the main camera is the camera that displays the image before switching. Image A0 can be an image located in the Raw domain captured by the main camera.

[0121] S302, Acquire image B0 captured by telephoto camera.

[0122] For example, the telephoto camera is the camera that is switched and displayed. Image B0 can be an image located in the Raw domain captured by the telephoto camera.

[0123] S303. Register and align the brightness of images A0 and B0.

[0124] For example, image A0 can be registered with image B0 as a reference. The registration method can be any registration method provided by related technologies, and this application embodiment does not limit it.

[0125] For example, statistical methods can be used to align image brightness.

[0126] S304, downsampling and blurring processing.

[0127] The downsampling and blurring methods can be methods provided by related technologies, and the embodiments of this application do not limit them.

[0128] It should be noted that subsequent processing requires pixel-by-pixel traversal of the image to construct a 3Dlut table. However, the two frames of images acquired by the two cameras have different FOVs and cannot be aligned. Therefore, the two frames can be blurred first to obscure the pixel-level differences, or it can be called noise reduction, to make the color and brightness smoother and ensure that the overall color and brightness of the image remain unchanged.

[0129] S305. Perform pixel-by-pixel traversal on the processed image A0 and processed image B0 to determine the mapping relationship, update the mapping relationship on the basic 3Dlut table, and obtain the target 3Dlut table.

[0130] S306. Using the target 3Dlut table, perform color and brightness alignment processing on the processed image A0 and processed image B0.

[0131] For example, such as Figure 8 As shown, by applying the target 3Dlut table to image B0, the fully applied image B can be obtained. A This should be understood as equivalent to applying the mapping relationship between image A0 and image B0 entirely to image B0, thereby obtaining image B. A It can be considered to be completely consistent with the color and brightness of image A0, and is the closest.

[0132] Then, in order to gradually transition from the color and brightness of the camera before switching to the color and brightness of the camera after switching, image B is... A Alphablending is performed on each frame of the image stream captured by the switched telephoto camera, and image B... A The corresponding weights decrease sequentially, while the weights of each frame in the image stream increase sequentially. Thus, the processed image stream will completely transition to the color and brightness of the switched camera when the corresponding weight is 1. It should be understood that image color and brightness alignment processing is the process described above involving the use of the target 3Dlut table and fusion.

[0133] For example, such as Figure 8 As shown, assuming the camera switches from the main camera to the telephoto camera at 2.5X, the image captured by the main camera at 2.49X is image A0, and the image captured by the telephoto camera at 2.51X is image B0. Assuming the preset transition frame count is 30 frames, applying the obtained target 3DLUT table to image B0 using the method described above yields image B0. A Then, for the first frame image B0 from the telephoto camera, combined with image B... A Image B'0 is obtained by fusing images based on weights 0 and 1; for the second frame image B1 captured by the telephoto camera, it is combined with image B... A Image B'1 is obtained by fusing images with weights of 1 / 30 and 29 / 30; for the third frame image B2 captured by the telephoto camera, it is combined with image B... A Image B'2 is obtained by fusing images with weights of 2 / 30 and 28 / 30; similarly, images B'2 are obtained by fusing images acquired by telephoto cameras. 29 Combined with image B A Image B' is obtained by fusing images based on weights 1 and 0. 29 Based on this, the acquired images Bˊ0 to Bˊ 29 These 30 frames represent the transition from the colors and brightness corresponding to the main camera to those corresponding to the telephoto camera.

[0134] Additionally, it should be understood that the image Bˊ0 obtained after switching cameras is based on the image B with a weight of 1. A The resulting image B'0 can be considered equivalent to image B. A Image B A Since it is obtained by conversion according to the target 3Dlut table, the color and brightness of image B'0 are basically the same as those of image A0. Therefore, the MCC algorithm provided in this application embodiment can avoid the problem of color and brightness jumps in the image before and after zoom switching.

[0135] For example, Figure 9 The diagram illustrates the images captured before and after the camera switching process. For example... Figure 9 As shown, when switching from the main camera to the telephoto camera, the main camera captures and displays the image before the switch. The displayed image is as follows: Figure 9 As shown in (a); after switching, the image is captured by the telephoto camera and sent to the display, and the displayed image is as follows. Figure 9As shown in (b) above. During this switching process, because the image acquired by the telephoto camera is processed using the MCC algorithm described above, the color and brightness of the image after the switch are aligned with the color and brightness of the image before the switch. Therefore, the color and brightness of the images before and after the switch are basically the same, and there is no jump in color or brightness.

[0136] based on Figure 8 As illustrated in the example, the MCC algorithm is based on the idea of ​​smooth transition, ensuring a gradual and smooth transition from the color and brightness of the previous camera to the corresponding color and brightness of the current camera before and after camera switching. However, in reality, zooming is not necessarily a uniform and smooth process. If the user performs multiple zoom operations quickly and repeatedly, due to the imaging differences between the two cameras, the center positions of the images captured by the two cameras cannot be guaranteed to be perfectly aligned. Therefore, the target 3Dlut table calculated before and after each switch will have a certain error. After multiple repeated zoom operations, the error will accumulate and increase, causing the MCC algorithm to fail after rapid, multiple zoom operations. The displayed image color and brightness will be inconsistent with the colors and brightness of the images captured by the two cameras, and there will be no correspondence; that is, the displayed image will be distorted.

[0137] For example, Figure 10 This is a schematic diagram showing two frames of images before zooming and after multiple rapid back-and-forth zoom operations. (See diagram below.) Figure 10 As shown, assuming 2.5X is the switching point, the displayed camera can switch from the main camera to the telephoto camera. If the user rapidly zooms back and forth between 1X and 3X multiple times, the camera will correspondingly switch from the main camera to the telephoto camera and then back to the main camera multiple times. Before the zoom switch, the displayed image might look like... Figure 10 As shown in (a), the tea box normally appears green, and the central area of ​​the tea box normally appears yellow. However, after rapidly switching between multiple zooms, the image may appear as follows. Figure 10 As shown in (b), at this time, the entire image is yellowish and dark, and the color and brightness of the entire image have a certain deviation.

[0138] In view of this, embodiments of this application provide an image processing method and related apparatus. In embodiments of this application, in order to combine the intensity of color and brightness alignment processing with the FOV corresponding to the current image to achieve rapid convergence, the intensity of color and brightness alignment processing can be combined with the zoom ratio to construct a relationship between the weight and the current zoom ratio. Thus, during rapid zooming, the intensity of color and brightness alignment processing can be adaptively adjusted according to the current FOV, rather than changing uniformly in a fixed manner, thereby avoiding the problem of color and brightness deviation after rapid zooming in related technologies.

[0139] The following is combined with Figure 11 A schematic flowchart illustrating the image processing method provided in the embodiments of this application is described in detail.

[0140] Figure 11 This is a schematic flowchart of an image processing method provided in an embodiment of this application. The method can be... Figure 1 The electronic device shown executes the method 400, which includes steps S410 to S440, and steps S410 to S440 are described in detail below.

[0141] It should be understood that in the embodiments of this application, the electronic device includes a first camera and a second camera; wherein the first camera and the second camera are cameras of different types.

[0142] S410, launch the camera application.

[0143] For example, a user can instruct an electronic device to launch the camera application by clicking the icon of the "Camera" application.

[0144] For example, when an electronic device is locked, a user can instruct the device to launch the camera application by swiping right on the screen. Alternatively, if the device is locked and the lock screen includes a camera application icon, the user can instruct the device to launch the camera application by tapping the icon. Or, if the device is running another application with permission to access the camera application, the user can instruct the device to launch the camera application by tapping the corresponding control. For instance, if the device is running an instant messaging application, the user can instruct the device to launch the camera application by selecting a control that enables camera functionality.

[0145] It should be understood that the above is an example of how to open a camera application; the camera application can also be opened by voice commands or other methods; this application does not limit this in any way.

[0146] It should also be understood that launching the camera application can mean running the camera application.

[0147] S420. Display the first image, which is obtained by capturing an image from the first camera.

[0148] For example, the first camera can be Figure 3 The wide-angle camera shown can be a main camera, or it can be any other camera with a wider field of view than a telephoto camera.

[0149] For example, the first image can be an RGB image captured by the main camera, or the first image can be an RGB image processed by a series of camera algorithms from a Raw image captured by the main camera.

[0150] For example, Figure 15 The preview image shown in (a) is the first image captured and displayed by the first camera. For example, Figure 16 The video image shown in (a) is the first image captured and displayed by the first camera.

[0151] It should be understood that the first image may include one or more subjects, and when displayed, each subject identified based on AI detection can have a detection box displayed simultaneously.

[0152] It should also be understood that the type of the subject can be preset as needed. For example, if the type is set to a face, the target object can be used to indicate a face identified in the first image. The type of subject can also include plants, animals, etc. This application embodiment does not impose any limitations in this regard.

[0153] S430, received zoom operation.

[0154] Optionally, receiving a zoom operation may include receiving a zoom operation for the first image.

[0155] It should be understood that zooming on the first image can be a user instruction, or it can be an instruction for zooming automatically triggered by the electronic device based on AI detection. The target object is one of the subjects included in the first image.

[0156] For example, when the zoom operation indicates the operation command triggered by the user, the zoom operation may include a two-finger reverse swipe operation, a click operation, a voice operation, an air gesture operation, etc., for the first image. This application embodiment does not impose any limitations on this.

[0157] For example, such as Figure 16 (a) and Figure 16As shown in (b), when the electronic device displays one or more objects to be photographed and a zoom control, the zoom operation may include a click operation on the target object in the first image and a swipe operation on the zoom control; or, the zoom operation may also include a click operation on the target object in the first image and a two-finger reverse swipe operation. It should be understood that when the zoom operation includes two operations, the user must first perform the click operation, and then perform the swipe operation on the zoom control or the two-finger reverse swipe operation on the target object. The zoom operation may also be other operations, or it may include three or more sub-operations; this application embodiment does not impose any limitations on this.

[0158] Optionally, when the zoom operation indicates a sliding operation against the displayed zoom control, the zoom operation may include one or more back-and-forth sliding operations.

[0159] For example, such as Figure 12 Line ③ shows the zoom operation, which refers to the repeated back-and-forth sliding operation of the zoom control, causing the zoom ratio to slide from 1X to 3.5X, then from 3.5X to 2X, from 2X to 3.4X, and then from 3.4X to 2.3X, etc.

[0160] S440, Display and save the second image.

[0161] The second image is obtained by color and brightness alignment processing of the image captured by the second camera and the last frame image captured by the first camera before switching. The weight assigned to the image captured by the second camera in the color and brightness alignment processing includes the minimum value of the first parameter, the second parameter and the third parameter. The first parameter corresponds to the current zoom ratio.

[0162] It should be understood that the first parameter will be different depending on the zoom level, and the minimum value determined among the first, second, and third parameters will be different. As a result, the weights assigned to the image captured by the second camera in the color and brightness alignment processing will be different.

[0163] In this embodiment, since the image captured by the second camera after switching is color and brightness aligned with the last frame image captured by the first camera before switching, the color and brightness of the images before and after the camera switch can be kept consistent.

[0164] Optionally, before performing color and brightness alignment processing between the image captured by the second camera and the last frame image captured by the first camera before switching, the method may further include:

[0165] Step 1: Preprocess the first frame image captured by the second camera and the last frame image captured by the first camera before switching. The preprocessing includes at least one of the following: registration, brightness alignment, downsampling, and blurring.

[0166] It should be understood that the items and order of preprocessing can be set and adjusted as needed, and this application embodiment does not limit this. In this application embodiment, by performing preprocessing, the content, brightness, and other information of the image captured by the second camera and the last frame image captured by the first camera before switching can be aligned, and the amount of data to be processed subsequently can be reduced.

[0167] Step 2: Traverse the two processed images pixel by pixel to determine the mapping relationship.

[0168] Step 3: Based on the mapping relationship, update the base 3Dlut table to obtain the target 3Dlut table.

[0169] The target 3Dlut table is used to indicate the color correspondence between the first frame image captured by the second camera and the last frame image captured by the first camera before switching.

[0170] Here, the target 3Dlut table is a mapping of RGB fields.

[0171] It should be noted that since the target 3Dlut table is mapped to the RGB domain, subsequent transformations and processing of the image based on the target 3Dlut table can align both color and brightness accordingly.

[0172] Step 4: Combine the target 3Dlut table to transform the first frame image captured by the second camera after switching, and obtain the reference frame.

[0173] For example, combining Figure 8 The first frame image captured by the second camera is image B0. Using the target 3Dlut table, the first frame image captured by the second camera after switching is transformed to obtain the reference frame, which is image B0. A .

[0174] Step 5: Fuse the M-th frame image captured by the second camera with the reference frame using the assigned weights to obtain the M-th target image. The second image includes the reference frame and the (M-1)-th target image, where M is an integer greater than or equal to 2. The assigned weights are the minimum values ​​among the first, second, and third parameters, with the first parameter corresponding to the current zoom level.

[0175] Optionally, the first parameter is determined by the first formula: 1-(Curzr-Refzr) / (0.5*Curzr).

[0176] Among them, Curzr is used to indicate the current zoom level, and Refzr is used to indicate the zoom level corresponding to the last frame image captured by the first camera before switching.

[0177] For example, if during a rapid zoom process, one of the zoom levels is a switch from 0.99X to 1.5X (e.g., switching from an ultra-wide-angle camera to the main camera, with 1X as the switching point), then, in order to ensure that the weights allocated in the color and brightness alignment processing correspond to the zoom ratio and achieve rapid convergence, this application presupposes that the transition to the main camera's color and brightness occurs at twice the zoom ratio corresponding to the last frame image captured by the first camera before the switch, that is, at a zoom ratio of 1.98X. Therefore, by utilizing the change ratio of the field of view (FOV), the following formula can be obtained:

[0178] 1 - (1 / Curzr - 1 / Refzr) / (1 / 2Refzr - 1 / Refzr); Substituting the values, we get: 1 - (1 / 1.5X - 1 / 0.99X) / (1 / 1.98X - 1 / 0.99X); where 1 / 1.5X - 1 / 0.99X indicates the change between the FOV corresponding to the current zoom level and the FOV corresponding to the last frame image captured by the first camera before the switch, and 1 / 1.98X - 1 / 0.99X indicates the change between the FOV corresponding to the target zoom level when the transition is complete and the FOV corresponding to the last frame image captured by the first camera before the switch. Combining this example, the above formula can be rearranged to obtain the first formula for determining the first parameter.

[0179] Alternatively, the 2x in the above example is an example. When i represents the multiple of the zoom ratio after the transition relative to the zoom ratio corresponding to the last frame image captured by the first camera before the switch, the following formula can be obtained using the change ratio of FOV:

[0180] 1 - (1 / Curzr - 1 / Refzr) / (1 / iRefzr - 1 / Refzr); Rearranging the above formula, we get 1 - (Curzr - Refzr) / ((1 - 1 / i) * Curzr). Where i is greater than 0. For example, i can be a decimal, fraction, or integer greater than 0.

[0181] In this embodiment, to combine the intensity of color-brightness alignment processing with the FOV of the current image, and to enable the MCC algorithm to converge quickly during rapid zooming, and since there is an inverse relationship between FOV and zoom ratio, this application combines the intensity of color-brightness alignment processing with the zoom ratio to construct a relationship between weights and the current zoom ratio. When the current zoom ratio changes, the first parameter changes, and the weights assigned to the image undergoing color-brightness alignment processing also change accordingly. Therefore, during rapid zooming, the weights assigned in the MCC algorithm adaptively adjust with the FOV, rather than changing uniformly.

[0182] Optionally, the second parameter is determined by the second formula: PreIntensity-1 / N; where PreIntensity is used to indicate the weight of the previous frame image captured by the second camera, and N is the preset total number of frames.

[0183] In this embodiment of the application, considering that there may be a scenario where zooming stops after rapid zooming, in order to continue to make a uniform transition in the color and brightness of the image after zooming stops, the weight can be determined by combining the weight corresponding to the previous frame image captured by the second camera and the preset total number of frames, so that the weight has a linear decreasing trend.

[0184] It should be noted that since PreIntensity is less than 1, and the weight corresponding to each frame decreases by 1 / N, and N and PreIntensity do not have a direct correspondence, it is possible that during the process of the weight decreasing to 0, the number of transition frames is less than the preset total number of frames, that is, the actual number of transition frames is less than the preset total number of frames.

[0185] For example, such as Figure 12 As shown by line ②, assuming the zoom stops after changing from 1X to 3X, the first parameter, determined by the first formula, is 2 / 3 and remains unchanged. To ensure a smooth transition in color and brightness after zooming stops, a 20-frame transition can be preset. Taking 2 / 3 as the weight corresponding to the previous frame captured by the second camera, the second parameter is determined as follows: for the first transition frame, the determined second parameter is 2 / 3 - 1 / 20 = 37 / 60; for the second transition frame, it is (2 / 3 - 1 / 20) - 1 / 20 = 34 / 60; for the third transition frame, it is 34 / 60 - 1 / 20 = 31 / 60; and so on. For the 13th transition frame, the determined second parameter is 1 / 60. The transition is completed in the 14th frame, and the weight changes to 0. It should be understood that the actual number of transition frames is less than the preset total number of frames.

[0186] Optionally, the third parameter is determined by the third formula: 1-CurIdx / N; where CurIdx is used to indicate the Mth frame image captured by the second camera after switching, N is the preset total number of frames, and M is less than or equal to N.

[0187] It should be understood that the third parameter retains its original logic. That is, during normal camera switching, the weight assigned to the reference frame decreases in the order of increasing image frame counts captured by the second camera, while the weight assigned to the images captured by the second camera increases. The reference frame is the image obtained after the first frame captured by the second camera after switching, processed by the target 3Dlut table.

[0188] For example, if the preset total number of frames is 30, the weight assigned to the first frame image captured by the second camera after switching is 1 / 30, and the weight assigned to the reference frame is 29 / 30; the weight assigned to the second frame image captured by the second camera after switching is 2 / 30, and the weight assigned to the reference frame is 28 / 30; and so on, which will not be elaborated here.

[0189] In this embodiment, during color and brightness alignment processing, the first, second, and third parameters can be determined based on the three formulas mentioned above. The minimum value of the first, second, and third parameters is used as the final weight assigned to the reference frame. Then, the weight assigned to the Mth frame image captured by the second camera is the difference between 1 and the minimum value, and the weight assigned to the reference frame is the minimum value. The image and the assigned weights are combined for weighted fusion, so that the fused image can be displayed as the second image.

[0190] Example 1: Suppose the user zooms from 0.9x to 1x and then stops. In this case, the first parameter determined by the first formula will always be 1, while the second parameter determined by the second formula and the third parameter determined by the third formula will gradually decrease.

[0191] Example 2: Suppose the user zooms from 0.9x to 2x quickly. Then, the first parameter determined by the first formula will drop rapidly, the second parameter determined by the second formula will be larger than the first parameter, and the third parameter determined by the third formula will be related to the frame rate.

[0192] Example 3: Suppose the user zooms from 0.9x to 1.5x and then stops. The zoom process is similar to the first example. Since the zoom level no longer changes, the first parameter determined by the first formula no longer changes. The second parameter determined by the second formula will slowly decrease based on the first parameter, while the third parameter determined by the third formula will continuously decrease slowly.

[0193] This application embodiment takes into account the above three situations, and therefore constructs the usage logic for these three parameters.

[0194] Here, it should also be noted that the reason for selecting the minimum value among the first, second, and third parameters is that the MCC algorithm requires the weight to transition from 1 to 0. Assuming that the above three formulas are three curves (curves of weight changing with the zoom multiplier, with an overall downward trend), always using the minimum value under the same zoom multiplier can ensure that the effect of 3Dlut decreases from large to small.

[0195] Combination Figure 12 As shown in the example, when the zoom operation includes multiple back-and-forth sliding operations on the zoom control, the color brightness alignment processing steps provided in this application can be used for processing each time the camera switches from the first camera to the second camera and each time the camera switches from the second camera to the first camera.

[0196] Optionally, when switching from the second camera to the first camera, the second image is obtained by color and brightness alignment processing of the image captured by the first camera and the last frame image captured by the second camera before the switch. The weight assigned to the image captured by the first camera in the color and brightness alignment processing includes the minimum value among the first parameter, the second parameter and the third parameter. The first parameter corresponds to the current zoom ratio.

[0197] For example, the first image can be an RGB image, and the corresponding second image can be an RGB image.

[0198] It should be understood that S440 can be executed simultaneously with S430, or S440 can be executed after S430. This application embodiment does not impose any restrictions on this.

[0199] For example, a second image can be displayed when in a preview scene. These preview shooting modes include, but are not limited to, shooting modes such as night scene preview mode, video preview mode, photo preview mode, and portrait preview mode.

[0200] For example, when in a recording scene, a second image can be displayed and saved.

[0201] This application provides an image processing method. In this embodiment, zoom switching is subdivided into three cases, and the weighting logic for color and brightness alignment processing differs in each case. For example, in normal zoom (such as a single or slow zoom switch), the color and brightness of the camera before the switch can be gradually transitioned to the color and brightness of the camera after the switch, according to the current frame number. In fast zoom, to combine the strength of the color and brightness alignment processing with the FOV of the current image and achieve rapid convergence, the strength of the color and brightness alignment processing can be combined with the zoom ratio to construct a relationship between the weights and the current zoom ratio. Therefore, during fast zoom, the strength of the color and brightness alignment processing can be adaptively adjusted according to the current FOV, rather than a fixed, uniform change, thus avoiding the problem of color and brightness deviations after fast zoom in related technologies.

[0202] In addition, when zooming stops after a fast zoom, the weights can be allocated by combining the weights of the previous frame and the number of transition frames to ensure a smooth transition of weights between each frame and the previous frame after editing stops.

[0203] For example, Figure 13 This is a schematic diagram showing two frames of images before zooming and after multiple rapid back-and-forth zoom operations. (See diagram below.) Figure 13 As shown, assuming 2.5X is the switching point, the displayed camera can switch from the main camera to the telephoto camera. If the user rapidly zooms back and forth between 1X and 3X multiple times, the camera will correspondingly switch from the main camera to the telephoto camera and then back to the main camera multiple times. Before the zoom switch, the displayed image might look like... Figure 13 As shown in (a), the tea box is normally displayed as green, and the central area of ​​the tea box is normally displayed as yellow. After multiple rapid zoom switching, and after processing with the weights provided in the embodiments of this application, the image may appear as follows. Figure 13 As shown in (b), at this point, the color and brightness of the entire image remain consistent with the image before the switch, and there is no deviation problem.

[0204] Figure 16 An exemplary flowchart of an image processing method for another electronic device 100 is shown. The method 600 includes steps S601 to S609; steps S601 to S609 are described in detail below.

[0205] S601. Acquire the image captured by the main camera.

[0206] For example, the image captured by the main camera can be a raw image captured by the wide-angle camera; or, the image captured by the main camera can be a raw image captured by the ultra-wide-angle camera.

[0207] S602, First front-end processing.

[0208] Optionally, the first front-end processing may include algorithms for converting the Raw image captured by the main camera into a YUV image; this application does not limit the algorithm in any way.

[0209] For example, the first front-end processing may refer to the image processing algorithm performed in the ISP to convert a Raw image to a YUV image. The first front-end processing may also include subsequent operations such as cropping and magnification based on the zoom ratio in response to zoom operations.

[0210] S603: Acquire images captured by the telephoto camera.

[0211] For example, the image captured by the telephoto camera can be a raw image captured by the telephoto camera.

[0212] Optionally, S603 and S601 can be executed simultaneously, or S603 and S601 can be executed sequentially.

[0213] S604, Second Front-End Processing.

[0214] Optionally, the second front-end processing may include algorithms for converting Raw images captured by a telephoto camera into YUV images; this application does not impose any limitations on the algorithms.

[0215] For example, the second front-end processing may refer to the image processing algorithm performed in the ISP to convert the Raw image to a YUV image. The second front-end processing may also include subsequent operations such as cropping and magnification based on the zoom ratio in response to zoom operations.

[0216] After S602 is executed, the processed image undergoes a first backend processing; the implementation of the first backend processing can be found in the relevant description of S605.

[0217] S605, First backend processing.

[0218] For example, the first backend processing includes, but is not limited to: brightening processing, noise reduction processing, saturation adjustment processing, cropping processing, or deformation processing.

[0219] S606, Smoothing (or Alignment).

[0220] For example, smoothing is performed on the image stream captured by the main camera and the image stream captured by the telephoto camera; this can be understood as the input data for smoothing being two image streams (e.g., the image stream captured by the main camera and the image stream captured by the telephoto camera), and the smoothed image stream being one image stream. This smoothing process may include the methods provided in the embodiments of this application. Figure 11 The image processing method shown.

[0221] In the embodiments of this application, smoothing processing can prevent obvious jumps in color and brightness in the image when switching cameras.

[0222] S607, Second Backend Processing.

[0223] For example, the second back-end processing includes stabilization processing; for instance, image frames in the image stream can be stabilized according to jitter parameters to obtain a processed image.

[0224] S608. Display the processed image.

[0225] For example, the processed image is the image obtained by color and brightness alignment between the image captured by the camera after zoom switching and the last frame image captured by the camera before switching.

[0226] S609, Zoom operation detected.

[0227] The zoom operation can be referred to in the description of S430 above.

[0228] Optionally, the electronic device may display an image captured by the main camera, or an image captured by the telephoto camera, and the electronic device may detect user interaction with the displayed image.

[0229] It should be understood that the above examples are provided to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments of this application to the specific values ​​or scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples, and such modifications or changes also fall within the scope of the embodiments of this application.

[0230] For example, Figure 15 This is a schematic diagram of an application scenario provided by an embodiment of this application.

[0231] like Figure 15 As shown in (a), in response to a user's operation on the camera application, the electronic device can display a preview interface 1301, which includes a preview window and shooting controls. The preview image 1302 displayed in the preview window can include a first subject, a second subject, and a third subject. Here, when the camera application is opened, the zoom level is 1x by default. Figure 15 As shown in (b), suppose a user wants to zoom in on the third subject located in the lower left corner of preview image 1302, the user can perform a two-finger reverse swipe on the screen; in response to this action, for example, as Figure 15 As shown in (c), the zoom ratio can be increased to 10x, the imaging size of the third subject increases from small to large, and the third subject can respond to the movement of the user's finger and be located in the center of the preview image 1303. In this embodiment, the user can repeatedly perform two-finger reverse swipe operations and two-finger relative swipe operations on the screen.

[0232] It should be understood that the above describes a preview scenario in photo shooting mode. This application embodiment can also be applied to preview scenarios in other shooting modes such as video recording. A preview scenario can refer to the scenario where the electronic device captures an image before clicking the shooting control in a certain shooting mode.

[0233] For example, Figure 16 This is a schematic diagram illustrating another application scenario provided by an embodiment of this application.

[0234] Once the camera app is opened, in response to the user's click on the recording controls, the electronic device can begin recording video and display the video recording interface 1401, such as... Figure 16 As shown in (a), the video recording interface 1401 may include a recording window, a pause control, an end control, and a zoom control. The video image 1402 displayed in the recording window may include a first subject, a second subject, a third subject, and a fourth subject. Here, the zoom ratio is set to 1x by default when video recording begins.

[0235] If the electronic device supports and enables AI detection, it can detect the content of the video image 1402 during acquisition and display multiple detection boxes when showing the video image 1402. Each detection box indicates the position information of a captured object in the video image 1402. Figure 16 As shown in (a), four detection boxes are displayed in video image 1402, which respectively indicate the position information of the first subject to the fourth subject.

[0236] Optionally, if the AI ​​detection function is only used for face detection, the electronic device may display only two detection boxes when displaying video image 1402, which respectively indicate the face of the second subject and the face of the fourth subject.

[0237] like Figure 16As shown in (a), during recording, assuming the user wants to zoom in on the face of the fourth subject located on the left side of video image 1402, the user can click on the detection box corresponding to the fourth subject on the screen. Figure 16 As shown in (b), in response to a user's click operation, the detection box of the fourth subject included in the video image 1403 can be selected, for example, the selected detection box can have a different color and / or style compared to other detection boxes. Then, the electronic device can receive a user's sliding operation on the zoom control, such as... Figure 16 As shown in (c), in response to a sliding operation on the zoom control (assuming a sliding to 8x), the electronic device can display a video image 1404 in which the face of the fourth subject included in the video image 1404 is larger than the face of the fourth subject included in the video image 1402, and the face of the fourth subject can be located in the center of the video image 1404 in response to the user's finger movement.

[0238] It should be noted that if the electronic device detects only one subject or face in the image, the user does not need to click on the detection box corresponding to that subject or face; the electronic device automatically selects it. Then, in response to the user's sliding operation on the zoom control, the electronic device zooms in and centers on that subject or face. In this embodiment, the user can repeatedly slide the zoom control back and forth on the screen multiple times.

[0239] It should be understood that the above is a video recording scenario, and the embodiments of this application can also be applied to scenarios such as video calls.

[0240] For example, video call scenarios may include, but are not limited to, the following: video calls, video conferencing applications, long and short video applications, live video applications, online video courses, portrait intelligent camera movement applications, video recording by system camera recording function, video surveillance, or shooting scenarios such as smart doorbell cameras.

[0241] The above text combined Figures 1 to 16 The image processing method provided in the embodiments of this application has been described in detail; the following will be combined with Figure 17 and Figure 18 The apparatus embodiments of this application are described in detail below. It should be understood that the apparatus in the embodiments of this application can perform the various methods described in the foregoing embodiments of this application, that is, the specific working processes of the various products described below can be referred to the corresponding processes in the foregoing method embodiments.

[0242] Figure 17This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 1500 includes a processing module 1510 and a display module 1520; the electronic device 1500 also includes a telephoto camera.

[0243] The processing module 1510 is used to: open the camera application; the display module 1520 is used to display a first image, which is an image captured by the first camera; receive a zoom operation; display and save a second image, which is an image captured by the second camera and the last frame image captured by the first camera before switching, after color and brightness alignment processing; the weight assigned to the image captured by the second camera in the color and brightness alignment processing includes the minimum value of a first parameter, a second parameter and a third parameter, and the first parameter corresponds to the current zoom ratio.

[0244] It should be noted that the aforementioned electronic device 1500 is embodied in the form of a functional unit. The term "module" here can be implemented in software and / or hardware, without specific limitations.

[0245] For example, a "module" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application-specific integrated circuit (ASIC), electronic circuitry, a processor (e.g., a shared processor, a proprietary processor, or a group processor) and memory for executing one or more software or firmware programs, integrated logic circuitry, and / or other suitable components that support the described functions.

[0246] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0247] Figure 18 A schematic diagram of the structure of an electronic device provided in this application is shown. Figure 18 The dashed lines indicate that the unit or module is optional. The electronic device 1600 can be used to implement the methods described in the above method embodiments.

[0248] Electronic device 1600 includes one or more processors 1601, which can support the implementation of the image processing method in the method embodiments of electronic device 1600. Processor 1601 can be a general-purpose processor or a special-purpose processor. For example, processor 1601 can be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, such as discrete gate, transistor logic devices, or discrete hardware components.

[0249] The processor 1601 can be used to control the electronic device 1600, execute software programs, and process data from the software programs. The electronic device 1600 may also include a communication unit 1605 for inputting (receiving) and outputting (transmitting) signals.

[0250] For example, electronic device 1600 may be a chip, communication unit 1605 may be the input and / or output circuit of the chip, or communication unit 1605 may be the communication interface of the chip, and the chip may be a component of terminal device or other electronic device.

[0251] For example, electronic device 1600 can be a terminal device, communication unit 1605 can be the transceiver of the terminal device, or communication unit 1605 can be the transceiver circuit of the terminal device.

[0252] The electronic device 1600 may include one or more memories 1602, which store a program 1604. The program 1604 can be executed by the processor 1601 to generate instructions 1603, causing the processor 1601 to execute the method described in the above method embodiments according to the instructions 1603.

[0253] Optionally, the memory 1602 may also store data. Optionally, the processor 1601 may also read the data stored in the memory 1602, which may be stored at the same memory address as the program 1604, or the data may be stored at a different memory address than the program 1604.

[0254] The processor 1601 and memory 1602 can be configured separately or integrated together, for example, integrated on the system on chip (SOC) of the terminal device.

[0255] For example, the memory 1602 can be used to store the relevant program 1604 of the image processing method provided in the embodiments of this application, and the processor 1601 can be used to call the relevant program 1604 of the image processing method stored in the memory 1602 during video processing to execute the image processing method of the embodiments of this application; for example, opening the camera application; displaying a first image, which is obtained by capturing an image from a first camera; receiving a zoom operation; displaying and saving a second image, which is obtained by color and brightness alignment processing of the image captured by the second camera and the last frame image captured by the first camera before switching; the weight assigned to the image captured by the second camera in the color and brightness alignment processing includes the minimum value of a first parameter, a second parameter, and a third parameter, and the first parameter corresponds to the current zoom ratio.

[0256] This application also provides a computer program product that, when executed by processor 1601, implements the methods described in any of the method embodiments of this application.

[0257] The computer program product can be stored in memory 1602, for example, program 1604. Program 1604 is finally converted into an executable object file that can be executed by processor 1601 after processing such as preprocessing, compilation, assembly and linking.

[0258] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer, implements the methods described in any of the method embodiments of this application. The computer program may be a high-level language program or an executable object program.

[0259] The computer-readable storage medium is, for example, memory 1602. Memory 1602 can be volatile memory or non-volatile memory, or memory 1602 can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0260] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process and technical effects of the above-described apparatus and equipment can be referred to the corresponding processes and technical effects in the foregoing method embodiments, and will not be repeated here.

[0261] In the several embodiments provided in this application, the systems, apparatuses, and methods disclosed can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not performed. The apparatus embodiments described above are merely illustrative; the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components can be combined or integrated into another system. Furthermore, the coupling between units or components can be direct coupling or indirect coupling, including electrical, mechanical, or other forms of connection.

[0262] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0263] Furthermore, the terms "system" and "network" are often used interchangeably in this paper. The term "and / or" in this paper merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " in this paper generally indicates that the preceding and following related objects have an "or" relationship.

[0264] In summary, the above description is merely a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image processing method, characterized in that, Applied to an electronic device, the electronic device including a first camera and a second camera, the method includes: Open the camera application; The first image is displayed, which is obtained by the first camera capturing the image. Zoom operation received; Display and save the second image, which is obtained by color and brightness alignment processing of the image captured by the second camera and the last frame image captured by the first camera before switching. The color and brightness alignment processing refers to traversing the first frame image captured by the second camera and the last frame image captured by the first camera before switching pixel by pixel to determine the mapping relationship. Based on the mapping relationship, a target 3DLUT table is obtained. Combined with the target 3DLUT table, the first frame image captured by the second camera after switching is transformed to obtain a reference frame. The Mth frame image captured by the second camera is fused with the reference frame and the assigned weights to obtain the Mth frame target image. The second image includes the Mth frame target image, where M is an integer greater than or equal to 1. The weights assigned to the image captured by the second camera in the color and brightness alignment processing include the minimum value of a first parameter, a second parameter, and a third parameter. The first parameter corresponds to the current zoom level. The second parameter is determined based on the weight corresponding to the previous frame image captured by the second camera and the preset total number of frames. The third parameter is determined based on the number of frames of the image captured by the second camera after switching and the preset total number of frames.

2. The image processing method according to claim 1, characterized in that, The first parameter is determined by the first formula: 1 - (Curzr - Refzr) / ((1 - 1 / i) * Curzr); Wherein, Curzr is used to indicate the current zoom ratio, Refzr is used to indicate the zoom ratio corresponding to the last frame image captured by the first camera before the switch, i is greater than 0, i is used to indicate the zoom ratio at which the transition to the second camera is completed, and is the multiple of the zoom ratio corresponding to the last frame image captured by the first camera before the switch.

3. The image processing method according to claim 1, characterized in that, The second parameter is determined by the second formula: PreIntensity - 1 / N; Wherein, PreIntensity is used to indicate the weight of the previous frame image captured by the second camera, and N is the preset total number of frames.

4. The image processing method according to claim 1, characterized in that, The third parameter is determined by the third formula: 1-CurIdx / N; Where CurIdx is used to indicate the number of frames in the M-th frame image captured by the second camera after switching, N is the preset total number of frames, and M is less than or equal to N.

5. The image processing method according to any one of claims 1 to 4, characterized in that, The weights assigned also include: the difference between 1 and the minimum value; The weight corresponding to the Mth frame image captured by the second camera is the difference, and the weight corresponding to the reference frame is the minimum value.

6. The image processing method according to any one of claims 1 to 5, characterized in that, Before traversing pixel by pixel, the method further includes: preprocessing the first frame image captured by the second camera with the last frame image captured by the first camera before switching; The preprocessing includes at least one of registration, downsampling, and blurring.

7. The image processing method according to any one of claims 1 to 6, characterized in that, The zoom operation includes: a two-finger reverse swipe, a swipe operation on the displayed zoom control, voice operation, or air gesture operation.

8. The image processing method according to claim 7, characterized in that, The zoom operation includes one or more back-and-forth sliding operations on the displayed zoom control.

9. The image processing method according to any one of claims 1 to 8, characterized in that, The first camera includes a wide-angle camera, and / or the second camera includes a telephoto camera or an ultra-wide-angle camera.

10. An electronic device, characterized in that, The electronic device includes: one or more processors, a memory, a first camera and a second camera; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to perform the image processing method as described in any one of claims 1 to 9.

11. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the processors being used to invoke computer instructions to cause the electronic device to perform the image processing method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the image processing method according to any one of claims 1 to 9.

Images