Image processing method and electronic device

Abstract

Disclosed in the present application are an image processing method and an electronic device. The method comprises: during operation of a first camera, receiving an operation of switching from the first camera to a second camera; acquiring M first image frames collected in real time by the first camera and M second image frames collected in real time by the second camera after receiving the operation, the Nth first image frame and the Nth second image frame being collected at the same time, M being a positive integer greater than 1, and N being a positive integer greater than or equal to 1 and less than or equal to M; performing image fusion on the M first image frames and the M second image frames, so as to generate M fused image frames, the Nth fused image frame being obtained by fusing the Nth first image frame and the Nth second image frame; and after displaying the M fused image frames, displaying the images collected by the second camera. By implementing the present application, an overall displayed picture exhibits a slow change effect visually during multi-camera switching, thereby achieving smooth transition of pictures.

Description

Image processing method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on December 29, 2023, with application number 202311862964.2 and application name “Image Processing Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of terminal technology, and in particular to image processing methods and electronic devices. Background Art

[0003] Currently, multiple cameras can be installed on the same terminal, providing different focal lengths to meet users' diverse shooting needs. For example, the wide-angle camera provides a focal length range of 0.4x to 1.0x, and the main camera provides a focal length range of 1.0x to 3.5x. If the user adjusts the shooting focal length from 0.9x to 1.2x, the terminal will switch from displaying the image captured by the wide-angle camera to displaying the image captured by the main camera. Due to the different shooting parameters and installation positions of these multiple cameras, the field of view obtained by these multiple cameras when photographing the same object will vary. Therefore, when the terminal switches from displaying the image captured by the previous camera to displaying the image captured by the next camera, a visual jump will occur.

[0004] Typically, a transformation matrix can be used to offset the entire image captured by the previous camera to align it as closely as possible with the image captured by the next camera. However, since the transformation matrix is ​​based on the same plane in the overall image captured by both cameras, when the overall image captured by the two cameras includes multiple planes, offsetting the entire image based on the transformation matrix will align the planes corresponding to the transformation matrix in the overall image, but the remaining planes will not be aligned. These unaligned planes will still visually jump when switching between displays. Therefore, how to achieve a smooth transition across the entire image when switching between multiple cameras has become a pressing issue.

[0005] Summary of the Invention

[0006] The present application provides an image processing method and an electronic device for achieving a smooth transition on the entire image when switching between multiple cameras.

[0007] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0008] During the operation of the first camera, an operation of switching from the first camera to the second camera is received; after receiving the operation, M frames of first images captured in real time by the first camera and M frames of second images captured in real time by the second camera are obtained, where the N-th frame of the first image and the N-th frame of the second image are captured simultaneously, M is a positive integer greater than 1, and N is a positive integer greater than or equal to 1 and less than or equal to M; image fusion is performed based on the M-frame first image and the M-frame second image to generate an M-frame fused image, where the N-frame fused image is obtained by fusing the N-frame first image and the N-frame second image; after displaying the M-frame fused image, the image captured by the second camera is displayed.

[0009] By implementing the method described in the first aspect, in the process of switching from displaying an image based on the first camera to displaying an image based on the second camera, the multi-frame fused image of the first camera and the second camera can be first displayed, and then the image of the second camera can be displayed separately. This method can make the overall image visually present a slowly changing effect when switching, thereby achieving a smooth transition of the image.

[0010] In a possible implementation, the above-mentioned image fusion is performed based on M frames of first images and M frames of second images to generate M frames of fused images, including: obtaining the target fused image based on the first display image corresponding to the target fused image, the second display image corresponding to the target fused image, and the weight pair corresponding to the target fused image; the first display image corresponding to the target fused image is obtained after image processing of the target first image, the second display image corresponding to the target fused image is obtained after image processing of the target second image, the target fused image is a frame image in the M frames of fused images, the target first image is a frame image in the M frames of first images corresponding to the target fused image, and the target second image is a frame image in the M frames of second images corresponding to the target fused image.

[0011] It can be seen that the target fused image can be obtained by performing image fusion based on the first display image, the second display image and the weight pair corresponding to the target fused image.

[0012] In a possible implementation, the target fusion image is obtained based on the first display image corresponding to the target fusion image, the second display image corresponding to the target fusion image, and the weight pair corresponding to the target fusion image, including: when the first field of view angle corresponding to the first camera is greater than or equal to the second field of view angle corresponding to the second camera, obtaining the weight pair corresponding to the target fusion image; based on the weight pair corresponding to the target fusion image, performing weighted fusion on the first display image corresponding to the target fusion image and the second display image corresponding to the target fusion image to obtain the target fusion image; the scale of the first display image is the same as the scale of the second display image, and the center point of the first display image is the same as the center point of the second display image.

[0013] Since, when the first field of view angle is greater than or equal to the second field of view angle (i.e., in the case of Zoomin), the scale of the first sent image is the same as the scale of the second sent image, and the center point of the first sent image is aligned to the two center points of the second sent image (this is because when obtaining the first sent image, scale alignment can be performed by cropping, and center point alignment can be performed by center point offset), there is no need to further align the scales and center points of the first sent image and the second sent image, and weighted fusion can be directly performed to obtain the target fused image.

[0014] In one possible implementation, the weight pair corresponding to the target fusion image includes a first weight corresponding to the target fusion image and a second weight corresponding to the target fusion image; the above-mentioned weight pair corresponding to the target fusion image performs weighted fusion on the first display image corresponding to the target fusion image and the second display image corresponding to the target fusion image to obtain the target fusion image, including: weighting the first display image corresponding to the target fusion image based on the first weight corresponding to the target fusion image to obtain a first weighted image; weighting the second display image corresponding to the target fusion image based on the second weight corresponding to the target fusion image to obtain a second weighted image; and fusing the first weighted image and the second weighted image to obtain the target fusion image.

[0015] In a possible implementation, the target fusion image is obtained based on the first display image corresponding to the target fusion image, the second display image corresponding to the target fusion image, and the weight pair corresponding to the target fusion image, including: when the first field of view angle corresponding to the first camera is smaller than the second field of view angle corresponding to the second camera, downsampling the first display image corresponding to the target fusion image to obtain a thumbnail image corresponding to the target fusion image, and the scale of the thumbnail image is the same as the scale of the second display image corresponding to the target fusion image; obtaining the parameters to be aligned corresponding to the target fusion image; based on the parameters to be aligned, aligning the thumbnail image to the center point of the second display image corresponding to the target fusion image to obtain an aligned image; and based on the weight pair corresponding to the target fusion image, weightedly fusing the aligned image and the second display image to obtain the target fusion image.

[0016] Since, when the first field of view angle is smaller than the second field of view angle (i.e., when zoomed out), the scale of the first image sent for display is different from the scale of the second image sent for display (this is because when the first image sent for display is obtained, the scales of the first image sent for display and the second image sent for display cannot be aligned by cropping), and the center point of the second image sent for display is not completely aligned with the two center points of the first image sent for display (this is because when the second image sent for display is obtained, partial alignment is performed by center point offset). Therefore, the first image sent for display can be downsampled first, and the scales of the first image sent for display and the second image sent for display can be aligned by downsampling, and then center point alignment and weighted fusion are performed to obtain the target fused image.

[0017] In one possible implementation, the weight pair corresponding to the target fusion image includes a first weight corresponding to the target fusion image and a second weight corresponding to the target fusion image; the above-mentioned weighted fusion of the aligned image and the second display image based on the weight pair corresponding to the target fusion image to obtain the target fusion image includes: weighting the aligned image based on the first weight corresponding to the target fusion image to obtain a first weighted image; weighting the second display image corresponding to the target fusion image based on the second weight corresponding to the target fusion image to obtain a second weighted image; and fusing the first weighted image and the second weighted image to obtain the target fusion image.

[0018] In one possible implementation, the above-mentioned acquisition of the parameters to be aligned corresponding to the target fusion image includes: acquiring the total alignment parameter and the completed alignment parameter corresponding to the target fusion image; the total alignment parameter is used to indicate the center point difference between the target first image and the target second image; the completed alignment parameter is used to indicate the parameter of the completed center point offset when the first display image and the second display image are obtained, and the value of the completed alignment parameter is less than the value of the total alignment parameter; based on the difference between the total alignment parameter and the completed alignment parameter, the parameters to be aligned corresponding to the target fusion image are determined.

[0019] It can be seen that the parameters to be aligned can reflect the difference that still exists between the center point of the first image sent for display and the center point of the second image sent for display. Therefore, based on the parameters to be aligned, the center points of the two images to be fused can be completely aligned before image fusion.

[0020] In one possible implementation, the first weight corresponding to the N-th frame fusion image is greater than the first weight corresponding to the N+1-th frame fusion image, and the second weight corresponding to the N-th frame fusion image is less than the second weight corresponding to the N+1-th frame fusion image, and the number of multiple-frame fusion images is M, M is a positive integer greater than 1, and N is a positive integer greater than or equal to 1 and less than M.

[0021] It can be seen that as the number of frames in the M-frame fusion image increases, the proportion of the picture captured by the first camera in the fusion image gradually decreases, and the proportion of the picture captured by the second camera in the frame fusion image gradually increases, which can achieve a reasonable transition of the displayed image.

[0022] In a second aspect, an embodiment of the present application provides an electronic device, comprising one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are configured to store computer program code, the computer program code comprising computer instructions. When the one or more processors execute the computer instructions, the electronic device performs the image processing method of the first aspect and any possible implementation thereof.

[0023] In a third aspect, an embodiment of the present application provides an image processing device, which includes a function / unit for executing the image processing method in the above-mentioned first aspect and any possible implementation thereof.

[0024] In a fourth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device. The chip system includes at least one processor and an interface, and the interface is used to receive computer instructions and transmit them to at least one processor; at least one processor runs the computer instructions so that the electronic device executes the image processing method in the above-mentioned first aspect and any possible implementation method thereof.

[0025] In a fifth aspect, the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the image processing method in the above-mentioned first aspect and any possible implementation thereof.

[0026] In a sixth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the image processing method in the first aspect and any possible implementation thereof.

[0027] It can be understood that the beneficial effects that can be achieved by the electronic device, image processing device, chip system, computer-readable storage medium, and computer program product provided above can be referred to the beneficial effects in the first aspect and any possible implementation method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] FIG1 is a schematic diagram of a preview interface before and after camera switching provided by an embodiment of the present application;

[0029] FIG2 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0030] FIG3 is a schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0031] FIG4 is a flow chart of an image processing method provided in an embodiment of the present application;

[0032] FIG5 is a schematic diagram of an interface of a camera APP provided in an embodiment of the present application;

[0033] FIG6 is a schematic diagram of software interaction of an image processing method provided in an embodiment of the present application;

[0034] FIG7 is a schematic diagram of the interaction between an IPE module and an image fusion module provided in an embodiment of the present application;

[0035] FIG8 is a schematic structural diagram of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The following is a clear and detailed description of the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship between related objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0037] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0038] The term "user interface (UI)" in the following embodiments of this application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device and finally presented as content that the user can recognize. The commonly used form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be a visual interface element such as time, date, text, icon, button, menu, tab, text box, dialog box, status bar, navigation bar, widget, etc. displayed on the display screen of an electronic device.

[0039] To facilitate understanding of the solutions provided by the embodiments of the present application, the following describes the relevant terms involved in the embodiments of the present application:

[0040] 1. Field of view

[0041] The field of view (FOV) refers to the angle formed by the two edges of the maximum range of the camera's lens, with the camera's lens as the vertex. In other words, the field of view is the angular range of the camera's image reception. A camera's field of view is proportional to its field of view. For example, when the field of view angle is larger, the camera's field of view is larger, and when the field of view angle is smaller, the camera's field of view is smaller. Generally, when the camera's focal length changes, the camera's field of view changes. For example, as the focal length increases, the camera's field of view decreases, the number of objects the camera can capture decreases, and the captured objects appear larger in the frame. Conversely, as the focal length decreases, the camera's field of view increases, the number of objects the camera can capture increases, and the captured objects appear smaller in the frame.

[0042] 2. Scale

[0043] The number of pixels occupied by an object in the image captured by the camera indicates the image's scale. A larger image's scale indicates a blurrier image, while a smaller image's scale indicates a more detailed image. For example, as the focal length increases, the captured object becomes larger (equivalent to the object occupying more pixels), and the image scale decreases, resulting in a more visually detailed image. Conversely, as the focal length decreases, the captured object becomes smaller (equivalent to the object occupying fewer pixels), and the image scale increases, resulting in a more blurry image.

[0044] 3. Difference between center points

[0045] When an electronic device is equipped with multiple cameras, these cameras are mounted in different locations, resulting in different optical centers. Consequently, the center points of the images captured by each camera are different. The difference in the center points of the images captured by the multiple cameras is called the center point difference.

[0046] 4. Multi-camera switching and multi-camera smoothing

[0047] In the process of an electronic device shooting and displaying a preview image through a camera, if the electronic device switches from displaying an image based on the image captured by the previous camera to displaying an image based on the image captured by the next camera, this process is called multi-camera switching. In the present application, multi-camera switching can be triggered based on methods such as user adjustment of focal length. For example, an electronic device is provided with three cameras, namely a wide-angle camera, a main camera and a telephoto camera, wherein the focal length range of the wide-angle camera is 0.4x to 1.0x, the focal length range of the main camera is 1.0x to 3.5x, and the focal length range of the telephoto camera is 3.5x to 30x; when the user adjusts the focal length from the focal length range corresponding to one camera to the focal length range corresponding to another camera, the electronic device triggers multi-camera switching.

[0048] Since the shooting parameters of the previous camera (such as focal length, brightness and color, etc.) are different from those of the next camera, and there is a center point difference between the center points of the image of the previous camera and the center points of the image of the next camera, the displayed image will jump when the electronic device switches between multiple cameras.

[0049] To minimize the jumps that occur when switching between multiple cameras, multi-camera smoothing is required. Traditional multi-camera smoothing methods can be implemented using transformation matrices (such as warp matrices). These transformation matrices are calculated by calculating the relationship between the images before and after the multi-camera switch. The transformation matrix can be used to offset the image before the switch, aligning the resulting image as closely as possible with the image after the switch. Smooth multi-camera switching can then be achieved by first displaying the offset image and then displaying the image after the switch. However, due to the homography of the transformation matrix, the image obtained by offsetting the transformation matrix cannot be completely aligned with the image after the switch. For example, as shown in Figure 1, the left side of Figure 1 shows the preview image from the previous camera, and the right side of Figure 1 shows the preview image from the next camera. Both preview images include a person, the road behind the person, and trees in the distance. The person, the road, and the trees are not on the same plane within the image. To perform multi-camera smoothing, a transformation matrix can be calculated based on the relationship between the people in the two preview images. The transformation matrix is ​​then used to align the left preview image with the right preview image. Because the transformation matrix is ​​derived based on the relationship between the people in the two preview images, the person in the left preview image can be successfully aligned with the person in the right preview image based on the transformation matrix. However, because the person and the road and trees are not on the same plane, the road and trees in the left preview image cannot be aligned with the road and trees in the right preview image based on the transformation matrix. Consequently, this multi-camera smoothing method cannot completely eliminate the visual jump phenomenon and achieve a smooth transition of the entire image.

[0050] In order to achieve a smooth transition of the overall image when switching between multiple cameras, an embodiment of the present application provides an image processing method and an electronic device. The electronic device can be a terminal device with multiple cameras, such as a smartphone, tablet computer, laptop computer, desktop computer, smart car, etc., but is not limited to these. The embodiment of the present application does not limit the type of electronic device. For example, the electronic device can be electronic device 200.

[0051] The hardware structure of electronic device 200 is described below. FIG2 shows a schematic diagram of the hardware structure of electronic device 200. It should be understood that electronic device 200 may have more or fewer components than shown in the figure, may combine two or more components, or may have a different component configuration. The various components shown in the figure may be implemented in hardware, including one or more signal processing and / or application-specific integrated circuits, software, or a combination of hardware and software.

[0052] As shown in FIG2 , the electronic device 200 may include: a processor 110, a memory 120, a camera 193, a display screen 194, a mobile communication module 150, and a wireless communication module 160. Among them:

[0053] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0054] The controller may be the nerve center and command center of the electronic device 200. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0055] Processor 110 may also include memory 120 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0056] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0057] The camera 193 is used to capture images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element, also known as an image sensor, can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal and transmits the converted electrical signal to the processor 110 so that the processor 110 can process the electrical signal. In an embodiment of the present application, the electronic device 200 may include multiple cameras 193, and the locations of the multiple cameras 193 are different.

[0058] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 200 can include one or more display screens 194.

[0059] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the terminal device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0060] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to terminal devices. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0061] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs the sound signal through an audio device or displays an image or video. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0062] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth (BT), BLE broadcast, global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. applied to terminal devices. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0063] In some embodiments, the antenna 1 of the terminal device is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation system (SBAS).

[0064] The software structure of the electronic device 200 is introduced below: As shown in Figure 3, Figure 3 is a software architecture diagram of the electronic device 200 provided in an embodiment of the present application. Among them, the software structure adopts a layered architecture, which divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. As shown in Figure 3, taking the Android system, the Android system running on the AP as an example, in some embodiments, the Android system is divided into four layers, from top to bottom, namely the application layer, the application framework layer (framework), the hardware abstraction layer (HAL) and the kernel layer (kernel). Among them:

[0065] ① Application Layer: The application layer can include a series of application packages. As shown in Figure 3, the application package includes applications such as the camera application. For example, the camera application can be the system's built-in camera app, a user-installed camera app, or other third-party applications that can call the camera app for shooting. In this application, the camera application can trigger multi-camera switching and multi-camera smoothing in response to user operations.

[0066] ② Application Framework Layer: The application framework layer provides application developers with an application programming interface (API) framework for accessing core functionality, as well as various services and management tools. The application framework layer includes predefined functions and algorithms. As shown in Figure 3, the application framework layer can include a camera service. After the camera application in the application layer receives a user action, it can call the camera service, enabling the camera service to manage the status of multiple cameras (e.g., camera on / off status), sessions, and collected data, enabling multi-camera switching and smooth multi-camera operation. For example, when switching between multiple cameras, the camera service needs to obtain relevant data collected by the previous and next cameras. Specifically, the camera service can obtain multiple frames of image data captured in real time by the previous camera after receiving the user action through the session corresponding to the previous camera. Simultaneously, the camera service needs to ensure that the next camera is in operation and, through the session corresponding to the next camera, obtain multiple frames of image data captured in real time by the next camera after receiving the user action. Optionally, the application framework layer may also include a window manager, a content provider, a view system, a resource manager, a notification manager, etc. The camera application may call the content provider, resource manager, notification manager, window manager, view system, etc. to assist in the management of the camera service. The embodiments of the present application do not impose any restrictions on this.

[0067] ③. Hardware Abstraction Layer: The hardware abstraction layer (HAL) is an interface layer between the operating system kernel and the hardware circuitry. Its purpose is to abstract the hardware. It hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system. As shown in Figure 3, the HAL includes image hardware abstraction, including the image signal processing front-end (IFE) module, the spatial alignment transform (SAT) module, the image signal processing back-end (IPE) module, the image fusion module, and the sensor nodes. After receiving image data collected simultaneously by different cameras, the HAL processes it sequentially through the IFE module, the SAT module, and the IPE module.

[0068] After receiving image data from each sensor, the IFE module performs a preliminary cropping of the initial image corresponding to the captured image data to ensure that the resulting image size is suitable for the screen display size. Optionally, the IFE module also performs pre-processing such as color correction and demosaicing on the initial image during the initial cropping process.

[0069] The SAT module is used to receive images processed by the IFE module from different cameras. The SAT module then calculates the transformation information between the different cameras based on these multiple images. The transformation information includes center point alignment parameters and one or more items in the cropping frame, and sends the transformation information to the IPE module. For example, taking the third frame as an example, the IFE module receives the third frame of image data captured in real time by the previous camera, and after preliminary cropping, obtains a cropped image and sends it to the SAT module. At the same time, the IFE module receives the third frame of image data captured in real time by the next camera, and after preliminary cropping, obtains a cropped image and sends it to the SAT module. The SAT module can calculate the transformation information based on the two cropped images corresponding to the third frame obtained.

[0070] The IPE module is used to offset the entire image corresponding to the cropped images from different cameras (i.e., align the center points) based on the transformation information calculated by the SAT module, and / or adjust the field of view corresponding to the cropped images from different cameras to the same size using the cropping frame (i.e., scale alignment). Optionally, the IPE module can also perform noise reduction, detail enhancement, and color processing on the cropped images from different cameras.

[0071] The image fusion module receives and fuses the images processed by the IPE module. Optionally, if the images processed by the IPE module are not spatially aligned (including scale and center point alignment), the image fusion module must perform spatial alignment again before fusion. The sensor node receives commands from the camera service in the application framework layer to control the camera's on / off function.

[0072] ④ The kernel layer is the layer between hardware and software. It includes components such as camera drivers and display drivers. These drivers respond to instructions from the hardware abstraction layer (HAL) to implement their respective functions. For example, the camera driver can receive instructions from the sensor node in the HAL to turn the camera on or off; the display driver can receive instructions from the image fusion module in the HAL to display a multi-frame fused image.

[0073] It is understandable that the above software architecture is only an example. In specific implementation, the electronic device 200 may further include more functional modules in the above layers, which will not be elaborated in this application.

[0074] The image processing method provided in the embodiment of the present application is further described in detail below:

[0075] As shown in Figure 4, Figure 4 is a flow chart of an image processing method provided by an embodiment of the present application. The subject of execution of the method shown in Figure 4 can be an electronic device, or the subject can be a chip in the electronic device. Figure 4 is illustrated by taking an electronic device as the execution subject of the method as an example. The execution subjects of the methods shown in other figures of the embodiment of the present application are similar, and will not be repeated hereafter. The hardware and software structure of the electronic device can refer to the structure shown in Figures 2 and 3 above. The image processing method shown in Figure 4 includes steps 401 to 404. Among them:

[0076] 401. While a first camera is operating, the electronic device receives an operation to switch from the first camera to the second camera.

[0077] In an embodiment of the present application, an electronic device is provided with multiple cameras, each of which has different shooting parameters, installation locations, etc. The first camera is any one of the multiple cameras, and the second camera is any one of the multiple cameras other than the first camera. For example, if the electronic device is equipped with a main camera, a wide-angle camera, and a telephoto camera, the first camera can be the main camera, and the second camera can be the wide-angle camera.

[0078] The first camera operation refers to the electronic device shooting with the first camera and continuously displaying a preview of the first camera shooting. While the first camera is operating, the electronic device may receive a user operation to switch from the first camera to the second camera. For example, when the focal length range supported by the first camera is different from the focal length range supported by the second camera, the operation may be an operation by the user to adjust from a first focal length to a second focal length, where the first focal length is a focal length within the focal length range supported by the first camera and the second focal length is a focal length within the focal length range supported by the second camera. For example, Figure 5 shows a schematic diagram of a camera app interface, wherein interface 501 is the interface displayed by the electronic device when shooting with the main camera. Interface 501 includes a preview interface and a shooting function area. The shooting function area includes multiple shooting modes, album controls, shooting controls, and front and rear camera switching controls. The multiple shooting modes include "Night Scene" mode, "Portrait" mode, "Wide Aperture" mode, "Photo" mode, "Video" mode, "Professional" mode, and "More" mode. The preview interface displays the subject and a focal length setting area. The focal length setting area in interface 501 indicates that the current main camera's shooting focal length is 1x. When the user swipes upward on the focal length setting area, as shown in interface 502, the electronic device increases the camera's focal length and determines whether the user's desired target focal length is still within the focal length supported by the main camera. For example, if the target focal length is 4x, the main camera's focal length range is 1x-3.5x, and the telephoto camera's focal length range is 3.5x to 30x, then the target focal length is not within the main camera's focal length range, but within the telephoto camera's focal length range. Therefore, the electronic device displays the interface shown in interface 503, which is the shooting preview interface corresponding to the telephoto camera, and the size of the subject in interface 503 becomes larger. It can be understood that when the focus setting area receives the user's downward sliding operation, the electronic device reduces the focal length of the camera and determines whether the target focal length expected by the user is still the focal length supported by the main camera. When the target focal length expected by the user is not the focal length supported by the main camera, it triggers the switch from the main camera to the remaining cameras and displays the shooting preview interface corresponding to the remaining cameras.

[0079] Optionally, after the electronic device receives the operation of switching from the first camera to the second camera, it can first determine the status of the second camera; if the second camera is in the activated state, the second camera is maintained in the activated state; if the second camera is in the deactivated state, the second camera is activated and updated from the deactivated state to the activated state. This is because the image signal processing in the electronic device can simultaneously receive image data collected from up to two cameras, so the electronic device can support up to two cameras to be activated at the same time. If the second camera is not activated before receiving the operation, the electronic device activates the second camera. If the second camera is already activated before receiving the operation, the electronic device does not need to activate the second camera again. It should be noted that after receiving the operation and before the second camera is successfully activated, the electronic device can continuously display the preview image taken by the first camera.

[0080] 402. The electronic device obtains, after receiving the operation, M frames of first images captured in real time by the first camera and M frames of second images captured in real time by the second camera.

[0081] After receiving the operation, the electronic device can capture multiple frames of images in real time through the first camera and the second camera. In this application, the image captured in real time by the first camera is referred to as the first image, which is the initial image without image processing. The image captured in real time by the second camera is referred to as the second image, which is the initial image without image processing.

[0082] Assume that the first camera captures M frames of first images and the second camera captures M frames of second images, where M is a positive integer greater than 1. Then the Nth first image in the M frames of first images and the Nth second image in the M frames of second images are captured simultaneously, where N is a positive integer greater than or equal to 1 and less than or equal to M.

[0083] After acquiring M frames of first images and M frames of second images, the electronic device may execute subsequent step 403 to obtain and display the M frames of fused images, and then execute subsequent step 404. In this way, the electronic device may first display the images fused by the multiple first cameras and the second camera, and then display the image captured by the second camera alone, thereby achieving camera switching transition.

[0084] 403. The electronic device performs image fusion based on the M frames of first image and the M frames of second image to generate M frames of fused image.

[0085] In an embodiment of the present application, the value of M can be pre-set. When the electronic device generates any one of the M fused images, it can be generated in real time, that is, after the electronic device acquires the Nth frame first image and the Nth frame second image in real time, it can perform image fusion based on the Nth frame first image and the Nth frame second image to obtain the Nth frame fused image; then, the electronic device can acquire the N+1th frame first image and the N+1th frame second image in real time, perform image fusion based on the N+1th frame first image and the N+1th frame second image to obtain the N+1th frame fused image; the electronic device repeats the above operations until the Mth frame fused image is obtained.

[0086] In one possible implementation, the target fusion image is a frame image in M ​​frames of fusion images, and the electronic device can obtain the target fusion image based on the first display image corresponding to the target fusion image, the second display image corresponding to the target fusion image, and the weight pair corresponding to the target fusion image.

[0087] The target first image is the image frame among the M first image frames that corresponds to the target fused image, and the target second image is the image frame among the M second image frames that corresponds to the target fused image. For example, if the target fused image is the third fused image, then the target first image is the third first image frame, and the target second image is the third second image frame.

[0088] The first display image corresponding to the target fusion image is obtained by performing image processing on the target first image, and the second display image corresponding to the target fusion image is obtained by performing image processing on the target second image.

[0089] Specifically, after the electronic device acquires the target first image and the target second image, it can first perform an initial cropping of the target first image and the target second image through the IFE module so that the size of the two images obtained by the initial cropping adapts to the size of the screen display. Then, the electronic device first performs image processing on the two images obtained by the initial cropping according to the frame number corresponding to the target fused image, and obtains a first display image corresponding to the target fused image and a second display image corresponding to the target fused image. Then, according to the frame number corresponding to the target fused image, the electronic device fuses the first display image corresponding to the target fused image and the second display image corresponding to the target fused image through the image fusion module and the weight pair corresponding to the target fused image to obtain the target fused image.

[0090] The following first introduces how to obtain the first display image corresponding to the target fusion image and the second display image corresponding to the target fusion image:

[0091] Assuming that the target fused image is the Nth fused image among M fused images, where N is a positive integer greater than, equal to, and less than M, the target first image is the Nth first image among the M first images, and the target second image is the Nth second image among the M second images.

[0092] First, the electronic device uses the IFE module to perform initial cropping on the first image of the Nth frame and the second image of the Nth frame. The SAT module then calculates the overall alignment parameter and the first alignment parameter corresponding to the Nth fused image of the two initially cropped images. Then, the electronic device uses the IPE module to offset the two initially cropped images based on the first alignment parameter corresponding to the Nth fused image, obtaining a first display image corresponding to the Nth fused image and a second display image corresponding to the Nth fused image.

[0093] Among them, the total alignment parameter is used to indicate the center point difference between the two images obtained by initial cropping (equivalent to the center point difference between the first image of the Nth frame and the second image of the Nth frame), and the first alignment parameter corresponding to the Nth frame fused image is used to indicate the parameters that the IPE module needs to align.

[0094] During the traditional multi-camera switching transition process, in the Zoomin scenario, the electronic device can determine that the first alignment parameter is the same as the total alignment parameter. Therefore, when the electronic device offsets the two images obtained by the initial cropping through the IPE module, the center point of the first display image corresponding to the obtained N-th frame fused image can be completely aligned with the center point of the second display image corresponding to the N-th frame fused image. However, in the Zoomout scenario, the electronic device does not directly set the first alignment parameter to be the same as the total alignment parameter. The electronic device gradually increases the value of the first alignment parameter as the value of N increases. Therefore, when the electronic device offsets the two images obtained by the initial cropping through the IPE module, the center point of the first display image corresponding to the N-th frame fused image cannot be completely aligned with the center point of the second display image corresponding to the N-th frame fused image.

[0095] For example, if the first field of view angle corresponding to the first camera is greater than or equal to the second field of view angle corresponding to the second camera, the multi-camera switching scene is a Zoomin scene. Assuming that M is 5 and the center point difference between the first and second images is 10 pixels (i.e., the total alignment parameter is 10 pixels), then the first alignment parameter corresponding to the first frame of the fused image, the first alignment parameter corresponding to the second frame of the fused image, the first alignment parameter corresponding to the third frame of the fused image, the first alignment parameter corresponding to the fourth frame of the fused image, and the first alignment parameter corresponding to the fifth frame of the fused image are all 10 pixels. Therefore, through the IPE module, the electronic device can completely align the center point of the first display image corresponding to the first frame fusion image with the center point of the second display image corresponding to the first frame fusion image, the center point of the first display image corresponding to the second frame fusion image is completely aligned with the center point of the second display image corresponding to the second frame fusion image, the center point of the first display image corresponding to the third frame fusion image is completely aligned with the center point of the second display image corresponding to the third frame fusion image, the center point of the first display image corresponding to the fourth frame fusion image is completely aligned with the center point of the second display image corresponding to the fourth frame fusion image, and the center point of the first display image corresponding to the fifth frame fusion image is completely aligned with the center point of the second display image corresponding to the fifth frame fusion image.

[0096] For example, if the first field of view angle corresponding to the first camera is smaller than the second field of view angle corresponding to the second camera, the multi-camera switching scene is a zoom-out scene. Assuming that M is 5 and the center point difference between the first and second images is 10 pixels (i.e., the total alignment parameter is 10 pixels), the first alignment parameter corresponding to the first frame fused image, the first alignment parameter corresponding to the second frame fused image, the first alignment parameter corresponding to the third frame fused image, the first alignment parameter corresponding to the fourth frame fused image, and the first alignment parameter corresponding to the fifth frame fused image can be 1 pixel, 2 pixels, 4 pixels, 6 pixels, and 8 pixels, respectively. Since the first alignment parameter is smaller than the total alignment parameter, the electronic device cannot, through the IPE module, completely align the center point of the first display image corresponding to the first frame fusion image with the center point of the second display image corresponding to the first frame fusion image; cannot completely align the center point of the first display image corresponding to the second frame fusion image with the center point of the second display image corresponding to the second frame fusion image; cannot completely align the center point of the first display image corresponding to the third frame fusion image with the center point of the second display image corresponding to the third frame fusion image; cannot completely align the center point of the first display image corresponding to the fourth frame fusion image with the center point of the second display image corresponding to the fourth frame fusion image; and cannot completely align the center point of the first display image corresponding to the fifth frame fusion image with the center point of the second display image corresponding to the fifth frame fusion image.

[0097] Optionally, if the first field of view angle corresponding to the first camera is greater than the second field of view angle corresponding to the second camera, the electronic device can also obtain re-cropping information through the SAT module when performing an offset based on the first alignment parameter, and re-crop the two images initially cropped according to the re-cropping information through the IPE module.

[0098] For example, if the first field of view angle is larger than the second field of view angle, the field of view of the Nth frame initial cropped image corresponding to the first camera (i.e., the image obtained by the IFE module initially cropping the Nth frame first image) is larger than the field of view of the Nth frame initial cropped image corresponding to the second camera (i.e., the image obtained by the IFE module initially cropping the Nth frame second image). Therefore, the re-cropping information of the Nth frame initial cropped image corresponding to the first camera can be obtained, and the IPE module can be used to crop an image with the same content as the Nth frame initial cropped image corresponding to the second camera from the Nth frame initial cropped image corresponding to the first camera, and the cropped image is enlarged, so that the scale of the first display image corresponding to the Nth frame fused image is the same as the scale of the second display image corresponding to the Nth frame fused image.

[0099] It should be noted that if the first field of view angle is smaller than the second field of view angle, the field of view of the Nth frame initial cropped image corresponding to the first camera (i.e., the image obtained by the IFE module performing initial cropping on the Nth frame first image) is smaller than the field of view of the Nth frame initial cropped image corresponding to the second camera (i.e., the image obtained by the IFE module performing initial cropping on the Nth frame second image). Therefore, the content of the Nth frame initial cropped image corresponding to the first camera is part of the content of the Nth frame initial cropped image corresponding to the second camera, and the Nth frame initial cropped image corresponding to the first camera cannot be cropped again to obtain an image with the same content as the Nth frame initial cropped image corresponding to the second camera. Therefore, when the first field of view angle is smaller than the second field of view angle, the electronic device does not need to obtain re-cropping information through the SAT module, and the IPE module does not need to re-crop the two images obtained by initial cropping according to the re-cropping information. If the first field of view angle is equal to the second field of view angle, the electronic device does not need to perform re-cropping, that is, the electronic device does not need to obtain re-cropping information through the SAT module, and the IPE module does not need to perform re-cropping.

[0100] After obtaining the first display image corresponding to the target fusion image and the second display image corresponding to the target fusion image based on the above operations, the target fusion image can be obtained based on the first display image corresponding to the target fusion image and the second display image corresponding to the target fusion image.

[0101] The following describes how to obtain the target fused image based on the first display image corresponding to the target fused image and the second display image corresponding to the target fused image:

[0102] (1) In a first possible implementation, when the first field of view angle is greater than or equal to the second field of view angle, the electronic device obtains a weight pair corresponding to the target fusion image, and based on the weight pair corresponding to the target fusion image, performs weighted fusion on the first display image corresponding to the target fusion image and the second display image corresponding to the target fusion image to obtain the target fusion image.

[0103] When the first field of view angle is greater than or equal to the second field of view angle, based on the operations of the SAT module and the IPE module, the scale of the first display image corresponding to the target fused image is the same as the scale of the second display image corresponding to the target fused image, and the center point of the first display image corresponding to the target fused image is completely aligned with the center point of the second display image corresponding to the target fused image. Therefore, the electronic device can perform weighted fusion of the first display image corresponding to the target fused image and the second display image corresponding to the target fused image based on the weight pair corresponding to the target fused image to obtain the target fused image.

[0104] Specifically, the weight pair corresponding to the target fused image includes a first weight corresponding to the target fused image and a second weight corresponding to the target fused image. The electronic device first weights the first display image corresponding to the target fused image based on the first weight corresponding to the target fused image to obtain a first weighted image, and then weights the second display image corresponding to the target fused image based on the first weight corresponding to the target fused image to obtain a second weighted image. The electronic device then fuses the first weighted image and the second weighted image to obtain the target fused image.

[0105] (2) In a second possible implementation, when the first field of view angle corresponding to the first camera is smaller than the second field of view angle corresponding to the second camera, the electronic device downsamples the first display image corresponding to the target fusion image to obtain a thumbnail image corresponding to the target fusion image, obtains the parameters to be aligned corresponding to the target fusion image, and aligns the thumbnail image to the center point of the second display image based on the parameters to be aligned to obtain an aligned image; and performs weighted fusion on the aligned image and the second display image based on the weight pair corresponding to the target fusion image to obtain the target fusion image.

[0106] In the case where the first field of view angle is smaller than the second field of view angle, based on the operations of the SAT module and the IPE module, the scale of the first display image corresponding to the target fused image is different from the scale of the second display image corresponding to the target fused image, and the center point of the second display image corresponding to the target fused image is not fully aligned with the center point of the first display image corresponding to the target fused image. Therefore, the electronic device may first perform scale and center point alignment on the first display image corresponding to the target fused image and the second display image corresponding to the target fused image, and then perform weighted fusion based on the scale-aligned and center point-aligned images to obtain the target fused image.

[0107] Specifically, the electronic device first downsamples the first display image corresponding to the target fusion image to obtain a thumbnail image, so that the scale of the thumbnail image is the same as the scale of the second display image corresponding to the target fusion image, thereby achieving scale alignment.

[0108] The electronic device then determines the parameters to be aligned corresponding to the target fused image based on the difference between the total alignment parameter and the completed alignment parameter corresponding to the target fused image (the completed alignment parameter is the first alignment parameter corresponding to the target fused image mentioned above). For example, if the total alignment parameter is 10 pixels and the first alignment parameter corresponding to the first frame of the fused image is 1 pixel, then the parameters to be aligned corresponding to the first frame of the image are 9 parameters. Based on the parameters to be aligned corresponding to the target fused image, the electronic device aligns the center point of the thumbnail image with the center point of the second transmitted image corresponding to the target fused image, obtaining an aligned image to achieve center point alignment.

[0109] Finally, the weight pair corresponding to the target fusion image includes a first weight corresponding to the target fusion image and a second weight corresponding to the target fusion image. The electronic device can weight the aligned images based on the first weight corresponding to the target fusion image to obtain a first weighted image, and weight the second displayed image corresponding to the target fusion image based on the first weight corresponding to the target fusion image to obtain a second weighted image; and fuse the first weighted image and the second weighted image to obtain the target fusion image.

[0110] Optionally, the electronic device may adjust the first weight and the second weight corresponding to the target fusion weights involved in (1) and (2) above according to the number of frames corresponding to the target fusion image. For example, the first weight corresponding to the Nth frame fusion image is greater than the first weight corresponding to the N+1th frame fusion image, and the second weight corresponding to the Nth frame fusion image is less than the second weight corresponding to the N+1th frame fusion image.

[0111] For example, the first weight may gradually decrease from 1 to 0 as the number of frames increases, and the second weight may gradually decrease from 0 to 1 as the number of frames increases. Since the first weight is used to weight the alignment image or the first display image, both of which are derived from the image captured by the first camera, and the second weight is used to weight the second display image, which is derived from the image captured by the second camera, as the first weight gradually decreases and the second weight gradually increases as the number of frames corresponding to the target fused image increases, the proportion of the image captured by the first camera in the target fused image gradually decreases, while the proportion of the image captured by the second camera in the target fused image gradually increases.

[0112] 404. After displaying the M-frame fused image, the electronic device displays the image captured by the second camera.

[0113] Among them, when the electronic device displays the M-frame fused image, it can be displayed in real time, that is, the electronic device displays the N-frame fused image after obtaining the N-frame fused image, and then displays the N+1-frame fused image after obtaining the N+1-frame fused image; the electronic device repeats this operation until the M-frame fused image is displayed, and the electronic device can display the picture captured by the second camera alone.

[0114] Optionally, if the first weight gradually decreases with the number of frames and the second weight gradually increases with the number of frames, when the electronic device displays M frame fused images in sequence, the user can visually feel that the picture captured by the first camera gradually fades and the picture captured by the second camera gradually becomes clear.

[0115] Based on the embodiment described in FIG4 , when switching between multiple cameras, the electronic device may first display a multi-frame fused image of the first camera and the second camera, and then display the image of the second camera alone. This method allows the overall image to visually present a slowly changing effect when switching, thereby achieving a smooth transition of the image.

[0116] The following, combined with the software architecture diagram in Figure 3, takes the Nth frame as an example to introduce in detail the software interaction for obtaining the Nth frame fused image. The software interaction in Figure 6 only introduces the interaction process of each module in the hardware abstraction layer, and the interaction with the other layers is not repeated here.

[0117] Step 601: The IFE module obtains the Nth frame of the first image and the Nth frame of the second image.

[0118] The IFE module may obtain the Nth frame of the first image captured by the first camera and the Nth frame of the second image captured by the second camera.

[0119] Step 602: The IFE module initially crops the Nth frame first image and the Nth frame second image to obtain the Nth frame first cropped image and the Nth frame second cropped image.

[0120] The IFE module in the electronic device supports simultaneous processing of two streams of data. Therefore, the IFE module can simultaneously perform initial cropping on the Nth frame's first image and the Nth frame's second image, obtaining the Nth frame's first cropped image and the Nth frame's second cropped image, respectively. The initial cropping is used to adapt the sizes of the obtained Nth frame's first cropped image and the Nth frame's second cropped image to the size of the preview image displayed on the screen.

[0121] Step 603: The IFE module sends the Nth frame of the first cropped image and the Nth frame of the second cropped image to the SAT module.

[0122] Step 604: The SAT module determines the overall alignment parameter and the first alignment parameter corresponding to the Nth fused image frame.

[0123] The SAT module can first determine the center coordinates of the first cropped image of the Nth frame and the center coordinates of the second cropped image of the Nth frame, and determine the overall alignment parameter based on the difference between these two center coordinates. Then, the SAT module determines the first alignment parameter corresponding to the Nth frame of the fused image based on the first field of view corresponding to the first camera and the second field of view corresponding to the second camera.

[0124] Optionally, when the first field of view angle is greater than or equal to the second field of view angle, the SAT module determines that the first alignment parameter corresponding to the Nth frame fused image is equal to the total alignment parameter.

[0125] Optionally, when the first field of view angle is smaller than the second field of view angle, the SAT module determines the first alignment parameter corresponding to the Nth fused image frame based on the frame number corresponding to the Nth fused image frame. For example, the SAT module may determine the first alignment parameter corresponding to the Nth fused image frame from the total alignment parameters based on the ratio of the frame number corresponding to the Nth fused image frame to the total number of fused image frames. For example, if the total number of fused image frames is M, a value smaller than the total alignment parameter may be determined from the total alignment parameters based on the result of N / M. Optionally, as the value of N increases, the value of the first alignment parameter corresponding to the Nth fused image frame may gradually increase.

[0126] Optionally, when the first field of view angle is greater than the second field of view angle, the SAT module also needs to determine re-cropping information corresponding to the Nth frame first cropped image, and the re-cropping information is used to re-crop the Nth frame first cropped image.

[0127] Step 605: The SAT module sends the first cropped image of the Nth frame, the second cropped image of the Nth frame, the total alignment parameter, and the first alignment parameter corresponding to the Nth fused image to the IPE module.

[0128] Optionally, if the SAT module further determines re-cropping information corresponding to the first cropped image of the Nth frame, the SAT module further sends the re-cropping information corresponding to the first cropped image of the Nth frame to the IPE module.

[0129] Step 606: The IPE module offsets the center points of the first cropped image of the Nth frame and the second cropped image of the Nth frame based on the first alignment parameter corresponding to the Nth frame fusion image to obtain the first display image corresponding to the Nth frame fusion image and the second display image corresponding to the Nth frame fusion image.

[0130] Specifically, when the IPE module offsets the first cropped image of the Nth frame and the second cropped image of the Nth frame based on the first alignment parameter corresponding to the fused image of the Nth frame, the operation may be performed on the image with the larger field of view angle between the first cropped image of the Nth frame and the second cropped image of the Nth frame. For example, if the first field of view angle is smaller than the second field of view angle, the center point of the second cropped image of the Nth frame is offset, while the center point of the first cropped image of the Nth frame remains unchanged. If the first field of view angle is greater than or equal to the second field of view angle, the center point of the first cropped image of the Nth frame is offset, while the center point of the second cropped image of the Nth frame remains unchanged.

[0131] Optionally, the IPE module further crops the first cropped image of the Nth frame again based on the re-cropping information corresponding to the first cropped image of the Nth frame.

[0132] Step 607: The IPE module sends the first display image corresponding to the Nth fused image, the second display image corresponding to the Nth fused image, the total alignment parameter, and the first alignment parameter corresponding to the Nth fused image to the image fusion module.

[0133] Step 608: The image fusion module determines the parameters to be aligned corresponding to the Nth fused image frame based on the first alignment parameters and the total alignment parameters corresponding to the Nth fused image frame.

[0134] The image fusion module may determine the difference between the total alignment parameter and the first alignment parameter corresponding to the N-th fused image as the parameter to be aligned corresponding to the N-th fused image.

[0135] Step 609 - 1 : When the alignment parameter is 0, the image fusion module performs weighted fusion on the first display image corresponding to the N-th fused image and the second display image corresponding to the N-th fused image to obtain the N-th fused image.

[0136] When the first field of view angle is greater than or equal to the second field of view angle, the first alignment parameter corresponding to the Nth fused image is equal to the total alignment parameter. Therefore, the pending alignment parameter corresponding to the Nth fused image is 0. Furthermore, the scale of the first display image corresponding to the Nth fused image is the same as the scale of the second display image corresponding to the Nth fused image. Therefore, the image fusion module directly performs weighted fusion to obtain the Nth fused image.

[0137] Step 609-2: When the alignment parameter is not 0, the image fusion module first performs scale alignment and center point alignment on the first display image corresponding to the N-th frame fusion image and the second display image corresponding to the N-th frame fusion image, and then performs weighted fusion to obtain the N-th frame fusion image.

[0138] When the first field of view angle is smaller than the second field of view angle, the first alignment parameter corresponding to the Nth fused image is smaller than the total alignment parameter. Therefore, the pending alignment parameter corresponding to the Nth fused image is not zero. Furthermore, the scale of the first display image corresponding to the Nth fused image is different from the scale of the second display image corresponding to the Nth fused image. Therefore, the image fusion module first performs scale alignment and center point alignment, and then performs weighted fusion to obtain the Nth fused image. For the specific implementation of step 609-2, please refer to the process shown in Figure 7.

[0139] During the weighted fusion in steps 609-1 and 609-2, the image fusion module may obtain a weight pair corresponding to the Nth fused image, where the weight pair includes a first weight and a second weight corresponding to the Nth fused image. Weighted fusion is then performed based on the first and second weights to obtain the Nth fused image.

[0140] Optionally, the image fusion module can determine the first weight and the second weight corresponding to the Nth frame fused image based on the ratio of the number of frames corresponding to the Nth frame fused image to the total number of frames of the fused image, and the first weight decreases as the ratio increases, and the second weight increases as the ratio increases.

[0141] For example, the number of frames corresponding to the Nth frame fused image is N, and the total number of frames of the fused image is M. Then, the first weight and the second weight corresponding to the Nth frame fused image can be determined based on the result of N / M. For example, the value of N / M can be directly determined as the second weight, and the value of 1-N / M can be determined as the first weight. If the value of M is 5, the first weight corresponding to the first frame fused image is 0.8, and the second weight is 0.2; the first weight corresponding to the first frame fused image is 0.6, and the second weight is 0.4. Alternatively, the first weight and the second weight corresponding to different values ​​of N can be pre-stored, so that the image fusion module can directly read the first weight and the second weight corresponding to the Nth frame fused image based on the value of N.

[0142] FIG7 shows the interaction process between the IPE module and the image fusion module when the first field of view angle is smaller than the second field of view angle:

[0143] As shown in Figure 7, when the first field of view angle corresponding to the first camera is smaller than the second field of view angle corresponding to the second camera, the field of view range of the first cropped image of the Nth frame is smaller than the field of view range of the second cropped image of the Nth frame, and the photographed object in the first cropped image of the Nth frame is larger than the photographed object in the second cropped image of the Nth frame. Therefore, when the IPE module offsets the center points of the first cropped image of the Nth frame and the second cropped image of the Nth frame, the center point of the first cropped image of the Nth frame can be kept unchanged to obtain the first display image corresponding to the Nth frame fusion image, and the first alignment parameter corresponding to the Nth frame fusion image is used to offset the center point of the second cropped image of the Nth frame to obtain the second display image corresponding to the Nth frame fusion image. As shown in Figure 7, since the scale of the first display image corresponding to the Nth fused image differs from the scale of the second display image corresponding to the Nth fused image, the size of the subject in the first display image corresponding to the Nth fused image differs from the size of the subject in the second display image corresponding to the Nth fused image. Therefore, the image fusion module downsamples the first display image corresponding to the Nth fused image to obtain a thumbnail image corresponding to the Nth fused image. The scale of the thumbnail image corresponding to the Nth fused image is equal to the scale of the second display image corresponding to the Nth fused image, indicating that the size of the subject in the thumbnail image is the same as the size of the subject in the second display image. Finally, the image fusion module obtains the parameters to be aligned and the weight pair corresponding to the Nth fused image, performs center point alignment on the first display image corresponding to the Nth fused image and the second display image corresponding to the Nth fused image, and then performs weighted fusion to obtain the Nth fused image. It can be understood that when aligning based on the parameters to be aligned corresponding to the Nth frame fusion image, the center point of the second display image corresponding to the Nth frame fusion image can be kept unchanged, and the center point of the thumbnail image corresponding to the Nth frame fusion image can be reversely offset.

[0144] An embodiment of the present application further provides an electronic device, which may include one or more processors and one or more memories. The one or more memories are coupled to the one or more processors and are configured to store computer program code. The computer program code includes computer instructions. When the one or more processors execute the computer instructions, the electronic device performs the functions or steps performed by the electronic device in the above-described method embodiments.

[0145] An embodiment of the present application further provides an image processing device, which includes a unit for executing the functions / units in the electronic device in the above embodiment.

[0146] An embodiment of the present application also provides a chip system, as shown in Figure 8, the chip system 800 includes at least one processor 801 and at least one interface circuit 802. The processor 801 and the interface circuit 802 can be interconnected via lines. For example, the interface circuit 802 can be used to receive signals from other devices (such as the memory of an electronic device). For another example, the interface circuit 802 can be used to send signals to other devices (such as the processor 801). Exemplarily, the interface circuit 802 can read instructions stored in the memory and send the instructions to the processor 801. When the instructions are executed by the processor 801, the electronic device can perform the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which is not specifically limited in the embodiment of the present application.

[0147] This embodiment further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the functions or steps executed by the mobile phone in the above method embodiment.

[0148] This embodiment further provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the functions or steps executed by the mobile phone in the above method embodiment.

[0149] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to perform the various functions or steps performed by the mobile phone in the above method embodiment.

[0150] Among them, the electronic device, communication system, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0151] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0152] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0153] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0154] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. An image processing method, characterized in that, The method includes: During the operation of the first camera, an operation to switch from the first camera to the second camera is received; After receiving the operation, M first images captured in real time by the first camera and M second images captured in real time by the second camera are obtained. The Nth first image and the Nth second image are captured simultaneously. M is a positive integer greater than 1, and N is a positive integer greater than or equal to 1 and less than or equal to M; Based on the M first images and the M second images, image fusion is performed to generate M fused images. The Nth fused image is obtained by fusing the Nth first image and the Nth second image; After displaying the M fused images, display is performed based on the images captured by the second camera.

2. The method according to claim 1, wherein The performing image fusion based on the M first images and the M second images to generate M fused images includes: Based on the first display image corresponding to the target fused image, the second display image corresponding to the target fused image, and the weight pair corresponding to the target fused image, the target fused image is obtained; The first display image corresponding to the target fused image is obtained by performing image processing on the target first image. The second display image corresponding to the target fused image is obtained by performing image processing on the target second image. The target fused image is one of the M fused images. The target first image is the image in the M first images corresponding to the target fused image. The target second image is the image in the M second images corresponding to the target fused image.

3. The method according to claim 2, wherein The obtaining the target fused image based on the first display image corresponding to the target fused image, the second display image corresponding to the target fused image, and the weight pair corresponding to the target fused image includes: When the first field of view angle corresponding to the first camera is greater than or equal to the second field of view angle corresponding to the second camera, the weight pair corresponding to the target fused image is obtained; Based on the weight pair corresponding to the target fused image, weighted fusion is performed on the first display image corresponding to the target fused image and the second display image corresponding to the target fused image to obtain the target fused image. The scale of the first display image is the same as that of the second display image, and the center point of the first display image is the same as the center point of the second display image.

4. The method according to claim 3, characterized in that, The weight pair corresponding to the target fused image includes the first weight corresponding to the target fused image and the second weight corresponding to the target fused image; The performing weighted fusion on the first display image corresponding to the target fused image and the second display image corresponding to the target fused image based on the weight pair corresponding to the target fused image to obtain the target fused image includes: Weighting the first display image corresponding to the target fused image based on the first weight corresponding to the target fused image to obtain a first weighted image; Weighting the second display image corresponding to the target fused image based on the second weight corresponding to the target fused image to obtain a second weighted image; Fuse the first weighted image and the second weighted image to obtain the target fused image.

5. The method according to claim 2, wherein Obtaining the target fused image based on the first display image corresponding to the target fused image, the second display image corresponding to the target fused image, and the weight pair corresponding to the target fused image includes: When the first field of view angle corresponding to the first camera is less than the second field of view angle corresponding to the second camera, downsample the first display image corresponding to the target fused image to obtain a thumbnail image corresponding to the target fused image, and the scale of the thumbnail image is the same as the scale of the second display image corresponding to the target fused image; Obtain the alignment parameter to be aligned corresponding to the target fused image; Based on the alignment parameter to be aligned, align the thumbnail image to the center point of the second display image corresponding to the target fused image to obtain an aligned image; Based on the weight pair corresponding to the target fused image, perform weighted fusion on the aligned image and the second display image to obtain the target fused image.

6. The method according to claim 5, wherein The weight pair corresponding to the target fused image includes the first weight corresponding to the target fused image and the second weight corresponding to the target fused image; Performing weighted fusion on the aligned image and the second display image based on the weight pair corresponding to the target fused image to obtain the target fused image includes: Weight the aligned image based on the first weight corresponding to the target fused image to obtain a first weighted image; Weight the second display image corresponding to the target fused image based on the second weight corresponding to the target fused image to obtain a second weighted image; Fuse the first weighted image and the second weighted image to obtain the target fused image.

7. The method according to claim 5 or 6, characterized in that, The obtaining the alignment parameter to be aligned corresponding to the target fused image includes: Obtain the total alignment parameter and the alignment parameter that has been completed corresponding to the target fused image; the total alignment parameter is used to indicate the difference in the center points of the target first image and the target second image; the alignment parameter that has been completed is used to indicate the parameter of the center point offset when obtaining the first display image and the second display image, and the value of the alignment parameter that has been completed is less than the value of the total alignment parameter; Based on the difference between the total alignment parameter and the alignment parameter that has been completed, determine the alignment parameter to be aligned corresponding to the target fused image.

8. The method according to claim 4 or 6, characterized in that The first weight corresponding to the Nth frame fused image is greater than the first weight corresponding to the (N + 1)th frame fused image, and the second weight corresponding to the Nth frame fused image is less than the second weight corresponding to the (N + 1)th frame fused image, where N is a positive integer greater than or equal to 1 and less than M.

9. An electronic device, characterized in that, Includes: One or more processors, one or more memories; wherein, the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the method according to any one of claims 1-8.

10. A chip system is applied to an electronic device, characterized in that, The chip system includes at least one processor and an interface, and the interface is configured to receive computer instructions and transmit them to the at least one processor; the at least one processor runs the computer instructions to cause the electronic device to execute the method according to any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, Computer instructions are stored in the computer-readable storage medium, and when the computer instructions run on the electronic device, they cause the electronic device to execute the method according to any one of claims 1-8.

Images