Image processing method and electronic device

Abstract

The application discloses an image processing method and an electronic device, and the method comprises the following steps: in the process of working of a first camera, an operation of switching from the first camera to a second camera is received; M frames of first images collected by the first camera in real time and M frames of second images collected by the second camera in real time are obtained after the operation is received, the Nth frame of the first images and the Nth frame of the second images are collected at the same time, M is a positive integer greater than 1, and N is a positive integer greater than or equal to 1 and less than or equal to M; image fusion is performed based on the M frames of the first images and the M frames of the second images, M frames of fused images are generated, and the Nth frame of the fused images is obtained by fusing the Nth frame of the first images and the Nth frame of the second images; and after the M frames of the fused images are displayed, images collected by the second camera are displayed. By implementing the application, the overall picture displayed can present a slow change effect in vision when multiple cameras are switched, so that the smooth transition of the picture is realized.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to image processing methods and electronic devices. Background Technology

[0002] Currently, multiple cameras can be installed on the same terminal, using the different focal length ranges provided by each camera to meet diverse user shooting needs. For example, a wide-angle camera provides a focal length range of 0.4x to 1.0x, while the main camera provides a focal length range of 1.0x to 3.5x. If the user adjusts the shooting focal length from 0.9x to 1.2x, the terminal will switch from displaying the image captured by the wide-angle camera to displaying the image captured by the main camera. Due to the different shooting parameters and installation positions of these multiple cameras, the field of view obtained by these cameras when shooting the same object varies. Therefore, when the terminal switches from displaying the image captured by one camera to displaying the image captured by another, a visual jump will occur.

[0003] Typically, a transformation matrix can be used to offset the entire image captured by the previous camera to align it as closely as possible with the entire image captured by the subsequent camera. However, since the transformation matrix is ​​based on the same plane within the overall images captured by both cameras, when the overall images from the two cameras include multiple planes, offsetting based on the transformation matrix can align the planes corresponding to the transformation matrix within the overall image, but other planes cannot be aligned. These misaligned planes still create a visual abruptness when switching displays. Therefore, achieving a smooth transition across the entire image during multi-camera switching has become a pressing problem to solve. Summary of the Invention

[0004] This application provides an image processing method and an electronic device that achieves a smooth transition in the overall image when switching between multiple cameras.

[0005] In a first aspect, embodiments of this application provide an image processing method, the method comprising:

[0006] During the operation of the first camera, an operation to switch from the first camera to the second camera is received; after receiving the operation, M frames of the first image captured in real time by the first camera and M frames of the second image captured in real time by the second camera are acquired, where the Nth frame of the first image and the Nth frame of the second image are acquired simultaneously, where M is a positive integer greater than 1 and N is a positive integer greater than or equal to 1 and less than or equal to M; based on the M frames of the first image and the M frames of the second image, image fusion is performed to generate an M-frame fused image, and the Nth frame fused image is obtained by fusing the Nth frame of the first image and the Nth frame of the second image; after displaying the M-frame fused image, the image captured by the second camera is displayed.

[0007] By implementing the method described in the first aspect, during the process of switching from displaying an image based on the first camera to displaying an image based on the second camera, a multi-frame fused image from the first and second cameras is first displayed, and then the image from the second camera is displayed separately. This method allows the overall image to visually change slowly during the switching, achieving a smooth transition of the image.

[0008] In one possible implementation, the above-mentioned image fusion based on M-frame first images and M-frame second images to generate M-frame fused images includes: obtaining a target fused image based on a first display image corresponding to the target fused image, a second display image corresponding to the target fused image, and a weight pair corresponding to the target fused image; the first display image corresponding to the target fused image is obtained by image processing the target first image, the second display image corresponding to the target fused image is obtained by image processing the target second image, the target fused image is a frame in the M-frame fused images, the target first image is a frame in the M-frame first images that corresponds to the target fused image, and the target second image is a frame in the M-frame second images that corresponds to the target fused image.

[0009] As can be seen, the target fused image can be obtained by performing image fusion based on the first and second displayed images corresponding to the target fused image and the weight pairs.

[0010] In one possible implementation, obtaining the target fused image based on the first display image corresponding to the target fused image, the second display image corresponding to the target fused image, and the weight pair corresponding to the target fused image includes: when the first field of view corresponding to the first camera is greater than or equal to the second field of view corresponding to the second camera, obtaining the weight pair corresponding to the target fused image; and performing weighted fusion on the first display image corresponding to the target fused image and the second display image corresponding to the target fused image based on the weight pair corresponding to the target fused image to obtain the target fused image; the scale of the first display image is the same as the scale of the second display image, and the center point of the first display image is the same as the center point of the second display image.

[0011] Since the scale of the first and second displayed images is the same when the first field of view is greater than or equal to the second field of view (i.e., in the case of Zoomin), and the center point of the first displayed image is aligned to the second center point of the second displayed image (because when the first displayed image is obtained, scale alignment can be performed by cropping and center point alignment can be performed by center point offset), there is no need to perform scale alignment and center point alignment on the first and second displayed images again. Weighted fusion can be performed directly to obtain the target fused image.

[0012] In one possible implementation, the weight pair corresponding to the target fused image includes a first weight corresponding to the target fused image and a second weight corresponding to the target fused image; the above-mentioned weighted fusion of the first display image and the second display image corresponding to the target fused image based on the weight pair corresponding to the target fused image to obtain the target fused image includes: weighting the first display image corresponding to the target fused image based on the first weight corresponding to the target fused image to obtain a first weighted image; weighting the second display image corresponding to the target fused image based on the second weight corresponding to the target fused image to obtain a second weighted image; and fusing the first weighted image and the second weighted image to obtain the target fused image.

[0013] In one possible implementation, obtaining the target fused image based on the first display image corresponding to the target fused image, the second display image corresponding to the target fused image, and the weight pair corresponding to the target fused image includes: when the first field of view corresponding to the first camera is smaller than the second field of view corresponding to the second camera, downsampling the first display image corresponding to the target fused image to obtain a thumbnail image corresponding to the target fused image, the scale of the thumbnail image being the same as the scale of the second display image corresponding to the target fused image; obtaining the alignment parameters corresponding to the target fused image; aligning the thumbnail image to the center point of the second display image corresponding to the target fused image based on the alignment parameters to obtain an aligned image; and performing weighted fusion of the aligned image and the second display image based on the weight pair corresponding to the target fused image to obtain the target fused image.

[0014] Since the scale of the first and second displayed images is different when the first field of view is smaller than the second field of view (i.e., in the case of zoom out), (this is because the scale of the first and second displayed images cannot be aligned by cropping when the first displayed image is obtained), and the center point of the second displayed image is not completely aligned with the two center points of the first displayed image (this is because partial alignment was performed by center point offset when the second displayed image is obtained), the first displayed image can be downsampled first. The scale of the first and second displayed images can be aligned by downsampling, and then the center point alignment and weighted fusion can be performed to obtain the target fused image.

[0015] In one possible implementation, the weight pair corresponding to the target fused image includes a first weight corresponding to the target fused image and a second weight corresponding to the target fused image; the above-mentioned weighted fusion of the aligned image and the second display image based on the weight pair corresponding to the target fused image to obtain the target fused image includes: weighting the aligned image based on the first weight corresponding to the target fused image to obtain a first weighted image; weighting the second display image corresponding to the target fused image based on the second weight corresponding to the target fused image to obtain a second weighted image; and fusing the first weighted image and the second weighted image to obtain the target fused image.

[0016] In one possible implementation, the above-mentioned acquisition of the alignment parameters corresponding to the target fused image includes: acquiring the total alignment parameter and the completed alignment parameter corresponding to the target fused image; the total alignment parameter is used to indicate the difference between the center points of the target first image and the target second image; the completed alignment parameter is used to indicate the parameter of the completed center point offset when the first display image and the second display image are obtained, and the value of the completed alignment parameter is less than the value of the total alignment parameter; based on the difference between the total alignment parameter and the completed alignment parameter, the alignment parameters corresponding to the target fused image are determined.

[0017] As can be seen, the alignment parameters can reflect the differences between the center points of the first and second images. Therefore, based on the alignment parameters, the center points of the two images to be merged can be completely aligned before image fusion.

[0018] In one possible implementation, the first weight corresponding to the Nth frame fused image is greater than the first weight corresponding to the N+1th frame fused image, and the second weight corresponding to the Nth frame fused image is less than the second weight corresponding to the N+1th frame fused image. The number of multi-frame fused images is M, where M is a positive integer greater than 1 and N is a positive integer greater than or equal to 1 and less than M.

[0019] As can be seen, as the number of frames in the M-frame fused image increases, the proportion of the image captured by the first camera in the fused image gradually decreases, while the proportion of the image captured by the second camera in the frame fused image gradually increases, thus achieving a reasonable transition of the displayed image.

[0020] Secondly, embodiments of this application provide an electronic device including one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, which includes computer instructions. When the one or more processors execute the computer instructions, the electronic device performs the image processing method described in the first aspect and any of its possible implementations.

[0021] Thirdly, embodiments of this application provide an image processing apparatus, which includes functions / units for performing the image processing method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes at least one processor and an interface for receiving computer instructions and transmitting them to the at least one processor. The at least one processor executes the computer instructions to cause the electronic device to perform the image processing method described in the first aspect and any of its possible implementations.

[0023] Fifthly, this application provides a computer-readable storage medium storing computer instructions that, when executed on an electronic device, cause the electronic device to perform the image processing method described in the first aspect and any of its possible implementations.

[0024] Sixthly, this application provides a computer program product that, when run on a computer, causes the computer to execute the image processing method described in the first aspect and any possible implementation thereof.

[0025] It is understood that the beneficial effects that the electronic devices, image processing devices, chip systems, computer-readable storage media, and computer program products provided above can be referred to in the first aspect and any possible implementation thereof, and will not be repeated here. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of a preview interface before and after camera switching provided in an embodiment of this application;

[0027] Figure 2 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0028] Figure 3 This is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0029] Figure 4 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0030] Figure 5 This is a schematic diagram of the interface of a camera app provided in an embodiment of this application;

[0031] Figure 6 This is a software interaction diagram of an image processing method provided in an embodiment of this application;

[0032] Figure 7This is a schematic diagram illustrating the interaction between an IPE module and an image fusion module provided in an embodiment of this application;

[0033] Figure 8 This is a schematic diagram of the structure of a chip system provided in an embodiment of this application. Detailed Implementation

[0034] The technical solutions in the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0035] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0036] The term "user interface (UI)" used in the following embodiments of this application refers to the medium interface through which an application or operating system interacts and exchanges information with a user. It realizes the conversion between the internal form of information and the form that the user can accept. The user interface is source code written in a specific computer language such as Java or Extensible Markup Language (XML). The interface source code is parsed and rendered on the electronic device, ultimately presenting content that the user can recognize. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be visual interface elements such as time, date, text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets displayed on the screen of an electronic device.

[0037] To facilitate understanding of the solutions provided in the embodiments of this application, the relevant terms involved in the embodiments of this application are introduced below:

[0038] I. Field of view

[0039] The field of view (FOV) is the angle between the two edges of a camera, representing the maximum range through which an object can be captured by the lens. In other words, the FOV is the angular range of images a camera can receive. A camera's FOV is directly proportional to its field of view. For example, a larger FOV results in a larger field of view, and a smaller FOV results in a smaller field of view. Generally, the FOV changes when the camera's focal length changes. For instance, as the focal length increases, the FOV decreases, the number of objects the camera can capture decreases, and the captured objects appear larger in the image; conversely, as the focal length decreases, the FOV increases, the number of objects the camera can capture increases, and the captured objects appear smaller in the image.

[0040] II. Scale

[0041] The number of pixels an object occupies in a frame indicates the image's scale. A larger image scale results in a blurrier image, while a smaller image scale results in a more detailed image. For example, as the focal length increases, the captured object appears larger in the frame (equivalent to the object occupying more pixels), resulting in a smaller image scale and a visually more detailed image. Conversely, as the focal length decreases, the captured object appears smaller in the frame (equivalent to the object occupying fewer pixels), resulting in a larger image scale and a visually blurrier image.

[0042] III. Difference of Center Points

[0043] When an electronic device is equipped with multiple cameras, and these cameras are installed in different positions, their optical centers are different. Therefore, the center point of the image captured by each of these cameras will be different. The difference in the center points of the images captured by multiple cameras is called the center point difference.

[0044] IV. Multi-camera switching and smooth multi-camera operation

[0045] In the process of an electronic device capturing and displaying a preview image using a camera, if the device switches from displaying an image captured by one camera to displaying an image captured by a subsequent camera, this process is called multi-camera switching. In this application, multi-camera switching can be triggered by the user adjusting the focal length, etc. For example, the electronic device has three cameras: a wide-angle camera, a main camera, and a telephoto camera. The wide-angle camera has a focal length range of 0.4x to 1.0x, the main camera has a focal length range of 1.0x to 3.5x, and the telephoto camera has a focal length range of 3.5x to 30x. When the user adjusts the focal length to switch from the focal length range corresponding to one camera to the focal length range corresponding to another camera, the electronic device will trigger multi-camera switching.

[0046] Because the shooting parameters (such as focal length, brightness, and color) of the previous camera are different from those of the subsequent camera, and there is a difference in the center point between the center point of the image from the previous camera and the center point of the image from the subsequent camera, the displayed image will exhibit a jump phenomenon when the electronic device switches between multiple cameras.

[0047] To minimize the abrupt transitions during multi-camera switching, multi-camera smoothing is necessary. Traditional multi-camera smoothing methods utilize transformation matrices (such as warp matrices), which are derived by calculating the relationship between the frames before and after the switch. The transformation matrix can be used to offset the frame before the switch, aligning it as closely as possible to the frame after the switch. By displaying the offset frame first, followed by the frame after the switch, smooth multi-camera switching is achieved. However, due to the homography of the transformation matrix, the frame offset based on the transformation matrix cannot be perfectly aligned to the frame after the switch. For example, ... Figure 1 As shown, Figure 1 The left side shows a preview of the previous camera. Figure 1 The image on the right is a preview from the next camera. Both previews include a person, the road behind the person, and trees in the distance, but the person, road, and trees are not on the same plane in the image. When performing multi-camera smoothing, a transformation matrix is ​​first calculated based on the relationship between the person in the two previews. This matrix is ​​then used to align the left preview to the right preview. Since the transformation matrix is ​​based on the relationship between the person in the two previews, the person in the left preview can be successfully aligned to the person in the right preview. However, because the person, road, and trees are not on the same plane, the transformation matrix cannot align the road and trees in the left preview to the road and trees in the right preview. Therefore, this multi-camera smoothing method cannot completely eliminate visual jumps and achieve a smooth transition in the overall image.

[0048] To achieve a smooth transition in the overall image when switching between multiple cameras, embodiments of this application provide an image processing method and an electronic device. The electronic device can be a terminal device with multiple cameras, such as a smartphone, tablet computer, laptop computer, desktop computer, or smart vehicle, but is not limited to these. Embodiments of this application do not limit the type of electronic device. For example, the electronic device can be electronic device 200.

[0049] The hardware structure of electronic device 200 is described below: Figure 2A schematic diagram of the hardware structure of electronic device 200 is shown. It should be understood that electronic device 200 may have more or fewer components than shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0050] like Figure 2 As shown, the electronic device 200 may include: a processor 110, a memory 120, a camera 193, a display screen 194, a mobile communication module 150, and a wireless communication module 160. Wherein:

[0051] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0052] The controller can be the nerve center and command center of the electronic device 200. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0053] The processor 110 may also include a memory 120 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0054] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0055] Camera 193 is used to capture images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element, also known as an image sensor, can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal and transmits the converted electrical signal to the processor 110 so that the processor 110 can process the electrical signal. In this embodiment, the electronic device 200 may include multiple cameras 193, and the multiple cameras 193 are located in different positions.

[0056] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 200 may have one or more display screens 194.

[0057] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the terminal device can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0058] The mobile communication module 150 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G on terminal devices. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0059] The modem processor may include a modulator and a demodulator. The modulator modulates a low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device or displays images or videos. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0060] The wireless communication module 160 can provide solutions for wireless communication applications on terminal devices, including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth (BT), BLE broadcasting, global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0061] In some embodiments, antenna 1 of the terminal device is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling the terminal device to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. GNSS can include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0062] The software structure of electronic device 200 is described below: Figure 3 As shown, Figure 3 This is a software architecture diagram of the electronic device 200 provided in an embodiment of this application. The software structure adopts a layered architecture, dividing the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. Figure 3 As shown, taking the Android system, which runs on an application processing platform (AP), as an example, in some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the hardware abstraction layer (HAL), and the kernel layer. Among them:

[0063] ① Application Layer: The application layer can include a series of application packages. For example... Figure 3 As shown, the application package includes applications such as a camera app. For example, the camera app can be the system's built-in camera app, a user-installed camera app, or other third-party applications that can call the camera app for shooting. In this application, the camera app can trigger multi-camera switching and multi-camera smoothing in response to user operations.

[0064] ② Application Framework Layer: The application framework layer provides application developers with an application programming interface (API) framework for accessing core functionalities, as well as various services and management tools. The application framework layer includes predefined functions and algorithms. For example... Figure 3 As shown, the application framework layer may include a camera service. When the camera application in the application layer receives a user's operation, it can call the camera service to enable multi-camera switching and smooth multi-camera operation by managing the status (e.g., camera on / off), sessions, and collected data of multiple cameras. For example, during multi-camera switching, the camera service needs to obtain relevant data from the previous and next cameras. Specifically, the camera service can obtain multiple frames of image data captured in real time by the previous camera after receiving the user's operation through the session corresponding to the previous camera. Simultaneously, the camera service needs to determine that the next camera is active and obtain multiple frames of image data captured in real time by the next camera after receiving the user's operation through the session corresponding to the next camera. Optionally, the application framework layer may also include a window manager, content provider, view system, resource manager, notification manager, etc. The camera application can call the content provider, resource manager, notification manager, window manager, view system, etc., to assist the camera service in management. This embodiment does not impose any limitations on this.

[0065] ③ Hardware Abstraction Layer (HAL): The HAL is the interface layer located between the operating system kernel and the hardware circuitry. Its purpose is to abstract the hardware. It hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system. For example... Figure 3As shown, the hardware abstraction layer includes image hardware abstraction, which comprises an image signal processing front end (IFE) module, a spatial alignment transform (SAT) module, an image signal processing back end (IPE) module, an image fusion module, and sensor nodes. After receiving image data simultaneously acquired by different cameras, the hardware abstraction layer processes the data sequentially through the IFE, SAT, and IPE modules.

[0066] After receiving image data from each camera (sensor), the IFE module performs preliminary cropping on the initial image corresponding to the acquired image data, so that the size of the pre-cropped image is suitable for the screen display size. Optionally, the IFE module will also perform preprocessing such as color correction and depixelation on the initial image during the preliminary cropping.

[0067] The SAT module receives images from different cameras processed by the IFE module. Based on these images, the SAT module calculates transformation information between the different cameras. This transformation information includes center point alignment parameters and one or more parameters within the cropping frame. This transformation information is then sent to the IPE module. For example, taking the third frame as an example, the IFE module receives the third frame image data captured in real-time by the preceding camera, performs preliminary cropping, and sends the resulting cropped image to the SAT module. Simultaneously, the IFE module receives the third frame image data captured in real-time by the following camera, performs preliminary cropping, and sends the resulting cropped image to the SAT module. The SAT module can then calculate transformation information based on the two cropped images corresponding to the obtained third frame.

[0068] The IPE module is used to offset the overall image corresponding to the cropped images from different cameras based on the transformation information calculated by the SAT module (i.e., center point alignment), and / or adjust the field of view corresponding to the cropped images from different cameras to the same size (i.e., scale alignment) through the cropping frame. Optionally, the IPE module can also perform noise reduction, detail enhancement, color processing, etc. on the cropped images from different cameras.

[0069] The image fusion module receives the image processed by the IPE module and performs image fusion on the received image. Optionally, if the image obtained after processing by the IPE module is still not spatially aligned (including scale alignment and center point alignment), the image fusion module needs to perform spatial alignment again before image fusion. The sensor node receives instructions from the camera service in the application framework layer to control the camera to turn on or off.

[0070] ④ The kernel layer is the layer between hardware and software. The kernel layer includes drivers such as camera drivers and display drivers. These drivers respond to instructions from the corresponding hardware abstraction in the hardware abstraction layer to implement their respective functions. For example, the camera driver can receive instructions from the sensor node in the hardware abstraction layer to turn the camera on or off; the display driver can receive instructions from the image fusion module in the hardware abstraction layer to display multi-frame fused images.

[0071] It is understood that the above software architecture is only an example. In actual implementation, the electronic device 200 may include more functional modules in the above layers, which will not be elaborated in this application.

[0072] The image processing method provided in the embodiments of this application will be described in further detail below:

[0073] like Figure 4 As shown, Figure 4 This is a flowchart illustrating an image processing method provided in an embodiment of this application. Figure 4 The subject of the method shown can be an electronic device, or the subject can be a chip in the electronic device. Figure 4 The method will be explained using an electronic device as the execution subject as an example. The execution subjects of the methods shown in other figures of this application are similar and will not be repeated hereafter. The hardware and software structure of the electronic device can be referred to the above. Figure 2 and Figure 3 The structure shown. Figure 4 The image processing method shown includes steps 401 to 404. Wherein:

[0074] 401. During the operation of the first camera, the electronic device receives an operation to switch from the first camera to the second camera.

[0075] In this embodiment, the electronic device is equipped with multiple cameras, each with different shooting parameters and installation positions. The first camera is any one of the multiple cameras, and the second camera is any one of the multiple cameras other than the first camera. For example, if the electronic device is equipped with a main camera, a wide-angle camera, and a telephoto camera, then the first camera can be the main camera, and the second camera can be the wide-angle camera.

[0076] In this context, "first camera operation" refers to the electronic device taking a picture through the first camera and continuously displaying a preview of the picture taken by the first camera. During the operation of the first camera, the electronic device can receive a user's command to switch from the first camera to the second camera. For example, when the focal length range supported by the first camera differs from that supported by the second camera, this command can be a user's adjustment from the first focal length to the second focal length, where the first focal length is a focal length within the focal length range supported by the first camera, and the second focal length is a focal length within the focal length range supported by the second camera. For example... Figure 5 A schematic diagram of a camera app interface is shown. Interface 501 is the interface displayed by the electronic device when shooting with the main camera. Interface 501 includes a preview interface and a shooting function area. The shooting function area includes multiple shooting modes, album controls, shooting controls, and front / rear camera switching controls. The multiple shooting modes include "Night Scene" mode, "Portrait" mode, "Large Aperture" mode, "Photo" mode, "Video" mode, "Pro" mode, and "More" modes. The preview interface displays the subject and a focal length setting area. The focal length setting area in interface 501 indicates that the current shooting focal length of the main camera is 1x. When the focus setting area receives an upward swipe from the user, as shown in interface 502, the electronic device increases the camera's focal length and determines whether the user's desired target focal length is still within the range supported by the main camera. For example, if the target focal length is 4x, the main camera's focal length range is 1x-3.5x, and the telephoto camera's focal length range is 3.5x to 30x, then the target focal length is not within the main camera's focal length range but is within the telephoto camera's focal length range. Therefore, the electronic device displays the interface shown in interface 503, which is the shooting preview interface corresponding to the telephoto camera, and the size of the subject in interface 503 increases. Alternatively, when the focus setting area receives a downward swipe from the user, the electronic device decreases the camera's focal length and determines whether the user's desired target focal length is still within the range supported by the main camera. If the user's desired target focal length is not within the range supported by the main camera, it triggers a switch from the main camera to another camera and displays the shooting preview interface corresponding to that other camera.

[0077] Optionally, after receiving an operation to switch from the first camera to the second camera, the electronic device can first determine the status of the second camera. If the second camera is active, it remains active; if the second camera is off, it is activated and updated from off to active. This is because the image signal processing in the electronic device can simultaneously receive image data from up to two cameras, thus supporting up to two cameras operating simultaneously. If the second camera was not active before receiving the operation, the electronic device activates it; if it was already active before receiving the operation, the electronic device does not need to activate it again. It should be noted that after receiving the operation but before the second camera successfully activates, the electronic device can continuously display the preview image captured by the first camera.

[0078] 402. After receiving the operation, the electronic device acquires M frames of the first image captured in real time by the first camera and M frames of the second image captured in real time by the second camera.

[0079] In this application, after receiving an operation, the electronic device can capture multiple frames of images in real time through a first camera and a second camera. The image captured in real time by the first camera is referred to as the first image, which is an initial image without image processing. Similarly, the image captured in real time by the second camera is referred to as the second image, which is also an initial image without image processing.

[0080] Assuming the first camera captures M frames of the first image and the second camera captures M frames of the second image, where M is a positive integer greater than 1, then the Nth frame of the first image in the M frames and the Nth frame of the second image in the M frames are captured simultaneously, where N is a positive integer greater than or equal to 1 and less than or equal to M.

[0081] After acquiring M frames of the first image and M frames of the second image, the electronic device can perform subsequent step 403 to obtain and display the M-frame fused image, and then perform subsequent step 404. In this way, the electronic device can first display the images fused from multiple first and second cameras, and then display the images captured by the second camera separately, so as to realize the camera switching transition.

[0082] 403. The electronic device performs image fusion based on the first image of M frames and the second image of M frames to generate an M-frame fused image.

[0083] In this embodiment, the value of M can be preset. When generating any frame of the M-frame fused image, the electronic device can generate it in real time. That is, after the electronic device acquires the first image of the Nth frame and the second image of the Nth frame in real time, it can perform image fusion based on the first image of the Nth frame and the second image of the Nth frame to obtain the Nth frame fused image; then, the electronic device can acquire the first image of the N+1th frame and the second image of the N+1th frame in real time, and perform image fusion based on the first image of the N+1th frame and the second image of the N+1th frame to obtain the N+1th frame fused image; the electronic device repeats the above operation until the M-frame fused image is obtained.

[0084] In one possible implementation, the target fused image is a frame in the M-frame fused image. The electronic device can obtain the target fused image based on the first display image corresponding to the target fused image, the second display image corresponding to the target fused image, and the weight pair corresponding to the target fused image.

[0085] Here, the first target image is a frame in the M-frame first images that corresponds to the target fused image, and the second target image is a frame in the M-frame second images that corresponds to the target fused image. For example, if the target fused image is the third frame fused image, then the first target image is the third frame first image, and the second target image is the third frame second image.

[0086] The first display image corresponding to the target fusion image is obtained by image processing the first target image, and the second display image corresponding to the target fusion image is obtained by image processing the second target image.

[0087] Specifically, after acquiring the target first image and the target second image, the electronic device can first perform initial cropping on the target first image and the target second image through the IFE module to make the size of the two images obtained by initial cropping adapt to the screen display size. Then, according to the frame number corresponding to the target fused image, the electronic device first performs image processing on the two images obtained by initial cropping through the SAT module and the IPE module to obtain the first display image and the second display image corresponding to the target fused image. Then, according to the frame number corresponding to the target fused image, the first display image and the second display image corresponding to the target fused image are fused through the image fusion module and the weight pair corresponding to the target fused image to obtain the target fused image.

[0088] The following section explains how to obtain the first display image and the second display image corresponding to the target fused image:

[0089] Assuming the target fused image is the Nth frame of the M-frame fused images, where N is a positive integer greater than or equal to and less than M, then the target first image is the Nth frame of the M-frame first images, and the target second image is the Nth frame of the M-frame second images.

[0090] First, the electronic device performs initial cropping on the first image and the second image of the Nth frame using the IFE module. Then, the SAT module calculates and determines the overall alignment parameter and the first alignment parameter corresponding to the fused image of the Nth frame using the two initially cropped images. Next, the electronic device uses the IPE module to offset the two initially cropped images based on the first alignment parameter corresponding to the fused image of the Nth frame, resulting in the first display image and the second display image corresponding to the fused image of the Nth frame.

[0091] The total alignment parameter is used to indicate the center point difference between the two images obtained from the initial cropping (equivalent to the center point difference between the first image in frame N and the second image in frame N), and the first alignment parameter corresponding to the fused image in frame N is used to indicate the parameters that the IPE module needs to align.

[0092] In traditional multi-camera switching transitions, in Zoomin scenarios, the electronic device can determine that the first alignment parameter is the same as the overall alignment parameter. Therefore, when the electronic device offsets the two initially cropped images through the IPE module, the center point of the first displayed image corresponding to the Nth frame fused image is perfectly aligned with the center point of the second displayed image corresponding to the Nth frame fused image. However, in Zoomout scenarios, the electronic device does not directly set the first alignment parameter to be the same as the overall alignment parameter. Instead, the electronic device gradually increases the value of the first alignment parameter as the value of N increases. Therefore, when the electronic device offsets the two initially cropped images through the IPE module, the center point of the first displayed image corresponding to the Nth frame fused image cannot be perfectly aligned with the center point of the second displayed image corresponding to the Nth frame fused image.

[0093] For example, if the first field of view corresponding to the first camera is greater than or equal to the second field of view corresponding to the second camera, then the multi-camera switching scenario is a Zoomin scenario. Assuming M is 5, and the difference between the center points of the first image and the second image is 10 pixels (i.e., the total alignment parameter is 10 pixels), then the first alignment parameter corresponding to the first frame fused image, the first alignment parameter corresponding to the second frame fused image, the first alignment parameter corresponding to the third frame fused image, the first alignment parameter corresponding to the fourth frame fused image, and the first alignment parameter corresponding to the fifth frame fused image are all 10 pixels. Therefore, through the IPE module, the electronic device can completely align the center point of the first display image corresponding to the first frame of the fused image with the center point of the second display image corresponding to the first frame of the fused image, completely align the center point of the first display image corresponding to the second frame of the fused image with the center point of the second display image corresponding to the second frame of the fused image, completely align the center point of the first display image corresponding to the third frame of the fused image with the center point of the second display image corresponding to the third frame of the fused image, completely align the center point of the first display image corresponding to the fourth frame of the fused image with the center point of the second display image corresponding to the fourth frame of the fused image, and completely align the center point of the first display image corresponding to the fifth frame of the fused image with the center point of the second display image corresponding to the fifth frame of the fused image.

[0094] For example, if the first field of view corresponding to the first camera is smaller than the second field of view corresponding to the second camera, then the multi-camera switching scenario is a zoom-out scenario. Assuming M is 5, and the difference between the center points of the first image and the second image is 10 pixels (i.e., the total alignment parameter is 10 pixels), then the first alignment parameter corresponding to the first frame fused image, the first alignment parameter corresponding to the second frame fused image, the first alignment parameter corresponding to the third frame fused image, the first alignment parameter corresponding to the fourth frame fused image, and the first alignment parameter corresponding to the fifth frame fused image can be 1 pixel, 2 pixels, 4 pixels, 6 pixels, and 8 pixels, respectively. Because the first alignment parameter is less than the total alignment parameter, the electronic device, through the IPE module, cannot completely align the center point of the first sent-to-display image corresponding to the first frame fused image to the center point of the second sent-to-display image corresponding to the first frame fused image, cannot completely align the center point of the first sent-to-display image corresponding to the second frame fused image to the center point of the second sent-to-display image corresponding to the second frame fused image, cannot completely align the center point of the first sent-to-display image corresponding to the third frame fused image to the center point of the second sent-to-display image corresponding to the third frame fused image, cannot completely align the center point of the first sent-to-display image corresponding to the fourth frame fused image to the center point of the second sent-to-display image corresponding to the fourth frame fused image, and cannot completely align the center point of the first sent-to-display image corresponding to the fifth frame fused image to the center point of the second sent-to-display image corresponding to the fifth frame fused image.

[0095] Optionally, if the first field of view corresponding to the first camera is greater than the second field of view corresponding to the second camera, the electronic device can also obtain the re-cropping information through the SAT module when offsetting based on the first alignment parameter, and then re-crop the two images obtained from the initial cropping through the IPE module according to the re-cropping information.

[0096] For example, if the first field of view is larger than the second field of view, the field of view of the Nth frame initial cropped image corresponding to the first camera (i.e., the image obtained by the IFE module performing initial cropping on the Nth frame first image) is larger than the field of view of the Nth frame initial cropped image corresponding to the second camera (i.e., the image obtained by the IFE module performing initial cropping on the Nth frame second image). Therefore, the re-cropping information of the Nth frame initial cropped image corresponding to the first camera can be obtained. Through the IPE module, an image with the same content as the Nth frame initial cropped image corresponding to the second camera is cropped from the Nth frame initial cropped image corresponding to the first camera. The cropped image is then enlarged, so that the scale of the first display image corresponding to the Nth frame fused image is the same as the scale of the second display image corresponding to the Nth frame fused image.

[0097] It should be noted that if the first field of view is smaller than the second field of view, the field of view of the Nth frame initial cropped image corresponding to the first camera (i.e., the image obtained by the IFE module from the Nth frame first image) is smaller than the field of view of the Nth frame initial cropped image corresponding to the second camera (i.e., the image obtained by the IFE module from the Nth frame second image). Therefore, the content of the Nth frame initial cropped image corresponding to the first camera is a part of the content of the Nth frame initial cropped image corresponding to the second camera, and it is impossible to crop the Nth frame initial cropped image corresponding to the first camera again to obtain an image with the same content as the Nth frame initial cropped image corresponding to the second camera. Therefore, when the first field of view is smaller than the second field of view, the electronic device does not need to obtain the re-cropping information through the SAT module, and the IPE module does not need to re-crop the two images obtained from the initial cropping based on the re-cropping information. If the first field of view is equal to the second field of view, the electronic device also does not need to perform re-cropping, that is, the electronic device does not need to obtain the re-cropping information through the SAT module, and the IPE module does not need to perform re-cropping.

[0098] After obtaining the first display image and the second display image corresponding to the target fused image based on the above operations, the target fused image can be obtained based on the first display image and the second display image corresponding to the target fused image.

[0099] The following describes how to obtain the target fused image based on the first display image corresponding to the target fused image and the second display image corresponding to the target fused image:

[0100] (1) In the first possible implementation, when the first field of view is greater than or equal to the second field of view, the electronic device acquires the weight pair corresponding to the target fused image, and performs weighted fusion on the first display image corresponding to the target fused image and the second display image corresponding to the target fused image based on the weight pair corresponding to the target fused image to obtain the target fused image.

[0101] Specifically, when the first field of view is greater than or equal to the second field of view, according to the operations of the SAT and IPE modules, the scale of the first displayed image corresponding to the target fused image is the same as the scale of the second displayed image corresponding to the target fused image, and the center point of the first displayed image corresponding to the target fused image is completely aligned with the center point of the second displayed image corresponding to the target fused image. Therefore, the electronic device can perform weighted fusion of the first displayed image corresponding to the target fused image and the second displayed image corresponding to the target fused image based on the weight pairs corresponding to the target fused image to obtain the target fused image.

[0102] Specifically, the weight pair corresponding to the target fused image includes a first weight corresponding to the target fused image and a second weight corresponding to the target fused image. The electronic device first weights the first display image corresponding to the target fused image based on the first weight to obtain a first weighted image, and then weights the second display image corresponding to the target fused image based on the first weight to obtain a second weighted image; then the first weighted image and the second weighted image are fused to obtain the target fused image.

[0103] (2) In the second possible implementation, when the first field of view corresponding to the first camera is smaller than the second field of view corresponding to the second camera, the electronic device downsamples the first display image corresponding to the target fused image to obtain a thumbnail image corresponding to the target fused image. The electronic device obtains the alignment parameters corresponding to the target fused image. Based on the alignment parameters, the thumbnail image is aligned to the center point of the second display image to obtain an aligned image. Based on the weight pair corresponding to the target fused image, the aligned image and the second display image are weighted and fused to obtain the target fused image.

[0104] Specifically, when the first field of view is smaller than the second field of view, according to the operations of the SAT and IPE modules, the scale of the first and second displayed images corresponding to the target fused image are different, and the center point of the second displayed image is not completely aligned with the center point of the first displayed image. Therefore, the electronic device can first perform scale alignment and center point alignment on the first and second displayed images corresponding to the target fused image, and then perform weighted fusion based on the scale-aligned and center-point-aligned images to obtain the target fused image.

[0105] Specifically, the electronic device first downsamples the first display image corresponding to the target fused image to obtain a thumbnail image, and makes the scale of the thumbnail image the same as the scale of the second display image corresponding to the target fused image, thereby achieving scale alignment.

[0106] Then, the electronic device determines the alignment parameters to be aligned for the target fused image based on the difference between the total alignment parameters and the completed alignment parameters corresponding to the target fused image (the completed alignment parameters are the first alignment parameters corresponding to the target fused image mentioned above). For example, if the total alignment parameters are 10 pixels and the first alignment parameters corresponding to the first frame fused image are 1 pixel, then the alignment parameters to be aligned for the first frame image are 9 parameters. According to the alignment parameters to be aligned for the target fused image, the electronic device aligns the center point of the thumbnail image to the center point of the second display image corresponding to the target fused image, thus obtaining an aligned image and achieving center point alignment.

[0107] Finally, the weight pair corresponding to the target fused image includes a first weight corresponding to the target fused image and a second weight corresponding to the target fused image. The electronic device can weight the aligned image based on the first weight corresponding to the target fused image to obtain a first weighted image, and weight the second display image corresponding to the target fused image based on the first weight corresponding to the target fused image to obtain a second weighted image; the first weighted image and the second weighted image are fused to obtain the target fused image.

[0108] Optionally, the electronic device may adjust the first weight and the second weight corresponding to the target fusion weight mentioned in (1) and (2) above according to the number of frames corresponding to the target fused image. For example, the first weight corresponding to the Nth frame fused image is greater than the first weight corresponding to the N+1th frame fused image, and the second weight corresponding to the Nth frame fused image is less than the second weight corresponding to the N+1th frame fused image.

[0109] For example, the first weight can gradually decrease from 1 to 0 as the frame rate increases, and the second weight can gradually decrease from 0 to 1 as the frame rate increases. Since the first weight is used to weight the aligned image or the first displayed image (both originating from the first camera), and the second weight is used to weight the second displayed image (originating from the second camera), as the first weight gradually decreases and the second weight gradually increases with the frame rate of the target fused image, the proportion of the image captured by the first camera in the target fused image gradually decreases, while the proportion of the image captured by the second camera gradually increases.

[0110] 404. After displaying the M-frame fused image, the electronic device displays the image based on the image captured by the second camera.

[0111] When displaying the M-frame fused image, the electronic device can display it in real time. That is, the electronic device displays the N-frame fused image after obtaining the N-frame fused image, and then displays the N+1-frame fused image after obtaining the N+1-frame fused image. The electronic device repeats this operation until the M-frame fused image is completely displayed. After that, the electronic device can display the image captured by the second camera alone.

[0112] Optionally, if the first weight gradually decreases with the number of frames and the second weight gradually increases with the number of frames, then when the electronic device displays M frames of fused images in sequence, the user can visually perceive that the image captured by the first camera gradually fades and the image captured by the second camera gradually becomes clearer.

[0113] based on Figure 4 In the described embodiment, when switching between multiple cameras, the electronic device can first display a multi-frame fused image from the first camera and the second camera, and then display the image from the second camera separately. This method can make the overall image visually change slowly when switching, achieving a smooth transition of the image.

[0114] The following is combined with Figure 3 The software architecture diagram, taking the Nth frame as an example, provides a detailed introduction to the software interactions for obtaining the fused image of the Nth frame. Figure 6 The software interaction described here only introduces the interaction process of each module in the hardware abstraction layer; the interaction with other layers is not described in detail.

[0115] Step 601: The IFE module acquires the first image of the Nth frame and the second image of the Nth frame.

[0116] The IFE module can acquire the first image (Nth frame) captured by the first camera and the second image (Nth frame) captured by the second camera.

[0117] Step 602: The IFE module performs initial cropping on the first image of the Nth frame and the second image of the Nth frame to obtain the first cropped image of the Nth frame and the second cropped image of the Nth frame.

[0118] The IFE module in the electronic device supports simultaneous processing of two data streams. Therefore, the IFE module can perform initial cropping of the Nth frame first image and the Nth frame second image simultaneously, resulting in the Nth frame first cropped image and the Nth frame second cropped image, respectively. The initial cropping is used to adapt the sizes of the resulting Nth frame first cropped image and Nth frame second cropped image to the size of the screen display preview.

[0119] Step 603: The IFE module sends the first cropped image of frame N and the second cropped image of frame N to the SAT module.

[0120] Step 604: The SAT module determines the total alignment parameters and the first alignment parameters corresponding to the Nth frame fused image.

[0121] The SAT module first determines the center point coordinates of the first cropped image in frame N and the center point coordinates of the second cropped image in frame N, and then determines the overall alignment parameter based on the difference between these two center point coordinates. Next, the SAT module determines the first alignment parameter corresponding to the fused image in frame N based on the first field of view corresponding to the first camera and the second field of view corresponding to the second camera.

[0122] Optionally, when the first field of view is greater than or equal to the second field of view, the SAT module determines that the first alignment parameter corresponding to the Nth frame fused image is equal to the total alignment parameter.

[0123] Optionally, when the first field of view is smaller than the second field of view, the SAT module determines the first alignment parameter corresponding to the Nth frame of the fused image based on the number of frames corresponding to the Nth frame of the fused image. For example, the SAT module can determine the first alignment parameter corresponding to the Nth frame of the fused image from the total alignment parameters based on the proportion of the number of frames corresponding to the Nth frame of the fused image to the total number of frames in the fused image. For example, if the total number of frames in the fused image is M, then a value smaller than the total alignment parameter can be determined from the total alignment parameters based on the result of N / M. Optionally, as the value of N increases, the value of the first alignment parameter corresponding to the Nth frame of the fused image can gradually increase.

[0124] Optionally, when the first field of view is greater than the second field of view, the SAT module also needs to determine the re-cropping information corresponding to the first cropped image of the Nth frame. The re-cropping information is used to crop the first cropped image of the Nth frame again.

[0125] Step 605: The SAT module sends the first cropped image of the Nth frame, the second cropped image of the Nth frame, the total alignment parameter, and the first alignment parameter corresponding to the fused image of the Nth frame to the IPE module.

[0126] Optionally, if the SAT module also determines the re-cropping information corresponding to the first cropped image of the Nth frame, the SAT module also sends the re-cropping information corresponding to the first cropped image of the Nth frame to the IPE module.

[0127] Step 606: The IPE module offsets the center points of the first cropped image and the second cropped image of the Nth frame based on the first alignment parameter corresponding to the Nth frame fused image, to obtain the first display image and the second display image corresponding to the Nth frame fused image.

[0128] Specifically, when the IPE module offsets the first cropped image of frame N and the second cropped image of frame N based on the first alignment parameter corresponding to the fused image of frame N, it can operate on the image with the larger field of view between the first cropped image of frame N and the second cropped image of frame N. For example, if the first field of view is smaller than the second field of view, the center point of the second cropped image of frame N is offset, while the center point of the first cropped image of frame N remains unchanged. If the first field of view is greater than or equal to the second field of view, the center point of the first cropped image of frame N is offset, while the center point of the second cropped image of frame N remains unchanged.

[0129] Optionally, the IPE module may further crop the first cropped image of the Nth frame based on the re-cropping information corresponding to the first cropped image of the Nth frame.

[0130] Step 607: The IPE module sends the first display image corresponding to the Nth frame fused image, the second display image corresponding to the Nth frame fused image, the total alignment parameter, and the first alignment parameter corresponding to the Nth frame fused image to the image fusion module.

[0131] Step 608: The image fusion module determines the alignment parameters corresponding to the Nth frame of the fused image based on the first alignment parameters and the total alignment parameters corresponding to the Nth frame of the fused image.

[0132] The image fusion module can determine the difference between the total alignment parameter and the first alignment parameter corresponding to the Nth frame fused image as the alignment parameter to be aligned for the Nth frame fused image.

[0133] Step 609-1: When the alignment parameter is 0, the image fusion module performs weighted fusion on the first display image corresponding to the Nth frame fused image and the second display image corresponding to the Nth frame fused image to obtain the Nth frame fused image.

[0134] Specifically, when the first field of view is greater than or equal to the second field of view, the first alignment parameter corresponding to the Nth frame fused image is equal to the total alignment parameter. Therefore, the alignment parameter to be aligned for the Nth frame fused image is 0. Simultaneously, the scale of the first display image corresponding to the Nth frame fused image is the same as the scale of the second display image corresponding to the Nth frame fused image. Therefore, the image fusion module directly performs weighted fusion to obtain the Nth frame fused image.

[0135] Step 609-2: When the alignment parameter is not 0, the image fusion module first performs scale alignment and center point alignment on the first display image corresponding to the Nth frame fused image and the second display image corresponding to the Nth frame fused image, and then performs weighted fusion to obtain the Nth frame fused image.

[0136] Specifically, when the first field of view is smaller than the second field of view, the first alignment parameter corresponding to the Nth frame fused image is smaller than the total alignment parameter, therefore the alignment parameter corresponding to the Nth frame fused image is not 0. Furthermore, the scale of the first display image corresponding to the Nth frame fused image is different from the scale of the second display image corresponding to the Nth frame fused image. Therefore, the image fusion module first performs scale alignment and center point alignment, and then performs weighted fusion to obtain the Nth frame fused image. For the specific implementation of step 609-2, please refer to... Figure 7 The process is shown below.

[0137] During the weighted fusion in steps 609-1 and 609-2, the image fusion module can obtain the weight pair corresponding to the Nth frame fused image. The weight pair corresponding to the Nth frame fused image includes the first weight and the second weight corresponding to the Nth frame fused image. Then, weighted fusion is performed based on the first weight and the second weight to obtain the Nth frame fused image.

[0138] Optionally, the image fusion module can determine the first weight and the second weight corresponding to the Nth frame of the fused image based on the proportion of the number of frames corresponding to the Nth frame of the fused image to the total number of frames of the fused image. The first weight decreases as the proportion increases, and the second weight increases as the proportion increases.

[0139] For example, if the number of frames corresponding to the Nth frame of the fused image is N, and the total number of frames in the fused image is M, then the first weight and the second weight corresponding to the Nth frame of the fused image can be determined based on the result of N / M. For instance, the value of N / M can be directly determined as the second weight, and the value of 1-N / M can be determined as the first weight. If M is 5, then the first weight corresponding to the first frame of the fused image is 0.8, and the second weight is 0.2; if the first weight is 0.6, the second weight is 0.4. Alternatively, the first weight and the second weight corresponding to different values ​​of N can be pre-stored, so that the image fusion module can directly read the first weight and the second weight corresponding to the Nth frame of the fused image based on the value of N.

[0140] Figure 7 The interaction flow between the IPE module and the image fusion module is shown when the first field of view is smaller than the second field of view:

[0141] like Figure 7 As shown, when the first field of view corresponding to the first camera is smaller than the second field of view corresponding to the second camera, the field of view of the first cropped image in frame N is smaller than the field of view of the second cropped image in frame N, and the object in the first cropped image in frame N is larger than the object in the second cropped image in frame N. Therefore, when the IPE module shifts the center point of the first cropped image in frame N and the second cropped image in frame N, the center point of the first cropped image in frame N can remain unchanged to obtain the first display image corresponding to the fused image in frame N. Then, the center point of the second cropped image in frame N is shifted using the first alignment parameter corresponding to the fused image in frame N to obtain the second display image corresponding to the fused image in frame N. Figure 7 As shown, since the scale of the first display image corresponding to the Nth frame fused image is different from the scale of the second display image corresponding to the Nth frame fused image, the size of the object in the first display image corresponding to the Nth frame fused image is different from the size of the object in the second display image corresponding to the Nth frame fused image. Therefore, the image fusion module downsamples the first display image corresponding to the Nth frame fused image to obtain a thumbnail image corresponding to the Nth frame fused image. The scale of the thumbnail image corresponding to the Nth frame fused image is equal to the scale of the second display image corresponding to the Nth frame fused image, indicating that the size of the object in the thumbnail image is the same as the size of the object in the second display image. Finally, the image fusion module obtains the alignment parameters and weight pairs corresponding to the Nth frame fused image, performs center point alignment on the first display image and the second display image corresponding to the Nth frame fused image, and then performs weighted fusion to obtain the Nth frame fused image. It is understandable that when aligning based on the alignment parameters corresponding to the Nth frame fused image, the center point of the second display image corresponding to the Nth frame fused image can be kept unchanged, and the center point of the thumbnail image corresponding to the Nth frame fused image can be shifted in the opposite direction.

[0142] This application also provides an electronic device that may include one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, which includes computer instructions. When the one or more processors execute the computer instructions, the electronic device performs the various functions or steps described in the method embodiments above.

[0143] This application also provides an image processing apparatus, which includes functions / units for performing the electronic devices described above.

[0144] This application also provides a chip system, such as... Figure 8 As shown, the chip system 800 includes at least one processor 801 and at least one interface circuit 802. The processor 801 and the interface circuit 802 are interconnected via lines. For example, the interface circuit 802 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 802 can be used to send signals to other devices (e.g., the processor 801). Exemplarily, the interface circuit 802 can read instructions stored in the memory and send those instructions to the processor 801. When the instructions are executed by the processor 801, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application embodiment does not specifically limit this.

[0145] This embodiment also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the various functions or steps performed by the mobile phone in the above method embodiment.

[0146] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps performed by the mobile phone in the above method embodiment.

[0147] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory. The memory is used to store computer execution instructions. When the apparatus is running, the processor can execute the computer execution instructions stored in the memory to cause the chip to perform the various functions or steps performed by the mobile phone in the above method embodiments.

[0148] In this embodiment, the electronic device, communication system, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0151] The unit described as a separate component may or may not be physically separate. The component shown as a unit can be one physical unit or multiple physical units, that is, it can be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of the solution in this embodiment according to actual needs.

[0152] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0153] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. An image processing method, characterized in that, The method includes: During the operation of the first camera, an operation to switch from the first camera to the second camera is received; After receiving the operation, the first camera captures M frames of first images in real time and the second camera captures M frames of second images in real time. The Nth frame of first images and the Nth frame of second images are captured simultaneously. M is a positive integer greater than 1 and N is a positive integer greater than or equal to 1 and less than or equal to M. When the first field of view corresponding to the first camera is greater than or equal to the second field of view corresponding to the second camera, the weight pair corresponding to the target fused image is obtained; based on the weight pair corresponding to the target fused image, the first display image corresponding to the target fused image and the second display image corresponding to the target fused image are weighted and fused to obtain the target fused image; the scale of the first display image is the same as the scale of the second display image, and the center point of the first display image is aligned with the center point of the second display image; When the first field of view corresponding to the first camera is smaller than the second field of view corresponding to the second camera, the first display image corresponding to the target fused image is downsampled to obtain a thumbnail image corresponding to the target fused image. The scale of the thumbnail image is the same as the scale of the second display image corresponding to the target fused image. The alignment parameters corresponding to the target fused image are obtained. Based on the alignment parameters, the thumbnail image is aligned to the center point of the second display image corresponding to the target fused image to obtain an aligned image. Based on the weight pair corresponding to the target fused image, the aligned image and the second display image are weighted and fused to obtain the target fused image. The first display image corresponding to the target fused image is obtained by cropping the target first image and shifting the center point once. The second display image corresponding to the target fused image is obtained by cropping the target second image and shifting the center point once. The target fused image is a frame in the M-frame fused image. The target first image is a frame in the M-frame first image that corresponds to the target fused image. The target second image is a frame in the M-frame second image that corresponds to the target fused image. After displaying the M-frame fused image, the image captured by the second camera is displayed.

2. The method according to claim 1, characterized in that, The weight pair corresponding to the target fused image includes a first weight corresponding to the target fused image and a second weight corresponding to the target fused image; The step of weightedly fusing the first display image corresponding to the target fused image and the second display image corresponding to the target fused image based on the weight pair corresponding to the target fused image to obtain the target fused image includes: Based on the first weight corresponding to the target fused image, the first display image corresponding to the target fused image is weighted to obtain the first weighted image; The second display image corresponding to the target fused image is weighted based on the second weight corresponding to the target fused image to obtain the second weighted image; The first weighted image and the second weighted image are fused together to obtain the target fused image.

3. The method according to claim 1, characterized in that, The weight pair corresponding to the target fused image includes a first weight corresponding to the target fused image and a second weight corresponding to the target fused image; The step of weightedly fusing the aligned image and the second displayed image based on the weight pairs corresponding to the target fused image to obtain the target fused image includes: The aligned image is weighted based on the first weight corresponding to the target fused image to obtain a first weighted image; The second display image corresponding to the target fused image is weighted based on the second weight corresponding to the target fused image to obtain the second weighted image; The first weighted image and the second weighted image are fused together to obtain the target fused image.

4. The method according to any one of claims 1-3, characterized in that, The step of obtaining the alignment parameters corresponding to the target fused image includes: Obtain the total alignment parameter and the completed alignment parameter corresponding to the target fused image; the total alignment parameter is used to indicate the center point difference between the target first image and the target second image; the completed alignment parameter is used to indicate the parameter of completed center point offset when the first display image and the second display image are obtained, and the value of the completed alignment parameter is less than the value of the total alignment parameter; Based on the difference between the total alignment parameter and the completed alignment parameter, the alignment parameters to be aligned for the target fused image are determined.

5. The method according to claim 2 or 3, characterized in that, The first weight corresponding to the Nth frame fused image is greater than the first weight corresponding to the N+1th frame fused image, and the second weight corresponding to the Nth frame fused image is less than the second weight corresponding to the N+1th frame fused image, where N is a positive integer greater than or equal to 1 and less than M.

6. An electronic device, characterized in that, include: One or more processors, one or more memories; wherein the one or more memories are coupled to the one or more processors, the one or more memories being used to store computer program code, the computer program code including computer instructions, which, when the one or more processors execute the computer instructions, cause the electronic device to perform the method as described in any one of claims 1-5.

7. A chip system applied to electronic devices, characterized in that, The chip system includes at least one processor and an interface for receiving computer instructions and transmitting them to the at least one processor; the at least one processor executes the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-5.

Images