Image processing method and electronic equipment

Abstract

The invention discloses an image processing method and electronic equipment, and the method comprises the steps: receiving an operation of switching from a first camera to a second camera in a working process of the first camera; m frames of first images collected by the first camera in real time and M frames of second images collected by the second camera in real time after the operation is received are obtained, the Nth frame of first image and the Nth frame of second image are collected at the same time, M is a positive integer larger than 1, and N is a positive integer larger than or equal to 1 and smaller than or equal to M; performing image fusion based on the M frames of first images and the M frames of second images to generate M frames of fused images, the Nth frame of fused image being obtained by fusing the Nth frame of first image and the Nth frame of second image; and after the M frames of fused images are displayed, performing display based on the images acquired by the second camera. By implementing the method and the device, the displayed whole picture can visually present a slow change effect during multi-camera switching, so that the smooth transition of the picture is realized.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of terminals, and in particular, to an image processing method and an electronic device. Background Art

[0002] Currently, multiple cameras can be installed on the same terminal, and different focal length ranges provided by the multiple cameras are used to meet the diverse shooting needs of users. For example, the focal length range provided by a wide-angle camera is from 0.4x to 1.0x, and the focal length range provided by a main camera is from 1.0x to 3.5x. If a user adjusts the shooting focal length from 0.9x to 1.2x, the terminal will switch from displaying the image captured by the wide-angle camera to displaying the image captured by the main camera. Since the shooting parameters and installation positions of these multiple cameras are different, there are deviations in the field of view angles obtained by these multiple cameras when shooting the same object. Therefore, when the terminal switches from displaying the image captured by the previous camera to displaying the image captured by the next camera, a jump phenomenon will occur visually.

[0003] Generally, an overall offset can be performed on the overall image captured by the previous camera through a transformation matrix to align the overall image captured by the next camera as much as possible. However, since the transformation matrix is obtained based on the same plane in the overall images captured by the two cameras, when the overall images captured by the two cameras include multiple planes, performing an overall offset based on the transformation matrix can align the plane corresponding to the transformation matrix in the overall image, but the remaining planes cannot be aligned. When the non-aligned planes are switched and displayed, there is still a jump phenomenon visually. Therefore, how to achieve a smooth transition on the overall image during multi-camera switching has become one of the problems to be solved urgently. Summary of the Invention

[0004] The present application provides an image processing method and an electronic device, which achieve a smooth transition on the overall image during multi-camera switching.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, which includes:

[0006] During the operation of a first camera, an operation to switch from the first camera to a second camera is received; M frames of first images captured in real time by the first camera and M frames of second images captured in real time by the second camera after receiving the operation are obtained. The Nth frame of the first images and the Nth frame of the second images are captured simultaneously. M is a positive integer greater than 1, and N is a positive integer greater than or equal to 1 and less than or equal to M; based on the M frames of first images and the M frames of second images, image fusion is performed to generate M frames of fused images. The Nth frame of the fused images is obtained by fusing the Nth frame of the first images and the Nth frame of the second images; after displaying the M frames of fused images, the image captured by the second camera is displayed.

[0007] When implementing the method described in the first aspect, during the process of switching the display from the image based on the first camera to the image based on the second camera, multiple frames of fused images of the first camera and the second camera can be displayed first, and then the image of the second camera can be displayed alone. This method can make the overall image present a slow-changing effect visually during the switching process, achieving a smooth transition of the image.

[0008] In a possible implementation manner, the above-mentioned image fusion based on M frames of the first image and M frames of the second image to generate M frames of fused images includes: obtaining a target fused image based on the first transmitted image corresponding to the target fused image, the second transmitted image corresponding to the target fused image, and the weight pair corresponding to the target fused image; the first transmitted image corresponding to the target fused image is obtained by performing image processing on the target first image, the second transmitted image corresponding to the target fused image is obtained by performing image processing on the target second image, the target fused image is one frame of the M frames of fused images, the target first image is one frame of the M frames of the first images corresponding to the target fused image, and the target second image is one frame of the M frames of the second images corresponding to the target fused image.

[0009] It can be seen that by performing image fusion based on the first transmitted image, the second transmitted image, and the weight pair corresponding to the target fused image, the target fused image can be obtained.

[0010] In a possible implementation manner, the above-mentioned obtaining the target fused image based on the first transmitted image corresponding to the target fused image, the second transmitted image corresponding to the target fused image, and the weight pair corresponding to the target fused image includes: when the first field of view angle corresponding to the first camera is greater than or equal to the second field of view angle corresponding to the second camera, obtaining the weight pair corresponding to the target fused image; based on the weight pair corresponding to the target fused image, performing weighted fusion on the first transmitted image corresponding to the target fused image and the second transmitted image corresponding to the target fused image to obtain the target fused image; the scale of the first transmitted image is the same as the scale of the second transmitted image, and the center point of the first transmitted image is the same as the center point of the second transmitted image.

[0011] Since when the first field of view angle is greater than or equal to the second field of view angle (i.e., in the case of Zoomin), the scale of the first transmitted image is the same as the scale of the second transmitted image, and the center point of the first transmitted image is aligned with the center point of the second transmitted image (this is because when obtaining the first transmitted image, scale alignment can be performed by cropping, and center point alignment can be performed by center point offset), there is no need to perform scale alignment and center point alignment on the first transmitted image and the second transmitted image anymore, and weighted fusion can be directly performed to obtain the target fused image.

[0012] In a possible implementation, the weight pair corresponding to the target fusion image includes a first weight corresponding to the target fusion image and a second weight corresponding to the target fusion image; the above-mentioned weighted fusion of the first display image corresponding to the target fusion image and the second display image corresponding to the target fusion image based on the weight pair corresponding to the target fusion image to obtain the target fusion image includes: weighting the first display image corresponding to the target fusion image based on the first weight corresponding to the target fusion image to obtain a first weighted image; weighting the second display image corresponding to the target fusion image based on the second weight corresponding to the target fusion image to obtain a second weighted image; fusing the first weighted image and the second weighted image to obtain the target fusion image.

[0013] In a possible implementation, the above-mentioned obtaining of the target fusion image based on the first display image corresponding to the target fusion image, the second display image corresponding to the target fusion image, and the weight pair corresponding to the target fusion image includes: when the first field of view angle corresponding to the first camera is less than the second field of view angle corresponding to the second camera, downsampling the first display image corresponding to the target fusion image to obtain a thumbnail image corresponding to the target fusion image, and the scale of the thumbnail image is the same as the scale of the second display image corresponding to the target fusion image; obtaining the alignment parameter to be aligned corresponding to the target fusion image; based on the alignment parameter to be aligned, aligning the thumbnail image to the center point of the second display image corresponding to the target fusion image to obtain an aligned image; based on the weight pair corresponding to the target fusion image, performing weighted fusion on the aligned image and the second display image to obtain the target fusion image.

[0014] Since when the first field of view angle is less than the second field of view angle (i.e., in the case of Zoom out), the scales of the first display image and the second display image are not the same (this is because when obtaining the first display image, the first display image and the second display image cannot be aligned in scale by cropping), and the center point of the second display image is not completely aligned to the center point of the first display image (this is because when obtaining the second display image, partial alignment is performed through the center point offset), the first display image can be downsampled first. By downsampling, the first display image and the second display image are aligned in scale first, and then the center point alignment and weighted fusion are performed, so as to obtain the target fusion image.

[0015] In a possible implementation, the weight pair corresponding to the target fusion image includes a first weight corresponding to the target fusion image and a second weight corresponding to the target fusion image; the above-mentioned weighted fusion of the aligned image and the second display image based on the weight pair corresponding to the target fusion image to obtain the target fusion image includes: weighting the aligned image based on the first weight corresponding to the target fusion image to obtain a first weighted image; weighting the second display image corresponding to the target fusion image based on the second weight corresponding to the target fusion image to obtain a second weighted image; fusing the first weighted image and the second weighted image to obtain the target fusion image.

[0016] In a possible implementation, the above-mentioned acquisition of the parameters to be aligned corresponding to the target fused image includes: acquiring the total alignment parameter and the completed alignment parameter corresponding to the target fused image; the total alignment parameter is used to indicate the center point difference between the target first image and the target second image; the completed alignment parameter is used to indicate the parameter of the completed center point offset when the first display image and the second display image are obtained, and the value of the completed alignment parameter is less than the value of the total alignment parameter; based on the difference between the total alignment parameter and the completed alignment parameter, determine the parameters to be aligned corresponding to the target fused image.

[0017] It can be seen that the parameters to be aligned can reflect the difference between the center point of the first image sent for display and the center point of the second image sent for display. Therefore, based on the parameters to be aligned, the center points of the two images to be fused can be completely aligned before the images are fused.

[0018] In a possible implementation, the first weight corresponding to the Nth frame fusion image is greater than the first weight corresponding to the N+1th frame fusion image, and the second weight corresponding to the Nth frame fusion image is less than the second weight corresponding to the N+1th frame fusion image, and the number of multiple frame fusion images is M, M is a positive integer greater than 1, and N is a positive integer greater than or equal to 1 and less than M.

[0019] It can be seen that as the number of frames in the M-frame fused image increases, the proportion of the picture captured by the first camera in the fused image gradually decreases, and the proportion of the picture captured by the second camera in the frame fused image gradually increases, which can achieve a reasonable transition of the displayed image.

[0020] In a second aspect, an embodiment of the present application provides an electronic device, the electronic device comprising one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, the one or more memories are used to store computer program codes, the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the image processing method in the first aspect and any possible implementation thereof.

[0021] In a third aspect, an embodiment of the present application provides an image processing apparatus, which includes functions / units for performing the image processing method in the first aspect and any possible implementation manner thereof described above.

[0022] In a fourth aspect, an embodiment of the present application provides a chip system. The chip system is applied to an electronic device and includes at least one processor and an interface. The interface is configured to receive computer instructions and transmit them to at least one processor; the at least one processor runs the computer instructions to cause the electronic device to perform the image processing method in the first aspect and any possible implementation manner thereof described above.

[0023] In a fifth aspect, the present application provides a computer-readable storage medium in which computer instructions are stored. When the computer instructions run on an electronic device, the electronic device is caused to perform the image processing method in the first aspect and any possible implementation manner thereof described above.

[0024] In a sixth aspect, the present application provides a computer program product. When the computer program product runs on a computer, the computer is caused to perform the image processing method in the first aspect and any possible implementation manner thereof described above.

[0025] It can be understood that the beneficial effects that can be achieved by the above-provided electronic device, image processing apparatus, chip system, computer-readable storage medium, and computer program product can refer to the beneficial effects in the first aspect and any possible implementation manner thereof, which will not be elaborated here. Description of the Drawings

[0026] Figure 1 is a schematic diagram of a preview interface before and after switching the camera provided by an embodiment of the present application;

[0027] Figure 2 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application;

[0028] Figure 3 is a schematic diagram of the software structure of an electronic device provided by an embodiment of the present application;

[0029] Figure 4 is a schematic diagram of the flowchart of an image processing method provided by an embodiment of the present application;

[0030] Figure 5 is a schematic diagram of the interface of a camera APP provided by an embodiment of the present application;

[0031] Figure 6 is a schematic diagram of the software interaction of an image processing method provided by an embodiment of the present application;

[0032] Figure 7It is an interaction schematic diagram of an IPE module and an image fusion module provided by an embodiment of the present application;

[0033] Figure 8 It is a structural schematic diagram of a chip system provided by an embodiment of the present application. Detailed implementation manners

[0034] Next, the technical solutions in the embodiments of the present application will be clearly and elaborately described with reference to the accompanying drawings. Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B may represent A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality" means two or more than two.

[0035] Hereinafter, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as implying or indicating relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality" is two or more.

[0036] The term "user interface (UI)" in the following embodiments of the present application is a media interface for interaction and information exchange between an application program or an operating system and a user, and it realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is source code written in a specific computer language such as Java or Extensible Markup Language (XML). The interface source code is parsed and rendered on an electronic device and finally presented as content recognizable by the user. The common manifestation form of the user interface is a graphical user interface (GUI), which refers to a user interface related to computer operations displayed in a graphical manner. It can be visible interface elements such as time, date, text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets displayed on the display screen of an electronic device.

[0037] To facilitate the understanding of the solution provided by the embodiments of the present application, the following introduces the related terms involved in the embodiments of the present application:

[0038] I. Field of view angle

[0039] The field of view (FOV) refers to the angle formed by the two edges of the maximum range through which the object being photographed can pass through the lens with the lens of the camera as the vertex, that is, the field of view is the angular range within which the camera can receive images. The field of view of the camera is directly proportional to the field of view range of the camera. For example, when the field of view is larger, the field of view range of the camera is larger; when the field of view is smaller, the field of view range of the camera is smaller. Usually, when the focal length of the camera changes, the field of view of the camera changes. For example, as the focal length increases, the field of view of the camera shrinks, the number of objects that the camera can capture decreases, and the objects captured become larger in the picture; conversely, as the focal length decreases, the field of view of the camera increases, the number of objects that the camera can capture increases, and the objects captured become smaller in the picture.

[0040] II. Scale

[0041] The number of pixel points occupied by the object captured by the camera in the picture is used to indicate the scale size of the picture. When the scale of the picture is larger, the picture is more blurred; when the scale of the picture is smaller, the picture is more detailed. For example, when the focal length increases, the object captured becomes larger in the picture (equivalent to the object occupying more pixel points), then the image scale is smaller, and visually the picture becomes more detailed; conversely, when the focal length decreases, the object captured becomes smaller in the picture (equivalent to the object occupying fewer pixel points), then the image scale is larger, and visually the picture becomes more blurred.

[0042] III. Difference between center points

[0043] When an electronic device is equipped with multiple cameras, the installation positions of these multiple cameras are different, so the optical centers of these multiple cameras are different. Therefore, for the pictures captured by these multiple cameras respectively, the center points of the pictures are different. The difference between the center points of the pictures captured by multiple cameras is called the difference between center points.

[0044] IV. Multi-camera switching and multi-camera smoothing

[0045] During the process of the electronic device capturing and displaying a preview picture through the camera, if the electronic device switches from displaying the picture collected by the previous camera to displaying the picture collected by the next camera, this process is called multi-camera switching. In this application, multi-camera switching can be triggered by means such as the user adjusting the focal length. For example, the electronic device is provided with three cameras, namely a wide-angle camera, a main camera, and a telephoto camera. Among them, the focal length range of the wide-angle camera is 0.4x to 1.0x, the focal length range of the main camera is 1.0x to 3.5x, and the focal length range of the telephoto camera is 3.5x to 30x; when the user adjusts the focal length to switch from the focal length range corresponding to one camera to the focal length range corresponding to another camera, the electronic device will trigger multi-camera switching.

[0046] Since the shooting parameters (such as focal length, brightness, and color) of the previous camera are different from those of the next camera, and there is a difference in the center points between the center point of the picture of the previous camera and the center point of the picture of the next camera, when the electronic device switches between multiple cameras, the displayed picture will have a jumping phenomenon.

[0047] In order to eliminate the jumping phenomenon during multi-camera switching as much as possible, multi-camera smoothing is required: The traditional multi-camera smoothing method can be implemented by means of a transformation matrix (such as a warp matrix), etc. Among them, the transformation matrix is obtained by finding the relationship between the pictures before and after multi-camera switching. The transformation matrix can be used to perform an offset process on the picture before switching, so that the picture obtained by the offset process is aligned with the picture after switching as much as possible. Then, by first displaying the picture obtained by the offset process and then displaying the picture after switching, smooth multi-camera switching can be achieved. However, due to the homography of the transformation matrix, the picture obtained by the offset process based on the transformation matrix cannot be completely aligned with the picture after switching. For example, as Figure 1 shown, Figure 1 the left side in Figure 1 is the preview picture of the previous camera,

[0048] and the right side in

[0049] is the preview picture of the next camera. Both of these preview pictures include a person, the road behind the person, and the trees in the distance, and the person, the road, and the trees are not on the same plane in the picture. When performing multi-camera smoothing, the transformation matrix can be calculated first according to the relationship between the people in the two preview pictures, and then the transformation matrix is used to align the left preview picture with the right preview picture. Since the transformation matrix is obtained based on the relationship between the people in the two preview pictures, the people in the left preview picture can be successfully aligned with the people in the right preview picture according to the transformation matrix; however, since the person, the road, and the trees are not on the same plane, the road and trees in the left preview picture cannot be aligned with the road and trees in the right preview picture based on the transformation matrix. Furthermore, this multi-camera smoothing method cannot completely eliminate the visual jumping phenomenon and achieve a smooth transition of the overall picture. Figure 2The figure shows a schematic diagram of the hardware structure of the electronic device 200. It should be understood that the electronic device 200 may have more or fewer components than those shown in the figure, may combine two or more components, or may have a different component configuration. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0050] As Figure 2 shown, the electronic device 200 may include: a processor 110, a memory 120, a camera 193, a display screen 194, a mobile communication module 150, and a wireless communication module 160. Among them:

[0051] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or may be integrated in one or more processors.

[0052] Among them, the controller may be the nerve center and command center of the electronic device 200. The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0053] A memory 120 may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data just used or recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0054] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0055] The camera 193 is used to capture images or videos. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element, also known as an image sensor, may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal and transmits the converted electrical signal to the processor 110 so that the processor 110 can process the electrical signal. In the embodiments of the present application, the electronic device 200 may include multiple cameras 193, and the multiple cameras 193 are located at different positions.

[0056] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 200 may include one or more display screens 194.

[0057] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the terminal device can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0058] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc. applied to the terminal device. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by Antenna 1, and perform processing such as filtering and amplification on the received electromagnetic waves, and then transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves through Antenna 1 and radiate them out. In some embodiments, at least some functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 can be disposed in the same device.

[0059] The modulation and demodulation processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through the audio device, or displays images or videos. In some embodiments, the modulation and demodulation processor may be an independent device. In other embodiments, the modulation and demodulation processor may be independent of the processor 110 and be disposed in the same device as the mobile communication module 150 or other functional modules.

[0060] The wireless communication module 160 may provide wireless communication solutions applied to the terminal device, including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth (BT), BLE broadcast, global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 may also receive the signal to be transmitted from the processor 110, perform frequency modulation on it, amplify it, and convert it into electromagnetic waves through the antenna 2 and radiate it out.

[0061] In some embodiments, antenna 1 of the terminal device is coupled to the mobile communication module 150, and antenna 2 is coupled to the wireless communication module 160, enabling the terminal device to communicate with the network and other devices through wireless communication technologies. The wireless communication technologies may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Beidou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0062] The software structure of the electronic device 200 will be introduced below: As Figure 3 shown, Figure 3 This is the software architecture diagram of the electronic device 200 provided by the embodiment of the present application. Among them, the software structure adopts a layered architecture, which divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. As Figure 3 shown, taking the Android system as an example where the Android system runs on the AP, in some embodiments, the Android system is divided into four layers, from top to bottom, namely the application layer, the application framework layer (framework), the hardware abstraction layer (HAL), and the kernel layer (kernel). Among them:

[0063] ①. Application layer: The application layer may include a series of application packages. For example, Figure 3 as shown, the application packages include application programs such as the camera application. Exemplarily, the camera application may be the built-in camera APP of the system, the camera APP installed by the user himself / herself, or other third-party applications that can call the camera APP for shooting. In this application, the camera application can trigger multi-camera switching and multi-camera smoothing in response to the user's operation.

[0064] ②. Application framework layer: The application framework layer provides application developers with an application programming interface (API) framework for accessing core functions, as well as various services and management tools. The application framework layer includes some predefined functions and algorithms. For example, Figure 3 as shown, the application framework layer may include a camera service. After the camera application in the application layer obtains the user's operation, the camera application can call the camera service, so that the camera service can manage the states of multiple cameras (such as the start state / stop state of the cameras), sessions, and the collected data, etc., to achieve multi-camera switching and multi-camera smoothing. For example, during multi-camera switching, the camera service needs to obtain the relevant data collected by the previous camera and the next camera. Specifically, the camera service can obtain, through the session corresponding to the previous camera, multiple frames of image data collected in real time by the previous camera after receiving the user's operation. At the same time, the camera service needs to determine that the next camera is in a working state, and obtain, through the session corresponding to the next camera, multiple frames of image data collected in real time by the next camera after receiving the user's operation. Optionally, the application framework layer may further include a window manager, a content provider, a view system, a resource manager, a notification manager, etc. The camera application can call the content provider, the resource manager, the notification manager, the window manager, the view system, etc. to assist the camera service in management. The embodiments of this application do not make any restrictions on this.

[0065] ③. Hardware abstraction layer: The hardware abstraction layer is an interface layer located between the operating system kernel and the hardware circuit, and its purpose is to abstract the hardware. It hides the hardware interface details of a specific platform and can provide a virtual hardware platform for the operating system. For example, Figure 3As shown in the figure, the hardware abstraction layer includes image hardware abstraction, etc. The image hardware abstraction includes an image signal processing front end (IFE) module, a spatial alignment transform (SAT) module, an image signal processing post end (IPE) module, an image fusion module, and a sensor node. After receiving the image data collected by different cameras simultaneously, the hardware abstraction layer is processed sequentially through the IFE module, the SAT module, and the IPE module.

[0066] After receiving the image data collected by each camera (Sensor), the IFE module performs preliminary cropping on the initial image corresponding to the collected image data so that the size of the image obtained after preliminary cropping is applicable to the size of the screen display. Optionally, the IFE module also performs preprocessing such as color correction and demosaicing on the initial image during preliminary cropping.

[0067] The SAT module is used to receive the images processed by different cameras via the IFE module, and then the SAT module calculates the transformation information between different cameras based on these multiple images. The transformation information includes one or more of the center point alignment parameters and the cropping frame, and sends the transformation information to the IPE module. For example, taking the third frame as an example, the IFE module receives the third frame image data collected in real time by the previous camera, and after preliminary cropping, sends the cropped image to the SAT module. At the same time, the IFE module receives the third frame image data collected in real time by the next camera, and after preliminary cropping, sends the cropped image to the SAT module. The SAT module can calculate the transformation information based on the two cropped images corresponding to the third frame obtained.

[0068] The IPE module is used to offset the overall picture corresponding to the cropped images of different cameras according to the transformation information calculated by the SAT module (that is, center point alignment), and / or adjust the field of view angles corresponding to the cropped images of different cameras to the same size through the cropping frame (that is, scale alignment). Optionally, the IPE module can also perform noise reduction, detail enhancement, color processing, etc. on the cropped images of different cameras.

[0069] The image fusion module is used to receive the images obtained after being processed by the IPE module and perform image fusion on the received images. Optionally, if the images obtained after being processed by the IPE module are still not spatially aligned (including scale alignment and center point alignment), the image fusion module needs to perform spatial alignment again before image fusion, and then perform image fusion. The sensor node is used to receive the instructions of the camera service in the application framework layer to control the opening or closing of the camera.

[0070] ④. The kernel layer is the layer between the hardware and the software. The kernel layer includes camera drivers, display drivers, etc. Each of these drivers is used to respond to instructions corresponding to the hardware abstractions in the hardware abstraction layer, so as to implement corresponding functions. For example, the camera driver can receive instructions from the sensor node in the hardware abstraction layer to turn on or off the camera; the display driver can receive instructions from the image fusion module in the hardware abstraction layer to display multi-frame fused images.

[0071] It can be understood that the above software architecture is only an example. In specific implementations, the electronic device 200 may further include more functional modules in the above layers, which will not be elaborated in this application.

[0072] The image processing method provided by the embodiments of the present application will be further described in detail below:

[0073] As Figure 4 shown, Figure 4 is a flowchart of an image processing method provided by an embodiment of the present application. Figure 4 The subject of execution of the method shown can be an electronic device, or the subject can be a chip in the electronic device. Figure 4 Taking the electronic device as the execution subject of the method as an example for illustration. The subject of execution of the methods shown in other drawings of the embodiments of the present application is the same by analogy, which will not be elaborated later. The software and hardware structure of the electronic device can refer to the structures shown in the above Figure 2 and Figure 3 shown. Figure 4 The image processing method shown includes steps 401 to 404. Among them:

[0074] 401. During the operation of the first camera, the electronic device receives an operation to switch from the first camera to the second camera.

[0075] In the embodiments of the present application, the electronic device is provided with multiple cameras, and the shooting parameters, installation positions, etc. of these multiple cameras are different. The first camera is any one of the multiple cameras, and the second camera is any one of the multiple cameras other than the first camera. For example, if the electronic device is configured with a main camera, a wide-angle camera, and a telephoto camera, the first camera can be the main camera, and the second camera can be the wide-angle camera.

[0076] Among them, the operation of the first camera means that the electronic device captures images through the first camera and continuously displays the preview screen during the capture by the first camera. During the operation of the first camera, the electronic device can receive an operation from the user to switch from the first camera to the second camera. By way of example, when the focal length ranges supported by the first camera and the second camera are different, this operation can be an operation by the user to adjust from a first focal length to a second focal length, where the first focal length is a focal length within the focal length range supported by the first camera, and the second focal length is a focal length within the focal length range supported by the second camera. For example, Figure 5 FIG. Figure 5 shows a schematic diagram of the interface of a camera APP. Among them, interface 501 is the interface displayed by the electronic device when capturing images using the main camera. Interface 501 includes a preview interface and a shooting function area. The shooting function area includes various shooting modes, an album control, a shooting control, and a front / back camera switching control. The various shooting modes include "night scene" mode, "portrait" mode, "large aperture" mode, "photo" mode, "video recording" mode, "professional" mode, and "more" mode. Among them, in the preview interface, a shooting object and a focal length setting area are displayed. The focal length setting area in interface 501 indicates that the shooting focal length of the current main camera is 1x. When the focal length setting area receives an operation of the user swiping upward, as shown in interface 502, the electronic device increases the focal length of the camera and determines whether the target focal length desired by the user is still within the focal length range supported by the main camera. For example, if the target focal length is 4x, the focal length range of the main camera is 1x - 3.5x, and the focal length range of the telephoto camera is 3.5x to 30x, it means that the target focal length is not within the focal length range of the main camera but within the focal length range of the telephoto camera. Therefore, the electronic device displays the interface shown in interface 503. Interface 503 is the shooting preview interface corresponding to the telephoto camera, and the size of the shooting object in interface 503 becomes larger. It can be understood that when the focal length setting area receives an operation of the user swiping downward, the electronic device decreases the focal length of the camera and determines whether the target focal length desired by the user is still within the focal length range supported by the main camera. When the target focal length desired by the user is not within the focal length range supported by the main camera, it triggers a switch from the main camera to the other camera, and the shooting preview interface corresponding to the other camera is displayed.

[0077] Optionally, after the electronic device receives the operation of switching from the first camera to the second camera, it may first determine the state of the second camera. If the second camera is in the startup state, the startup state of the second camera is maintained. If the second camera is in the shutdown state, the second camera is started, and the state of the second camera is updated from the shutdown state to the startup state. This is because the image signal processing in the electronic device can receive at most the image data collected from two cameras simultaneously. Therefore, the electronic device can support at most two cameras to start simultaneously. If the second camera is not started before receiving the operation, the electronic device starts the second camera. If the second camera is already started before receiving the operation, the electronic device does not need to start the second camera again. It should be noted that before the second camera is successfully started after receiving the operation, the electronic device can continuously display the preview image captured by the first camera.

[0078] 402. After the electronic device receives the operation, it acquires M frames of first images captured by the first camera in real time and M frames of second images captured by the second camera in real time.

[0079] Among them, after the electronic device receives the operation, it can capture multiple frames of images in real time through the first camera and the second camera. In this application, the images captured by the first camera in real time are called first images, and the first images are initial images that have not undergone image processing. The images captured by the second camera in real time are called second images, and the second images are initial images that have not undergone image processing.

[0080] Assume that the first camera captures M frames of first images and the second camera captures M frames of second images, where M is a positive integer greater than 1. Then the Nth first image in the M frames of first images and the Nth second image in the M frames of second images are captured simultaneously, where N is a positive integer greater than or equal to 1 and less than or equal to M.

[0081] After the electronic device acquires the M frames of first images and the M frames of second images, it can execute the subsequent step 403 to obtain M frames of fused images and display them, and then execute the subsequent step 404. In this way, the electronic device can first display the images fused by the first camera and the second camera, and then display the images captured by the second camera alone to achieve the switching transition of the camera.

[0082] 403. The electronic device performs image fusion based on the M frames of first images and the M frames of second images to generate M frames of fused images.

[0083] In an embodiment of the present application, the value of M can be set in advance. When the electronic device generates any one of the M-frame fused images, it can be generated in real time. That is, after the electronic device obtains the Nth frame of the first image and the Nth frame of the second image in real time, it can perform image fusion based on the Nth frame of the first image and the Nth frame of the second image to obtain the Nth frame of the fused image; then, the electronic device can obtain the (N + 1)th frame of the first image and the (N + 1)th frame of the second image in real time, and perform image fusion based on the (N + 1)th frame of the first image and the (N + 1)th frame of the second image to obtain the (N + 1)th frame of the fused image; the electronic device repeats the above operations until the Mth frame of the fused image is obtained.

[0084] In a possible implementation manner, the target fused image is one of the M-frame fused images, and the electronic device can obtain the target fused image based on the first display image corresponding to the target fused image, the second display image corresponding to the target fused image, and the weight pair corresponding to the target fused image.

[0085] Wherein, the target first image is one of the M-frame first images corresponding to the target fused image, and the target second image is one of the M-frame second images corresponding to the target fused image. For example, if the target fused image is the third frame of the fused image, the target first image is the third frame of the first image, and the target second image is the third frame of the second image.

[0086] The first display image corresponding to the target fused image is obtained by performing image processing on the target first image, and the second display image corresponding to the target fused image is obtained by performing image processing on the target second image.

[0087] Specifically, after the electronic device obtains the target first image and the target second image, it can first perform initial cropping on the target first image and the target second image through the IFE module so that the sizes of the two images obtained by the initial cropping are adapted to the size of the screen display. Then, according to the frame number corresponding to the target fused image, the electronic device first performs image processing on the two images obtained by the initial cropping through the SAT module and the IPE module to obtain the first display image corresponding to the target fused image and the second display image corresponding to the target fused image. Then, according to the frame number corresponding to the target fused image, through the image fusion module and the weight pair corresponding to the target fused image, the first display image corresponding to the target fused image and the second display image corresponding to the target fused image are fused to obtain the target fused image.

[0088] Next, how to obtain the first display image corresponding to the target fused image and the second display image corresponding to the target fused image will be introduced first:

[0089] Assume that the target fused image is the Nth fused image among the M fused images, where N is a positive integer greater than or equal to 1 and less than M. Then the target first image is the Nth first image among the M first images, and the target second image is the Nth second image among the M second images.

[0090] First, the electronic device performs initial cropping on the Nth first image and the Nth second image respectively through the IFE module, and calculates the total alignment parameter and the first alignment parameter corresponding to the Nth fused image for the two images obtained by the initial cropping through the SAT module. Then, the electronic device uses the IPE module to offset the two images obtained by the initial cropping based on the first alignment parameter corresponding to the Nth fused image, to obtain the first display image corresponding to the Nth fused image and the second display image corresponding to the Nth fused image.

[0091] Among them, the total alignment parameter is used to indicate the difference in the center points of the two images obtained by the initial cropping (equivalent to the difference in the center points of the Nth first image and the Nth second image), and the first alignment parameter corresponding to the Nth fused image is used to indicate the parameter that the IPE module needs to align.

[0092] During the traditional multi-camera switching transition process, in the Zoomin scenario, the electronic device can determine that the first alignment parameter is the same as the total alignment parameter. Therefore, when the electronic device uses the IPE module to offset the two images obtained by the initial cropping, the center point of the first display image corresponding to the Nth fused image can be completely aligned to the center point of the second display image corresponding to the Nth fused image. However, in the Zoom out scenario, the electronic device does not directly set the first alignment parameter to be the same as the total alignment parameter. Instead, as the value of N increases, the electronic device gradually increases the value of the first alignment parameter. Therefore, when the electronic device uses the IPE module to offset the two images obtained by the initial cropping, the center point of the first display image corresponding to the Nth fused image cannot be completely aligned to the center point of the second display image corresponding to the Nth fused image.

[0093] For example, if the first field of view corresponding to the first camera is greater than or equal to the second field of view corresponding to the second camera, it means that the multi-camera switching scene is a Zoomin scene. Assuming that the value of M is 5, and the center point difference between the first image and the second image is 10 pixels (that is, the total alignment parameter is 10 pixels), the first alignment parameter corresponding to the first frame fused image, the first alignment parameter corresponding to the second frame fused image, the first alignment parameter corresponding to the third frame fused image, the first alignment parameter corresponding to the fourth frame fused image, and the first alignment parameter corresponding to the fifth frame fused image are all 10 pixels. Therefore, through the IPE module, the electronic device can make the center point of the first display image corresponding to the first frame fusion image completely aligned with the center point of the second display image corresponding to the first frame fusion image, the center point of the first display image corresponding to the second frame fusion image completely aligned with the center point of the second display image corresponding to the second frame fusion image, the center point of the first display image corresponding to the third frame fusion image completely aligned with the center point of the second display image corresponding to the third frame fusion image, the center point of the first display image corresponding to the fourth frame fusion image completely aligned with the center point of the second display image corresponding to the fourth frame fusion image, and the center point of the first display image corresponding to the fifth frame fusion image completely aligned with the center point of the second display image corresponding to the fifth frame fusion image.

[0094] For another example, if the first field of view corresponding to the first camera is smaller than the second field of view corresponding to the second camera, it means that the multi-camera switching scene is a Zoom out scene. Assuming that the value of M is 5, and the center point difference between the first image and the second image is 10 pixels (that is, the total alignment parameter is 10 pixels), the first alignment parameter corresponding to the first frame fusion image, the first alignment parameter corresponding to the second frame fusion image, the first alignment parameter corresponding to the third frame fusion image, the first alignment parameter corresponding to the fourth frame fusion image, and the first alignment parameter corresponding to the fifth frame fusion image may be 1 pixel, 2 pixels, 4 pixels, 6 pixels, and 8 pixels, respectively. Since the first alignment parameter is smaller than the total alignment parameter, the electronic device cannot, through the IPE module, completely align the center point of the first display image corresponding to the first frame fusion image with the center point of the second display image corresponding to the first frame fusion image, cannot completely align the center point of the first display image corresponding to the second frame fusion image with the center point of the second display image corresponding to the second frame fusion image, cannot completely align the center point of the first display image corresponding to the third frame fusion image with the center point of the second display image corresponding to the third frame fusion image, cannot completely align the center point of the first display image corresponding to the fourth frame fusion image with the center point of the second display image corresponding to the fourth frame fusion image, and cannot completely align the center point of the first display image corresponding to the fifth frame fusion image with the center point of the second display image corresponding to the fifth frame fusion image.

[0095] Optionally, if the first field of view angle corresponding to the first camera is greater than the second field of view angle corresponding to the second camera, when the electronic device performs offset based on the first alignment parameter, the SAT module can also be used to obtain re-clipping information, and the IPE module can re-clip the two images obtained by the initial clipping according to the re-clipping information.

[0096] For example, if the first field of view angle is greater than the second field of view angle, the field of view range of the Nth frame of the initial clipped image corresponding to the first camera (i.e., the image obtained by the IFE module initially clipping the Nth frame of the first image) is greater than the field of view range of the Nth frame of the initial clipped image corresponding to the second camera (i.e., the image obtained by the IFE module initially clipping the Nth frame of the second image). Therefore, the re-clipping information of the Nth frame of the initial clipped image corresponding to the first camera can be obtained, and through the IPE module, an image with the same content as the Nth frame of the initial clipped image corresponding to the second camera can be clipped from the Nth frame of the initial clipped image corresponding to the first camera, and the clipped image can be enlarged, so that the scale of the first display image corresponding to the Nth frame of the fused image is the same as the scale of the second display image corresponding to the Nth frame of the fused image.

[0097] It should be noted that if the first field of view angle is less than the second field of view angle, the field of view range of the Nth frame of the initial clipped image corresponding to the first camera (i.e., the image obtained by the IFE module initially clipping the Nth frame of the first image) is less than the field of view range of the Nth frame of the initial clipped image corresponding to the second camera (i.e., the image obtained by the IFE module initially clipping the Nth frame of the second image). Therefore, the content of the Nth frame of the initial clipped image corresponding to the first camera is a part of the content of the Nth frame of the initial clipped image corresponding to the second camera, and it is impossible to re-clip the Nth frame of the initial clipped image corresponding to the first camera to obtain an image with the same content as the Nth frame of the initial clipped image corresponding to the second camera. Therefore, when the first field of view angle is less than the second field of view angle, the electronic device does not need to obtain re-clipping information through the SAT module, and the IPE module does not need to re-clip the two images obtained by the initial clipping according to the re-clipping information. If the first field of view angle is equal to the second field of view angle, the electronic device also does not need to perform re-clipping, that is, the electronic device does not need to obtain re-clipping information through the SAT module, and the IPE module does not need to perform re-clipping.

[0098] After obtaining the first display image corresponding to the target fused image and the second display image corresponding to the target fused image based on the above operations, the target fused image can be obtained based on the first display image corresponding to the target fused image and the second display image corresponding to the target fused image.

[0099] The following describes how to obtain the target fused image based on the first display image corresponding to the target fused image and the second display image corresponding to the target fused image:

[0100] (1) In the first possible implementation, when the first field of view angle is greater than or equal to the second field of view angle, the electronic device obtains the weight pair corresponding to the target fusion image, and based on the weight pair corresponding to the target fusion image, performs weighted fusion on the first display image corresponding to the target fusion image and the second display image corresponding to the target fusion image to obtain the target fusion image.

[0101] Among them, when the first field of view angle is greater than or equal to the second field of view angle, according to the operations of the above SAT module and IPE module, the scale of the first display image corresponding to the target fusion image is the same as the scale of the second display image corresponding to the target fusion image, and the center point of the first display image corresponding to the target fusion image is completely aligned with the center point of the second display image corresponding to the target fusion image. Therefore, the electronic device can perform weighted fusion on the first display image corresponding to the target fusion image and the second display image corresponding to the target fusion image based on the weight pair corresponding to the target fusion image to obtain the target fusion image.

[0102] Specifically, the weight pair corresponding to the target fusion image includes the first weight corresponding to the target fusion image and the second weight corresponding to the target fusion image. The electronic device first weights the first display image corresponding to the target fusion image based on the first weight corresponding to the target fusion image to obtain a first weighted image, and weights the second display image corresponding to the target fusion image based on the first weight corresponding to the target fusion image to obtain a second weighted image; then fuses the first weighted image and the second weighted image to obtain the target fusion image.

[0103] (2) In the second possible implementation, when the first field of view angle of the first camera is less than the second field of view angle of the second camera, the electronic device downsamples the first display image corresponding to the target fusion image to obtain a thumbnail image corresponding to the target fusion image. The electronic device obtains the alignment parameter to be aligned corresponding to the target fusion image, and based on the alignment parameter, aligns the thumbnail image to the center point of the second display image to obtain an aligned image; based on the weight pair corresponding to the target fusion image, performs weighted fusion on the aligned image and the second display image to obtain the target fusion image.

[0104] Among them, when the first field of view angle is smaller than the second field of view angle, according to the operations of the above SAT module and IPE module, the scale of the first display image corresponding to the target fusion image is different from the scale of the second display image corresponding to the target fusion image, and the center point of the second display image corresponding to the target fusion image is not completely aligned with the center point of the first display image corresponding to the target fusion image. Therefore, the electronic device can first perform scale alignment and center point alignment on the first display image corresponding to the target fusion image and the second display image corresponding to the target fusion image, and then perform weighted fusion based on the images after scale alignment and center point alignment to obtain the target fusion image.

[0105] Specifically, the electronic device first downsamples the first display image corresponding to the target fusion image to obtain a thumbnail image, so that the scale of the thumbnail image is the same as the scale of the second display image corresponding to the target fusion image, thereby achieving scale alignment.

[0106] Then, the electronic device determines the alignment parameter to be aligned corresponding to the target fusion image based on the difference between the total alignment parameter and the completed alignment parameter corresponding to the target fusion image (the completed alignment parameter is the first alignment parameter corresponding to the target fusion image in the above content); for example, if the total alignment parameter is 10 pixels and the first alignment parameter corresponding to the first frame of the fusion image is 1 pixel, then the alignment parameter to be aligned corresponding to the first frame of the image is 9 pixels. According to the alignment parameter to be aligned corresponding to the target fusion image, the electronic device aligns the center point of the thumbnail image with the center point of the second display image corresponding to the target fusion image to obtain an aligned image, so as to achieve center point alignment.

[0107] Finally, the weight pair corresponding to the target fusion image includes the first weight corresponding to the target fusion image and the second weight corresponding to the target fusion image. The electronic device can weight the aligned image based on the first weight corresponding to the target fusion image to obtain a first weighted image, and weight the second display image corresponding to the target fusion image based on the first weight corresponding to the target fusion image to obtain a second weighted image; fuse the first weighted image and the second weighted image to obtain the target fusion image.

[0108] Optionally, the electronic device can adjust the first weight and the second weight corresponding to the target fusion weight involved in the above (1) and (2) according to the number of frames corresponding to the target fusion image. For example, the first weight corresponding to the Nth frame of the fusion image is greater than the first weight corresponding to the N + 1th frame of the fusion image, and the second weight corresponding to the Nth frame of the fusion image is less than the second weight corresponding to the N + 1th frame of the fusion image.

[0109] For example, the first weight can gradually decrease from 1 to 0 as the number of frames increases, and the second weight can gradually decrease from 0 to 1 as the number of frames increases. Since the first weight is used to weight the aligned image or the first transmitted display image, both the aligned image and the first transmitted display image are derived from the images captured by the first camera, and the second weight is used to weight the second transmitted display image, which is derived from the images captured by the second camera. Therefore, when the first weight gradually decreases and the second weight gradually increases as the number of frames corresponding to the target fused image increases, the proportion of the images captured by the first camera in the target fused image gradually decreases, and the proportion of the images captured by the second camera in the target fused image gradually increases.

[0110] 404. After the electronic device displays M frames of fused images, it displays based on the images captured by the second camera.

[0111] Among them, when the electronic device displays M frames of fused images, it can be displayed in real time, that is, after the electronic device obtains the Nth frame of fused image, it displays the Nth frame of fused image, and then after the electronic device obtains the (N + 1)th frame of fused image, it displays the (N + 1)th frame of fused image; the electronic device repeats this operation until after M frames of fused images are displayed, the electronic device can display the images captured by the second camera alone.

[0112] Optionally, if the first weight gradually decreases as the number of frames increases and the second weight gradually increases as the number of frames increases, when the electronic device sequentially displays M frames of fused images, the user can visually feel that the images captured by the first camera gradually fade and the images captured by the second camera gradually become clearer.

[0113] Based on Figure 4 the described embodiments, when the electronic device switches between multiple cameras, it first displays multiple frames of fused images of the first camera and the second camera, and then separately displays the images captured by the second camera. This method can make the overall image present a slow-changing effect visually during the switch, achieving a smooth transition of the image.

[0114] Below, in combination with Figure 3 the software architecture diagram, taking the Nth frame as an example, the software interaction for obtaining the Nth frame of fused image will be introduced in detail. Figure 6 The software interaction in

[0115] only introduces the interaction process of each module in the hardware abstraction layer, and the interaction with the remaining layers will not be elaborated here.

[0116] Among them, the IFE module can obtain the Nth frame of the first image captured by the first camera and the Nth frame of the second image captured by the second camera.

[0117] Step 602: The IFE module performs initial cropping on the first image of the Nth frame and the second image of the Nth frame to obtain the first cropped image of the Nth frame and the second cropped image of the Nth frame.

[0118] Among them, the IFE module in the electronic device supports processing two-way data simultaneously. Therefore, the IFE module can perform initial cropping on the first image of the Nth frame and the second image of the Nth frame at the same time, and obtain the first cropped image of the Nth frame and the second cropped image of the Nth frame respectively. Initial cropping is used to make the sizes of the obtained first cropped image of the Nth frame and the second cropped image of the Nth frame adapt to the size of the screen display preview picture.

[0119] Step 603: The IFE module sends the first cropped image of the Nth frame and the second cropped image of the Nth frame to the SAT module.

[0120] Step 604: The SAT module determines the total alignment parameter and the first alignment parameter corresponding to the Nth frame fused image.

[0121] Among them, the SAT module can first determine the center point coordinates of the first cropped image of the Nth frame and the center point coordinates of the second cropped image of the Nth frame, and determine the total alignment parameter according to the difference between these two center point coordinates. Then, the SAT module determines the first alignment parameter corresponding to the Nth frame fused image according to the first field of view angle corresponding to the first camera and the second field of view angle corresponding to the second camera.

[0122] Optionally, when the first field of view angle is greater than or equal to the second field of view angle, the SAT module determines that the first alignment parameter corresponding to the Nth frame fused image is equal to the total alignment parameter.

[0123] Optionally, when the first field of view angle is less than the second field of view angle, the SAT module determines the first alignment parameter corresponding to the Nth frame fused image according to the number of frames corresponding to the Nth frame fused image. For example, the SAT module can determine the first alignment parameter corresponding to the Nth frame fused image from the total alignment parameter according to the ratio of the number of frames corresponding to the Nth frame fused image to the total number of frames of the fused image. For example, if the total number of frames of the fused image is M, then a value less than the total alignment parameter can be determined from the total alignment parameter according to the result of N / M. Optionally, as the value of N increases, the value of the first alignment parameter corresponding to the Nth frame fused image can gradually increase.

[0124] Optionally, when the first field of view angle is greater than the second field of view angle, the SAT module also needs to determine the re-cropping information corresponding to the first cropped image of the Nth frame, and the re-cropping information is used to perform re-cropping on the first cropped image of the Nth frame.

[0125] Step 605: The SAT module sends the first cropped image of the Nth frame, the second cropped image of the Nth frame, the total alignment parameter, and the first alignment parameter corresponding to the Nth frame fused image to the IPE module.

[0126] Optionally, if the SAT module also determines the re-clipping information corresponding to the first clipped image of the Nth frame, the SAT module also sends the re-clipping information corresponding to the first clipped image of the Nth frame to the IPE module.

[0127] Step 606: The IPE module offsets the center points of the first clipped image of the Nth frame and the second clipped image of the Nth frame based on the first alignment parameter corresponding to the Nth frame fused image, to obtain the first display image corresponding to the Nth frame fused image and the second display image corresponding to the Nth frame fused image.

[0128] Wherein, when the IPE module offsets the first clipped image of the Nth frame and the second clipped image of the Nth frame based on the first alignment parameter corresponding to the Nth frame fused image, it can operate on the image with a larger field of view angle in the first clipped image of the Nth frame and the second clipped image of the Nth frame. For example, if the first field of view angle is less than the second field of view angle, the center point of the second clipped image of the Nth frame is offset, and the center point of the first clipped image of the Nth frame remains unchanged; if the first field of view angle is greater than or equal to the second field of view angle, the center point of the first clipped image of the Nth frame is offset, and the center point of the second clipped image of the Nth frame remains unchanged.

[0129] Optionally, the IPE module also re-clips the first clipped image of the Nth frame based on the re-clipping information corresponding to the first clipped image of the Nth frame.

[0130] Step 607: The IPE module sends the first display image corresponding to the Nth frame fused image, the second display image corresponding to the Nth frame fused image, the total alignment parameter, and the first alignment parameter corresponding to the Nth frame fused image to the image fusion module.

[0131] Step 608: The image fusion module determines the alignment parameter to be aligned corresponding to the Nth frame fused image based on the first alignment parameter corresponding to the Nth frame fused image and the total alignment parameter.

[0132] Wherein, the image fusion module can determine the difference between the total alignment parameter and the first alignment parameter corresponding to the Nth frame fused image as the alignment parameter to be aligned corresponding to the Nth frame fused image.

[0133] Step 609-1: When the alignment parameter to be aligned is 0, the image fusion module performs weighted fusion on the first display image corresponding to the Nth frame fused image and the second display image corresponding to the Nth frame fused image, to obtain the Nth frame fused image.

[0134] Wherein, when the first field of view angle is greater than or equal to the second field of view angle, the first alignment parameter corresponding to the Nth frame of the fused image is equal to the total alignment parameter. Therefore, the alignment parameter to be aligned corresponding to the Nth frame of the fused image is 0. At the same time, the scale of the first display image corresponding to the Nth frame of the fused image is the same as the scale of the second display image corresponding to the Nth frame of the fused image. Therefore, the image fusion module directly performs weighted fusion to obtain the Nth frame of the fused image.

[0135] Step 609-2: When the alignment parameter to be aligned is not 0, the image fusion module first performs scale alignment and center point alignment on the first display image corresponding to the Nth frame of the fused image and the second display image corresponding to the Nth frame of the fused image, and then performs weighted fusion to obtain the Nth frame of the fused image.

[0136] Wherein, when the first field of view angle is less than the second field of view angle, the first alignment parameter corresponding to the Nth frame of the fused image is less than the total alignment parameter. Therefore, the alignment parameter to be aligned corresponding to the Nth frame of the fused image is not 0. Moreover, the scale of the first display image corresponding to the Nth frame of the fused image is not the same as the scale of the second display image corresponding to the Nth frame of the fused image. Therefore, the image fusion module first performs scale alignment and center point alignment, and then performs weighted fusion to obtain the Nth frame of the fused image. The specific implementation manner corresponding to Step 609-2 can refer to Figure 7 the process shown.

[0137] During the weighted fusion in Step 609-1 and Step 609-2, the image fusion module can obtain the weight pair corresponding to the Nth frame of the fused image. The weight pair corresponding to the Nth frame of the fused image includes the first weight and the second weight corresponding to the Nth frame of the fused image. Then, based on the first weight and the second weight, weighted fusion is performed to obtain the Nth frame of the fused image.

[0138] Optionally, the image fusion module can determine the first weight and the second weight corresponding to the Nth frame of the fused image according to the proportion of the number of frames corresponding to the Nth frame of the fused image in the total number of frames of the fused image. The first weight decreases as the proportion increases, and the second weight increases as the proportion increases.

[0139] For example, if the number of frames corresponding to the Nth frame of the fused image is N and the total number of frames of the fused image is M, then the first weight and the second weight corresponding to the Nth frame of the fused image can be determined according to the result of N / M. Exemplarily, the value of N / M can be directly determined as the second weight, and the value of 1 - N / M can be determined as the first weight. If M is 5, then the first weight corresponding to the first frame of the fused image is 0.8 and the second weight is 0.2; the first weight corresponding to the second frame of the fused image is 0.6 and the second weight is 0.4. Or, the first weight and the second weight corresponding to different values of N can be stored in advance, so that the image fusion module can directly read the first weight and the second weight corresponding to the Nth frame of the fused image according to the value of N.

[0140] Figure 7 Shows the interaction process between the IPE module and the image fusion module when the first field of view angle is smaller than the second field of view angle:

[0141] As Figure 7 shown, when the first field of view angle corresponding to the first camera is smaller than the second field of view angle corresponding to the second camera, the field of view range of the Nth frame of the first cropped image is smaller than that of the Nth frame of the second cropped image, and the photographed object in the Nth frame of the first cropped image is larger than the photographed object in the Nth frame of the second cropped image. Therefore, when the IPE module offsets the center points of the Nth frame of the first cropped image and the Nth frame of the second cropped image, the center point of the Nth frame of the first cropped image can be kept unchanged to obtain the first display image corresponding to the Nth frame of the fused image, and the center point of the Nth frame of the second cropped image is offset by using the first alignment parameter corresponding to the Nth frame of the fused image to obtain the second display image corresponding to the Nth frame of the fused image. As Figure 7 shown, since the scales of the first display image corresponding to the Nth frame of the fused image and the second display image corresponding to the Nth frame of the fused image are different, the size of the photographed object in the first display image corresponding to the Nth frame of the fused image is different from the size of the photographed object in the second display image corresponding to the Nth frame of the fused image. Therefore, the image fusion module downsamples the first display image corresponding to the Nth frame of the fused image to obtain the thumbnail image corresponding to the Nth frame of the fused image, and the scale of the thumbnail image corresponding to the Nth frame of the fused image is equal to the scale of the second display image corresponding to the Nth frame of the fused image, indicating that the size of the photographed object in the thumbnail image is the same as the size of the photographed object in the second display image. Finally, the image fusion module obtains the alignment parameter and weight pair corresponding to the Nth frame of the fused image, and performs weighted fusion after centering alignment on the first display image corresponding to the Nth frame of the fused image and the second display image corresponding to the Nth frame of the fused image to obtain the Nth frame of the fused image. It can be understood that when aligning based on the alignment parameter corresponding to the Nth frame of the fused image, the center point of the second display image corresponding to the Nth frame of the fused image can be kept unchanged, and the center point of the thumbnail image corresponding to the Nth frame of the fused image is offset in the reverse direction.

[0142] An embodiment of this application also provides an electronic device, which may include: one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions. When the one or more processors execute the computer instructions, the electronic device performs each function or step executed by the electronic device in the above method embodiment.

[0143] An embodiment of this application also provides an image processing device, which includes units for performing the functions in the electronic device in the above embodiment.

[0144] The embodiments of the present application further provide a chip system, as Figure 8 shown. The chip system 800 includes at least one processor 801 and at least one interface circuit 802. The processor 801 and the interface circuit 802 can be interconnected by a line. For example, the interface circuit 802 can be used to receive signals from other devices (such as the memory of an electronic device). Again, for example, the interface circuit 802 can be used to send signals to other devices (such as the processor 801). Exemplarily, the interface circuit 802 can read the instructions stored in the memory and send the instructions to the processor 801. When the instructions are executed by the processor 801, the electronic device can execute each step in the above embodiments. Of course, the chip system can also include other discrete devices, and the embodiments of the present application do not make specific limitations on this.

[0145] The embodiments of the present application further provide a computer-readable storage medium. Computer instructions are stored in the computer-readable storage medium. When the computer instructions run on an electronic device, the electronic device is enabled to execute each function or step that the mobile phone executes in the above method embodiments.

[0146] The embodiments of the present application further provide a computer program product. When the computer program product runs on a computer, the computer is enabled to execute each function or step that the mobile phone executes in the above method embodiments.

[0147] In addition, the embodiments of the present application further provide a device, which can specifically be a chip, a component or a module. The device can include a processor and a memory connected to each other. Wherein, the memory is used to store computer execution instructions. When the device runs, the processor can execute the computer execution instructions stored in the memory, so that the chip executes each function or step that the mobile phone executes in the above method embodiments.

[0148] Among them, the electronic device, communication system, computer-readable storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be elaborated here.

[0149] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and conciseness of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0150] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the module or unit is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.

[0151] The unit described as a separate component may or may not be physically separated. The component displayed as a unit may be a physical unit or multiple physical units, that is, it can be located in one place, or it can be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] In addition, in each embodiment of the present application, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods of the embodiments of the present application. The aforementioned storage medium includes: USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disc and other various media that can store program codes.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solution of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solution of the present application.

Claims

1. An image processing method, characterized in that, The method includes: During the operation of the first camera, an operation to switch from the first camera to the second camera is received; After receiving the operation, M first images captured in real time by the first camera and M second images captured in real time by the second camera are obtained. The Nth first image and the Nth second image are captured simultaneously, M is a positive integer greater than 1, and N is a positive integer greater than or equal to 1 and less than or equal to M; Based on the M first images and the M second images, image fusion is performed to generate M fused images. The Nth fused image is obtained by fusing the Nth first image and the Nth second image; After displaying the M fused images, display is performed based on the images captured by the second camera.

2. The method according to claim 1, characterized in that, The performing image fusion based on the M first images and the M second images to generate M fused images includes: Based on the first display image corresponding to the target fused image, the second display image corresponding to the target fused image, and the weight pair corresponding to the target fused image, the target fused image is obtained; The first display image corresponding to the target fused image is obtained by performing image processing on the target first image. The second display image corresponding to the target fused image is obtained by performing image processing on the target second image. The target fused image is one of the M fused images. The target first image is one of the M first images corresponding to the target fused image. The target second image is one of the M second images corresponding to the target fused image.

3. The method according to claim 2, wherein The obtaining the target fused image based on the first display image corresponding to the target fused image, the second display image corresponding to the target fused image, and the weight pair corresponding to the target fused image includes: When the first field of view angle corresponding to the first camera is greater than or equal to the second field of view angle corresponding to the second camera, the weight pair corresponding to the target fused image is obtained; Based on the weight pair corresponding to the target fused image, weighted fusion is performed on the first display image corresponding to the target fused image and the second display image corresponding to the target fused image to obtain the target fused image; the scale of the first display image is the same as the scale of the second display image, and the center point of the first display image is the same as the center point of the second display image.

4. The method according to claim 3, wherein The weight pair corresponding to the target fused image includes the first weight corresponding to the target fused image and the second weight corresponding to the target fused image; The performing weighted fusion on the first display image corresponding to the target fused image and the second display image corresponding to the target fused image based on the weight pair corresponding to the target fused image to obtain the target fused image includes: Weighting the first display image corresponding to the target fused image based on the first weight corresponding to the target fused image to obtain a first weighted image; Weighting the second display image corresponding to the target fused image based on the second weight corresponding to the target fused image to obtain a second weighted image; Fuse the first weighted image and the second weighted image to obtain the target fused image.

5. The method according to claim 2, characterized in that, Obtaining the target fused image based on the first display image corresponding to the target fused image, the second display image corresponding to the target fused image, and the weight pair corresponding to the target fused image includes: When the first field of view angle corresponding to the first camera is smaller than the second field of view angle corresponding to the second camera, downsample the first display image corresponding to the target fused image to obtain a thumbnail image corresponding to the target fused image, and the scale of the thumbnail image is the same as the scale of the second display image corresponding to the target fused image; Obtain the alignment parameter to be aligned corresponding to the target fused image; Based on the alignment parameter to be aligned, align the thumbnail image to the center point of the second display image corresponding to the target fused image to obtain an aligned image; Based on the weight pair corresponding to the target fused image, perform weighted fusion on the aligned image and the second display image to obtain the target fused image.

6. The method according to claim 5, characterized in that, The weight pair corresponding to the target fused image includes the first weight corresponding to the target fused image and the second weight corresponding to the target fused image; Performing weighted fusion on the aligned image and the second display image based on the weight pair corresponding to the target fused image to obtain the target fused image includes: Weight the aligned image based on the first weight corresponding to the target fused image to obtain a first weighted image; Weight the second display image corresponding to the target fused image based on the second weight corresponding to the target fused image to obtain a second weighted image; Fuse the first weighted image and the second weighted image to obtain the target fused image.

7. The method according to claim 5 or 6, characterized in that, Obtaining the alignment parameter to be aligned corresponding to the target fused image includes: Obtain the total alignment parameter and the alignment parameter that has been completed corresponding to the target fused image; the total alignment parameter is used to indicate the difference in the center points of the target first image and the target second image; the alignment parameter that has been completed is used to indicate the parameter of the center point offset when obtaining the first display image and the second display image, and the value of the alignment parameter that has been completed is less than the value of the total alignment parameter; Based on the difference between the total alignment parameter and the alignment parameter that has been completed, determine the alignment parameter to be aligned corresponding to the target fused image.

8. The method according to claim 4 or 6, characterized in that, The first weight corresponding to the Nth frame of fused image is greater than the first weight corresponding to the (N + 1)th frame of fused image, and the second weight corresponding to the Nth frame of fused image is less than the second weight corresponding to the (N + 1)th frame of fused image, where N is a positive integer greater than or equal to 1 and less than M.

9. An electronic device, characterized in that, Including: One or more processors, one or more memories; wherein, the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the method according to any one of claims 1-8.

10. A chip system is applied to an electronic device, characterized in that, The chip system includes at least one processor and an interface, the interface is used to receive computer instructions and transmit them to the at least one processor; the at least one processor runs the computer instructions to cause the electronic device to execute the method described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, Computer instructions are stored in the computer-readable storage medium, and when the computer instructions run on an electronic device, the electronic device is caused to execute the method described in any one of claims 1-8.

Images