Dual-camera video fusion method and electronic device

Abstract

A dual-camera video fusion method and electronic device can achieve the integration of the advantages of different cameras during video recording, thereby improving video quality. The method includes: obtaining a first image captured by a first camera at a first moment and a second image captured by a second camera at a second moment; processing the first image and the second image to obtain a processed first image and a processed second image; fusing the processed first image with the processed second image to obtain a first fused image at the first moment; obtaining a third image captured by the first camera at a third moment; processing the third image to obtain a processed third image; obtaining a homography matrix between the first moment and the third moment; obtaining a fourth image based on the first fused image and the homography matrix; and fusing the processed third image with the fourth image to obtain a second fused image at the third moment.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and more specifically, to a method and electronic device for dual-camera video fusion. Background Art

[0002] As users' demands for imaging capabilities on mobile devices increase, current smartphones often have multiple cameras, including wide-angle, telephoto, and main cameras. Each of these lenses has different advantages in terms of dynamic range, color, and clarity. Usually, different cameras are enabled separately according to the focal lengths set by the user. Multi-camera fusion technology has been explored in academic research in recent years. It usually requires more complex algorithms to fuse two frames of images obtained by different cameras. However, due to the computing power of the mobile phone itself, the relevant algorithms can only be applied to scenes where images are captured, and cannot be applied to video recording. The current dual-view recording function only involves stitching the images of the two cameras together, and does not involve the function of effect fusion. Therefore, how to integrate the advantages of different cameras in the video recording process is of great significance. Summary of the Invention

[0003] The present application provides a method and electronic device for dual-camera video fusion, which can integrate the advantages of different cameras during video recording and improve the quality of the video.

[0004] In a first aspect, a method for dual-camera video fusion is provided, including: obtaining a first image captured by a first camera at a first moment and a second image captured by a second camera at a second moment; processing the first image and the second image to obtain a processed first image and a processed second image; fusing the processed first image with the processed second image to obtain a first fused image at the first moment; obtaining a third image captured by the first camera at a third moment; processing the third image to obtain a processed third image; obtaining a homography matrix between the first moment and the third moment; obtaining a fourth image based on the first fused image and the homography matrix; and fusing the processed third image with the fourth image to obtain a second fused image at the third moment.

[0005] The dual-camera video fusion method of the present application can process each frame of the image during the video shooting process, thereby obtaining a video that combines the advantages of the first camera and the second camera, thereby improving the quality of video shooting.

[0006] With reference to the first aspect, in a possible implementation, the first moment is the same as the second moment.

[0007] The first camera and the second camera can shoot at the same time when acquiring images, or can shoot successively within a preset time, which can be flexibly set according to specific circumstances.

[0008] In combination with the first aspect, in a possible implementation, processing the first image and the second image is processing the first image and the second image through an image signal processor, and processing the third image is processing the third image through an image signal processor.

[0009] In combination with the first aspect, in one possible implementation, processing the first image and the second image, and processing the third image, includes: the first image and the second image are in RAW format; converting the first image and the second image from RAW format to RGB format according to an image signal processor, and the third image is in RAW format; converting the third image from RAW format to RGB format according to the image signal processor.

[0010] The dual-camera video fusion method of the present application only requires the acquired real image to be processed by an image signal processor, while the generated image does not need to be processed by an image signal processor, which greatly reduces the device performance requirements and memory consumption of the image signal processor for processing images.

[0011] In combination with the first aspect, in a possible implementation, before fusing the processed first image and the processed second image, the method further includes: performing image registration on the processed first image and the processed second image.

[0012] In combination with the first aspect, in a possible implementation, fusing the processed first image and the processed second image includes: fusing the aligned regions in the processed first image and the processed second image.

[0013] In combination with the first aspect, in one possible implementation, the processed first image and the processed second image are fused, including: aligning the non-aligned areas in the processed first image and the processed second image according to a local registration method; and fusing the aligned areas in the processed first image and the processed second image.

[0014] The dual-camera video fusion method of the present application only requires image registration of the acquired real image, and the generated fourth image is aligned with the real image acquired by the first camera. Therefore, there is no need to perform image registration before the fourth image and the third image are fused, which reduces the image processing time and realizes the real-time image processing during the video shooting process.

[0015] In combination with the first aspect, in one possible implementation, obtaining a homography matrix between a first moment and a third moment includes: obtaining first data of an inertial measurement unit at the first moment; obtaining second data of the inertial measurement unit at the third moment; and obtaining a homography matrix based on the first data and the second data.

[0016] In combination with the first aspect, in a possible implementation, the first camera and the second camera have different clarity, dynamic range, or color.

[0017] The first camera and the second camera each have advantages in clarity, dynamic range or color. The dual-camera video fusion method of the present application can fuse the advantages of the first camera and the second camera to obtain a video with better clarity, dynamic range or color.

[0018] In combination with the first aspect, in a possible implementation method, the processed first image and the processed second image are fused, including: using an image fusion algorithm or an image fusion neural network to fuse the processed first image and the processed second image, and the image fusion neural network is a pre-trained neural network.

[0019] In a second aspect, an electronic device is provided, comprising: one or more processors; a memory, wherein the memory includes instructions; when the instructions are executed by the one or more processors, the electronic device performs the following steps: obtaining a first image captured by a first camera at a first moment and a second image captured by a second camera at a second moment; processing the first image and the second image to obtain a processed first image and a processed second image; fusing the processed first image and the processed second image to obtain a first fused image at the first moment; obtaining a third image captured by the first camera at a third moment; processing the third image to obtain a processed third image; obtaining a homography matrix between the first moment and the third moment; obtaining a fourth image based on the first fused image and the homography matrix; and fusing the processed third image and the fourth image to obtain a second fused image at the third moment.

[0020] In conjunction with the second aspect, in a possible implementation, the first moment is the same as the second moment.

[0021] In combination with the second aspect, in a possible implementation, processing the first image and the second image is processing the first image and the second image through an image signal processor, and processing the third image is processing the third image through an image signal processor.

[0022] In combination with the second aspect, in one possible implementation, the processing unit processes the first image and the second image, and processes the third image, including: the first image and the second image are in RAW format; according to the image signal processor, the first image and the second image are converted from RAW format to RGB format, and the third image is in RAW format; according to the image signal processor, the third image is converted from RAW format to RGB format.

[0023] In combination with the second aspect, in a possible implementation, before fusing the processed first image and the processed second image, the electronic device is further used to: perform image registration on the processed first image and the processed second image.

[0024] In combination with the second aspect, in a possible implementation, fusing the processed first image and the processed second image includes: fusing aligned regions in the processed first image and the processed second image.

[0025] In combination with the second aspect, in one possible implementation, the processed first image and the processed second image are fused, including: aligning the non-aligned areas in the processed first image and the processed second image according to a local registration method; and fusing the aligned areas in the processed first image and the processed second image.

[0026] In combination with the second aspect, in one possible implementation, obtaining a homography matrix between a first moment and a third moment includes: obtaining first data of an inertial measurement unit at the first moment; obtaining second data of the inertial measurement unit at the third moment; and obtaining a homography matrix based on the first data and the second data.

[0027] In conjunction with the second aspect, in a possible implementation, the first camera and the second camera have different clarity, dynamic range, or color.

[0028] In combination with the second aspect, in one possible implementation, the processed first image and the processed second image are fused, including: using an image fusion algorithm or an image fusion neural network to fuse the processed first image and the processed second image, where the image fusion neural network is a pre-trained neural network.

[0029] In a third aspect, a chip system is provided, comprising at least one processor, which, when program instructions are executed in the at least one processor, enables the at least one processor to execute the method in the above-mentioned first aspect and any possible implementation of the first aspect.

[0030] In a fourth aspect, a computer-readable storage medium is provided, which stores program code, wherein the program code includes instructions for executing part or all of the steps in the above-mentioned first aspect and any possible implementation of the first aspect.

[0031] In a fifth aspect, a computer program product is provided, which, when run on a computer, enables the computer to execute instructions of part or all of the steps in the above-mentioned first aspect and any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram illustrating the application of the homography matrix of this application;

[0033] Figure 2 is a schematic diagram of an electronic device of the present application;

[0034] Figure 3 is a schematic exploded diagram of a lens module of the present application;

[0035] Figure 4 is a schematic cross-sectional view of a lens module of the present application;

[0036] Figure 5 This is a schematic diagram illustrating the application of combining two cameras to improve clarity;

[0037] Figure 6 is a schematic diagram illustrating the application of combining two cameras to improve the dynamic range;

[0038] Figure 7 This is a schematic diagram illustrating the application of combining two cameras to improve brightness and clarity;

[0039] Figure 8 is a schematic block diagram of the dual-camera video fusion method of the present application;

[0040] Figure 9 This is a schematic block diagram of the specific operation of the dual-camera video fusion method of the present application;

[0041] Figure 10 is a schematic flow chart of the dual-camera video fusion method of the present application;

[0042] Figure 11 This is a schematic diagram of a possible composition of the electronic device of the present application. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0044] The electronic devices involved in the embodiments of the present application may include handheld devices, vehicle-mounted devices, wearable devices, computing devices, or other processing devices connected to a wireless modem. They may also include cellular phones, smart phones, personal digital assistants (PDAs), tablet computers, laptop computers, machine type communication (MTC) terminals, point of sales (POS), cameras, video recorders, cameras, smart watches, smart wristbands, vehicle-mounted computers, and other electronic devices with imaging capabilities.

[0045] In the following, the terms "first," "second," etc. are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the quantity of the technical features indicated. Therefore, a feature specified as "first," "second," etc. may explicitly or implicitly include one or more of the features.

[0046] In addition, in this application, directional terms such as "center", "up", "down", "inside", and "outside" are defined relative to the orientation or position of the components schematically placed in the drawings. It should be understood that these directional terms are relative concepts. They are used for relative description and clarification, rather than indicating or implying that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. They may change accordingly according to changes in the orientation of the components placed in the drawings, and therefore cannot be understood as limitations on this application.

[0047] It should also be noted that, in the embodiments of the present application, the same figure mark represents the same component or the same part. For the same parts in the embodiments of the present application, the figure may only mark one of the parts or components as an example. It should be understood that the figure mark also applies to other identical parts or components.

[0048] The following is a brief introduction to the concepts that may be involved in the embodiments of this application.

[0049] Homography matrix: Homography is a concept in projective geometry, also known as projective transformation. It maps a point (three-dimensional homogeneous vector) on a projective plane to another projective plane, and maps a straight line to a straight line, with the property of preserving the line. In general, homography is a linear transformation about three-dimensional homogeneous vectors, which can be represented by the homography matrix H (3×3 non-singular matrix). Figure 1As shown, according to the homography matrix between Figure (a) and Figure (b), Figure (a) can be transformed to the perspective of Figure (b) to obtain Figure (a) from the perspective of Figure (b), that is, Figure (c).

[0050] Image registration is the process of converting different images of the same scene into the same coordinate system. These images can be taken at different times (multi-temporal registration), with different sensors (multi-modal registration), or from different viewpoints. The spatial relationship between these images can be rigid (translation and rotation), radial (shear), homographic, or based on large deformable models.

[0051] Inertial measurement unit (IMU): An IMU is a device that measures an object's three-axis attitude angle (or angular rate) and acceleration. Typically, an IMU consists of three single-axis accelerometers and three single-axis gyroscopes. The accelerometers detect the object's acceleration signals along three independent axes in the carrier's coordinate system, while the gyroscopes detect the carrier's angular velocity signals relative to the navigation coordinate system. After processing these signals, the object's attitude can be calculated.

[0052] Figure 2 The following is a schematic diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may be a device with a video or photo function, and the embodiment of the present application does not impose any special restrictions on the specific form of the electronic device 100. For the convenience of explanation and understanding, the following description is based on the example of the electronic device 100 being a mobile phone. For example, Figure 2 (a) and (b) schematically illustrate the front and back sides of the electronic device 100 , respectively.

[0053] like Figure 2 As shown, the electronic device 100 may include a housing 101 , a display panel (DP) 102 , and a camera compact module (CCM) 103 .

[0054] The housing 101 is formed with a storage space for arranging the various components of the electronic device 100. The housing 101 also serves to protect the electronic device 100 and support the entire device. The display screen 102 and the lens module 103 are disposed in the storage space of the housing 101 and are connected to the housing 101. In some embodiments, the housing 101 includes a back cover and a middle frame, and the display screen 102 and the lens module 103 can be fixed to the middle frame. The material of the housing 101 can be metal, plastic, ceramic, or glass.

[0055] The display screen 102 can be a liquid crystal display (LCD) screen, an organic light emitting diode (OLED) display screen, etc., wherein the OLED display screen can be a flexible display screen or a rigid display screen. The display screen 102 can be an ordinary regular screen, or it can be an irregular screen, a folding screen, etc. For example, the display screen 102 can be relatively freely rotated or folded to form an arc, a sphere, a cylinder, etc. The display screen 102 can be set on the front and / or back of the electronic device 100. The front of the electronic device 100 can be understood as the side facing the user when the user uses the electronic device 100, and the back of the electronic device 100 can be understood as the side facing away from the user when the user uses the electronic device 100.

[0056] The lens module 103 is used to capture still images or videos. It should be understood that Figure 2 The electronic device in may include multiple lens modules, which may have the same function or different functions. Only one of the lens modules is taken as an example for introduction here. When the lens module 103 is set on the front of the electronic device 100, it can be used to shoot the scene on the front side of the electronic device 100. In some embodiments, it can be called a front camera. When the lens module 103 is set on the back of the electronic device 100, it can be used to shoot the scene on the back side of the electronic device 100. In some embodiments, it can be called a rear camera. When shooting, the user can select the corresponding lens module according to the shooting requirements. The lens module 103 can be used to shoot scenes at different distances, such as far, close or macro, and the embodiments of the present application are not particularly limited.

[0057] It should be understood that Figure 2The installation position of the middle lens module 103 is only schematic. When the lens module 103 is used as a front camera, it can be installed at any position on the front of the electronic device 100 except the display screen 102, such as the left side of the receiver, the upper middle of the electronic device 100, the lower part of the electronic device 100 (or chin), or the four corners of the electronic device 100. When the lens module 103 is used as a rear camera, it can be installed at any position on the back of the electronic device 100, such as the upper left corner or the upper right corner. In some other embodiments, the lens module 103 may not be set on the main body of the electronic device 100, but on an edge protruding relative to the main body of the electronic device 100, or on a component that is movable or rotatable relative to the electronic device 100, which can be extended or rotated from the main body of the electronic device 100. When the lens module 103 is rotatable relative to the electronic device 100, the lens module 103 functions as both a front-facing camera and a rear-facing camera. That is, by rotating the same lens module 103, it can capture both the scene on the front side and the scene on the rear side of the electronic device 100. In other embodiments, when the display screen 101 is foldable, the lens module 103 can function as either a front-facing camera or a rear-facing camera as the display screen 102 is folded.

[0058] The embodiment of the present application does not limit the number of lens modules 103 provided, and can be one, two, four, or even more. For example, one or more lens modules 103 can be provided on the front of the electronic device 100, and / or one or more lens modules 103 can be provided on the back of the electronic device 100. When multiple lens modules 103 are provided, the multiple lens modules 103 can be completely identical or different, for example, the multiple lens modules 103 have different lens optical parameters, different lens installation positions, different lens shapes, etc. The embodiment of the present application also does not impose any restrictions on the relative positions of the multiple lens modules when they are provided.

[0059] Optionally, the electronic device 100 may further include a protective lens 104 for protecting the lens module 103. The protective lens 104 is arranged on the housing 101 and covers the lens module 103. When the protective lens 104 is used to protect the front camera, the protective lens 104 may only cover the front lens module or cover the entire front of the electronic device 100. When the protective lens 104 covers the entire front of the electronic device 100, it can be used to protect the front lens module and the display screen 102 at the same time. The protective lens 104 is a cover glass (CG). When the protective lens 104 is used to protect the rear camera, the protective lens 104 may cover the entire back of the electronic device 100, or it may only be arranged at a position corresponding to the rear lens module. The material of the protective lens 104 may be glass, sapphire, ceramic, etc., and is not specifically limited in the embodiments of the present application. In some embodiments, the protective lens 104 is transparent, and light outside the electronic device 100 can enter the lens module 103 through the protective lens 104.

[0060] It should be understood that Figure 2 The structure illustrated in the figure does not constitute a specific limitation on the electronic device 100. The electronic device 100 may include more or fewer components than shown in the figure. For example, the electronic device 100 may also include one or more components such as a battery, a flash, a fingerprint recognition module, an earpiece, buttons, and a sensor. The electronic device 100 may also be set with a component arrangement different from that shown in the figure.

[0061] Figure 3 A schematic exploded view of a lens module provided in an embodiment of the present application is shown. Figure 4 Shown Figure 3 The lens module 200 may be a schematic cross-sectional view of the lens module. Figure 2 An exemplary structure of the lens module 103 in FIG. Figure 3 and Figure 4 The structure of the lens module 200 is briefly introduced.

[0062] For the convenience of description, the optical axis direction of the lens module 200 is defined as the Z direction, the side in the direction of the subject in the optical axis direction is the front side, and the side opposite to the subject is the rear side. The first direction perpendicular to the optical axis is the X direction, and the second direction perpendicular to the optical axis and the first direction is the Y direction. The direction close to the optical axis in the X and Y directions is the inner side, and the direction opposite to the optical axis is the outer side. Similarly, the definitions of the X, Y, Z directions and the front, back, inner and outer sides also apply to the various drawings to be described later. It should be noted that the above definitions of the X, Y, Z directions and the front, back, inner and outer sides are only for the convenience of describing the positional relationship and connection relationship between the various components in the embodiments of the present application, and should not be understood as limitations on the embodiments of the present application.

[0063] As shown in the figure, the lens module 200 may include a housing 210 , a lens assembly 220 , an auto focus (AF) assembly 230 , an optical image stabilization (OIS) assembly 240 and an image sensor assembly 250 .

[0064] The lens assembly 220 mainly includes an optical lens 211 and a lens barrel 212. The lens assembly 220 is used to image the scene on the object side onto the imaging surface on the image side. The optical lens 211 may include at least one lens, which may be different or the same. The at least one lens may include a solid lens and / or a liquid lens. The solid lens may be an optical element with a surface that is part of a spherical surface made of a transparent material such as plastic or glass, and has fixed lens parameters. The liquid lens is an optical element made of one or more liquids without mechanical connections, and its lens parameters can be dynamically adjusted by external control. In the embodiment of the present application, the number of lenses included in the optical lens 211 is not specifically limited. Those skilled in the art can set the number of lenses accordingly according to actual needs, such as 1, 2, 3, 5, 8 or more. The combination of solid lenses and / or liquid lenses can also be set according to actual needs, which will not be described in detail here.

[0065] The focal length of the optical lens 211 can be fixed, in which case the lens assembly 220 is a fixed-focus lens. Alternatively, the focal length of the optical lens 211 can be adjusted, in which case the lens assembly 220 is a zoom lens. For example, the focal length of the optical lens 211 can be adjusted by adjusting the relative positions of the lenses of the optical lens 211, adjusting the refractive index of the liquid lens, or changing the surface shape (curvature) of the liquid lens.

[0066] The lens barrel 212 is formed with a receiving space, which is mainly used to accommodate the optical lens 211. The lens barrel 212 can be a whole body, and the optical lens 211 is accommodated in the lens barrel 212 of the whole body, but the relative positions between the lenses of the optical lens 211 can be adjusted by other structures. The lens barrel 212 can also include multiple lens barrel parts, and the lenses of the optical lens 211 are grouped in the multiple lens barrel parts. The relative positions between the multiple lens barrel parts can be adjusted, thereby achieving the adjustment of the relative positions between the lenses. Therefore, it should be understood that Figure 3 and Figure 4 The structure of the lens barrel 212 and the connection method between the optical lens 211 and the lens barrel 212 are merely exemplary and do not impose any limitation on the embodiments of the present application.

[0067] The AF component 230 is used to realize automatic focusing. Figure 4As shown, AF assembly 230 is connected to lens barrel 212 in lens assembly 220. During autofocus, AF assembly 230 can push lens barrel 212 up and down along the optical axis, thereby changing the distance between the optical center of optical lens 211 and the imaging plane (i.e., changing the image distance) to obtain a clear image. It should be understood that the figure only schematically illustrates the location of AF assembly 230 and does not limit the specific structure of AF assembly 230 in any way.

[0068] The OIS component 240 is used to implement optical image stabilization. Figure 4 As shown, OIS assembly 240 is connected to lens barrel 212 in lens assembly 220. During optical image stabilization, OIS assembly 240 can drive lens barrel 212 to move perpendicular to the optical axis, thereby shifting the focus of optical lens 211 off the optical axis to obtain a clear image. It should be understood that the figure only schematically illustrates the location of SMA assembly 240 and does not limit the specific structure of SMA assembly 240 in any way.

[0069] In the embodiment of the present application, the AF component 230 can be referred to as an AF motor, and the OIS component can be referred to as an OIS motor. In some embodiments, the lens module 200 can include an AF motor and / or an OIS motor, that is, the lens module 200 can achieve autofocus and / or optical image stabilization. In some embodiments, the AF motor and the SMA motor can also be integrated into one in the lens module 200, so that autofocus and optical image stabilization can be achieved simultaneously by one motor. The embodiment of the present application is only described by taking the lens module 200 including two components, the AF motor and the OIS motor, as an example, but it should be understood that the embodiment of the present application is not limited to this.

[0070] The sensor assembly 250 is arranged on the rear side of the lens assembly 210 and is mainly used for imaging. Specifically, the sensor assembly 250 may include a filter (such as an infrared cutoff filter or a filter that filters out other light bands), a sensor, a circuit board, etc. Among them, the infrared cutoff filter can eliminate unnecessary light projected onto the sensor, preventing the sensor from having problems such as ghosting, stray light, and color cast when imaging. The sensor is a semiconductor chip with hundreds of thousands to millions of photodiodes on its surface. When exposed to light, it generates an electric charge, which is converted into a digital signal by an analog-to-digital converter chip. The sensor can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The circuit board can be a flexible printed circuit (FPC) or a printed circuit board (PCB) for transmitting electrical signals. Among them, the FPC can be a single-sided flexible board, a double-sided flexible board, a multi-layer flexible board, a rigid-flexible board, or a flexible circuit board with a hybrid structure.

[0071] Housing 210 defines an interior cavity for housing lens assembly 220, AF assembly 230, OIS assembly 240, and the like. Housing 210 also provides protection and support. The structure of housing 210 shown in the figure is merely illustrative and does not limit the present invention. Persons skilled in the art may design the shape of housing 210 based on practical needs.

[0072] The lens module 200 may further include a connector, peripheral electronic components and other components (not shown in the figure), which will not be described in detail here.

[0073] Different cameras (wide-angle, telephoto, main camera, etc.) have different advantages in terms of dynamic range, color, and clarity. It is of great significance to integrate the advantages of different cameras in a single imaging.

[0074] An existing imaging method that fuses the ordinary visible light band RGB and the near-infrared light band NIR uses a camera with the infrared filter removed to obtain a fused image of RGB and NIR, and then uses another camera without removing the infrared filter to obtain an RGB image. By fusing these two images, better night scene imaging effects can be achieved.

[0075] An existing multi-camera fusion algorithm framework uses a unified convolutional neural network (CNN) to achieve end-to-end frame-by-frame fusion of different cameras. This includes fusing images from black and white cameras to colorize black and white images; fusing images with different exposures to create high-dynamic range images; and fusing SLR and stereo cameras to produce higher-definition stereo images. In general, each frame captured by the two cameras is fused to achieve optimal imaging results.

[0076] In the above-mentioned existing technologies, most image signal processors (ISPs) for camera recording only support high frame rates for a single camera, such as dual-view recording with a maximum resolution of 1080p and a frame rate of 30 fps. Therefore, if the ISP processes both high-frame-rate and high-resolution images from the dual camera, such as 4K and 30 fps, it will place an excessive burden on the mobile device. In addition, existing image fusion methods require two real image frames as input for each fusion, which places extremely high demands on the power consumption and memory of the mobile device. Even the most advanced mobile processors cannot meet the real-time and power consumption constraints.

[0077] Therefore, an embodiment of the present application provides a method for recording videos with multiple cameras, which integrates the advantages of different cameras through the cooperation of multiple cameras, thereby improving the quality of the recorded video.

[0078] Figure 5 A schematic block diagram of an embodiment of the present application showing how to improve clarity by combining two cameras is shown in FIG. Figure 5 As shown, camera 2 generally has a higher dynamic range and better color, but lacks clarity. Combining camera 2 with the higher-resolution camera 1 can improve clarity while preserving the high dynamic range and better color. For example, a mobile phone's built-in camera lens has a higher dynamic range than a DSLR lens, while a DSLR lens has higher clarity. Combining a mobile phone's built-in camera lens with a DSLR lens can improve clarity while preserving the high dynamic range. For another example, a color lens has better color than a black and white lens, while a black and white lens has higher clarity. Combining a color lens with a black and white lens can improve clarity while preserving better color.

[0079] Figure 6 A schematic block diagram showing an embodiment of the present application of improving the dynamic range by combining two cameras is shown in FIG. Figure 6 As shown, camera 2 (main lens) has higher clarity but insufficient dynamic range. Combining camera 2 with camera 1 (wide-angle lens) with better dynamic range can improve the dynamic range while retaining high clarity, and can record richer picture details.

[0080] Figure 7 FIG2 shows a schematic block diagram of an embodiment of the present application for improving brightness and clarity by combining two cameras, as shown in FIG2 . Figure 7 As shown, camera 2 (telephoto lens) can capture distant scenes and focus on local areas, but the amount of light entering is low, and the shooting effect is poor in dark conditions. Combining camera 2 with camera 1 (main lens) can improve brightness and clarity in dark conditions.

[0081] Therefore, an embodiment of the present application provides a method for dual-camera video fusion, which can achieve the fusion of the advantages of two cameras during the video recording process, thereby improving the quality of the recorded video.

[0082] Figure 8 A schematic block diagram of a dual-camera video fusion method according to an embodiment of the present application is shown, which can improve the quality of recorded videos. Figure 8 The method includes the fusion of the main camera and the auxiliary camera, where the main camera can be Figure 4 、 5 、Camera 2 in 6, the auxiliary camera can be Figure 4 、 5 , Camera 1 in 6, the main camera and auxiliary camera can be Figure 2 、 3 The camera module in Figure 8 The method can be applied to Figure 2 Electronic devices shown. Figure 8 The method in

[15] combines the main camera and the auxiliary camera, which can retain the advantages of the main camera while integrating the advantages of the auxiliary camera, thereby obtaining better video recording effects.

[0083] like Figure 8As shown, during the video capture process, a frame of image A1 captured by the main camera and a frame of image B1 captured by the auxiliary camera are first acquired. The main and auxiliary cameras can capture the images simultaneously or sequentially within a preset time interval. For example, the preset time interval can be 0.1 milliseconds, with the auxiliary camera capturing the first frame of image B1 0.1 millisecond after the main camera captures the first frame of image A1; or it can be 0.1 millisecond after the auxiliary camera captures the first frame of image B1. The specific time interval can be flexibly set based on actual conditions. Both images A1 and B1 can be in RAW format, which is the raw data obtained by converting the captured light source signal by the image sensor into a digital signal. Simultaneously, at the moment the main camera captures image A1, position data I1 from the inertial measurement unit in the electronic device is acquired. The position data includes the electronic device's current angular velocity and linear acceleration, totaling six degrees of freedom. Specifically, the accelerometer in the inertial measurement unit measures acceleration along three axes, and the gyroscope measures angular velocity around three axes.

[0084] Both image A1 and image B1 are processed by an image signal processor, and the RAW format is converted into RGB format using an existing ISP demosaicing algorithm, thereby obtaining converted RGB format image A1 and converted RGB format image B1. Converting the RAW format into RGB format requires the electronic unit to perform a large amount of digital calculations, which has a large impact on the power consumption and performance overhead of the device.

[0085] Since the converted image A1 and the converted image B1 come from different cameras, image registration is required before fusion. Image registration refers to the process of matching and superimposing two or more images acquired at different times, using different sensors, or under different conditions (weather, illumination, camera position and angle, etc.). For example, the existing scale-invariant feature transform (SIFT) image matching algorithm can be used. This algorithm extracts SIFT feature points from two digital images respectively, and then aligns the feature points of the two images through a certain method to obtain images with the same spatial position.

[0086] The images A1 and B1 after format conversion and image registration are fused. Because the optical centers of the shooting lenses of images A1 and B1 are not completely consistent in space, there is parallax. In addition, the image recording time of the two lenses may also be inconsistent. The positions of the moving objects in the moving scene will also be inconsistent in the two images. Therefore, in both cases, there will be some areas where the image content is not completely consistent. The areas with consistent content are called aligned areas, and the areas with inconsistent content are called non-aligned areas. In the dual-camera video fusion method of the embodiment of the present application, the aligned areas of the two images in image A1 and image B1 can be directly fused, while the non-aligned areas are not processed; or a local registration method such as optical flow features can be used to align the non-aligned areas in image A1 and image B1, and then a detail enhancement network is used to restore the details, wherein the detail enhancement network is a pre-trained neural network that can restore image details. Finally, the aligned areas and the processed non-aligned areas in image A1 and image B1 are fused. In the embodiment of the present application, the fusion of image A1 and image B1 can be performed using a traditional image fusion algorithm, such as merging the black and white channels of image A1 and the color channels of image B1 into one image, that is, fusing image A1 and image B1. Alternatively, the fusion of image A1 and image B1 can be performed using an image fusion neural network, that is, inputting image A1 and image B1 into the image fusion neural network, and the image fusion neural network outputs the fused image, wherein the image fusion neural network is a pre-trained neural network capable of image fusion. After the fusion of image A1 and image B1, a fused image AB1 can be obtained, which combines the advantages of the main camera and the auxiliary camera.

[0087] The second frame of image A2 captured by the main camera is captured. Image A2 is still in RAW format. The auxiliary camera is not in use at this time, i.e., image capture is stopped and the operating frequency is reduced. Simultaneously, when the main camera captures image A2, the position and posture data I2 of the inertial measurement unit in the electronic device is acquired.

[0088] The image A2 is processed by an image signal processor, and the RAW format is converted into an RGB format by an ISP demosaicing algorithm, thereby obtaining a converted image A2.

[0089] The homography matrix H1 between the image A1 and the image A2 can be calculated based on the posture data I1 and the posture data I2, where the homography matrix represents the mapping relationship between points on the same plane between different images.

[0090] Based on image AB1 and the homography matrix H1, the H1 matrix calculation can be performed on each pixel in AB1. The resulting pixels form the virtual image B2: B2 = H1 × AB1. Since image AB1 is derived from image A1, images AB1 and A1 are aligned. The homography between images A1 and A2 is the homography matrix H1. Therefore, image B2, derived from image AB1 based on the homography matrix H1, is aligned with image A2.

[0091] Since image B2 and image A2 are aligned, image B2 and image A2 do not need to be registered and can be directly fused to obtain image AB2.

[0092] According to the above method, the third frame image A3 can be processed again.

[0093] In the dual-camera video fusion method of the present embodiment, the auxiliary camera has a lower frame rate than the primary camera. That is, while the primary camera captures multiple frames of real images, the auxiliary camera captures one real frame. The missing images of the auxiliary camera are supplemented by automatically generated virtual images, thereby achieving a one-to-one correspondence between the auxiliary camera image and the primary camera image. The ratio of the frame rate of the primary camera to the frame rate of the auxiliary camera is pre-set by the system, and the specific ratio is not limited in the present embodiment.

[0094] After the auxiliary camera captures a frame of image B1, the timing of the next capture of the second frame is determined by the ratio of the main camera's frame rate to the auxiliary camera's frame rate. For example, assuming that the main camera captures the first frame of image A1 and the auxiliary camera captures the first frame of image B1 simultaneously, and the ratio of the main camera's frame rate to the auxiliary camera's frame rate is 5:1, then the images A2, A3, A4, and A5 captured by the main camera are fused with the generated virtual images B2, B3, B4, and B5 accordingly. When the main camera captures image A6, the auxiliary camera simultaneously captures the second frame of image B6. Alternatively, if the main camera captures the first frame of image A1 and the auxiliary camera captures the first frame of image B1 at different times, for example, 0.1 seconds after the main camera captures the first frame of image A1, the auxiliary camera captures the first frame of image B1, and the ratio of the main camera's frame rate to the auxiliary camera's frame rate is 3:1, then the images A2 and A3 captured by the main camera are fused with the generated virtual images B2 and B3 accordingly. 0.1 seconds after the main camera captures image A4, the auxiliary camera captures the second frame of image B4.

[0095] According to the above Figure 8 The method in can process each frame of the video during shooting, thereby obtaining a video that combines the advantages of the main camera and the auxiliary camera. Figure 8In this method, only the acquired real image needs to be processed by the image signal processor, and the generated virtual image does not need to be processed by the image signal processor. The RAW image acquisition and ISP demosaicing process are much more computationally intensive than the simple virtual synthesis process. The generated virtual image consumes less energy, thereby greatly reducing the device performance requirements and memory consumption of the image signal processor for image processing. The generated virtual image is aligned with the real image acquired by the main camera, so there is no need for image registration before image fusion, which reduces image processing time and achieves real-time image processing during video capture. Image AB1 combines the advantages of image A1 and image B1. Image B2 is obtained from image AB1 through the homography matrix, so image B2 inherits the advantages of image AB1. Image B2 is then fused with image A2. The resulting image AB2 combines the advantages of the previous images, so the obtained video has a better effect. In addition, because the subsequent images inherit the information of the previous images, the resulting video also has good continuity.

[0096] Figure 8 The detail enhancement network and image fusion network involved in the method are both pre-trained neural networks. The detail enhancement network can be trained by inputting existing, un-detailed images as training data into the neural network model to be trained, using corresponding detail-enhanced images as training labels to complete the training of the neural network. The image fusion network can be trained by inputting two images to be fused as training data into the neural network model to be trained, using the fusion result of the two images to be fused as training labels to complete the training of the neural network model.

[0097] Figure 9 A schematic block diagram of the specific method operation of dual-camera video fusion in an embodiment of the present application is shown, taking the fusion of a black and white camera and a color camera on a mobile phone as an example, and is introduced step by step below.

[0098] 1. The user turns on the camera and enters the video recording mode;

[0099] 2. After detecting the user clicking the record button, the color camera module obtains a color image C1. The color camera module can be Figure 3The camera model in the image sensor is used to generate the color image C1. The initial format of the color image C1 is RAW format. RAW format is the original data of the image sensor converting the captured light source signal into a digital signal. At the same time, the posture data I1 of the inertial measurement unit (IMU) corresponding to the moment when the color image C1 is generated is recorded. The posture data includes the current angular velocity and linear acceleration of the electronic device, totaling six-dimensional degrees of freedom information. Specifically, the accelerometer in the inertial measurement unit measures the acceleration on the three axes, and the gyroscope measures the angular velocity around the three axes. The black and white camera is opened in the background to obtain the black and white image M1, where the black and white camera module can be Figure 3 In the camera model, the format of the black and white image M1 is RAW format; the color camera and the black and white camera can shoot at the same time or successively within a preset time, and the color image C1 and the black and white image M1 are stored in the mobile phone memory at the same time.

[0100] 3. The color image C1 and the black-and-white image M1 stored in the mobile phone memory are subjected to ISP processing through the camera's built-in software and hardware image processing algorithm modules. For example, the color image C1 and the black-and-white image M1 can be converted from the original RAW format to the RGB format through the existing ISP demosaicing algorithm.

[0101] 4. Use an existing image registration algorithm to align the RGB color image C1 and the RAW black-and-white image M1. Then, output the aligned color and black-and-white images. Specifically, crop the black-and-white image's field of view to match that of the color image. Image registration refers to the process of matching and overlaying two or more images acquired at different times, using different sensors, or under different conditions (such as weather, illumination, camera position, and angle). For example, the existing SIFT image matching algorithm can be used to extract SIFT feature points from each of the two digital images. The feature points of the two images are then aligned using a specific method to obtain images with identical spatial locations. Finally, use an existing frame-by-frame fusion algorithm to fuse the color image C1 and the black-and-white image M1, thereby obtaining a fused image MCF1, where the field of view of MCF1 is completely consistent with that of the color image C1.

[0102] 5. The color camera continues to acquire the next frame of color image C2, where the color image C2 is still in RAW format. At the same time, the IMU data I2 corresponding to the moment when image C2 is generated is recorded. The black and white camera temporarily stops working, that is, stops image capture and reduces the working frequency.

[0103] 6. Based on the IMU data I1 corresponding to the time when the color image C1 is generated and the IMU data I2 corresponding to the time when the image C2 is generated, the homography matrix H1 can be calculated, where the homography matrix represents the mapping relationship between points in the same plane between different images.

[0104] 7. According to the homography matrix H1, the fused image MCF1 and the geometric principle, the virtual image M2 can be obtained:

[0105] M2=H1×MCF1

[0106] That is, each pixel in the fused image MCF1 is mapped and transformed through the homography matrix H1, and the transformed pixels form a virtual image M2. At this time, the field of view of the virtual image M2 is completely consistent with that of the color image C2.

[0107] 8. The virtual image M2 and the color image C2 can directly obtain the fusion result MCF2 without additional registration.

[0108] Therefore, according to Figure 9 The method can achieve the fusion of the advantages of color lens and black and white lens during video recording, thereby obtaining a clearer video.

[0109] In combination with the above embodiments and related drawings, the embodiment of the present application provides a method for dual-camera video fusion. The method can be used in Figure 2 The present invention is implemented in the electronic device shown (such as a mobile phone, tablet computer, etc.). Figure 10 This is a schematic flow chart of a method for dual-camera video fusion provided in an embodiment of the present application, such as Figure 10 As shown, the method includes steps 1001 to 1008.

[0110] S1001: Acquire a first image captured by a first camera at a first moment and a second image captured by a second camera at a second moment.

[0111] In one example, the first time instant is the same as the second time instant.

[0112] S1002: Process the first image and the second image to obtain a processed first image and a processed second image.

[0113] S1003: Fusing the processed first image and the processed second image to obtain a first fused image at a first moment.

[0114] In one example, before fusing the processed first image with the processed second image, the method of the present application further includes: performing image registration on the processed first image and the processed second image. Specifically, the aligned regions in the processed first image and the processed second image are directly fused. Alternatively, if non-aligned regions exist in the processed first image and the processed second image, the non-aligned regions in the processed first image and the processed second image are aligned according to a local registration method, and then the aligned regions in the processed first image and the processed second image are fused.

[0115] S1004: Acquire a third image captured by the first camera at a third moment.

[0116] S1005: Process the third image to obtain a processed third image.

[0117] In one example, processing the first image and the second image involves processing the first image and the second image through an image signal processor, and processing the third image involves processing the third image through an image signal processor. Specifically, the first image, the second image, and the third image are in RAW format, and the image signal processor converts the first image, the second image, and the third image from RAW format to RGB format.

[0118] S1006: Obtain a homography matrix between the first moment and the third moment.

[0119] Specifically, first data of the inertial measurement unit at a first moment is obtained; second data of the inertial measurement unit at a third moment is obtained; and a homography matrix is ​​obtained according to the first data and the second data.

[0120] S1007 : Acquire a fourth image according to the first fused image and the homography matrix.

[0121] S1008: Fusing the processed third image and the fourth image to obtain a second fused image at a third moment.

[0122] In one example, the first camera and the second camera have different resolution, dynamic range, or color.

[0123] In one example, the image fusion in the present application, including the fusion of the processed first image and the processed second image, and the fusion of the processed third image and the fourth image, can be performed using an existing image fusion algorithm or an image fusion neural network.

[0124] It should be understood that Figure 10 This is just a brief introduction to the process of the dual-camera video fusion method of this application. For details, please refer to the above Figure 8 and Figure 9For the sake of brevity, the embodiments of the present application will not be described in detail here.

[0125] It is understandable that the electronic device of the present application includes hardware and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0126] In this embodiment, the electronic device can be divided into functional modules according to the above-mentioned method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into a single processing module. The above-mentioned integrated modules can be implemented in the form of hardware. It should be noted that the module division in this embodiment is illustrative and is only a logical functional division. In actual implementation, other division methods may be used.

[0127] In the case of dividing each functional module into corresponding functional modules, Figure 11 A possible schematic diagram of the composition of the electronic device involved in the above embodiment is shown. Figure 11 As shown, the electronic device may include: an acquisition unit 1101 and a processing unit 1102.

[0128] The acquisition unit 1101 can be used to support the electronic device 1100 to perform the above Figure 10 Steps 1001, 1004, etc., and / or other processes for the technology described herein.

[0129] The processing unit 1102 can be used to support the electronic device 1100 to perform the above Figure 10 Steps 1002, 1003, 1005 to 1008, etc., and / or other processes for the technology described herein.

[0130] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0131] The electronic device provided in this embodiment is used for the method of dual-camera video fusion, and therefore can achieve the same effect as the above-mentioned implementation method.

[0132] When integrated, the electronic device may include a processing module, a storage module, and a communication module. The processing module may be used to control and manage the electronic device's operations. For example, it may be used to support the electronic device in executing the aforementioned acquisition unit 1101 and processing unit 1102. The storage module may be used to support the electronic device in executing and storing program code and data. The communication module may be used to support communication between the electronic device and other devices.

[0133] The processing module may be a processor or a controller. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module may be a memory. The communication module may specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, or a Wi-Fi chip.

[0134] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment may be a Figure 2 Device with the structure shown.

[0135] This embodiment also provides a computer storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the dual-camera video fusion method in the above-mentioned embodiment.

[0136] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the dual-camera video fusion method in the above-mentioned embodiment.

[0137] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the dual-camera video fusion method in the above-mentioned method embodiments.

[0138] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0139] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0140] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0141] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0142] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0143] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0144] The above content is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for dual-camera video fusion, characterized in that: include: Acquire a first image captured by the first camera at a first moment and a second image captured by the second camera at a second moment; Processing the first image and the second image to obtain a processed first image and a processed second image; fusing the processed first image and the processed second image to obtain a first fused image at the first moment; Acquire a third image captured by the first camera at a third moment; processing the third image to obtain a processed third image; Obtaining a homography matrix between the first moment and the third moment; Acquire a fourth image according to the first fused image and the homography matrix; The processed third image and the fourth image are fused to obtain a second fused image at the third moment.

2. The method according to claim 1, characterized in that The first moment is the same as the second moment.

3. The method according to claim 1 or 2, characterized in that The processing of the first image and the second image is to process the first image and the second image through an image signal processor, and the processing of the third image is to process the third image through an image signal processor.

4. The method according to claim 1 or 2, characterized in that The processing of the first image and the second image, and the processing of the third image, include: The first image and the second image are in RAW format; converting the first image and the second image from a RAW format to an RGB format according to an image signal processor; The third image is in RAW format; The image signal processor converts the third image from a RAW format to an RGB format.

5. The method according to claim 1 or 2, characterized in that Before fusing the processed first image and the processed second image, the method further includes: Perform image registration on the processed first image and the processed second image.

6. The method according to claim 5, characterized in that The fusing the processed first image and the processed second image comprises: The aligned regions in the processed first image and the processed second image are fused.

7. The method according to claim 5, characterized in that The fusing the processed first image and the processed second image comprises: aligning non-aligned regions in the processed first image and the processed second image according to a local registration method; The aligned regions in the processed first image and the processed second image are fused.

8. The method according to claim 1 or 2, characterized in that The obtaining of the homography matrix between the first moment and the third moment includes: Acquiring first data of an inertial measurement unit at the first moment; Acquiring second data of the inertial measurement unit at the third moment; The homography matrix is ​​obtained according to the first data and the second data.

9. The method according to claim 1 or 2, characterized in that The first camera and the second camera have different clarity, dynamic range, or color.

10. The method according to claim 1 or 2, characterized in that The fusing the processed first image and the processed second image comprises: The processed first image and the processed second image are fused using an image fusion algorithm or an image fusion neural network, where the image fusion neural network is a pre-trained neural network.

11. An electronic device, characterized in that: include: one or more processors; A memory comprising instructions; when the instructions are executed by the one or more processors, the electronic device performs the following steps: Acquire a first image captured by the first camera at a first moment and a second image captured by the second camera at a second moment; Processing the first image and the second image to obtain a processed first image and a processed second image; fusing the processed first image and the processed second image to obtain a first fused image at the first moment; Acquire a third image captured by the first camera at a third moment; processing the third image to obtain a processed third image; Obtaining a homography matrix between the first moment and the third moment; Acquire a fourth image according to the first fused image and the homography matrix; The processed third image and the fourth image are fused to obtain a second fused image at the third moment.

12. The electronic device according to claim 11, wherein: The first moment is the same as the second moment.

13. The electronic device according to claim 11 or 12, characterized in that: The processing of the first image and the second image is to process the first image and the second image through an image signal processor, and the processing of the third image is to process the third image through an image signal processor.

14. The electronic device according to claim 11 or 12, characterized in that: The processing of the first image and the second image, and the processing of the third image, include: The first image and the second image are in RAW format; converting the first image and the second image from a RAW format to an RGB format according to an image signal processor; The third image is in RAW format; The image signal processor converts the third image from a RAW format to an RGB format.

15. The electronic device according to claim 11 or 12, characterized in that: Before fusing the processed first image and the processed second image, the electronic device is further configured to: Perform image registration on the processed first image and the processed second image.

16. The electronic device according to claim 15, characterized in that The fusing the processed first image and the processed second image comprises: The aligned regions in the processed first image and the processed second image are fused.

17. The electronic device according to claim 15, characterized in that The fusing the processed first image and the processed second image comprises: aligning non-aligned regions in the processed first image and the processed second image according to a local registration method; The aligned regions in the processed first image and the processed second image are fused.

18. The electronic device according to claim 11 or 12, characterized in that: The obtaining of the homography matrix between the first moment and the third moment includes: Acquiring first data of an inertial measurement unit at the first moment; Acquiring second data of the inertial measurement unit at the third moment; The homography matrix is ​​obtained according to the first data and the second data.

19. The electronic device according to claim 11 or 12, characterized in that: The first camera and the second camera have different clarity, dynamic range, or color.

20. The electronic device according to claim 11 or 12, characterized in that: The fusing the processed first image and the processed second image comprises: The processed first image and the processed second image are fused using an image fusion algorithm or an image fusion neural network, where the image fusion neural network is a pre-trained neural network.

21. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program codes including instructions for executing part or all of the steps of the method according to any one of claims 1 to 10.

22. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute instructions of part or all of the steps of the method according to any one of claims 1 to 10.

Images