Shooting methods, devices, electronic equipment, and readable storage media

By using a combination of color and black and white cameras in dark light environments, the color channel and brightness information of the image are integrated, the problem of poor image quality in dark light environments is solved, high-quality shooting effects are achieved and power consumption is controlled.

CN116156334BActive Publication Date: 2024-11-26VIVO MOBILE COMM CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310182887.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-02-28
Publication Date
2024-11-26
Estimated Expiration
2043-02-28

AI Technical Summary

Technical Problem

In dark light environments, the images captured in the prior art are of poor quality and are greatly affected by light.

Method used

Using a combination of color cameras and black and white cameras, the high sensitivity and high frame rate of black and white cameras are used to collect images, and fuse them with the low frame rate of color cameras. By fusing the color channel information and brightness information of the image, the target image is output to ensure that The image has rich colors, high brightness, many details and low noise.

Benefits of technology

It effectively improves the quality of captured images in dark light environments, reduces the need for synchronous acquisition of images, thereby controlling the power consumption increment and improving the battery life of electronic devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116156334B_ABST
    Figure CN116156334B_ABST
Patent Text Reader

Abstract

The present application discloses a shooting method, device, electronic device and readable storage medium, which belongs to the field of electronic technology. The method includes: fusing the color channel information of the first image captured by the first camera of the electronic device with the brightness information of the second image captured by the second camera of the electronic device to obtain a first fused image; fusing the color channel information of the first fused image with the brightness information of the third image captured by the second camera to obtain a second fused image; the third image is an adjacent frame image of the second image, and at the moment when the second camera captures the third image, the first camera does not capture the image; outputting the target image according to the first fused image and the second fused image; the first camera is a color camera, and the second camera is a black and white camera; the frame rate of the first camera is less than the frame rate of the second camera; the sensitivity of the second camera is N times the sensitivity of the first camera, N≥2.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of electronic technology, and specifically relates to a shooting method, device, electronic device and readable storage medium. Background Art

[0002] Currently, electronic devices have increasingly stronger shooting capabilities, but in some special environments, the shooting function is still lacking.

[0003] For example, when shooting in a dark environment such as at night, the brightness of the captured image is low due to the influence of light, thereby reducing the image quality.

[0004] It can be seen that in the prior art, there is a problem of poor image quality due to a dark shooting environment. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a shooting method that can solve the problem in the prior art that the image quality obtained by shooting is poor due to the influence of the dark light environment.

[0006] In a first aspect, an embodiment of the present application provides a shooting method, the method comprising: fusing color channel information of a first image captured by a first camera of an electronic device with brightness information of a second image captured by a second camera of the electronic device to obtain a first fused image; fusing color channel information of the first fused image with brightness information of a third image captured by the second camera to obtain a second fused image; wherein the third image is an adjacent frame image of the second image, and at the moment when the second camera captures the third image, the first camera does not capture an image; outputting a target image according to the first fused image and the second fused image; wherein the first camera is a color camera and the second camera is a black and white camera; the frame rate of the first camera is less than the frame rate of the second camera; the sensitivity of the second camera is N times the sensitivity of the first camera, N≥2.

[0007] In a second aspect, an embodiment of the present application provides a shooting device, which includes: a first fusion module, used to fuse color channel information of a first image captured by a first camera of an electronic device with brightness information of a second image captured by a second camera of the electronic device to obtain a first fused image; a second fusion module, used to fuse color channel information of the first fused image with brightness information of a third image captured by the second camera to obtain a second fused image; wherein the third image is an adjacent frame image of the second image, and at the moment when the second camera captures the third image, the first camera does not capture an image; an output module, used to output a target image based on the first fused image and the second fused image; wherein the first camera is a color camera and the second camera is a black and white camera; the frame rate of the first camera is less than the frame rate of the second camera; the sensitivity of the second camera is N times the sensitivity of the first camera, N≥2.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0012] Thus, in the embodiment of the present application, when shooting, the electronic device first fuses the first image captured by the first camera (color camera) and the second image captured by the second camera (black and white camera) to obtain a first fused image. Among them, the sensitivity of the second camera is twice or more than twice the sensitivity of the first camera. Therefore, the second image has the advantages of high brightness, more details and low noise relative to the first image, so that in the fusion process, the brightness information of the second image is fused with the color channel information of the first image to obtain the first fused image. Furthermore, the frame rate of the second camera is greater than the frame rate of the first camera. After the second camera captures the adjacent frame image of the second image, that is, the third image, the color channel information of the first fused image is fused with the brightness information of the third image to obtain the second fused image. Finally, based on the first fused image and the second fused image, the target image is output. It can be seen that based on the embodiment of the present application, the output target image has the characteristics of high brightness, more details and low noise on the basis of rich colors, which effectively improves the quality of the captured image. In addition, in this embodiment, based on the different frame rates of the two cameras, compared with the case where the frame rates are the same, the step of fusing two frames of images captured synchronously can be reduced, thereby controlling the power consumption increment. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flow chart of a photographing method according to an embodiment of the present application;

[0014] Figures 2 to 6 is a schematic diagram for explaining a photographing method according to an embodiment of the present application;

[0015] Figure 7 is a block diagram of a photographing device according to an embodiment of the present application;

[0016] Figure 8 It is one of the hardware structure diagrams of the electronic device according to the embodiment of the present application;

[0017] Fig. 9 This is the second schematic diagram of the hardware structure of the electronic device according to the embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings of the embodiments of the present application to clearly describe the technical solutions of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present application belong to the scope of protection of this application.

[0019] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0020] The shooting method provided in the embodiment of the present application may be executed by the shooting device provided in the embodiment of the present application, or an electronic device integrating the shooting device, wherein the shooting device may be implemented in hardware or software.

[0021] The following is a detailed description of the shooting method provided in the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0022] Figure 1 A flowchart of a shooting method according to an embodiment of the present application is shown, and the method is applied to an electronic device as an example, including:

[0023] Step 110: Fusing the color channel information of the first image captured by the first camera of the electronic device with the brightness information of the second image captured by the second camera of the electronic device to obtain a first fused image.

[0024] In this embodiment, the electronic device includes a first camera and a second camera.

[0025] Optionally, the first camera and the second camera are arranged in a horizontal direction, and the optical axes of the two cameras are parallel.

[0026] Optionally, the first camera and the second camera are arranged in a vertical direction, and the optical axes of the two cameras are parallel.

[0027] The first camera is a color camera, and the second camera is a black and white camera; the frame rate of the first camera is lower than the frame rate of the second camera; and the sensitivity of the second camera is N times that of the first camera, where N≥2.

[0028] Optionally, the first camera is composed of a lens, a motor, a motor drive, an infrared (IR) filter, a color filter array and a photosensitive pixel array.

[0029] Among them, the color filter array can sense three colors and output three color channel images, and the combination of the three color channels includes but is not limited to: red, green and blue, so that the image captured by the first camera is a color image; the color filter array can adopt a Bayer (bayer) or quad Bayer (quad bayer) arrangement, specifically but not limited to: red-green-green-blue (Red-green-green-blue, RGGB), red-green-blue-white (red-green-blue-white, RGBW) and other arrangements.

[0030] Optionally, the second camera is composed of a lens, a motor and a motor drive, an IR filter, and a photosensitive pixel array.

[0031] The photosensitive pixel array senses visible light without distinguishing colors, so that the image captured by the second camera is a black and white image.

[0032] In the present application, the two cameras capture images at different frame rates, the first camera captures images at a low frame rate, and the second camera captures images at a high frame rate.

[0033] For example, the frame rate of the second camera is 24fps, 30fps, and 60fps; the frame rate of the first camera is 1fps and 2fps.

[0034] Optionally, the frame rate of the second camera is M times the frame rate of the first camera, where M>1.

[0035] This embodiment is applied in shooting scenes.

[0036] For example, in a scene of shooting a video, the captured video image is displayed on the screen in real time; for another example, in a scene of shooting a photo, the captured preview image is displayed on the screen in real time.

[0037] Optionally, the frame rate of the second camera is set to a common frame rate for shooting videos.

[0038] Optionally, during the shooting process, the two cameras are controlled to be exposed simultaneously, and the two cameras are controlled to focus at the same distance.

[0039] The first image and the second image are two images taken at the same time.

[0040] Optionally, the first image and the second image are preprocessed images, and the preprocessing method includes: image signal processing (Image Signal Processor, ISP) methods such as bad pixel removal, noise reduction, and lens correction.

[0041] In the fusion process of this step, the color space conversion is performed on the first image to transfer the color channel information of the first image to the second image, so as to realize the fusion of the color channel information of the first image and the brightness information of the second image.

[0042] Optionally, based on the first image, separate channels for brightness and color are output.

[0043] For example, a first image in a red (R), green (G), and blue (B) format is converted into a first image in a YUV format, where Y is used to represent brightness information, and U and V are used to represent color channel information.

[0044] See the formula for conversion:

[0045] Y = 0.299R + 0.587G + 0.114B;

[0046] U = 0.596R - 0.274G - 0.322B;

[0047] V = 0.211R - 0.523G + 0.312B.

[0048] In this embodiment, the sensitivity of the second camera is N times that of the first camera, where N≥2. Therefore, the sensitivity of the second camera is greater than that of the first camera, so that in a dark environment, compared with the first image, the second image has higher brightness, more details, and lower noise, and then the color channel information of the first image and the brightness information of the second image are fused into one image, which can improve the brightness of the image while ensuring the color of the image, and better handle the details and noise of the image.

[0049] Step 120: Fuse the color channel information of the first fused image with the brightness information of the third image captured by the second camera to obtain a second fused image; wherein the third image is an adjacent frame image of the second image, and when the second camera captures the third image, the first camera does not capture the image.

[0050] Because the frame rate of the second camera is greater than that of the first camera, if the first image and the second image are captured at a certain moment, the first camera does not capture the image when the second camera captures the adjacent frame image of the second image, that is, the third image.

[0051] In this step, the color channel information of the first fused image and the brightness information of the third image are fused together, which can ensure that the current frame image is a color image while retaining the advantages of a black and white image.

[0052] The fusion in this step may refer to the fusion in step 110, which will not be described in detail here.

[0053] Optionally, the third image is a next frame image of the second image.

[0054] In this scenario, the fusion process in step 120 is performed synchronously with image acquisition. That is, at a certain moment, the second camera acquires an image and performs image fusion synchronously.

[0055] Optionally, the third image is a previous frame image of the second image.

[0056] In this scenario, the fusion process in step 120 is performed asynchronously with image acquisition. That is, at the first moment, the second camera acquires an image and image fusion may not be performed temporarily; when the second camera continues to acquire an image at the next moment (i.e., the second moment), and the first camera also acquires an image at the second moment, after the two images acquired at the second moment are fused, that is, the fusion process in step 110 is performed, and the obtained fused image is then used to fuse with the image acquired by the second camera at the first moment.

[0057] Step 130: Output a target image according to the first fused image and the second fused image.

[0058] Optionally, after obtaining the first fused image, the first fused image in YUV format is converted into a first fused image in RGB format; and after obtaining the second fused image, the second fused image in YUV format is converted into a second fused image in RGB format.

[0059] Furthermore, based on the fused image after the format conversion, a target image is output.

[0060] Optionally, the target image is captured video content.

[0061] Optionally, the target image is a captured preview content.

[0062] Optionally, in an output solution, the first fused image and the second fused image are directly output as the target image. Optionally, the first fused image and the second fused image are in RGB format.

[0063] Optionally, in one scenario, for the second camera, if the adjacent frame image of the third image still does not have a corresponding image captured by the first camera, the second fused image continues to be fused with the frame image, and so on, until at a certain moment both the first camera and the second camera capture images, and the fusion step in step 110 can be performed.

[0064] Optionally, in another scenario, the first camera captures an image, and the shooting scene does not change within a certain period of time before or after the image is captured. Therefore, within this period of time, all frame images captured by the second camera do not contain any changes in the captured object. Therefore, after a fused image is obtained based on the image captured by the first camera and the corresponding image captured by the second camera, the fused image can be repeatedly used to fuse the fused image with multiple frame images captured by the second camera within the period of time in sequence, until the captured images of the first camera and the second camera are fused to form a new image that can be used.

[0065] Thus, in the embodiment of the present application, when shooting, the electronic device first fuses the first image captured by the first camera (color camera) and the second image captured by the second camera (black and white camera) to obtain a first fused image. Among them, the sensitivity of the second camera is twice or more than twice the sensitivity of the first camera. Therefore, the second image has the advantages of high brightness, more details and low noise relative to the first image, so that in the fusion process, the brightness information of the second image is fused with the color channel information of the first image to obtain the first fused image. Furthermore, the frame rate of the second camera is greater than the frame rate of the first camera. After the second camera captures the adjacent frame image of the second image, that is, the third image, the color channel information of the first fused image is fused with the brightness information of the third image to obtain the second fused image. Finally, based on the first fused image and the second fused image, the target image is output. It can be seen that based on the embodiment of the present application, the output target image has the characteristics of high brightness, more details and low noise on the basis of rich colors, which effectively improves the quality of the captured image. In addition, in this embodiment, based on the different frame rates of the two cameras, compared with the case where the frame rates are the same, the step of fusing two frames of images captured synchronously can be reduced, thereby controlling the power consumption increment.

[0066] In the process of the shooting method of another embodiment of the present application, step 110 includes:

[0067] Sub-step A1: according to a first matching relationship between pixel points of the first image and pixel points of the second image, setting the color channel information of the first image as the color channel information of the second image.

[0068] Optionally, based on the brightness information of the first image and the brightness information of the second image, the first image and the second image are stereo matched at the pixel level to obtain a one-to-one correspondence between pixels of the first image and pixels of the second image, that is, a first matching relationship.

[0069] Among them, pixel-level stereo matching is similar to the stereo registration between two sensors. It completes the correspondence or mapping of pixel points (coordinates) on the first image to pixel points (coordinates) on the second image through a method similar to image block similarity search and matching.

[0070] Optionally, after matching, a disparity map or an optical flow map mapped from the first image to the second image is output.

[0071] Optionally, the matching method adopts: traditional computer vision such as block matching method, semi-global matching method or neural network (NN)-based stereo registration network method.

[0072] Optionally, in this step, a warp operation is performed on the color channel information of the first image according to the disparity map or the optical flow map to complete the alignment of the two images.

[0073] Sub-step A2: According to the brightness information of the second image and the brightness information of the first image, the color channel information of the second image is adjusted to obtain a first fused image.

[0074] Optionally, before the adjustment, color noise suppression processing is performed on the color channel information of the second image.

[0075] Optionally, the brightness information of the second image and the brightness information of the first image are used as guiding parameters to adjust the color saturation of the second image so that the brightness information of the second image and the color channel information are naturally integrated.

[0076] Specifically, the brightness information of the second image is different from the brightness information of the first image. Therefore, in the second image, the brightness information of a local area is also different from the brightness information of the corresponding area in the first image, and presents a proportional relationship, which is defined as the stretching ratio. Therefore, the color channel information of each area in the second image can be adjusted based on the stretching ratio of the area.

[0077] For example, in the first image, the brightness of a certain area is high, and in the second image, the brightness of the area is low. Therefore, in the second image, after setting the color channel information of the corresponding area of ​​the first image for the area, the set color channel information is reduced and adjusted in proportion. The proportion is the brightness ratio of the same area in different images.

[0078] Finally, the first fused image is obtained.

[0079] For example, see Figure 2The first picture represents the first image 201, and the second picture represents the second image 202. In the second image 202, the dotted line object 2021 and the solid line object 2022 are the same object, and the dotted line object 2021 is the position of the object in the first image. Here, the dotted line object 2021 and the solid line object 2022 are used to represent the parallax between the two images. The third picture represents the first fused image 203.

[0080] In this embodiment, a fusion process is shown. In the fusion process, in order to naturally fuse the brightness information and color channel information in the fused image, after the second image has the color channel information of the first image, the color channel information in the second image is adaptively adjusted based on the correspondence between the brightness information of the second image and the brightness information of the first image, thereby further improving the quality of the captured image.

[0081] In the process of the shooting method of another embodiment of the present application, after sub-step A1 and before obtaining the first fused image, the method further includes:

[0082] Sub-step B1: when the color channel information of the first pixel point in the first area of ​​the second image is not set, setting the color channel information of the first pixel point according to the color channel information of the second pixel point in the first area.

[0083] There is a first correlation relationship between the brightness information of the first pixel and the brightness information of the second pixel.

[0084] In this step, after the warp operation, there may be parts of the second image where color channel information is missing. For example, in the first area, some pixels are still black and white. Therefore, the overall brightness information of the first area can be used to set the missing color channel information.

[0085] Optionally, in the second image, the brightness of the local area is basically consistent, so based on this, when the color channel information is missing in the local area, the missing color feature information of several pixels (such as the first pixel) can be filled in according to the color channel information of the surrounding pixels (such as the second pixel).

[0086] Correspondingly, the first association relationship may represent that, when the first pixel point and the second pixel point are located in the same area, the brightness information of the first pixel point and the brightness information of the second pixel point are consistent or close.

[0087] Correspondingly, after filling, the color channel information of the first pixel is consistent with or close to the color channel information of the second pixel.

[0088] Optionally, in the second image, the brightness of the local area has a certain changing pattern, and based on this, when the color channel information is missing in the local area, the missing color feature information of several pixels (such as the first pixel) can be filled in according to the changing pattern of the color channel information of the surrounding pixels (such as the second pixel).

[0089] Correspondingly, the first association relationship may represent that when the first pixel point and the second pixel point are located in the same area, the brightness information of the first pixel point and the brightness information of the second pixel point present a certain change pattern.

[0090] In this embodiment, the local brightness information of the second image can be used to complete the color filling of the color missing area in the second image, thereby ensuring the high quality of the captured image.

[0091] In the process of the shooting method of another embodiment of the present application, step 120 includes:

[0092] Sub-step C1: according to a second matching relationship between pixels of the first fused image and pixels of the third image, setting the color channel information of the first fused image as the color channel information of the third image.

[0093] Optionally, the first fused image and the third image are densely matched according to brightness information of the first fused image and brightness information of the third image, so as to obtain a one-to-one correspondence between pixels of the first fused image and pixels of the third image, that is, a second matching relationship.

[0094] Optionally, a dense optical flow map is output after matching.

[0095] Optionally, the matching method adopts: a block matching method, a variational optimization-based method or a NN-based optical flow registration network method.

[0096] It should be noted that, in the aforementioned embodiment, pixel-level stereo matching is adopted, and for two images at the same time, the two images have a position deviation in one direction, such as the horizontal direction or the vertical direction; in the present embodiment, dense matching is adopted, and for two images at different times, the two images have a position deviation in multiple directions, such as the horizontal direction and the vertical direction.

[0097] Optionally, according to the dense optical flow map, a warp operation is performed on the color channel information of the first fused image to complete the alignment of the two images.

[0098] Sub-step C2: adjusting the color channel information of the third image according to the brightness information of the third image and the brightness information of the first fused image to obtain a second fused image.

[0099] Optionally, before the adjustment, color noise suppression processing is performed on the color channel information of the third image.

[0100] Optionally, the brightness information of the third image and the brightness information of the first fused image are used as guiding parameters to adjust the color saturation of the third image so that the brightness information of the third image is naturally integrated with the color channel information.

[0101] Specifically, due to the difference in shooting time, there is a difference in the brightness information of the first fused image and the brightness information of the third image. Therefore, in the third image, the brightness information of a local area is different from the brightness information of the corresponding area in the first fused image, and presents a proportional relationship, which is defined as the stretching ratio. Therefore, the color channel information of each area in the third image can be adjusted based on the stretching ratio of the area.

[0102] For example, in the first fused image, the brightness of a certain area is high, and in the third image, the brightness of the area is low. Therefore, in the third image, after setting the color channel information of the corresponding area of ​​the first fused image for the area, the set color channel information is reduced and adjusted in proportion. The proportion is the brightness ratio of the same area in different images.

[0103] Finally, a second fused image is obtained.

[0104] For example, see Figure 3 The first picture represents the first fused image 301, and the second picture represents the third image 302. In the third image 302, the dotted line object 3021 and the solid line object 3022 are the same object. The dotted line object 3021 is the position of the object in the first fused image. Here, the dotted line object 3021 and the solid line object 3022 are used to represent the parallax between the two images. The third picture represents the second fused image 303.

[0105] In this embodiment, a fusion process is shown. In the fusion process, in order to naturally fuse the brightness information and color channel information in the fused image, after the third image has the color channel information of the first fused image, the color channel information in the third image is adaptively adjusted based on the corresponding relationship between the brightness information of the third image and the brightness information of the first fused image, thereby further improving the quality of the captured image.

[0106] In the process of the shooting method of another embodiment of the present application, after sub-step C1 and before obtaining the second fused image, the method further includes:

[0107] Sub-step D1: when the color channel information of the third pixel point in the second area of ​​the third image is not set, the color channel information of the third pixel point is set according to the color channel information of the fourth pixel point in the second area.

[0108] The brightness information of the third pixel point and the brightness information of the fourth pixel point have a second association relationship.

[0109] In this step, after the warp operation, there may be parts of the third image where color channel information is missing. For example, in the second area, some pixels are still black and white. Therefore, the overall brightness information of the second area can be used to set the missing color channel information.

[0110] Optionally, in the third image, the brightness of the local area is basically consistent, and based on this, when color channel information is missing in the local area, the missing color channel information of several pixels (such as the third pixel) can be filled in according to the color channel information of surrounding pixels (such as the fourth pixel).

[0111] Correspondingly, the first association relationship may represent that when the third pixel point and the fourth pixel point are located in the same area, the brightness information of the third pixel point is consistent with or close to the brightness information of the fourth pixel point.

[0112] Correspondingly, after filling, the color channel information of the third pixel is consistent with or close to the color channel information of the fourth pixel.

[0113] Optionally, in the third image, the brightness of the local area has a certain changing pattern, and based on this, when the color channel information is missing in the local area, the missing color feature information of several pixels (such as the third pixel) can be filled in according to the changing pattern of the color channel information of the surrounding pixels (such as the fourth pixel).

[0114] Correspondingly, the second association relationship may represent that when the third pixel point and the fourth pixel point are located in the same area, the brightness information of the third pixel point and the brightness information of the fourth pixel point present a certain change pattern.

[0115] In this embodiment, the local brightness information of the third image can be used to complete the color filling of the color missing area in the third image, thereby ensuring the high quality of the captured image.

[0116] In the process of the shooting method of another embodiment of the present application, step 130 includes:

[0117] Sub-step E1: Perform high-dynamic synthesis on the first fused image and the second fused image to output a target image.

[0118] When the second image and the third image of the second camera are collected by long and short exposure, after obtaining the first fused image and the second fused image, the first fused image and the second fused image are synthesized with high dynamic range to output a target image in a high dynamic range scene.

[0119] Optionally, the high dynamic synthesis superimposes the first fused image and the second fused image according to a certain weight.

[0120] In this embodiment, a high dynamic synthesis method is adopted to synthesize the first fused image and the second fused image before outputting the target image, so that adjacent frames of the target image are naturally merged, thereby improving the quality of the target image.

[0121] In the process of the shooting method of another embodiment of the present application, before step 110, the method further includes:

[0122] Step F1: when the frame rate of the second camera has been determined, the frame rate of the first camera is determined according to the shooting scene information.

[0123] The change frequency information of the shooting scene information is positively correlated with the frame rate of the first camera.

[0124] In this embodiment, the frame rate of the first camera can be adjusted dynamically.

[0125] Optionally, a target motion detection method or a field of view change detection method is used to dynamically adjust the frame rate of the first camera.

[0126] For example, based on the target motion detection method or the field of view change detection method, it is detected that the shooting scene information changes frequently, such as someone running in the scene, etc., the frame rate of the first camera is dynamically increased to capture more frames of images to ensure the continuity of the captured images.

[0127] Optionally, based on the image captured by the second camera, a target motion detection method or a field of view change detection method is used to obtain change frequency information of the shooting scene information.

[0128] In this embodiment, in combination with the shooting scene information, a solution of dynamically adjusting the frame rate of the first camera is adopted to ensure that the images captured by the first camera are coherent, and at the same time, the power consumption increment is dynamically controlled to improve the battery life of the electronic device while ensuring the high quality of the captured images.

[0129] In a shooting method of another embodiment of the present application, in a video shooting scene, all images captured by the first camera are saved together, and all images captured by the second camera are saved together. After shooting is completed, the shot video is output. The two image sets can be used to automatically process the video to obtain a higher quality video.

[0130] For example, see Figure 4 During the shooting process, the user zooms out, resulting in the color loss of the third area 401 (the area marked by the diagonal lines) in the captured black and white image. Therefore, the saved color image is used for re-matching, and the matched color is migrated to the color-missing area.

[0131] For example, see Figure 5 During the shooting process, the electronic device moves, resulting in color loss in the fourth area 501 (the area marked by diagonal lines) in the captured black and white image, so the saved color image is used for re-matching and the matched color is migrated to the color-missing area.

[0132] For example, see Figure 6 During the shooting process, the color of the fifth area 601 (the area marked by the oblique lines) in the captured black and white image is lost due to the movement of the photographed object, so the saved color image is used for re-matching and the matched color is migrated to the color-missing area.

[0133] In this embodiment, after shooting is completed, the color-missing area in the shot image may be filled with color to further improve the shooting quality.

[0134] In summary, the electronic device of the present application adopts a side-by-side structure of black and white and color cameras, and the sensitivity of the black and white camera is twice or more than that of the color camera, so as to ensure that the details and brightness of the black and white image are better than the color image under the same exposure. Among them, the black and white camera collects at a high frame rate (such as 30fps), and the color camera collects at a low frame rate combined with a dynamic frame rate. The dynamic frame rate of the color camera is adjusted with the motion detection result of the black and white camera, thereby controlling the power consumption. In addition, for dark light environments, using the high transmittance of the black and white camera, taking the black and white image as the reference, when there is a color image output, the color channel information is migrated to the black and white image after registration, and the color propagation, color noise suppression, color restoration and other processing are performed to obtain the fused image; when there is no color image, the front and rear black and white images are continuously registered, and the color channel information is migrated from the fused image to the new black and white image, and a similar fusion operation is performed to achieve the purpose of continuously outputting full-color images, and the image quality of the output image has the characteristics of more details and low noise. In addition, the two cameras have different frame rates, which can reduce the frequency of dual-camera registration, thereby controlling the power consumption increment and improving the endurance of electronic equipment.

[0135] The shooting method provided in the embodiment of the present application can be executed by a shooting device. In the embodiment of the present application, the shooting method is executed by a shooting device as an example to illustrate the shooting device provided in the embodiment of the present application.

[0136] Figure 7 A block diagram of a photographing device according to an embodiment of the present application is shown, and the device includes:

[0137] A first fusion module 10 is used to fuse the color channel information of the first image captured by the first camera of the electronic device with the brightness information of the second image captured by the second camera of the electronic device to obtain a first fused image;

[0138] The second fusion module 20 is used to fuse the color channel information of the first fused image with the brightness information of the third image captured by the second camera to obtain a second fused image; wherein the third image is an adjacent frame image of the second image, and when the second camera captures the third image, the first camera does not capture an image;

[0139] An output module 30, configured to output a target image according to the first fused image and the second fused image;

[0140] The first camera is a color camera, and the second camera is a black and white camera; the frame rate of the first camera is lower than the frame rate of the second camera; and the sensitivity of the second camera is N times that of the first camera, where N≥2.

[0141] Thus, in the embodiment of the present application, when shooting, the electronic device first fuses the first image captured by the first camera (color camera) and the second image captured by the second camera (black and white camera) to obtain a first fused image. Among them, the sensitivity of the second camera is twice or more than twice the sensitivity of the first camera. Therefore, the second image has the advantages of high brightness, more details and low noise relative to the first image, so that in the fusion process, the brightness information of the second image is fused with the color channel information of the first image to obtain the first fused image. Furthermore, the frame rate of the second camera is greater than the frame rate of the first camera. After the second camera captures the adjacent frame image of the second image, that is, the third image, the color channel information of the first fused image is fused with the brightness information of the third image to obtain the second fused image. Finally, based on the first fused image and the second fused image, the target image is output. It can be seen that based on the embodiment of the present application, the output target image has the characteristics of high brightness, more details and low noise on the basis of rich colors, which effectively improves the quality of the captured image. In addition, in this embodiment, based on the different frame rates of the two cameras, compared with the case where the frame rates are the same, the step of fusing two frames of images captured synchronously can be reduced, thereby controlling the power consumption increment.

[0142] Optionally, the first fusion module 10 includes:

[0143] A first setting unit, configured to set the color channel information of the first image as the color channel information of the second image according to a first matching relationship between the pixel points of the first image and the pixel points of the second image;

[0144] The adjustment unit is used to adjust the color channel information of the second image according to the brightness information of the second image and the brightness information of the first image to obtain a first fused image.

[0145] Optionally, the device further comprises:

[0146] A second setting unit is used to set the color channel information of the first pixel point according to the color channel information of the second pixel point in the first area when the color channel information of the first pixel point in the first area of ​​the second image is not set;

[0147] There is a first correlation relationship between the brightness information of the first pixel and the brightness information of the second pixel.

[0148] Optionally, the output module 30 includes:

[0149] The output unit is used to perform high-dynamic synthesis on the first fused image and the second fused image to output a target image.

[0150] Optionally, the device further comprises:

[0151] A determination module, configured to determine the frame rate of the first camera according to the shooting scene information when the frame rate of the second camera has been determined;

[0152] The change frequency information of the shooting scene information is positively correlated with the frame rate of the first camera.

[0153] The shooting device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not make specific limitations.

[0154] The shooting device of the embodiment of the present application may be a device with an action system. The action system may be an Android action system, an iOS action system, or other possible action systems, which are not specifically limited in the embodiment of the present application.

[0155] The shooting device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0156] Alternatively, if Figure 8 As shown, an embodiment of the present application also provides an electronic device 100, including a processor 101, a memory 102, and a program or instruction stored in the memory 102 and executable on the processor 101. When the program or instruction is executed by the processor 101, each step of any of the above-mentioned shooting method embodiments is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0157] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0158] Fig. 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0159] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0160] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0161] The processor 1010 is used to fuse the color channel information of a first image captured by a first camera of the electronic device with the brightness information of a second image captured by a second camera of the electronic device to obtain a first fused image; fuse the color channel information of the first fused image with the brightness information of a third image captured by the second camera to obtain a second fused image; wherein the third image is an adjacent frame image of the second image, and at the moment when the second camera captures the third image, the first camera does not capture an image; output a target image based on the first fused image and the second fused image; wherein the first camera is a color camera and the second camera is a black and white camera; the frame rate of the first camera is less than the frame rate of the second camera; the sensitivity of the second camera is N times the sensitivity of the first camera, and N≥2.

[0162] Thus, in the embodiment of the present application, when shooting, the electronic device 1000 first fuses the first image captured by the first camera (color camera) and the second image captured by the second camera (black and white camera) to obtain a first fused image. Among them, the sensitivity of the second camera is twice or more than twice the sensitivity of the first camera. Therefore, the second image has the advantages of high brightness, more details and low noise relative to the first image, so that in the fusion process, the brightness information of the second image is fused with the color channel information of the first image to obtain the first fused image. Furthermore, the frame rate of the second camera is greater than the frame rate of the first camera. After the second camera captures the adjacent frame image of the second image, that is, the third image, the color channel information of the first fused image is fused with the brightness information of the third image to obtain the second fused image. Finally, based on the first fused image and the second fused image, the target image is output. It can be seen that based on the embodiment of the present application, the output target image has the characteristics of high brightness, more details and low noise on the basis of rich colors, which effectively improves the quality of the captured image. In addition, in this embodiment, based on the different frame rates of the two cameras, compared with the case where the frame rates are the same, the step of fusing two frames of images captured synchronously can be reduced, thereby controlling the power consumption increment.

[0163] Optionally, the processor 1010 is further used to set the color channel information of the first image as the color channel information of the second image according to a first matching relationship between pixel points of the first image and pixel points of the second image; and adjust the color channel information of the second image according to brightness information of the second image and the brightness information of the first image to obtain the first fused image.

[0164] Optionally, the processor 1010 is also used to set the color channel information of the first pixel point according to the color channel information of the second pixel point in the first area when the color channel information of the first pixel point is not set; wherein there is a first correlation relationship between the brightness information of the first pixel point and the brightness information of the second pixel point.

[0165] Optionally, the processor 1010 is further configured to perform high-dynamic synthesis on the first fused image and the second fused image, and output the target image.

[0166] Optionally, the processor 1010 is further used to determine the frame rate of the first camera according to the shooting scene information when the frame rate of the second camera has been determined; wherein the change frequency information of the shooting scene information is positively correlated with the frame rate of the first camera.

[0167] In summary, the electronic device 1000 of the present application adopts a side-by-side structure of black and white and color cameras, and the sensitivity of the black and white camera is twice or more than that of the color camera, so as to ensure that the details and brightness of the black and white image are better than the color image under the same exposure. Among them, the black and white camera collects at a high frame rate (such as 30fps), and the color camera collects at a low frame rate combined with a dynamic frame rate. The dynamic frame rate of the color camera is adjusted with the motion detection result of the black and white camera, so as to control the power consumption. In addition, for the dark light environment, the black and white camera has a high light transmittance, and the black and white image is used as a reference. When there is a color image output, the color channel information is migrated to the black and white image after registration, and the fused image is obtained after color propagation, color noise suppression, color restoration and other processing; when there is no color image, the front and back black and white images are continuously registered, and the color channel information is migrated from the fused image to the new black and white image, and a similar fusion operation is performed to achieve the purpose of continuously outputting full-color images, and the image quality of the output image has the characteristics of more details and low noise. In addition, the different frame rates of the two cameras can reduce the frequency of dual-camera alignment, thereby controlling the power consumption increase and improving the battery life of the electronic device 1000.

[0168] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video image obtained by the image capture device (such as a camera) in the video image capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and an action rod, which will not be repeated here. The memory 1009 can be used to store software programs and various data, including but not limited to applications and action systems. The processor 1010 may integrate an application processor and a modem processor, wherein the application processor mainly processes the action system, user pages and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 1010.

[0169] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0170] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0171] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned shooting method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0172] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0173] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned shooting method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0174] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0175] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned shooting method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0176] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0177] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0178] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A shooting method, characterized in that: The method comprises: Fusing color channel information of a first image captured by a first camera of an electronic device with brightness information of a second image captured by a second camera of the electronic device to obtain a first fused image; Fusing the color channel information of the first fused image with the brightness information of the third image captured by the second camera to obtain a second fused image; wherein the third image is an adjacent frame image of the second image, and when the second camera captures the third image, the first camera does not capture an image; outputting a target image according to the first fused image and the second fused image; Among them, the first camera is a color camera, and the second camera is a black and white camera; the frame rate of the first camera is lower than the frame rate of the second camera; the sensitivity of the second camera is N times the sensitivity of the first camera, N≥2.

2. The method according to claim 1, characterized in that The step of fusing the color channel information of the first image acquired by the first camera of the electronic device with the brightness information of the second image acquired by the second camera of the electronic device to obtain a first fused image includes: According to a first matching relationship between pixels of the first image and pixels of the second image, setting color channel information of the first image as color channel information of the second image; According to the brightness information of the second image and the brightness information of the first image, the color channel information of the second image is adjusted to obtain the first fused image.

3. The method according to claim 2, characterized in that After setting the color channel information of the first image as the color channel information of the second image and before obtaining the first fused image, the method further includes: When the color channel information of the first pixel point in the first area of ​​the second image is not set, setting the color channel information of the first pixel point according to the color channel information of the second pixel point in the first area; There is a first correlation relationship between the brightness information of the first pixel and the brightness information of the second pixel.

4. The method according to claim 1, characterized in that: The step of outputting a target image according to the first fused image and the second fused image comprises: The first fused image and the second fused image are synthesized with high dynamic range to output the target image.

5. The method according to claim 1, characterized in that Before fusing the color channel information of the first image acquired by the first camera of the electronic device with the brightness information of the second image acquired by the second camera of the electronic device to obtain the first fused image, the method further includes: When the frame rate of the second camera has been determined, determining the frame rate of the first camera according to the shooting scene information; The change frequency information of the shooting scene information is positively correlated with the frame rate of the first camera.

6. A photographing device, characterized in that: The device comprises: A first fusion module, configured to fuse color channel information of a first image captured by a first camera of an electronic device with brightness information of a second image captured by a second camera of the electronic device to obtain a first fused image; a second fusion module, configured to fuse the color channel information of the first fused image with the brightness information of the third image captured by the second camera, to obtain a second fused image; wherein the third image is an adjacent frame image of the second image, and when the second camera captures the third image, the first camera does not capture an image; An output module, configured to output a target image according to the first fused image and the second fused image; Among them, the first camera is a color camera, and the second camera is a black and white camera; the frame rate of the first camera is lower than the frame rate of the second camera; the sensitivity of the second camera is N times the sensitivity of the first camera, N≥2.

7. The device according to claim 6, characterized in that The first fusion module includes: A first setting unit, configured to set the color channel information of the first image as the color channel information of the second image according to a first matching relationship between pixels of the first image and pixels of the second image; An adjustment unit is used to adjust the color channel information of the second image according to the brightness information of the second image and the brightness information of the first image to obtain the first fused image.

8. The device according to claim 7, characterized in that The device also includes: a second setting unit, configured to set the color channel information of the first pixel point according to the color channel information of the second pixel point in the first area when the color channel information of the first pixel point in the first area of ​​the second image is not set; There is a first correlation relationship between the brightness information of the first pixel and the brightness information of the second pixel.

9. The device according to claim 6, characterized in that The output module comprises: An output unit is used to perform high-dynamic synthesis on the first fused image and the second fused image, and output the target image.

10. The device according to claim 6, characterized in that The device also includes: a determination module, configured to determine the frame rate of the first camera according to shooting scene information when the frame rate of the second camera has been determined; The change frequency information of the shooting scene information is positively correlated with the frame rate of the first camera.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the shooting method according to any one of claims 1 to 5 are implemented.

12. A chip, characterized in that: The chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the shooting method according to any one of claims 1 to 5.