Image generation method, device and electronic device

Abstract

The present application discloses an image generation method, device, and electronic device, belonging to the field of communication technology. The camera module in the electronic device includes: N monochrome cameras arranged in parallel, where N is a positive integer greater than 2; the monochrome cameras include a pixel array, the pixel array includes a plurality of minimal repeating units, the minimal repeating unit includes a first pixel and a second pixel, the first pixel is a first visible light pixel or an invisible light pixel, and the second pixel is a second visible light pixel; the wavelength band of light transmitted through the light-transmitting portion of the first visible light pixel includes a wavelength band ranging from blue to red, and the wavelength band of light transmitted through the light-transmitting portion of the second visible light pixel is narrower than the wavelength band of light transmitted through the light-transmitting portion of the first visible light pixel.

Description

Technical Field

[0001] The present application belongs to the field of communication technology, and specifically relates to an image generation method, device and electronic device. Background Art

[0002] Currently, consumer electronic products such as mobile phones are equipped with front and rear camera modules, and the number of camera modules has evolved from a single camera to the current multiple cameras, and the photography method has also evolved from single-camera photography to binocular or even more camera modules working together to achieve photography or video. Camera modules are divided into main camera, telephoto, wide-angle, ultra-wide-angle, black and white, macro, telephoto macro, time-of-flight (TOF), etc. in terms of function. Among them, as far as visible light modules are concerned, they can be divided into color and black and white. The color acquisition method of a common color camera module is to use a spectroscopic prism and a dichroic filter to divide the incident light into three beams of red, green, and blue, and to form images on three image sensor arrays respectively. Because it is a coaxial optical path, the three red, green, and blue (RGB) images can be aligned through internal reference calibration, and then fused into an image in which each pixel has complete color information.

[0003] However, due to the use of a beam splitter prism, the size of the beam splitter prism is compressed, resulting in a limited sensor area and low image quality. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide an image generation method, device and electronic device that can solve the problem of low image quality in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a camera module, comprising:

[0006] N monochrome cameras arranged in parallel, where N is a positive integer greater than 2;

[0007] The monochrome camera includes a pixel array, the pixel array includes a plurality of minimal repeating units, the minimal repeating unit includes a first pixel and a second pixel, the first pixel is a first visible light pixel or an invisible light pixel, and the second pixel is a second visible light pixel;

[0008] Among them, the wavelength band of light transmitted through the light-transmitting part of the first visible light pixel includes a wavelength band within the range from blue to red, and the wavelength band of light transmitted through the light-transmitting part of the second visible light pixel is narrower than the wavelength band of light transmitted through the light-transmitting part of the first visible light pixel.

[0009] In a second aspect, an embodiment of the present application provides an electronic device comprising the camera module as described above.

[0010] In a third aspect, an embodiment of the present application provides an image generation method, which is applied to the above-mentioned electronic device, and the method includes:

[0011] Acquire N first images through N monochrome cameras;

[0012] Taking one of the N first images as a reference, performing calculations on the other first images respectively to obtain N-1 offset pixel matrices;

[0013] The pixels of the N first images are superimposed according to the N-1 offset pixel matrices to obtain a target image.

[0014] In a fourth aspect, an embodiment of the present application provides an image generating device, which is applied to the above-mentioned electronic device, and the image generating device includes:

[0015] A first acquisition module, configured to acquire N first images through N monochrome cameras;

[0016] A first processing module is configured to use one of the N first images as a reference and calculate the other first images using a stereo registration algorithm to obtain N-1 offset pixel matrices;

[0017] The second processing module is configured to perform pixel superposition on the pixels of the N first images according to the N-1 offset pixel matrices to obtain a target image.

[0018] In a fifth aspect, an embodiment of the present application provides an electronic device comprising a processor, a memory, and a camera module as described in the first aspect, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the third aspect are implemented.

[0019] In a sixth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the third aspect are implemented.

[0020] In the seventh aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method described in the third aspect.

[0021] In an eighth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the third aspect.

[0022] In an embodiment of the present application, since the camera module is composed of N monochrome cameras arranged in parallel, the thickness of the camera module and the Z-direction height can be reduced; and each monochrome camera includes a pixel array, the pixel array includes multiple minimum repeating units, the minimum repeating unit includes a first pixel and a second pixel, the first pixel is a first visible light pixel or an invisible light pixel, and the second pixel is a second visible light pixel. Since the pixel arrays of different monochrome cameras contain different second pixels, the role of the dichroic prism is replaced, and the problem of limited sensor area due to the size of the compressed dichroic prism is solved; and a first pixel is added to the pixel array of each monochrome camera, and the pixel alignment between the monochrome cameras is assisted by the first pixel, so that a finer color image can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of the structure of the camera module provided in the embodiment of the present application;

[0024] Figure 2 This is one of the pixel arrangement diagrams of the color filter array provided in an embodiment of the present application;

[0025] Figure 3 This is the second pixel arrangement diagram of the color filter array provided in an embodiment of the present application;

[0026] Figure 4 This is the third pixel arrangement diagram of the color filter array provided in an embodiment of the present application;

[0027] Figure 5 This is the fourth pixel arrangement diagram of the color filter array provided in an embodiment of the present application;

[0028] Figure 6 is a flowchart of the image generation method provided in an embodiment of the present application;

[0029] Figure 7 This is one of the pixel arrangement diagrams of the grayscale image provided in the embodiment of the present application;

[0030] Figure 8 This is the second schematic diagram of the pixel arrangement of the grayscale image provided in the embodiment of the present application;

[0031] Figure 9 This is the third schematic diagram of the pixel arrangement of the grayscale image provided in the embodiment of the present application.

[0032] Figure 10 is a structural diagram of an image generating device provided in an embodiment of the present application;

[0033] Figure 11 This is a structural block diagram of an electronic device provided in an embodiment of the present application;

[0034] Figure 12 This is a structural block diagram of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0036] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0037] Currently, common color camera modules use a repetitive Bayer pattern and its variations based on a single imaging array, such as Red-Green-Green-Blue (RGGB), Red-Yellow-Yellow-Blue (RYYB), and Red-Green-Blue-White (RGBW), which adds pixels sensing visible light. Because these images require demosaicing to restore an image with complete color information at every pixel, demosaicing can cause color confusion in high-frequency color details in scenes with multiple textures, high-frequency foregrounds, and different background colors.

[0038] There are two main reasons:

[0039] First, because the Bayer pattern is used to capture colors on the imaging pixel array, only one color can be captured at each pixel. The color of each pixel is restored by using the colors of the surrounding pixels through the demosaic method. This is an interpolation process, which is inevitable in scenes with multiple textures, high-frequency foregrounds, and different-color backgrounds. Color confusion may occur.

[0040] Second, there is crosstalk between adjacent pixels in the imaging pixel array, including two types of crosstalk: optically, wide-angle incident light crosstalks to adjacent pixels; electrically, crosstalk between charge collection wells between pixels.

[0041] Therefore, in order to solve the above problems, the embodiment of the present application proposes a camera module. The camera module provided by the embodiment of the present application is described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0042] An embodiment of the present application provides a camera module, which includes:

[0043] N monochrome cameras arranged in parallel, where N is a positive integer greater than 2;

[0044] The monochrome camera includes a pixel array, the pixel array includes a plurality of minimal repeating units, the minimal repeating unit includes a first pixel and a second pixel, the first pixel is a first visible light pixel or an invisible light pixel, and the second pixel is a second visible light pixel;

[0045] Among them, the wavelength band of light transmitted through the light-transmitting part of the first visible light pixel includes a wavelength band within the range from blue to red, and the wavelength band of light transmitted through the light-transmitting part of the second visible light pixel is narrower than the wavelength band of light transmitted through the light-transmitting part of the first visible light pixel.

[0046] Specifically, in order to avoid the lack of common information for registration between monochrome cameras in scenes with monochromatic light sources or monochrome areas, a first pixel is added to each monochrome camera to transmit visible light or invisible light. It can be used as a stereo matching reference between single monochrome cameras to assist in pixel registration between monochrome camera images. If the first pixel is a first visible light pixel, then under low-light conditions, since the first visible light pixel has a higher transmittance, it can capture more texture details of the color image. If the first pixel is an invisible light pixel, the monochrome camera also needs to include a fill light device. Under low-light conditions, fill light can be provided by the fill light device, which can provide higher signal-to-noise ratio support for stereo matching and provide more details.

[0047] The value of N is a positive integer greater than 2. The following describes this application solution by taking the value of N as 3 as an example:

[0048] If the value of N is 3, then Figure 1 The figure shows a schematic structural diagram of the camera module 11, wherein the camera module 11 includes three monochrome cameras 111 arranged in parallel. This arrangement can reduce the thickness of the camera module 11, thereby reducing the Z-direction height.

[0049] It should be noted that Figure 1The number of monochrome cameras in is only an example, and the value of N can also be other values. Figure 1 The arrangement of the three monochrome cameras is only an example and is not limited to a horizontal side-by-side structure.

[0050] The pixel array is periodically arranged in a monochrome pixel pattern (i.e., a minimum repeating unit). The monochrome pixel pattern includes four 2*2 pixel units, each consisting of a first pixel and a second pixel, wherein there is at least one first pixel and at least two second pixels. The first pixel is a first visible light pixel or an invisible light pixel. The first visible light pixel accepts a spectral wavelength band ranging from blue to red light, while the invisible light pixel accepts a wavelength band of near infrared light. The second pixel accepts a spectral wavelength narrower than the wavelength band of the first visible light pixel, and may preferably be one of the red, green, and blue wavelength bands.

[0051] If three monochrome cameras primarily transmit red light R, green light G, and blue light B, respectively, then the monochrome camera primarily transmitting red light R is a red light camera, the monochrome camera primarily transmitting green light G is a green light camera, and the monochrome camera primarily transmitting blue light B is a blue light camera. The difference between the three monochrome cameras that transmit different colors of visible light is that the second pixel in each monochrome camera accepts a different spectral wavelength band.

[0052] Each monochrome camera also includes: a motor, a motor driver, an infrared (IR) filter, a photosensitive pixel array, etc.

[0053] For example: Figure 2 As shown in , if each minimum repeating unit contains at least 2 second pixels and 1 first pixel in a 2*2 pixel arrangement. If the pixel arrangement of the minimum repeating unit in the red light camera contains 3 second pixels and 1 first pixel, then the pixel arrangement format of the minimum repeating unit of the red light camera is as follows: Figure 2 As shown, R represents the second red pixel and W represents the first pixel. If the pixel arrangement of the minimum repeating unit in the red light camera includes 2 second pixels and 2 first pixels, the pixel arrangement format of the minimum repeating unit of the red light camera is as follows: Figure 3 shown.

[0054] Similarly, if the pixel arrangement of the minimum repeating unit in the green light camera includes 3 second pixels and 1 first pixel, the pixel arrangement format of the minimum repeating unit of the green light camera is as follows: Figure 4 As shown, G represents the second green pixel, and W represents the first pixel.

[0055] Similarly, if the pixel arrangement of the minimum repeating unit in the blue light camera includes 3 second pixels and 1 first pixel, the pixel arrangement format of the minimum repeating unit of the blue light camera is as follows: Figure 5 As shown, B represents the second blue pixel, and W represents the first pixel.

[0056] In an embodiment of the present application, since the camera module is composed of N monochrome cameras arranged in parallel, the thickness of the camera module and the Z-direction height can be reduced; and each monochrome camera includes a pixel array, the pixel array includes multiple minimum repeating units, the minimum repeating unit includes a first pixel and a second pixel, the first pixel is a first visible light pixel or an invisible light pixel, and the second pixel is a second visible light pixel. Since the pixel arrays of different monochrome cameras contain different second pixels, the role of the dichroic prism is replaced, and the problem of limited sensor area due to the size of the compressed dichroic prism is solved; and a first pixel is added to the pixel array of each monochrome camera, and the pixel alignment between the monochrome cameras is assisted by the first pixel, so that a finer color image can be obtained.

[0057] As an optional embodiment, the parallel arrangement includes: parallel arrangement with equal intervals in the horizontal direction, or parallel arrangement with equal intervals in the vertical direction.

[0058] Specifically, the N monochrome cameras can be arranged in a horizontal direction, and can be further arranged so that the distance between each two adjacent monochrome cameras is equal, thereby reducing the thickness of the camera module and further reducing the Z-direction height. Figure 1 As shown in FIG, when N is 3, three monochrome cameras are arranged in parallel at equal intervals in the horizontal direction.

[0059] An embodiment of the present invention further provides an electronic device, comprising the camera module as described in any of the above embodiments.

[0060] The electronic device may further include a central processing unit, a display screen, a memory, an imaging device, and the like.

[0061] The electronic device may be a mobile phone, tablet computer, smart TV or other equipment.

[0062] The central processing unit is connected to the imaging device, sets up the imaging device and starts the acquisition instruction; the central processing unit controls the image output by the imaging device to be saved in the memory; and the central processing unit controls the display screen to display the image output by the imaging device.

[0063] like Figure 6 As shown, the embodiment of the present application provides an image generation method, which can be applied to the above electronic device. The above image generation method can specifically include the following steps:

[0064] Step 601: Acquire N first images through N monochrome cameras.

[0065] Specifically, N monochrome cameras are used to capture images respectively to obtain N captured images. During the image capture process, the exposure control parameters and focal length parameters of one of the monochrome cameras can be used as a reference, and the other monochrome cameras use the same exposure control parameters and focal length parameters.

[0066] The N captured images are preprocessed through at least N image preprocessing channels to obtain N first images, wherein the preprocessing includes at least one of bad pixel removal processing, noise reduction processing, and lens correction processing.

[0067] Step 602: Using one of the N first images as a reference, perform calculations on the other first images to obtain N-1 offset pixel matrices.

[0068] Specifically, one of the N first images is used as a reference benchmark, and an offset pixel matrix of each of the other first images relative to the first image used as the reference benchmark is calculated using an algorithm such as stereo registration.

[0069] For example, if N is 3 and the three first images are C1, C2, and C3, and C2 is used as a reference, a stereo registration algorithm is used to calculate the offset pixel matrix of C1 relative to C2, and a stereo registration algorithm is used to calculate the offset pixel matrix of C3 relative to C2.

[0070] Step 603: Superimpose the pixels of the N first images according to the N-1 offset pixel matrices to obtain a target image.

[0071] Specifically, in other first images, according to the offset pixel matrix of each first image relative to the reference benchmark, the pixels of the first image are superimposed on the pixels corresponding to the reference benchmark, thereby obtaining pixels after superposition of multi-color pixels, thereby obtaining the target image, that is, one pixel position of the target image has N color information.

[0072] In the above embodiment of the present application, N first images are acquired through N monochrome cameras. One of the N first images is used as a reference, and the other first images are calculated using a stereo registration algorithm to obtain N-1 offset pixel matrices. Based on the N-1 offset pixel matrices, the pixels of the N first images are pixel-overlaid to obtain a target image. Specifically, the pixel stereo registration algorithm is used to perform stereo registration between the monochrome cameras to obtain an offset pixel matrix. The first images of the N monochrome cameras are then merged based on the offset pixel matrix, resulting in a more refined color image. Furthermore, the N monochrome cameras increase the amount of light entering, resulting in a better signal-to-noise ratio.

[0073] As an optional embodiment, before step 603 superimposes the pixels of the N first images according to the N-1 offset pixel matrices to obtain the target image, the method may further include:

[0074] For each of the N first images, perform interpolation processing on a first pixel in the first image to obtain a first interpolation result;

[0075] The first pixel is replaced by the first interpolation result to obtain a monochrome image.

[0076] Furthermore, step 603 superimposes the pixels of the N first images according to the N-1 offset pixel matrices to obtain a target image, which specifically includes:

[0077] According to the N-1 offset pixel matrices, the N monochrome images are pixel-superimposed to obtain a target image.

[0078] Specifically, for each of the N first images, interpolation processing is performed on the first pixel in the first image. Specifically, a first interpolation result is calculated by averaging the second pixels surrounding the first pixel. The position of the first pixel is filled with the first interpolation result. After all first pixel positions are filled, a monochrome image is obtained. Based on the offset pixel matrix of the monochrome image corresponding to each other first image, the pixels of the monochrome image are superimposed on the corresponding pixel positions of the monochrome image of the reference image. After the superposition is completed, a multi-color target image is obtained.

[0079] As an optional embodiment, the step of performing pixel superposition on the N monochrome images according to the N-1 offset pixel matrices to obtain the target image specifically includes:

[0080] According to the N-1 offset pixel matrices, one of the N monochrome images is used as a reference image, and pixels in other monochrome images are superimposed on corresponding pixels of the reference image to obtain a multi-color image;

[0081] The multi-color image is fused with the reference image to obtain a target image.

[0082] Specifically, if the first pixel is a first visible light pixel, for each first image, the first visible light pixels of the first image are extracted to form a grayscale image, thereby obtaining N grayscale images. The grayscale image corresponding to one of the N monochrome images is used as a reference image. Using a stereo registration algorithm, the offset pixel matrix of each grayscale image relative to the reference image is calculated in the other grayscale images, thereby obtaining N-1 offset pixel matrices. Preferably, the first image captured by the middle monochrome camera among the N monochrome cameras is selected as the reference baseline.

[0083] For example: the number of the first image is 3, respectively as Figure 2 The first image C1 shown, Figure 4 The first image C2 shown and Figure 5 The first image C3 shown. Figure 7 As shown, Figure 2 The grayscale image W1 is obtained after extracting the first visible light pixel. Figure 8 As shown, Figure 4 The grayscale image W2 is obtained after extracting the first visible light pixel. Figure 9 As shown, Figure 5 After extracting the first visible light pixel, the grayscale image W3 is obtained. Assuming that the three monochrome cameras are aligned in the row direction, if W2 is the reference image, it is necessary to calculate the offset pixel matrix d1 of W1 relative to W2 and the offset pixel matrix d3 of W3 relative to W2.

[0084] Based on the offset pixel matrix of the monochrome image corresponding to each other grayscale image, the pixels of this monochrome image are superimposed on the corresponding pixel positions of the monochrome image of the reference image. After the superposition is completed, each pixel position has multiple color information, thus obtaining a multi-color image. By fusing the multi-color image with the reference image, the target image with greater detail and a higher signal-to-noise ratio can be obtained in low-light conditions.

[0085] As an optional embodiment, the step of fusing the multi-color image with the reference image to obtain the target image specifically includes:

[0086] Processing the multi-color image and the reference image into images with the same resolution;

[0087] Converting the multi-color image with the same resolution as the reference image into a second image in YUV format;

[0088] A reference image having the same resolution as the multi-color image is fused with the Y component of the second image to obtain a target image.

[0089] Specifically, if the multi-color image is a full-resolution image and the reference image is a 1 / 2 resolution or 1 / 4 resolution image, the reference image can be upscaled from its original 1 / 2 resolution or 1 / 4 resolution to full resolution through super-resolution processing, thereby obtaining a reference image with the same resolution as the multi-color image. Alternatively, the resolution of the color image can be reduced to obtain a multi-color image with the same resolution as the reference image.

[0090] If the reference image is enlarged to a full-resolution image, the full-resolution multi-color image is format-converted to obtain a second image in YUV format, and the Y component of the second image in YUV format is fused with the full-resolution reference image to obtain a target image with lower brightness noise.

[0091] Furthermore, the above-mentioned fusion processing of the reference image having the same resolution as the multi-color image and the Y component of the second image includes but is not limited to the following methods:

[0092] Replacing the Y component with a reference image having the same resolution as the multi-color image;

[0093] Alternatively, a Gaussian filter is used to decompose the base layer and detail layer of the Y component, and decompose the base layer and detail layer of a reference image having the same resolution as the multi-color image;

[0094] The detail layer of the Y component is replaced by the detail layer of a reference image having the same resolution as the multi-color image.

[0095] Specifically, one way may be to replace the Y component with a reference image; another way may be to replace the detail layer of the Y component with a detail layer of the reference image.

[0096] The pixel superposition process is described in detail below through a specific embodiment:

[0097] For example: Figures 7 to 9 As shown, if W2 is the reference image, the offset pixel matrix of W1 relative to W2 is d1, and the offset pixel matrix of W3 relative to W2 is d3. Through d1, the pixel C2(i, j) corresponding to C1(i, j1) on C1 is obtained, that is, C1(i, j1) is C1(i, j+d1 ij ); Get the pixel C2(i, j) corresponding to C3(i, j3) on C3 through d3, that is, C3(i, j3) is C3(i, j+d3 ij ). C1(i, j+d1 ij ) and C3(i, j+d3 ij) is superimposed on pixel C2(i, j). Then, position C2(i, j) has three color information. After superposition, the pixel information contained in position C2(i, j) is C1(i, j+d1ij), C2(i, j), and C3(i, j+d3ij). Where i represents the horizontal coordinate of the pixel and j represents the vertical coordinate of the pixel.

[0098] like Figure 2 、 Figures 4 to 9 As shown, x1, x2, x3, and x4 are the coordinate indices of the image in the X direction (column direction), that is, the x1 column, the x2 column, the x3 column, the x4 column, etc. y1, y2, y3, and y4 are the coordinate indices of the image in the Y direction (row direction), that is, the y1 row, the y2 row, the y3 row, and the y4 row. The position of the filled W is the matching point pair found by stereo registration, that is, the pixel with coordinates (y1, x2) in W2 corresponds to the coordinates (y1, x1) in the W1 map and (y1, x3) in the W3 map. The offset pixels of the corresponding disparity maps d1 and d3 at (y1, x2) are:

[0099] d1(y1, x2) = x1 - x2

[0100] d3(y1, x2)=x3-x2

[0101] The monochrome images corresponding to W1, W2, and W3 are A1, A2, and A3 respectively. Since the resolutions of W1, W2, and W3 are half of those of A1, A2, and A3, it is necessary to multiply the pixel coordinates and the offset pixels by 2 to obtain the offset pixels of A1 and A3 relative to A2.

[0102] Taking the position of W2 at (y1, x2) as an example, when it corresponds to the image size of A2, the corresponding pixel coordinates become (2*y1, 2*x2) and the corresponding offset pixels are also magnified to twice the original:

[0103] D1(2*y1, 2*x2) is 2*(x1-x2)

[0104] D3(2*y1, 2*x2) is 2*(x3-x2)

[0105] Pixels at odd coordinate positions, such as (2*y1,2*x2+1), (2*y1+1,2*x2), (2*y1+1,2*x2+1), can be obtained by interpolating the offset pixels at the even coordinate positions in D1 and D3 above.

[0106] Therefore, for the pixel coordinate (i, j) in A2, we can obtain two other color pixel values ​​from the A1 and A3 images respectively according to the offset pixels D1 and D3, and superimpose them to obtain the three color pixel values ​​at the coordinate position.

[0107] It should be noted that if the three monochrome sensors are not aligned in the row direction, it is also necessary to characterize the offset pixels in the y direction.

[0108] In summary, the above embodiment of the present application, through the parallel and side-by-side structure of N monochrome cameras and the addition of a first pixel in each camera, can assist in achieving pixel registration between monochrome camera images, thereby obtaining a more refined color image; and utilize pixel registration to merge the monochrome images of N monochrome cameras, thereby obtaining a more refined multi-color image; under low-light conditions, the multi-color image and the grayscale image are fused to obtain a target image with more details and a higher signal-to-noise ratio under low-light conditions. The above scheme not only improves color resolution, but also can obtain more accurate and detailed image information than a monocular camera. It utilizes N monochrome cameras to achieve more refined color reproduction through color separation, and utilizes N monochrome cameras to achieve the purpose of binocular ranging. For the online live broadcast mode of portraits or exhibits such as cultural relics, it can more prominently present the color details of the foreground portraits or exhibits.

[0109] The image generation method provided in the embodiment of the present application can be executed by an image generation device. In the embodiment of the present application, the image generation device provided in the embodiment of the present application is described by taking the image generation method executed by the image generation device as an example.

[0110] like Figure 10 As shown, an embodiment of the present application further provides an image generating device, which is applied to the above-mentioned electronic device, and the image generating device includes:

[0111] A first acquisition module 801 is configured to acquire N first images using N monochrome cameras;

[0112] A first processing module 802 is configured to use one of the N first images as a reference and perform calculations on the other first images to obtain N-1 offset pixel matrices;

[0113] The second processing module 803 is configured to perform pixel superposition on the pixels of the N first images according to the N-1 offset pixel matrices to obtain a target image.

[0114] Optionally, the device further includes:

[0115] a third processing module, configured to perform interpolation processing on a first pixel in each of the N first images to obtain a first interpolation result;

[0116] a fourth processing module, configured to replace the first pixel with the first interpolation result to obtain a monochrome image;

[0117] The second processing module 803 is specifically configured to:

[0118] According to the N-1 offset pixel matrices, the N monochrome images are pixel-superimposed to obtain a target image.

[0119] Optionally, when the second processing module 803 performs pixel superposition on the N monochrome images according to the N-1 offset pixel matrices to obtain the target image, it is specifically configured to:

[0120] According to the N-1 offset pixel matrices, one of the N monochrome images is used as a reference image, and pixels in other monochrome images are superimposed on corresponding pixels of the reference image to obtain a multi-color image;

[0121] The multi-color image is fused with the reference image to obtain a target image.

[0122] Optionally, when the second processing module 803 fuses the multi-color image with the reference image to obtain the target image, it is specifically configured to:

[0123] Processing the multi-color image and the reference image into images with the same resolution;

[0124] Converting the multi-color image with the same resolution as the reference image into a second image in YUV format;

[0125] A reference image having the same resolution as the multi-color image is fused with the Y component of the second image to obtain a target image.

[0126] Optionally, the fusing of a reference image having the same resolution as the multi-color image with a Y component in the second image includes:

[0127] Replacing the Y component with a reference image having the same resolution as the multi-color image;

[0128] Alternatively, a Gaussian filter is used to decompose the base layer and detail layer of the Y component, and decompose the base layer and detail layer of a reference image having the same resolution as the multi-color image;

[0129] The detail layer of the Y component is replaced by the detail layer of a reference image having the same resolution as the multi-color image.

[0130] In summary, the above embodiment of the present application, through the parallel and side-by-side structure of N monochrome cameras and the addition of a first pixel in each camera, can assist in achieving pixel registration between monochrome camera images, thereby obtaining a more refined color image; and utilize pixel registration to merge the monochrome images of N monochrome cameras, thereby obtaining a more refined multi-color image; under low-light conditions, the multi-color image and the grayscale image are fused to obtain a target image with more details and a higher signal-to-noise ratio under low-light conditions. The above scheme not only improves color resolution, but also can obtain more accurate and detailed image information than a monocular camera. It utilizes N monochrome cameras to achieve more refined color reproduction through color separation, and utilizes N monochrome cameras to achieve the purpose of binocular ranging. For the online live broadcast mode of portraits or exhibits such as cultural relics, it can more prominently present the color details of the foreground portraits or exhibits.

[0131] The image generating device in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or a device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc., and the embodiment of the present application does not specifically limit it.

[0132] The image generation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0133] The image generation device provided in the embodiment of the present application can achieve Figures 2 to 9 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0134] Alternatively, as Figure 11 As shown, an embodiment of the present application further provides an electronic device 900, including a processor 901 and a memory 902, wherein the memory 902 stores a program or instruction that can be run on the processor 901, and when the program or instruction is executed by the processor 901, the various steps of the above-mentioned image generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0135] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0136] Figure 12 A schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0137] The electronic device 1000 includes but is not limited to components such as a radio frequency unit 1001 , a network module 1002 , an audio output unit 1003 , an input unit 1004 , a sensor 1005 , a display unit 1006 , a user input unit 1007 , an interface unit 1008 , a memory 1009 , and a processor 1010 .

[0138] Those skilled in the art will understand that the electronic device 1000 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 12 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0139] The input unit 1004 acquires N first images through N monochrome cameras;

[0140] The processor 1010 is configured to use one of the N first images as a reference and perform calculations on the other first images to obtain N-1 offset pixel matrices.

[0141] The pixels of the N first images are superimposed according to the N-1 offset pixel matrices to obtain a target image.

[0142] In the above embodiment of the present application, N first images are acquired through N monochrome cameras. One of the N first images is used as a reference, and the other first images are calculated using a stereo registration algorithm to obtain N-1 offset pixel matrices. Based on the N-1 offset pixel matrices, the pixels of the N first images are pixel-overlaid to obtain a target image. Specifically, the pixel stereo registration algorithm is used to perform stereo registration between the monochrome cameras to obtain an offset pixel matrix. The first images of the N monochrome cameras are then merged based on the offset pixel matrix, resulting in a more refined color image. Furthermore, the N monochrome cameras increase the amount of light entering, resulting in a better signal-to-noise ratio.

[0143] Optionally, before performing pixel superposition on the pixels of the N first images according to the N-1 offset pixel matrices to obtain the target image, the processor 1010 is further configured to:

[0144] For each of the N first images, perform interpolation processing on a first pixel in the first image to obtain a first interpolation result;

[0145] replacing the first pixel with the first interpolation result to obtain a monochrome image;

[0146] The processor 1010, when performing pixel superposition on the pixels of the N first images according to the N-1 offset pixel matrices to obtain the target image, is specifically configured to:

[0147] According to the N-1 offset pixel matrices, the N monochrome images are pixel-superimposed to obtain a target image.

[0148] Optionally, when the processor 1010 performs pixel superposition on the N monochrome images according to the N-1 offset pixel matrices to obtain a target image, it is specifically configured to:

[0149] According to the N-1 offset pixel matrices, one of the N monochrome images is used as a reference image, and pixels in other monochrome images are superimposed on corresponding pixels of the reference image to obtain a multi-color image;

[0150] The multi-color image is fused with the reference image to obtain a target image.

[0151] Optionally, when the processor 1010 fuses the multi-color image with the reference image to obtain a target image, it is specifically configured to:

[0152] Processing the multi-color image and the reference image into images with the same resolution;

[0153] Converting the multi-color image with the same resolution as the reference image into a second image in YUV format;

[0154] A reference image having the same resolution as the multi-color image is fused with the Y component of the second image to obtain a target image.

[0155] Optionally, the fusing of a reference image having the same resolution as the multi-color image with a Y component in the second image includes:

[0156] Replacing the Y component with a reference image having the same resolution as the multi-color image;

[0157] Alternatively, a Gaussian filter is used to decompose the base layer and detail layer of the Y component, and decompose the base layer and detail layer of a reference image having the same resolution as the multi-color image;

[0158] The detail layer of the Y component is replaced by the detail layer of a reference image having the same resolution as the multi-color image.

[0159] In summary, the above embodiment of the present application, through the parallel and side-by-side structure of N monochrome cameras and the addition of a first pixel in each camera, can assist in achieving pixel registration between monochrome camera images, thereby obtaining a more refined color image; and utilize pixel registration to merge the monochrome images of N monochrome cameras, thereby obtaining a more refined multi-color image; under low-light conditions, the multi-color image and the grayscale image are fused to obtain a target image with more details and a higher signal-to-noise ratio under low-light conditions. The above scheme not only improves color resolution, but also can obtain more accurate and detailed image information than a monocular camera. It utilizes N monochrome cameras to achieve more refined color reproduction through color separation, and utilizes N monochrome cameras to achieve the purpose of binocular ranging. For the online live broadcast mode of portraits or exhibits such as cultural relics, it can more prominently present the color details of the foreground portraits or exhibits.

[0160] It should be understood that in an embodiment of the present application, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0161] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0162] Processor 1010 may include one or more processing units. Optionally, processor 1010 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1010.

[0163] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image generation method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0164] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0165] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0166] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0167] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0168] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0169] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0170] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A camera module, characterized in that: include: N monochrome cameras arranged in parallel, where N is a positive integer greater than 2; The monochrome camera includes a pixel array, the pixel array includes a plurality of minimal repeating units, the minimal repeating unit includes a first pixel and a second pixel, the first pixel is a first visible light pixel or an invisible light pixel, and the second pixel is a second visible light pixel; the first pixel is used to assist in pixel registration between the N monochrome cameras; Among them, the wavelength band of light transmitted through the light-transmitting part of the first visible light pixel includes a wavelength band within the range from blue to red, and the wavelength band of light transmitted through the light-transmitting part of the second visible light pixel is narrower than the wavelength band of light transmitted through the light-transmitting part of the first visible light pixel.

2. The camera module according to claim 1, wherein: The parallel arrangement includes: arranging in parallel at equal intervals in the horizontal direction, or arranging in parallel at equal intervals in the vertical direction.

3. An electronic device, characterized in that: Comprising the camera module as described in any one of claims 1 to 2.

4. An image generation method, characterized in that: Applied to the electronic device according to claim 3, the method comprises: Acquire N first images through N monochrome cameras; Taking one of the N first images as a reference, performing calculations on the other first images respectively to obtain N-1 offset pixel matrices; The pixels of the N first images are superimposed according to the N-1 offset pixel matrices to obtain a target image.

5. The method according to claim 4, characterized in that Before superimposing the pixels of the N first images according to the N-1 offset pixel matrices to obtain the target image, the method further includes: For each of the N first images, perform interpolation processing on a first pixel in the first image to obtain a first interpolation result; replacing the first pixel with the first interpolation result to obtain a monochrome image; The step of superimposing the pixels of the N first images according to the N-1 offset pixel matrices to obtain the target image includes: According to the N-1 offset pixel matrices, the N monochrome images are pixel-superimposed to obtain a target image.

6. The method according to claim 5, characterized in that The step of performing pixel superposition on the N monochrome images according to the N-1 offset pixel matrices to obtain a target image includes: According to the N-1 offset pixel matrices, one of the N monochrome images is used as a reference image, and pixels in other monochrome images are superimposed on corresponding pixels of the reference image to obtain a multi-color image; The multi-color image is fused with the reference image to obtain a target image.

7. The method according to claim 6, characterized in that The fusing the multi-color image with the reference image to obtain a target image comprises: Processing the multi-color image and the reference image into images with the same resolution; Converting the multi-color image with the same resolution as the reference image into a second image in YUV format; A reference image having the same resolution as the multi-color image is fused with the Y component of the second image to obtain a target image.

8. The method according to claim 7, characterized in that The fusing of the reference image having the same resolution as the multi-color image with the Y component in the second image comprises: Replacing the Y component with a reference image having the same resolution as the multi-color image; Alternatively, a Gaussian filter is used to decompose the base layer and detail layer of the Y component, and decompose the base layer and detail layer of a reference image having the same resolution as the multi-color image; The detail layer of the Y component is replaced by the detail layer of a reference image having the same resolution as the multi-color image.

9. An image generating device, characterized in that: The camera module according to any one of claims 1 to 2, wherein the image generating device further comprises: A first acquisition module, configured to acquire N first images through N monochrome cameras; A first processing module is configured to use one of the N first images as a reference and calculate the other first images using a stereo registration algorithm to obtain N-1 offset pixel matrices; The second processing module is configured to perform pixel superposition on the pixels of the N first images according to the N-1 offset pixel matrices to obtain a target image.

10. An electronic device, characterized in that: It includes a processor, a memory and a camera module as described in any one of claims 1-2, the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, it implements the steps of the image generation method as described in any one of claims 4-8.

11. A chip, characterized in that: The chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the image generation method according to any one of claims 4 to 8.

Images