Image processing chip, application processing chip, electronic device and image processing method

The image processing chip fusion and tone mapping of multiple original image data is solved, and the problems of high transmission bandwidth and large power consumption of multiple image data are reduced, and the data volume and power consumption are reduced, and zero-delay photography is supported. Application of technology.

CN115835011BActive Publication Date: 2024-12-03GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202111081125.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-15
Publication Date
2024-12-03
Estimated Expiration
2041-09-15

AI Technical Summary

Technical Problem

In the prior art, the transmission of multiple raw image data has high bandwidth requirements and high power consumption. Especially when shooting with multiple cameras, the number of data channels of MIPI hardware is insufficient, making it difficult to meet the data transmission requirements.

Method used

An image processing chip is designed to obtain N-channel fused image data by fusing the original M-channel fused image data, and tone mapping processing of the fused image data is reduced to reduce data transmission, bandwidth requirements and power consumption. The chip includes a first image signal processor for fusing the M-channel original image data and sending the fused image data to the application processing chip, which then calibrates the fused image data.

Benefits of technology

The image processing chip fusion and tone mapping of multiple original image data is used to significantly reduce the data transmission amount, reduce bandwidth requirements and power consumption, and supports the application of zero-delay photography technology, especially on low-end platforms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115835011B_ABST
    Figure CN115835011B_ABST
Patent Text Reader

Abstract

The present invention discloses an image processing chip, an application processing chip, an electronic device and an image processing method. The image processing chip comprises: a first image signal processor, which is used to perform fusion processing on M-channel original image data to obtain N-channel fused image data, wherein M and N are both positive integers and M>N; the image processing chip is also used to send the fused image data to the application processing chip. The image processing chip can reduce the amount of data transmission, reduce the bandwidth requirements during data transmission, and also has the function of reducing power consumption.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to an image processing chip, an application processing chip, an electronic device and an image processing method. Background Art

[0002] Cameras have become essential equipment for various digital products, such as mobile phones and tablet computers. To ensure the image acquisition effect, the number of cameras has increased from one to multiple, and the multiple channels of original RAW data obtained by the image sensor under the camera need to be transmitted to the application processing chip for processing. As a result, the amount of data transmitted is large, the bandwidth requirement is high, and the power consumption is also high. Summary of the invention

[0003] The present invention aims to solve one of the technical problems in the related art at least to a certain extent. To this end, one object of the present invention is to provide an image processing chip.

[0004] A second objective of the present invention is to provide an application processing chip.

[0005] A third objective of the present invention is to provide an electronic device.

[0006] A fourth objective of the present invention is to provide an image processing method.

[0007] To achieve the above-mentioned objectives, an embodiment of the first aspect of the present invention proposes an image processing chip, which includes: a first image signal processor, used to perform fusion processing on M channels of original image data to obtain N channels of fused image data, wherein M and N are both positive integers and M>N; the image processing chip is also used to send the fused image data to an application processing chip.

[0008] To achieve the above-mentioned purpose, the second aspect of the present invention proposes an application processing chip, which is used to obtain N-channel fused image data from an image processing chip, and the application processing chip includes: a second image signal processor, which is used to calibrate the N-channel fused image data; wherein the N-channel fused image is obtained by fusion processing of M-channel original image data, wherein M and N are both positive integers, and M>N.

[0009] To achieve the above-mentioned purpose, the third aspect of the present invention proposes an electronic device, which includes an image processing chip for fusing M channels of original image data to obtain N channels of fused image data, wherein M and N are both positive integers and M>N; an application processing chip for obtaining N channels of fused image data from the image processing chip and calibrating the N channels of fused image data.

[0010] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes an image processing method, which includes: acquiring M channels of original image data; fusing the M channels of original image data to obtain N channels of fused image data; and calibrating the N channels of fused image data.

[0011] According to the image processing chip, application processing chip, electronic device and image processing method of the embodiments of the present invention, N fused images are obtained by fusing M original images, and then the N fused image data are transmitted, which greatly reduces the data transmission volume, reduces the bandwidth requirements during data transmission, and also has the effect of reducing power consumption.

[0012] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a structural diagram of image data processing according to an embodiment of the present invention;

[0014] Figure 2 is a schematic diagram of the structure of an image processing chip according to an embodiment of the present invention;

[0015] Figure 3 is a schematic structural diagram of an image processing chip according to a specific embodiment of the present invention;

[0016] Figure 4 is a schematic diagram of image size comparison before and after fusion processing according to a specific embodiment of the present invention;

[0017] Figure 5 is a schematic diagram of image size comparison before and after tone mapping processing according to a specific embodiment of the present invention;

[0018] Figure 6 is a schematic diagram of the structure of an application processing chip according to an embodiment of the present invention

[0019] Figure 7 is a schematic diagram of the structure of an application processing chip according to a specific embodiment of the present invention;

[0020] Figure 8 is a flow chart of a calibration process according to a specific embodiment of the present invention;

[0021] Fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

[0022] Fig.10 is a schematic structural diagram of an electronic device according to another embodiment of the present invention;

[0023] Fig.11 It is a flowchart of an image processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0025] In one embodiment of the present invention, Figure 1 As shown, when an electronic device capable of collecting multiple channels of raw image data performs image acquisition, it is necessary to continuously transmit the multiple channels of raw RAW data obtained by the image sensor under the camera to the image processing chip and the application processing chip for processing. If the multiple channels of raw image data are transmitted to the application processing chip for processing, the amount of data transmitted will be large, the bandwidth requirement will be high, and the power consumption will also be high. In addition, see Figure 1 If MIPI (Mobile Industry Processor Interface) is used for data transmission, it is difficult to achieve too much data transmission due to hardware and cost limitations.

[0026] Specifically, as an example, when an electronic device captures images in a mode such as smooth zoom, multiple cameras capture images simultaneously, and multiple raw image data and 3A statistical information (3A stats) of each raw image need to be transmitted to the image processing chip and the application processing chip in sequence. The 3A statistical information includes automatic exposure statistical information, automatic white balance statistical information, and automatic focus statistical information. The data transmission volume is large, the transmission bandwidth requirement is high, and the power consumption of transmitting data is high.

[0027] As another example, when an electronic device captures an image in DOL (Digital overlap) mode, it needs to transmit multiple exposure images output by the camera's image sensor to the image processing chip and the application processing chip in sequence, as well as 3A statistical information and PD of each exposure image. Taking two cameras as an example, it is necessary to count 3 channels * 2 chips * 3A, and at least 18 categories of statistical data need to be used and transmitted, plus (3 channels of Raw images + 3 channels of PD) * 2 (chips) of data, for a total of 30 channels of data. Due to hardware and cost constraints, the number of data channels of MIPI (Mobile Industry Processor Interface) hardware cannot meet the demand, where PD stands for Phase Data, and PD is used for focusing.

[0028] To this end, the present invention proposes an image processing chip, an application processing chip, an electronic device and an image processing method, which are intended to solve the problem that the amount of data is large, the number of data paths of MIPI hardware is small, and the data transmission requirements cannot be met. Figure 2-11 The image processing chip, application processing chip, electronic device and image processing method of the embodiments of the present invention are described in detail in the specific implementation modes.

[0029] Figure 2 Schematic diagram of the structure of the image processing chip according to the embodiment of the present invention.

[0030] Figure 2 As shown, the image processing chip 2 includes a first image signal processor 21. The first image signal processor 21 is used to perform fusion processing on M channels of original image data to obtain N channels of fused image data, wherein M and N are both positive integers and M>N; the image processing chip 2 is also used to send the fused image data to the application processing chip 3.

[0031] Specifically, see Figure 2 , M channels of raw image data may be obtained by one or more image sensors. For example, M channels of raw image data may be obtained by an image sensor in a digital overlap DOL mode. If the number of image sensors is 2, M=2*3=6 channels of raw image data may be obtained. After the first image signal processor 21 fuses M channels (e.g., 6 channels) of raw image data into N channels (N<M, such as when M=6, N=2) of fused image data, the image processing chip 2 transmits the N channels of fused image data to the application processing chip 3. In this way, the requirements for transmission bandwidth when the image processing chip 2 transmits data back to the application processing chip 3 can be reduced, and the power consumption when transmitting data back can be reduced.

[0032] The image sensor may use photosensitive elements such as CMOS (Complementary Metal Oxide Semiconductor) and CCD (Charge-coupled Device).

[0033] In this embodiment, the raw image data is the unprocessed image data acquired by the image sensor, which is the raw data obtained by converting the captured light source signal into a digital signal by a photosensitive element such as CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge-coupled Device). The raw image data records the original information of the image sensor, and also records some metadata generated by the camera shooting, such as ISO settings, shutter speed, aperture value, white balance, etc. If the image sensor can work in the digital overlap DOL mode, the raw image data obtained by each image sensor includes multiple exposure images. For example, when the raw image data is acquired in the 3DOL mode, the acquired raw image data will include 3 exposure images, such as a long exposure image, an intermediate exposure image, and a short exposure image.

[0034] In an embodiment of the present invention, the number of image sensors may be one or more (greater than or equal to two) for acquiring M channels of raw image data. When the image sensor acquires raw image data in the DOL mode, the multiple channels of raw image data acquired by each image sensor are multiple channels of exposure image data.

[0035] As a feasible implementation, the image processing chip 2 can be used in an electronic device with a camera. To better support ZSL (Zero Shutter Lang) photography, the M-channel original image data collected by the camera's image sensor must be continuously input into the image processing chip 2. After the M-channel original image data is fused and processed into N (N<M)-channel fused image data by the first image signal processor 21, the image processing chip 2 transmits the N-channel fused image data to the application processing chip 3. As a result, the requirements for transmission bandwidth when the image processing chip 2 transmits data back to the application processing chip 3 can be reduced, and the power consumption when transmitting data back can be reduced, which is conducive to the implementation of zero-delay photography technology on low-end platforms.

[0036] In one embodiment of the present invention, the first image signal processor 21 is specifically used to divide M channels of original image data into N groups, where each group includes m channels of original image data, m is an integer, and 2≤m≤M; and perform fusion processing on the m channels of original image data in each group according to the following formula:

[0037] Pixel_Value_j_Fusioned = (Pixel_Value_i*k i )(1)

[0038] Among them, Pixel_Value_j_Fusioned represents the pixel value of the j-th fused image in the N-way fused image, Pixel_Value_i represents the pixel value of the i-th original image data in the m-way original image data, and k i It represents the ratio of the longest exposure time among the exposure times of m channels of original image data to the exposure time of the i-th channel of original image data, where i is an integer and 1<i≤m.

[0039] As a specific example, see Figure 3 The first image signal processor 21 may include a first ISP (Image Signal Processing) module and a fusion module, wherein the number of the first ISP module and the fusion module may be one or N. If there are N, the first ISP module and the fusion module correspond to each group of m-channel original image data one by one, and at this time, the m-channel original image data are sequentially input to the corresponding first ISP module and the fusion module for processing; if there is one, the first ISP module and the fusion module may process the N groups of original image data in parallel. In this way, the image processing efficiency can be guaranteed. See Figure 3 The image processing chip 2 may also include a neural network processor, denoted as an NPU (Neural-network Processing Unit) module.

[0040] In this embodiment, N first ISP modules are used to receive M channels of original image data, and pre-process the received original image data to obtain preview images of the channels.

[0041] Specifically, the first ISP module processes the raw image data transmitted by the image sensor to match image sensors of different models. At the same time, the first ISP module completes the effect processing of the raw image data through a series of digital image processing algorithms, which may mainly include 3A (automatic white balance, auto focus, automatic exposure), bad pixel correction, denoising, strong light suppression, backlight compensation, color enhancement, lens shading correction and other processing to obtain a preview image.

[0042] The NPU module is used to process each preview image separately using an AI algorithm.

[0043] Specifically, the NPU module uses AI algorithms to complete the demosaic difference algorithm, automatic white balance, color correction, noise reduction, HDR (High-Dynamic Range), super-resolution and other processing of each preview image.

[0044] The fusion module is used to fuse the corresponding preview images processed by the AI ​​algorithm to obtain N fused images.

[0045] Specifically, although the original image data transmitted by the image sensor is processed by the first ISP module and the NPU module, the data volume is not reduced. The fusion module fuses the images processed by the first ISP module and the NPU module, converting M channels of original image data into N channels of fused images, which can reduce the data transmission bandwidth and save power consumption.

[0046] As a specific example, see Figure 4 When acquiring raw image data in 3DOL mode, the raw image data acquired by each image sensor includes three exposure images (long exposure image, intermediate exposure image and short exposure image). Therefore, when the long exposure image, the intermediate exposure image and the short exposure image are fused, the raw image can be fused according to the following formula:

[0047] Pixel_Value_Fusioned = Pixel_Value_long + Pixel_Value_medium*4 + Pixel_Value_short*16,

[0048] Among them, Pixel_Value_Fusioned represents the pixel value of the fused image, Pixel_Value_long represents the pixel value of the long exposure image, Pixel_Value_medium represents the pixel value of the intermediate exposure image, and Pixel_Value_short represents the pixel value of the short exposure image.

[0049] In this embodiment, the exposure time t of the long exposure image is 长 , exposure time t of the intermediate exposure image 中 and the exposure time t of the short exposure image 短 Fourfold relationship: t 长 =4*t 中 =16*t 短 .

[0050] In this embodiment, when the fusion module processes the exposure image in the preview image, it rearranges the exposure image. As an example, Figure 4 As shown, the fusion module can fuse three 10-bit exposure images into a 30-bit fused image.

[0051] In an embodiment of the present invention, the first image signal processor 21 is also used to perform tone mapping processing on each channel of fused image data to obtain fused image data after tone mapping processing and tone mapping processing parameters; wherein, the image processing chip 2 is also used to send N channels of fused image data after tone mapping processing and their corresponding tone mapping processing parameters to the application processing chip 3.

[0052] As a specific embodiment, the first image signal processor 21 may include a tone mapping module. The tone mapping module may correspond one-to-one with the first ISP module and the fusion module, that is, the number of the tone mapping modules is the same as the number of the first ISP modules and the number of the fusion modules. When the number of the first ISP modules and the number of the fusion modules are both N, the number of the tone mapping modules is also N. When the number of the first ISP modules and the number of the fusion modules are both 1, the number of the tone mapping modules is also 1, so that the fused image processed by the first ISP module and the fusion module can be transmitted to the corresponding tone mapping module for processing, thereby ensuring the reliability of data processing. The tone mapping module is used to perform tone mapping on the fused image to obtain the fused image after the tone mapping and the tone mapping processing parameters. Specifically, the tone mapping module may use a tone mapping algorithm (tonemapping) to perform tone mapping on the high-bitwidth fused image obtained by the fusion process. Figure 5 As shown, the 30-bit fused image obtained after the fusion process can be converted into a 10-bit image after tone mapping.

[0053] In an embodiment of the present invention, when the first image signal processor 21 performs tone mapping processing on the fused image data, it is specifically used to: determine the region of interest of the fused image data; perform histogram equalization processing based on the region of interest to obtain a histogram equalization mapping relationship, wherein the histogram equalization mapping relationship is a tone mapping processing parameter; and map the histogram equalization mapping relationship to the entire image of the fused image data.

[0054] Specifically, the region of interest of the fused image is determined to enhance a certain part of the image in a targeted manner. The method for demarcating the region of interest can be a user input method. The number of demarcated images of interest can be one or more. The shape of the acquired image of interest can be a polygon, an ellipse, etc. Histogram equalization is to stretch the image nonlinearly and redistribute the pixel values ​​of the image to achieve a roughly the same number of pixels within a certain grayscale range, so that the given histogram distribution is transformed into a uniform histogram distribution, thereby obtaining the maximum contrast. When performing histogram equalization based on the region of interest, the histogram equalization mapping relationship is recorded. Based on the histogram equalization mapping relationship, it is mapped to the entire image of the fused image to perform histogram equalization on the entire image of the fused image to ensure that the information fidelity of the ROI area is the highest.

[0055] As an example, after obtaining the ROI area, the extended area can be further obtained, and the size of the extended area can be (width and height of the ROI area * 1.25). For example, if the ROI area is a rectangular area, the extended area is a rectangular area, and the length of the extended area is the length of the ROI area * 1.5, the width of the extended area is the width of the ROI area * 1.5, and the centers of the two areas coincide. Histogram equalization is performed based on the extended area to obtain a histogram equalization mapping relationship.

[0056] It should be noted that histogram equalization is very useful for images with too bright or too dark backgrounds and foregrounds, and can better show details in overexposed or underexposed photos. A major advantage of this method is that it is quite intuitive and reversible. If the equalization function is known, the original histogram can be restored, and the computational effort is not large.

[0057] In an embodiment of the present invention, the first image signal processor 21 is also used to obtain 3A statistical information of M channels of original image data, wherein the 3A statistical information includes automatic exposure statistical information, automatic white balance statistical information and automatic focus statistical information; wherein the image processing chip 2 is also used to send the 3A statistical information to the application processing chip 3.

[0058] Specifically, the first image signal processor 21 can obtain 3A statistical information of M-channel original image data by using the first ISP module, wherein the 3A statistical information includes automatic exposure statistical information (AE, Auto Exposure), automatic white balance statistical information (AWB, Auto White Balance) and auto focus statistical information (AF, Auto Focus).

[0059] In an embodiment of the present invention, the image processing chip 2 is also used to encode 3A statistical information, fused image data after tone mapping processing, tone mapping processing parameters and PD data to obtain encoding information, and send the encoding information to the application processing chip 3.

[0060] As a specific example, see Figure 3 , the image processing chip 2 may include a MIPI-TX encoding submodule, wherein the MIPI-TX encoding submodule may correspond to the above-mentioned tone mapping module one by one, that is, the number of MIPI-TX encoding submodules is the same as the number of tone mapping modules, which may be one or N. The MIPI-TX encoding submodule receives the 3A statistical information of the original image data, the fused image after tone mapping processing, the tone mapping processing parameters and the PD data to encode the 3A statistical information of the original image data, the fused image after tone mapping processing, the tone mapping processing parameters and the PD data, and transmits the encoded information to the application processing chip 3 through the MIPI protocol.

[0061] The image processing chip provided by the present invention performs fusion processing on M channels of original image data to obtain N channels of fused image data, and performs tone mapping processing on the N channels of fused image data, which greatly reduces the data transmission volume, reduces the bandwidth requirement during data transmission, and also has the effect of reducing power consumption, which is conducive to the application of zero-delay photography technology on low-end platforms.

[0062] The invention provides an application processing chip.

[0063] Figure 6 is a schematic diagram of the structure of an application processing chip according to an embodiment of the present invention. Figure 2 , Figure 6 The application processing chip 3 is used to obtain N-channel fused image data from the image processing chip 2.

[0064] like Figure 6 As shown, the application processing chip 3 includes a second image signal processor 31. The second image signal processor 31 is used to perform calibration processing on N-channel fused image data; wherein the N-channel fused image is obtained by fusion processing of M-channel original image data, wherein M and N are both positive integers, and M>N.

[0065] Specifically, the amount of data of the original image is greatly reduced after being fused or fused and tone-mapped by the image processing chip 2. However, after the image processing chip 2 performs tone mapping on the fused image, the 3A accuracy of the image is affected, so it is necessary to perform calibration on the fused image after tone mapping. As an example, the fused image after tone mapping, as well as 3A statistical information, tone mapping processing parameters and PD data can be obtained to perform calibration on the fused image data to obtain the target image.

[0066] As a possible implementation, see Figure 7 , the application processing chip 3 may include a MIPI-RX decoding submodule, and the second image signal processor 31 may include a second ISP module. The number of the MIPI-RX decoding submodule and the second ISP module may be one or N, which may be the same as the number of the MIPI-TX encoding submodules in the image processing chip 2.

[0067] In this embodiment, the MIPI-RX decoding submodule is used to receive the encoding information of the corresponding MIPI-TX encoding submodule, and decode the encoding information to obtain 3A statistical information, a fused image after tone mapping processing, tone mapping processing parameters and PD data, and then transmit the fused image after tone mapping processing to the second ISP module. The second ISP module is used to pre-process the fused image after tone mapping processing using a digital image processing algorithm after receiving the corresponding fused image after tone mapping processing. The pre-processing of the fused image after tone mapping processing by the second ISP module is the same as the pre-processing performed by the first ISP module, which will not be repeated here.

[0068] In the embodiments of the present invention, see Figure 6 , Figure 7 The application processing chip 3 also includes a second central processor 32. The number of the second central processors 32 can be one or N, which can be the same as the number of the MIPI-RX decoding submodule and the second ISP module. The second central processor 32 is used to obtain the AWB gain parameters and CCM parameters of the N-channel fused image data according to the 3A statistical information of the M-channel original image data and the tone mapping processing parameters of the N-channel fused image data using the 3A algorithm, and calibrate the AWB gain parameters according to the tone mapping processing parameters; wherein the second image signal processor 31 is specifically used to perform automatic white balance calibration and color calibration on the M-channel fused image data using the calibrated AWB gain calibration parameters and CCM parameters.

[0069] Specifically, after receiving the corresponding 3A statistical information, tone mapping processing parameters and PD data, the second central processor 32 is used to obtain AWB gain parameters and CCM (Color Correct Matrix) parameters based on the 3A statistical information, tone mapping processing parameters and PD data using the 3A algorithm, and calibrate the AWB gain parameters according to the tone mapping processing parameters.

[0070] As an example, see Figure 8 The second central processor 32 can compare the 3A statistical information before image fusion compression and the 3A statistical information after image fusion compression to calibrate the color of the RAW image received by the application processing chip 3, and obtain the ratio coefficient by comparing the RGB statistics before and after fusion compression. The ratio is used to correct the result (RGB Gain) of the AWB algorithm at the application processing chip end, and the color of the Raw image of the application processing chip 3 is calibrated using the corrected 3A algorithm result.

[0071] In an embodiment of the present invention, when the second central processor 32 calibrates the AWB gain parameter according to the tone mapping processing parameter, it can be specifically used to:

[0072] Performing inverse tone mapping processing on the fused image data after the tone mapping processing;

[0073] Calculate the AWB gain calibration parameters according to the following formula:

[0074] RGainCalibration = RGain / Cr / Cg;

[0075] BGain calibration = BGain / Cb / Cg;

[0076] Among them, RGain calibration is the R gain after calibration, BGain calibration is the B gain after calibration, RGain is the R gain before calibration, Cr / Cg is the relative G gain of R, Cb / Cg is the relative G gain of B, Cr=Rsum / Rsum_untonemapping, Cg=Gsum / Gsum_untonemapping, Cb=Bsum / Bsum_untonemapping, Rsum, Gsum, Bsum are the total values ​​of R, G, B components of the fused image after tone mapping, Rsum_untonemapping, Gsum_untonemapping, Bsum_untonemapping are the total values ​​of R, G, B components of the fused image after inverse tone mapping.

[0077] Furthermore, the calibrated AWB gain calibration parameters and CCM parameters are used to perform automatic white balance calibration and color calibration on the fused image after the tone mapping process.

[0078] In summary, the application processing chip according to the embodiment of the present invention can ensure the display effect of the image by performing calibration processing on N-channel fused image data obtained by fusing M-channel original image data.

[0079] The invention also provides an electronic device.

[0080] See also Fig. 9 and Fig.10 , the electronic device 10 includes an image processing chip 2 and an application processing chip 3.

[0081] In this embodiment, the image processing chip 2 is used to perform fusion processing on M channels of original image data to obtain N channels of fused image data, wherein M and N are both positive integers, and M>N.

[0082] Specifically, the image processing chip 2 is used to: divide M channels of original image data into N groups, where each group includes m channels of original image data, m is an integer, and 2≤m≤M; and perform fusion processing on the m channels of original image data in each group according to the following formula:

[0083] Pixel_Value_j_Fusioned= (Pixel_Value_i*k i ),

[0084] Among them, Pixel_Value_j_Fusioned represents the pixel value of the j-th fused image in the N-way fused image, Pixel_Value_i represents the pixel value of the i-th original image data in the m-way original image data, and k i It represents the ratio of the longest exposure time among the exposure times of m channels of original image data to the exposure time of the i-th channel of original image data, where i is an integer and 1<i≤m.

[0085] In one embodiment of the present invention, the image processing chip 2 is also used to: perform tone mapping processing on each channel of fused image data to obtain fused image data after tone mapping processing and tone mapping processing parameters, and send N channels of fused image data after tone mapping processing and their corresponding tone mapping processing parameters to the application processing chip 3.

[0086] The application processing chip 3 is used to obtain N-channel fused image data from the image processing chip and perform calibration processing on the N-channel fused image data.

[0087] The electronic device according to the embodiment of the present invention may be a mobile terminal, such as a smart phone, a tablet computer, etc.

[0088] It should be noted that other specific implementations of the image processing chip 2 and the application processing chip 3 in the electronic device 10 of the embodiment of the present invention may refer to the specific implementations of the image processing chip 2 and the application processing chip 3 of the above-mentioned embodiment of the present invention.

[0089] Also, see Fig. 9 The image processing chip 2 may also include a CPU, a memory, and a computer vision engine, wherein the CPU may be responsible for controlling the image processing chip 2, such as power on and off, firmware loading, and runtime control; the memory may be used to store data that needs to be stored during image data processing; the computer vision engine may be configured to process scenes, generate information flows that represent the observed activities, and transmit the information flows to other modules through the system bus to learn the object behaviors of the corresponding scenes. The application processing chip 3 may also include a memory to store data that needs to be stored during image data processing.

[0090] The electronic device provided by the embodiment of the present invention fuses or fuses and performs tone mapping on the original image transmitted from the image sensor through the image processing chip, and sends the compressed fused image to the application processing chip, which greatly reduces the amount of data transmission, reduces the bandwidth requirements during data transmission, and also has the effect of reducing power consumption. The electronic device provided by the embodiment of the present invention can be applied to a multi-camera (such as two cameras, one main camera and one auxiliary camera), and the main and auxiliary cameras synchronously use this method to reduce bandwidth, synchronize and integrate the tone mapping parameters when the main and auxiliary cameras are fused, so that the tone mapping is more accurate.

[0091] The invention also provides an image processing method.

[0092] Fig.11 FIG. 1 is a flow chart of an image processing method according to an embodiment of the present invention. Fig.11 As shown, the image processing method includes:

[0093] S1, obtain M channels of original image data.

[0094] Specifically, an image sensor may be used to obtain M channels of original image data, wherein the original image is obtained in a digital overlap DOL mode. The image sensor is a photosensitive element that utilizes the photoelectric conversion function of a photoelectric device to convert the light image on the photosensitive surface into an electrical signal that is proportional to the light image. The image sensor may be a photosensitive element such as CMOS, CCD, etc.

[0095] Specifically, a CMOS image sensor is essentially a chip, which mainly includes modules such as photosensitive area array (Bayer array), timing control, analog signal processing, and analog-to-digital conversion. Its main function is to convert optical signals into electrical signals, and then convert them into digital signals through ADC (Analog-to-digital converter).

[0096] S2, performing fusion processing on the M channels of original image data to obtain N channels of fused image data.

[0097] As a feasible implementation, the fusion processing of M channels of original image data may include:

[0098] Divide M channels of original image data into N groups, where each group includes m channels of original image data, m is an integer, and 2≤m≤M;

[0099] The m-channel original image data in each group are fused according to the following formula:

[0100] Pixel_Value_j_Fusioned= (Pixel_Value_i*k i),

[0101] Among them, Pixel_Value_j_Fusioned represents the pixel value of the j-th fused image in the N-way fused image, Pixel_Value_i represents the pixel value of the i-th original image data in the m-way original image data, and k i It represents the ratio of the longest exposure time among the exposure times of m channels of original image data to the exposure time of the i-th channel of original image data, where i is an integer and 1<i≤m.

[0102] In an embodiment of the present invention, the image processing method further comprises: performing tone mapping processing on each channel of fused image data to obtain fused image data after tone mapping processing and tone mapping processing parameters.

[0103] S3, calibrating the N-channel fused image data.

[0104] It should be noted that for other specific implementations of the image processing method of the embodiment of the present invention, reference may be made to the specific implementations of the image processing chip and the application processing chip of the above-mentioned embodiment of the present invention.

[0105] The image processing method provided by the embodiment of the present invention performs fusion or fusion and tone mapping processing on M original images, and corrects the fused image after tone mapping processing, which greatly reduces the amount of data transmission, reduces the bandwidth requirements during data transmission, and also has the effect of reducing power consumption. In addition, the image processing method provided by the embodiment of the present invention can be applied to a multi-camera (such as two cameras, one main camera and one auxiliary camera), and the main and auxiliary cameras use this method synchronously to reduce bandwidth, synchronize and integrate the tone mapping parameters when the main and auxiliary cameras are fused, so that the tone mapping is more accurate.

[0106] It should be noted that the logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0107] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0108] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0109] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0110] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0111] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0112] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0113] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An image processing chip, characterized in that: The image processing chip comprises: A first image signal processor is used to perform fusion processing on M channels of original image data to obtain N channels of fused image data, wherein M and N are both positive integers and M>N; The image processing chip is also used to send the fused image data to the application processing chip; The first image signal processor is specifically used for: Divide the M channels of original image data into N groups, wherein each group includes m channels of original image data, m is an integer, and 2≤m≤M; The m-channel original image data in each group are fused according to the following formula: Pixel_Value_j_Fusioned = (Pixel_Value_i*k i ), Wherein, Pixel_Value_j_Fusioned represents the pixel value of the j-th fused image in the N-way fused image, Pixel_Value_i represents the pixel value of the i-th original image in the m-way original image, and k i It represents the ratio of the longest exposure time among the exposure times of the m original images to the exposure time of the i-th original image, where i is an integer and 1<i≤m.

2. The image processing chip according to claim 1, characterized in that: The first image signal processor is further configured to: Performing tone mapping processing on each channel of the fused image data to obtain tone mapped fused image data and tone mapping processing parameters; The image processing chip is further used to send the fused image data after N-way tone mapping processing and its corresponding tone mapping processing parameters to the application processing chip.

3. The image processing chip according to claim 2, characterized in that: When the first image signal processor performs tone mapping processing on the fused image data, the first image signal processor is specifically used to: determining a region of interest of the fused image data; Performing histogram equalization processing based on the region of interest to obtain a histogram equalization mapping relationship, wherein the histogram equalization mapping relationship is the tone mapping processing parameter; The histogram equalization mapping relationship is mapped to the full image of the fused image data.

4. The image processing chip according to claim 2, characterized in that: The first image signal processor is further configured to: Obtaining 3A statistical information of the M channels of original image data by statistics, wherein the 3A statistical information includes automatic exposure statistical information, automatic white balance statistical information, and automatic focus statistical information; The image processing chip is further used to send the 3A statistical information to the application processing chip.

5. The image processing chip according to claim 4, characterized in that: The image processing chip is also used for: The 3A statistical information, the fused image data after the tone mapping process, and the tone mapping process parameters are encoded to obtain encoding information, and the encoding information is sent to the application processing chip.

6. An application processing chip, characterized in that: The application processing chip is used to obtain N-channel fused image data from the image processing chip, and the application processing chip includes: A second image signal processor, used for calibrating the N-channel fused image data; Wherein, the N-channel fused image is obtained by fusion processing of M-channel original image data, wherein M and N are both positive integers, and M>N; The application processing chip also includes: A second central processing unit is configured to obtain an AWB gain parameter and a CCM parameter of the N-channel fused image data according to the 3A statistical information of the M-channel original image data and the tone mapping processing parameters of the N-channel fused image data by using a 3A algorithm, and calibrate the AWB gain parameter according to the tone mapping processing parameters; The second image signal processor is specifically used to perform automatic white balance calibration and color calibration on the N-channel fused image data using the calibrated AWB gain calibration parameters and the CCM parameters; When the second central processor calibrates the AWB gain parameter according to the tone mapping processing parameter, it is specifically used to: Performing inverse tone mapping processing on the fused image data after the tone mapping processing; The AWB gain calibration parameters are calculated according to the following formula: R Gain calibration = R Gain / Cr / Cg; B Gain calibration = B Gain / Cb / Cg; Among them, R Gain calibration is the R gain after calibration, B Gain calibration is the B gain after calibration, R Gain is the R gain before calibration, Cr / Cg is the relative G gain of R, Cb / Cg is the relative G gain of B, Cr = Rsum / Rsum_untonemapping, Cg = Gsum / Gsum_untonemapping, Cb = Bsum / Bsum_untonemapping, Rsum, Gsum, and Bsum are respectively the total values ​​of the R, G, and B components of the fused image after the tone mapping process, and Rsum_untonemapping, Gsum_untonemapping, and Bsum_untonemapping are respectively the total values ​​of the R, G, and B components of the fused image after the reverse tone mapping process.

7. An electronic device, characterized in that: include: An image processing chip is used to perform fusion processing on M channels of original image data to obtain N channels of fused image data, wherein M and N are both positive integers and M>N; An application processing chip, used for obtaining N-channel fused image data from the image processing chip, and performing calibration processing on the N-channel fused image data; The image processing chip is specifically used for: Divide the M channels of original image data into N groups, wherein each group includes m channels of original image data, m is an integer, and 2≤m≤M; The m-channel original image data in each group are fused according to the following formula: Pixel_Value_j_Fusioned = (Pixel_Value_i*k i ), Wherein, Pixel_Value_j_Fusioned represents the pixel value of the j-th fused image in the N-way fused image, Pixel_Value_i represents the pixel value of the i-th original image in the m-way original image, and k i It represents the ratio of the longest exposure time among the exposure times of the m original images to the exposure time of the i-th original image, where i is an integer and 1<i≤m.

8. The electronic device according to claim 7, characterized in that: The image processing chip is also used for: Perform tone mapping processing on each channel of the fused image data to obtain tone mapped fused image data and tone mapping processing parameters, and send the N channels of tone mapped fused image data and their corresponding tone mapping processing parameters to the application processing chip.

9. An image processing method, characterized in that: The method comprises: Obtain M channels of original image data; Performing fusion processing on the M channels of original image data to obtain N channels of fused image data; Performing calibration processing on the N-channel fused image data; The fusing process of the M channels of original image data includes: Divide the M channels of original image data into N groups, wherein each group includes m channels of original image data, m is an integer, and 2≤m≤M; The m-channel original image data in each group are fused according to the following formula: Pixel_Value_j_Fusioned = (Pixel_Value_i*k i ), Wherein, Pixel_Value_j_Fusioned represents the pixel value of the j-th fused image in the N-way fused image, Pixel_Value_i represents the pixel value of the i-th original image in the m-way original image, and k i It represents the ratio of the longest exposure time among the exposure times of the m original images to the exposure time of the i-th original image, where i is an integer and 1<i≤m.

10. The image processing method according to claim 9, characterized in that: The method further comprises: Perform tone mapping processing on each channel of the fused image data to obtain tone mapped fused image data and tone mapping processing parameters.

Citation Information

Patent Citations

  • Method for tone mapping based on histogram equalization

    CN103353982A

  • Multipath image fusion system and image fusion method

    CN104424627A

  • Image light effect processing method and device, electronic device, and storage medium

    CN109118427A