Data processing method and system for image sensor

Abstract

The present invention relates to the field of image sensing technology, and in particular to a data processing method for an image sensor, which first acquires first image data, second image data, and third image data of a target object based on a target image sensor, wherein the first image data and the second image data have the same exposure, the second image data has a higher exposure than the third image, the first image data and the third image data have the same resolution, and the resolution of the second image data is an even multiple of the resolution of the first image data, and then a weight map for the fusion of the first image data is constructed by using the second image data with a higher resolution than the first image data, thereby ensuring the accuracy of the weight during fusion while reducing the amount of calculation. At the same time, the present invention also uses the second image data with a higher resolution as a reference during sampling, thereby reducing the pixel loss generated by the image data during sampling, and further improving the accuracy of image data processing.

Description

Technical Field

[0001] The present invention relates to the field of image sensor technology, and in particular to a data processing method and system for an image sensor. Background Art

[0002] An image sensor is an electronic device that functions like the eyes of a camera to convert light signals into digital images. These sensors are usually composed of many photosensitive elements, each of which is responsible for capturing a small part of the light signal. When light hits the sensor surface, the photosensitive elements convert the light signal into an electrical signal, which is then transmitted through circuits to a computer or other device for processing and display. Therefore, image sensors play a vital role in digital cameras, mobile phone cameras, and many other image acquisition devices.

[0003] However, image sensors have some deficiencies in low-light conditions, which poses a challenge to image quality. Due to insufficient light signals, the sensor may have difficulty capturing a clear image. In this case, the image may appear blurry, with lost details, or an overall lack of clarity, which has a negative impact on image quality.

[0004] Therefore, a method is needed to improve the quality of images captured by image sensors in low-light environments. Summary of the invention

[0005] Therefore, the present invention provides a data processing method for an image sensor, so as to solve the problem in the prior art of how to improve the quality of images collected by the image sensor in a low-light environment.

[0006] The present invention provides a data processing method for an image sensor, comprising:

[0007] Acquire first image data, second image data, and third image data of the target object based on the target image sensor, wherein the first image data and the second image data have the same exposure, the second image data has a higher exposure than the third image data, the first image data and the third image data have the same resolution, and the resolution of the second image data is an even multiple of the resolution of the first image data;

[0008] constructing a second Gaussian pyramid of the second image data;

[0009] According to the pixel attribute of each pixel in the second image data, a weight map corresponding to the second image data is obtained, and a weighted Gaussian pyramid of the weight map is constructed;

[0010] Using the image of each layer in the second Gaussian pyramid as a reference for sampling, constructing a first Gaussian pyramid and a first Laplacian pyramid of the first image data;

[0011] constructing a third Gaussian pyramid and a third Laplacian pyramid of third image data;

[0012] The weighted Gaussian pyramid is used as the weight of the first Laplacian pyramid, and the first image data and the third image data are exposed and fused according to the first Gaussian pyramid, the first Laplacian pyramid, the third Gaussian pyramid, and the third Laplacian pyramid to obtain a target output image of the target object.

[0013] In a preferred solution, the method of constructing a first Gaussian pyramid and a first Laplacian pyramid of the first image data by taking the image of each layer in the second Gaussian pyramid as a reference for sampling includes:

[0014] Using the brightness distribution of the image in each layer of the second Gaussian pyramid as a reference for downsampling, a first Gaussian pyramid of the first image data is constructed;

[0015] The pixels of the image in each layer of the second Gaussian pyramid are used as references for upsampling, and a first Laplacian pyramid of the first image data is constructed based on the first Gaussian pyramid.

[0016] In a preferred solution, the step of constructing a first Gaussian pyramid of the first image data using the brightness distribution of the image in each layer of the second Gaussian pyramid as a reference for downsampling includes:

[0017] using the first image data as a first initial image;

[0018] Convolving the first initial image based on a preset convolution kernel to obtain a Gaussian smoothed image;

[0019] The Gaussian smoothed image is divided into multiple 2×2 image blocks, and according to the brightness distribution of the image with the same resolution as the Gaussian smoothed image in the second Gaussian pyramid, the pixel with the highest brightness in each image block is retained to obtain a first Gaussian pyramid image;

[0020] The first Gaussian pyramid image is used as a new first initial image, and the first initial image is repeatedly convolved based on a preset convolution kernel to obtain a Gaussian smoothed image, and the Gaussian smoothed image is divided into multiple 2×2 image blocks, and the pixel with the highest brightness in each image block is retained according to the brightness distribution of an image with the same resolution as the Gaussian smoothed image in the second Gaussian pyramid to obtain a first Gaussian pyramid image, until a preset end condition is reached, and a first Gaussian pyramid consisting of multiple layers of first Gaussian pyramid images is obtained.

[0021] In a preferred solution, the Gaussian smoothed image is divided into a plurality of 2×2 image blocks, and according to the brightness distribution of an image with the same resolution as the Gaussian smoothed image in the second Gaussian pyramid, the pixel with the highest brightness in each image block is retained to obtain a first Gaussian pyramid image, including:

[0022] Divide the Gaussian smoothed image into multiple 2×2 image blocks;

[0023] Obtain an image block as a target image block, obtain a pixel in the target image block as a target pixel, and obtain an image in the second Gaussian pyramid with the same resolution as the Gaussian smoothed image as a down-sampling reference image;

[0024] Performing a weighted summation on the brightness values ​​of the pixels corresponding to the target pixel and the brightness values ​​of the pixels adjacent to the pixels corresponding to the target pixel in the downsampled reference image to obtain a brightness description value of the target pixel;

[0025] The pixel with the highest brightness value in each image block is retained to obtain a first Gaussian pyramid image.

[0026] In a preferred solution, the method of taking pixels of an image in each layer of the second Gaussian pyramid as a reference for upsampling and constructing a first Laplacian pyramid of the first image data based on the first Gaussian pyramid includes:

[0027] Using the first Gaussian pyramid image at the top layer of the first Gaussian pyramid as the second initial image;

[0028] The second initial image is expanded to twice its original value in the horizontal and vertical directions to obtain an expanded image, and the newly added pixels in the expanded image are filled with pixels at the same position in the image with the same resolution as the expanded image in the second Gaussian pyramid;

[0029] Convolve the expanded image based on the preset convolution kernel to obtain a Gaussian convolution image;

[0030] Obtaining a first Gaussian pyramid having the same resolution as the Gaussian convolution image in the first Gaussian pyramid as a new reference image, calculating a residual between the reference image and the Gaussian convolution image, and obtaining a first Laplacian pyramid image;

[0031] The reference image is used as a new second initial image, and the first Gaussian pyramid image at the top layer in the first Gaussian pyramid is repeatedly used as the second initial image, the second initial image is expanded to twice the original in the horizontal and vertical directions to obtain an extended image, the newly added pixels in the extended image are filled with pixels at the same position in the image with the same resolution as the extended image in the second Gaussian pyramid, the extended image is convolved based on a preset convolution kernel to obtain a Gaussian convolution image, the first Gaussian pyramid with the same resolution as the Gaussian convolution image in the first Gaussian pyramid is obtained as a new reference image, the residual between the reference image and the Gaussian convolution image is calculated, and a first Laplacian pyramid image is obtained, until the reference image is the bottom layer image in the first Gaussian pyramid, and a first Laplacian pyramid consisting of multiple layers of first Laplacian pyramid images is obtained.

[0032] In a preferred solution, the method uses the weighted Gaussian pyramid as the weight of the first Laplacian pyramid, performs exposure fusion on the first image data and the third image data according to the first Gaussian pyramid, the first Laplacian pyramid, the third Gaussian pyramid, and the third Laplacian pyramid to obtain a target output image of the target object, including:

[0033] Using the weighted Gaussian pyramid as the weight of the first Laplacian pyramid, fusing the first Laplacian pyramid and the third Laplacian pyramid to obtain a hybrid Laplacian pyramid;

[0034] Using the weighted Gaussian pyramid as the weight of the first Laplacian pyramid, fusing the top-layer image in the first Gaussian pyramid with the top-layer image in the third Gaussian pyramid to obtain an image fusion starting image;

[0035] The pixels of the image in each layer of the second Gaussian pyramid are used as the reference for upsampling, and upsampling is performed in sequence based on the image fusion starting picture, and is added layer by layer with the hybrid Laplacian pyramid to obtain an image with the same resolution as the third image data as the target output image of the target object.

[0036] In a preferred solution, the step of acquiring the first image data, the second image data and the third image data of the target object based on the target image sensor includes:

[0037] acquiring first image data and third image data of a target object based on a target image sensor;

[0038] The first image data is input into a preset super-resolution neural network model to obtain second image data whose resolution is greater than the first image data, output by the preset super-resolution neural network model.

[0039] The present invention also provides a data processing system for an image sensor, comprising:

[0040] a data acquisition module, configured to acquire first image data, second image data, and third image data of a target object based on a target image sensor, wherein the first image data and the second image data have the same exposure, the second image data has a higher exposure than the third image data, the first image data and the third image data have the same resolution, and the resolution of the second image data is an even multiple of the resolution of the first image data;

[0041] A first filtering module, used for constructing a second Gaussian pyramid of second image data;

[0042] A weight analysis module, used to obtain a weight map corresponding to the second image data according to the pixel attribute of each pixel in the second image data, and construct a weighted Gaussian pyramid of the weight map;

[0043] A fine filtering module, used to construct a first Gaussian pyramid and a first Laplacian pyramid of the first image data by taking the image of each layer in the second Gaussian pyramid as a reference for sampling;

[0044] A second filtering module, used for constructing a third Gaussian pyramid and a third Laplacian pyramid of the third image data;

[0045] The exposure fusion module is used to use the weighted Gaussian pyramid as the weight of the first Laplacian pyramid, and perform exposure fusion on the first image data and the third image data according to the first Gaussian pyramid, the first Laplacian pyramid, the third Gaussian pyramid, and the third Laplacian pyramid to obtain a target output image of the target object.

[0046] The beneficial effects of adopting the above embodiment are:

[0047] The present invention provides a data processing method for an image sensor. The method first acquires first image data, second image data and third image data of a target object based on a target image sensor, wherein the first image data and the second image data have the same exposure, the exposure of the second image data is higher than that of the third image, the first image data and the third image data have the same resolution, and the resolution of the second image data is an even multiple of the resolution of the first image data, then constructs a second Gaussian pyramid of the second image data, obtains a weight map corresponding to the second image data according to a pixel attribute of each pixel in the second image data, and constructs a weighted Gaussian pyramid of the weight map, then uses an image of each layer in the second Gaussian pyramid as a reference for sampling, constructs a first Gaussian pyramid and a first Laplacian pyramid of the first image data, and constructs a third Gaussian pyramid and a third Laplacian pyramid of the third image data, finally uses the weighted Gaussian pyramid as the weight of the first Laplacian pyramid, and performs exposure fusion on the first image data and the third image data according to the first Gaussian pyramid, the first Laplacian pyramid, the third Gaussian pyramid and the third Laplacian pyramid to obtain a target output image of the target object. The present invention achieves image quality enhancement in low-light environments by fusing first image data and third image data of different exposures. Compared with existing technologies such as HDR (High Dynamic Range Imaging), the present invention achieves image quality enhancement in low-light environments by fusing only the first image data and the third image data, thereby reducing the hardware computing overhead. On this basis, by using second image data with a higher resolution than the first image data to construct a weight map for the fusion of the first image data, the accuracy of the weights during fusion is ensured while reducing the amount of computation. At the same time, the present invention also uses the second image data with a higher resolution as a reference during sampling, thereby reducing the pixel loss of the image data during sampling and further improving the accuracy of image data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A method flow chart of an embodiment of a data processing method for an image sensor provided by the present invention;

[0049] Figure 2 for Figure 1 A method flow chart of an embodiment of step S104;

[0050] Figure 3 for Figure 2 A method flow chart of step S201 in an embodiment;

[0051] Figure 4 A schematic diagram of pixel distribution during downsampling in one embodiment;

[0052] Figure 5 for Figure 2 A method flow chart of an embodiment of step S202;

[0053] Figure 6 This is a method flow chart of an embodiment of a data processing method for an image sensor provided by the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Combination Figure 1 As shown, a specific embodiment of the present invention discloses a data processing method for an image sensor, comprising:

[0056] S101, acquiring first image data, second image data, and third image data of a target object based on a target image sensor, wherein the first image data and the second image data have the same exposure, the second image data has a higher exposure than the third image data, the first image data and the third image data have the same resolution, and the resolution of the second image data is an even multiple of the resolution of the first image data;

[0057] S102, constructing a second Gaussian pyramid for second image data;

[0058] S103, obtaining a weight map corresponding to the second image data according to the pixel attribute of each pixel in the second image data, and constructing a weighted Gaussian pyramid of the weight map;

[0059] S104, constructing a first Gaussian pyramid and a first Laplacian pyramid of the first image data by taking the image of each layer in the second Gaussian pyramid as a reference for sampling;

[0060] S105, constructing a third Gaussian pyramid and a third Laplacian pyramid of the third image data;

[0061] S106 , using the weighted Gaussian pyramid as the weight of the first Laplacian pyramid, performing exposure fusion on the first image data and the third image data according to the first Gaussian pyramid, the first Laplacian pyramid, the third Gaussian pyramid, and the third Laplacian pyramid to obtain a target output image of the target object.

[0062] It should be noted that the first Gaussian pyramid, the second Gaussian pyramid, and the third Gaussian pyramid involved in the present invention are Gaussian pyramids corresponding to the first image data, the second image data, and the third image data, respectively. The Gaussian pyramid is a multi-scale image representation generated by gradually downsampling and smoothing the image. It contains a series of images, each of which is smaller than the previous image, but retains the characteristics of the original image.

[0063] Similarly, the first Laplacian pyramid, the second Laplacian pyramid, and the third Laplacian pyramid involved in the present invention are Laplacian pyramids corresponding to the first image data, the second image data, and the third image data, respectively. The Laplacian pyramid is obtained from the Gaussian pyramid, which represents the detail information of the image. It is generated by subtracting the enlarged version of the previous level from each coarser image in the Gaussian pyramid. This pyramid contains detail information such as the edge and texture of the image.

[0064] The weight map represents the weight of each pixel in the image during exposure fusion, that is, in image exposure fusion, the weight usually specifies the proportion of each input image in the final composite image. These weights are adjusted according to factors such as the exposure level and quality of each image to ensure that the brightness, contrast and details of each part of the composite image are appropriately preserved and expressed.

[0065] It is understandable that the above-mentioned Gaussian pyramid, Laplacian pyramid and related concepts and implementation methods of weights are all existing technologies that can be understood by those skilled in the art, so they are not explained in detail herein.

[0066] Compared to existing technologies such as HDR (High Dynamic Range Imaging), the present invention achieves image quality enhancement in low-light environments only by fusing the first image data and the third image data, reducing the overhead of hardware operations. On this basis, by using the second image data with a higher resolution than the first image data to construct a weight map for the fusion of the first image data, the accuracy of the weights during fusion is ensured while reducing the amount of calculation. At the same time, the present invention also uses the second image data with a higher resolution as a reference for sampling, which reduces the pixel loss caused by the image data during sampling and further improves the accuracy of image data processing.

[0067] Specifically, the above step S101 can be implemented by hardware itself, and by hardware control of the target image sensor itself, to achieve image acquisition with different exposures and different resolutions, and the present invention further provides a preferred solution:

[0068] In a preferred embodiment, the above step S101, acquiring the first image data, the second image data and the third image data of the target object based on the target image sensor, specifically includes:

[0069] acquiring first image data and third image data of a target object based on a target image sensor;

[0070] The first image data is input into a preset super-resolution neural network model to obtain second image data whose resolution is greater than the first image data, output by the preset super-resolution neural network model.

[0071] The above process obtains a super-resolution image by presetting the super-resolution neural network model, so that the target image sensor only needs to obtain two images with the same resolution and different exposure images when acquiring images. Although the amount of calculation is increased, the hardware actions are further reduced and the image acquisition time is improved.

[0072] A preset super-resolution neural network model commonly used for super-resolution reconstruction is SRCNN (Super-Resolution Convolutional Neural Network), which is a classic convolutional neural network architecture specially designed to convert low-resolution images into high-resolution images.

[0073] The basic structure of SRCNN consists of three main parts:

[0074] Convolutional Layer: SRCNN contains multiple convolutional layers to extract features of the input low-resolution image.

[0075] ReLU activation function: ReLU (Rectified Linear Unit) is applied as the activation function after each convolutional layer to introduce nonlinearity and increase the expressive power of the network.

[0076] Upsampling layer: The last layer is an upsampling layer that maps the low-resolution features to the high-resolution space, thus generating a high-resolution image.

[0077] SRCNN learns how to accurately extract features from low-resolution images and generate high-resolution images by training on a large number of high-resolution and corresponding low-resolution image pairs. During the testing phase, SRCNN can process the input low-resolution image through the network and output a high-resolution image, effectively achieving super-resolution reconstruction of the image.

[0078] In addition to SRCNN, there are other popular neural network models, such as ESPCN (Efficient Sub-Pixel Convolutional Neural Network), VDSR (Very Deep Super-Resolution Network), etc., which can be used to achieve super-resolution reconstruction of images. These models have achieved remarkable results in the field of image processing and provide effective solutions for image enhancement and reconstruction.

[0079] In the above step S103, according to the pixel attributes of each pixel in the second image data, a weight map corresponding to the second image data is obtained, and a weighted Gaussian pyramid of the weight map is constructed. The pixel attribute refers to the relevant attribute that can evaluate the importance of a pixel, that is, contrast, brightness, saturation, etc. For example, if the weight of a pixel needs to be calculated, the brightness of the pixel can be directly normalized as the weight value of the pixel, or:

[0080] Obtaining the brightness value of the pixel as the first attribute value;

[0081] Calculate the variance of the saturation value (i.e., color channel value) of the pixel in the three channels of red, green, and blue (i.e., RGB channels) and the mean of the color value of the channel respectively, and calculate the average of the three variances as the second attribute value;

[0082] Performing Laplace filtering on the grayscale image of the second image data, and taking the absolute value of the Laplace filtering result as the third attribute value of the pixel;

[0083] The first attribute value, the second attribute value and the third attribute value are weighted and summed to obtain the weight value of the pixel, and the graph composed of the weight value of each pixel is the weight graph corresponding to the second image data.

[0084] Further, combined with Figure 2 As shown, in a preferred embodiment, the above step S104, using the image of each layer in the second Gaussian pyramid as a reference for sampling, constructing a first Gaussian pyramid and a first Laplacian pyramid of the first image data, specifically includes:

[0085] S201, constructing a first Gaussian pyramid of first image data by taking the brightness distribution of the image in each layer of the second Gaussian pyramid as a reference for downsampling;

[0086] S202 : Using pixels of the image in each layer of the second Gaussian pyramid as a reference for upsampling, construct a first Laplacian pyramid of the first image data based on the first Gaussian pyramid.

[0087] In the existing Gaussian pyramid construction process, each layer of the image needs to be downsampled, which reduces the image of each layer by a certain ratio to generate different levels of the pyramid. However, there is a defect of information loss in the downsampling process, because the pixels of the original image are compressed to a smaller size, resulting in the loss of some details and texture information, especially for smaller objects or areas with fine features.

[0088] In this embodiment, the brightness distribution of the image in each layer of the second Gaussian pyramid is used as a reference for downsampling to reduce the degree of information loss during the downsampling process and improve the image processing accuracy.

[0089] Similarly, in the process of constructing the existing Laplacian pyramid, each layer of the image needs to be upsampled. Upsampling amplifies the image of each layer so that it can be subtracted from the image of the next layer to obtain detail information. However, the upsampling process also has some defects, such as the possibility of introducing image blur or artifacts, especially when a simple interpolation method is used, which will lead to a decrease in image quality and affect the accuracy and reliability of the pyramid.

[0090] In this embodiment, the pixels of the image in each layer of the second Gaussian pyramid are also used as references for upsampling, which increases the scientific rationality of pixel supplementation and further improves the quality of image processing.

[0091] Specifically, combined Figure 3 As shown, in a preferred embodiment, the above step S201, using the brightness distribution of the image of each layer in the second Gaussian pyramid as a reference for downsampling, constructing a first Gaussian pyramid of the first image data, specifically includes:

[0092] S301, using the first image data as a first initial image;

[0093] S302, convolving the first initial image based on a preset convolution kernel to obtain a Gaussian smoothed image;

[0094] S303, dividing the Gaussian smoothed image into a plurality of 2×2 image blocks, retaining the pixel with the highest brightness in each image block according to the brightness distribution of an image with the same resolution as the Gaussian smoothed image in the second Gaussian pyramid, and obtaining a first Gaussian pyramid image;

[0095] S304, using the first Gaussian pyramid image as a new first initial image, repeatedly convolving the first initial image based on a preset convolution kernel to obtain a Gaussian smoothed image, dividing the Gaussian smoothed image into a plurality of 2×2 image blocks, retaining the pixel with the highest brightness in each image block according to the brightness distribution of an image with the same resolution as the Gaussian smoothed image in the second Gaussian pyramid, and obtaining a first Gaussian pyramid image, until a preset end condition is reached, and a first Gaussian pyramid consisting of multiple layers of first Gaussian pyramid images is obtained.

[0096] In the downsampling process of the prior art, the pixel data of odd rows / columns or even rows / columns are usually removed in a simple and rough manner to achieve downsampling, which obviously causes more information loss. In the above process, due to its high-resolution characteristics, the second image data makes the image of each layer in the second Gaussian pyramid more realistic and more referenceable. Therefore, in this embodiment, the second Gaussian pyramid is used as a reference, so that in the upsampling process, the Gaussian pyramid image of each layer is the image with the highest relative brightness, which means that the Gaussian pyramid image of each layer retains the most graphic information as much as possible, improves the compensation strength for low light when it is subsequently fused with the third image data, and improves the quality of image data processing.

[0097] Specifically, in a preferred embodiment, the above step of: dividing the Gaussian smoothed image into a plurality of 2×2 image blocks, retaining the pixel with the highest brightness in each image block according to the brightness distribution of an image with the same resolution as the Gaussian smoothed image in the second Gaussian pyramid, and obtaining a first Gaussian pyramid image, specifically includes:

[0098] Divide the Gaussian smoothed image into multiple 2×2 image blocks;

[0099] Obtain an image block as a target image block, obtain a pixel in the target image block as a target pixel, and obtain an image in the second Gaussian pyramid with the same resolution as the Gaussian smoothed image as a down-sampling reference image;

[0100] Performing a weighted summation on the brightness values ​​of the pixels corresponding to the target pixel and the brightness values ​​of the pixels adjacent to the pixels corresponding to the target pixel in the downsampled reference image to obtain a brightness description value of the target pixel;

[0101] The pixel with the highest brightness value in each image block is retained to obtain a first Gaussian pyramid image.

[0102] The brightness description value in the above process is used to describe the degree of brightness change at the location of a pixel. Downsampling is performed through the brightness description value, so that when deciding which pixel to retain, this embodiment not only refers to the brightness value information of the pixel itself, but also combines the brightness information of other pixels around the pixel to calculate the brightness description value of the target pixel, so that the downsampled pixel screening is more accurate. Figure 4 This is an example of pixel distribution in a specific downsampling process. In the figure, multiple pixels are divided by dotted lines, and the target image block is divided by solid lines, where pixel 1 is the target pixel, and pixels 2-8 are pixels adjacent to pixel 1. The weight of pixel 1 can be set to 0.6, and the weight of pixels 2-8 can be set to 0.05. The brightness of pixels 1-8 is weighted and summed to obtain the brightness description value of the target pixel. The larger the value, the higher the brightness of the target pixel in its area, and the more information is retained. The greater the role of the pixel in low light compensation, the more the pixel should be retained.

[0103] Further, combined with Figure 5 As shown, in a preferred embodiment, the above step S202, using the pixels of the image of each layer in the second Gaussian pyramid as a reference for upsampling, constructing a first Laplacian pyramid of the first image data based on the first Gaussian pyramid, specifically includes:

[0104] S501, using a first Gaussian pyramid image at the top layer of a first Gaussian pyramid as a second initial image;

[0105] S502, expanding the second initial image to twice its original value in the horizontal and vertical directions to obtain an expanded image, and filling the newly added pixels in the expanded image with pixels at the same position in the image with the same resolution as the expanded image in the second Gaussian pyramid;

[0106] S503, convolving the expanded image based on a preset convolution kernel to obtain a Gaussian convolution image;

[0107] S504, obtaining a first Gaussian pyramid with the same resolution as the Gaussian convolution image in the first Gaussian pyramid as a new reference image, calculating a residual between the reference image and the Gaussian convolution image, and obtaining a first Laplacian pyramid image;

[0108] S505: taking the reference image as a new second initial image, and repeatedly taking the first Gaussian pyramid image at the top layer in the first Gaussian pyramid as the second initial image, expanding the second initial image to twice the original value in the horizontal and vertical directions to obtain an extended image, filling the newly added pixels in the extended image with pixels at the same position in the image with the same resolution as the extended image in the second Gaussian pyramid, convolving the extended image based on a preset convolution kernel to obtain a Gaussian convolution image, obtaining a first Gaussian pyramid with the same resolution as the Gaussian convolution image in the first Gaussian pyramid as a new reference image, calculating a residual between the reference image and the Gaussian convolution image, and obtaining a first Laplacian pyramid image, until the reference image is the image at the bottom layer in the first Gaussian pyramid, and a first Laplacian pyramid consisting of multiple layers of first Laplacian pyramid images is obtained.

[0109] In the prior art, when upsampling is performed, the newly added pixels in the expanded image are usually filled with 0, which causes the image obtained by sequential upsampling to become increasingly blurred, and the error of the first Laplacian pyramid image obtained will also become increasingly larger. Therefore, in this embodiment, the pixels at the same position in the second Gaussian pyramid image are directly used for filling, which eliminates the uncertainty during image expansion and improves the accuracy of upsampling.

[0110] Further, in a preferred embodiment, the step S106, using the weighted Gaussian pyramid as the weight of the first Laplacian pyramid, performing exposure fusion on the first image data and the third image data according to the first Gaussian pyramid, the first Laplacian pyramid, the third Gaussian pyramid, and the third Laplacian pyramid to obtain a target output image of the target object, includes:

[0111] Using the weighted Gaussian pyramid as the weight of the first Laplacian pyramid, fusing the first Laplacian pyramid and the third Laplacian pyramid to obtain a hybrid Laplacian pyramid;

[0112] Using the weighted Gaussian pyramid as the weight of the first Laplacian pyramid, fusing the top-layer image in the first Gaussian pyramid with the top-layer image in the third Gaussian pyramid to obtain an image fusion starting image;

[0113] The pixels of the image in each layer of the second Gaussian pyramid are used as the reference for upsampling, and upsampling is performed in sequence based on the image fusion starting picture, and is added layer by layer with the hybrid Laplacian pyramid to obtain an image with the same resolution as the third image data as the target output image of the target object.

[0114] The above process is the process of performing exposure fusion through the Laplacian pyramid. Compared with the prior art, this embodiment has two main improvements over the prior art:

[0115] On the one hand, a weighted Gaussian pyramid is used as the weight of the first Laplacian pyramid. In the prior art, weights obtained by analyzing the first image data (specifically, a Gaussian pyramid formed by these weights) are usually used as the weights of the first Laplacian pyramid in the exposure fusion process. However, this embodiment uses weights obtained by analyzing second image data with higher resolution (i.e., a weighted Gaussian pyramid) as the weights of the first Laplacian pyramid, which improves the credibility of the weights and increases the accuracy.

[0116] On the other hand, in the upsampling process based on the hybrid Laplacian pyramid, this embodiment also adopts the method of "using the pixels of the image in each layer of the second Gaussian pyramid as a reference for upsampling" (the specific process may be the same as step S202 above), so as to improve the accuracy of low-light compensation of the final image.

[0117] Combination Figure 6 As shown, the present invention also provides a data processing system for an image sensor, comprising:

[0118] A data acquisition module 610 is configured to acquire first image data, second image data, and third image data of a target object based on a target image sensor, wherein the first image data and the second image data have the same exposure, the second image data has a higher exposure than the third image data, the first image data and the third image data have the same resolution, and the resolution of the second image data is an even multiple of the resolution of the first image data;

[0119] A first filtering module 620, configured to construct a second Gaussian pyramid for second image data;

[0120] A weight analysis module 630, for obtaining a weight map corresponding to the second image data according to the pixel attribute of each pixel in the second image data, and constructing a weighted Gaussian pyramid of the weight map;

[0121] A fine filtering module 640 is used to construct a first Gaussian pyramid and a first Laplacian pyramid of the first image data by taking the image of each layer in the second Gaussian pyramid as a reference for sampling;

[0122] A second filtering module 650, configured to construct a third Gaussian pyramid and a third Laplacian pyramid of the third image data;

[0123] The exposure fusion module 660 is used to use the weighted Gaussian pyramid as the weight of the first Laplacian pyramid, and perform exposure fusion on the first image data and the third image data according to the first Gaussian pyramid, the first Laplacian pyramid, the third Gaussian pyramid, and the third Laplacian pyramid to obtain a target output image of the target object.

[0124] It should be noted here that the corresponding system provided in the above embodiments can implement the technical solutions described in the above method embodiments. The specific implementation principles of the above modules or units can be found in the corresponding contents in the above method embodiments, which will not be repeated here.

[0125] The present invention provides a data processing method for an image sensor. The method first acquires first image data, second image data and third image data of a target object based on a target image sensor, wherein the first image data and the second image data have the same exposure, the exposure of the second image data is higher than that of the third image, the first image data and the third image data have the same resolution, and the resolution of the second image data is an even multiple of the resolution of the first image data, then constructs a second Gaussian pyramid of the second image data, obtains a weight map corresponding to the second image data according to a pixel attribute of each pixel in the second image data, and constructs a weighted Gaussian pyramid of the weight map, then uses an image of each layer in the second Gaussian pyramid as a reference for sampling, constructs a first Gaussian pyramid and a first Laplacian pyramid of the first image data, and constructs a third Gaussian pyramid and a third Laplacian pyramid of the third image data, finally uses the weighted Gaussian pyramid as the weight of the first Laplacian pyramid, and performs exposure fusion on the first image data and the third image data according to the first Gaussian pyramid, the first Laplacian pyramid, the third Gaussian pyramid and the third Laplacian pyramid to obtain a target output image of the target object. The present invention achieves image quality enhancement in low-light environments by fusing first image data and third image data of different exposures. Compared with existing technologies such as HDR (High Dynamic Range Imaging), the present invention achieves image quality enhancement in low-light environments by fusing only the first image data and the third image data, thereby reducing the hardware computing overhead. On this basis, by using second image data with a higher resolution than the first image data to construct a weight map for the fusion of the first image data, the accuracy of the weights during fusion is ensured while reducing the amount of computation. At the same time, the present invention also uses the second image data with a higher resolution as a reference during sampling, thereby reducing the pixel loss of the image data during sampling and further improving the accuracy of image data processing.

[0126] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0127] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method for an image sensor, characterized in that: include: Acquire first image data, second image data, and third image data of the target object based on the target image sensor, wherein the first image data and the second image data have the same exposure, the second image data has a higher exposure than the third image data, the first image data and the third image data have the same resolution, and the resolution of the second image data is an even multiple of the resolution of the first image data; constructing a second Gaussian pyramid of the second image data; According to the pixel attribute of each pixel in the second image data, a weight map corresponding to the second image data is obtained, and a weighted Gaussian pyramid of the weight map is constructed; Using the image of each layer in the second Gaussian pyramid as a reference for sampling, constructing a first Gaussian pyramid and a first Laplacian pyramid of the first image data; constructing a third Gaussian pyramid and a third Laplacian pyramid of third image data; Using the weighted Gaussian pyramid as the weight of the first Laplacian pyramid, performing exposure fusion on the first image data and the third image data according to the first Gaussian pyramid, the first Laplacian pyramid, the third Gaussian pyramid, and the third Laplacian pyramid to obtain a target output image of the target object; The weighted Gaussian pyramid is used as the weight of the first Laplacian pyramid, and the first image data and the third image data are exposed and fused according to the first Gaussian pyramid, the first Laplacian pyramid, the third Gaussian pyramid, and the third Laplacian pyramid to obtain a target output image of the target object, including: Using the weighted Gaussian pyramid as the weight of the first Laplacian pyramid, fusing the first Laplacian pyramid and the third Laplacian pyramid to obtain a hybrid Laplacian pyramid; Using the weighted Gaussian pyramid as the weight of the first Laplacian pyramid, fusing the top image of the first Gaussian pyramid with the top image of the third Gaussian pyramid to obtain an image fusion starting image; The pixels of the image in each layer of the second Gaussian pyramid are used as the reference for upsampling, and the image fusion starting image is upsampled in sequence, and is added layer by layer with the hybrid Laplacian pyramid to obtain an image with the same resolution as the third image data as the target output image of the target object.

2. The data processing method of the image sensor according to claim 1, characterized in that: The method of constructing a first Gaussian pyramid and a first Laplacian pyramid of the first image data by taking the image of each layer in the second Gaussian pyramid as a reference for sampling includes: Using the brightness distribution of the image in each layer of the second Gaussian pyramid as a reference for downsampling, a first Gaussian pyramid of the first image data is constructed; The pixels of the image in each layer of the second Gaussian pyramid are used as references for upsampling, and a first Laplacian pyramid of the first image data is constructed based on the first Gaussian pyramid.

3. The data processing method of the image sensor according to claim 2, characterized in that: The method of constructing a first Gaussian pyramid of the first image data by taking the brightness distribution of the image of each layer in the second Gaussian pyramid as a reference for downsampling includes: using the first image data as a first initial image; Convolving the first initial image based on a preset convolution kernel to obtain a Gaussian smoothed image; The Gaussian smoothed image is divided into multiple 2×2 image blocks, and according to the brightness distribution of the image with the same resolution as the Gaussian smoothed image in the second Gaussian pyramid, the pixel with the highest brightness in each image block is retained to obtain a first Gaussian pyramid image; The first Gaussian pyramid image is used as a new first initial image, and the first initial image is repeatedly convolved based on a preset convolution kernel to obtain a Gaussian smoothed image, and the Gaussian smoothed image is divided into multiple 2×2 image blocks, and the pixel with the highest brightness in each image block is retained according to the brightness distribution of an image with the same resolution as the Gaussian smoothed image in the second Gaussian pyramid to obtain a first Gaussian pyramid image, until a preset end condition is reached, and a first Gaussian pyramid consisting of multiple layers of first Gaussian pyramid images is obtained.

4. The data processing method of the image sensor according to claim 3, characterized in that: The Gaussian smoothed image is divided into a plurality of 2×2 image blocks, and according to the brightness distribution of an image with the same resolution as the Gaussian smoothed image in the second Gaussian pyramid, the pixel with the highest brightness in each image block is retained to obtain a first Gaussian pyramid image, including: Divide the Gaussian smoothed image into multiple 2×2 image blocks; Obtain an image block as a target image block, obtain a pixel in the target image block as a target pixel, and obtain an image in a second Gaussian pyramid with the same resolution as the Gaussian smoothed image as a down-sampling reference image; Performing a weighted summation on the brightness values ​​of the pixels corresponding to the target pixel and the brightness values ​​of the pixels adjacent to the pixels corresponding to the target pixel in the downsampled reference image to obtain a brightness description value of the target pixel; The pixel with the highest brightness description value in each image block is retained to obtain a first Gaussian pyramid image.

5. The data processing method of the image sensor according to claim 2, characterized in that: The method of taking pixels of an image in each layer of the second Gaussian pyramid as a reference for upsampling and constructing a first Laplacian pyramid of the first image data based on the first Gaussian pyramid includes: Using the first Gaussian pyramid image at the top layer of the first Gaussian pyramid as the second initial image; The second initial image is expanded to twice its original value in the horizontal and vertical directions to obtain an expanded image, and the newly added pixels in the expanded image are filled with pixels at the same position in the image with the same resolution as the expanded image in the second Gaussian pyramid; Convolve the expanded image based on the preset convolution kernel to obtain a Gaussian convolution image; Obtaining a first Gaussian pyramid having the same resolution as the Gaussian convolution image in the first Gaussian pyramid as a new reference image, calculating a residual between the reference image and the Gaussian convolution image, and obtaining a first Laplacian pyramid image; The reference image is used as a new second initial image, and the first Gaussian pyramid image at the top layer in the first Gaussian pyramid is repeatedly used as the second initial image, the second initial image is expanded to twice the original in the horizontal and vertical directions to obtain an extended image, the newly added pixels in the extended image are filled with pixels at the same position in the image with the same resolution as the extended image in the second Gaussian pyramid, the extended image is convolved based on a preset convolution kernel to obtain a Gaussian convolution image, the first Gaussian pyramid with the same resolution as the Gaussian convolution image in the first Gaussian pyramid is obtained as a new reference image, the residual between the reference image and the Gaussian convolution image is calculated, and a first Laplacian pyramid image is obtained, until the reference image is the bottom layer image in the first Gaussian pyramid, and a first Laplacian pyramid consisting of multiple layers of first Laplacian pyramid images is obtained.

6. The data processing method of the image sensor according to claim 1, characterized in that: The method of acquiring first image data, second image data, and third image data of a target object based on a target image sensor includes: acquiring first image data and third image data of a target object based on a target image sensor; The first image data is input into a preset super-resolution neural network model to obtain second image data whose resolution is greater than the first image data, output by the preset super-resolution neural network model.

7. A data processing system for an image sensor, characterized in that: include: a data acquisition module, configured to acquire first image data, second image data, and third image data of a target object based on a target image sensor, wherein the first image data and the second image data have the same exposure, the second image data has a higher exposure than the third image data, the first image data and the third image data have the same resolution, and the resolution of the second image data is an even multiple of the resolution of the first image data; A first filtering module, used for constructing a second Gaussian pyramid of second image data; A weight analysis module, used to obtain a weight map corresponding to the second image data according to the pixel attribute of each pixel in the second image data, and construct a weighted Gaussian pyramid of the weight map; A fine filtering module, used to construct a first Gaussian pyramid and a first Laplacian pyramid of the first image data by taking the image of each layer in the second Gaussian pyramid as a reference for sampling; A second filtering module, used for constructing a third Gaussian pyramid and a third Laplacian pyramid of the third image data; an exposure fusion module, configured to use the weighted Gaussian pyramid as the weight of the first Laplacian pyramid, perform exposure fusion on the first image data and the third image data according to the first Gaussian pyramid, the first Laplacian pyramid, the third Gaussian pyramid, and the third Laplacian pyramid, to obtain a target output image of the target object; The weighted Gaussian pyramid is used as the weight of the first Laplacian pyramid, and the first image data and the third image data are exposed and fused according to the first Gaussian pyramid, the first Laplacian pyramid, the third Gaussian pyramid, and the third Laplacian pyramid to obtain a target output image of the target object, including: Using the weighted Gaussian pyramid as the weight of the first Laplacian pyramid, fusing the first Laplacian pyramid and the third Laplacian pyramid to obtain a hybrid Laplacian pyramid; Using the weighted Gaussian pyramid as the weight of the first Laplacian pyramid, fusing the top image of the first Gaussian pyramid with the top image of the third Gaussian pyramid to obtain an image fusion starting image; The pixels of the image in each layer of the second Gaussian pyramid are used as the reference for upsampling, and the image fusion starting image is upsampled in sequence, and is added layer by layer with the hybrid Laplacian pyramid to obtain an image with the same resolution as the third image data as the target output image of the target object.

Images