Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: controlling a merging mode of four Bayer array image sensors based on an image amplification instruction of a user, so as to generate an original output image, carrying out the preset operation of the original output image, obtaining initial Bayer grating data corresponding to the original output image; performing a first reconstruction operation on the initial Bayer grid data to obtain first intermediate grid data, and performing a second reconstruction operation on the initial Bayer grid data to obtain second intermediate grid data; and integrating the first intermediate raster data and the second intermediate raster data into target raster data, and generating a target output image based on the target raster data. According to the technical scheme provided by the invention, the problem of image blurring in the digital zooming process can be solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] In existing digital cameras or mobile phone products, the zoom function is a very commonly used function. Through the zoom function, users can shoot objects as needed and accurately enlarge or reduce the picture.

[0003] Zoom is generally divided into optical zoom and digital zoom. Optical zoom requires the support of hardware devices such as optical lenses, which is often costly. Digital zoom scales the image based on an algorithm. It does not require additional hardware devices to the product and is lower in cost, so it has been widely used.

[0004] However, using algorithms to scale images will cause image distortion and reduced resolution, ultimately affecting image clarity and imaging quality. Summary of the invention

[0005] The purpose of the present application is to provide an image processing method, device, electronic device and computer-readable storage medium, which can solve the problem of image blur during digital zoom.

[0006] To achieve the above-mentioned purpose, the present application provides an image processing method on the one hand, which is applied to a device with a four-Bayer array image sensor, and the method includes: based on a user's image magnification instruction, controlling the merging mode of the four-Bayer array image sensor to generate an original output image, and performing a preset operation on the original output image to obtain initial Bayer grid data corresponding to the original output image, wherein the pixel arrangement of the original output image has a four-Bayer array structure; performing a first reconstruction operation on the initial Bayer grid data to obtain first intermediate grid data, and performing a second reconstruction operation on the initial Bayer grid data to obtain second intermediate grid data; integrating the first intermediate grid data and the second intermediate grid data into target grid data, and generating a target output image based on the target grid data, wherein the pixel arrangement of the target output image has a Bayer array structure.

[0007] To achieve the above-mentioned purpose, the present application further provides an image processing device on the other hand, the device has a four-Bayer array image sensor, and the device includes: an image preprocessing module, which is used to control the merging mode of the four-Bayer array image sensor based on the user's image magnification instruction to generate an original output image, and perform a preset operation on the original output image to obtain initial Bayer grid data corresponding to the original output image, wherein the pixel arrangement of the original output image has a four-Bayer array structure; a data reconstruction module, which is used to perform a first reconstruction operation on the initial Bayer grid data to obtain first intermediate grid data, and perform a second reconstruction operation on the initial Bayer grid data to obtain second intermediate grid data; an image output module, which is used to integrate the first intermediate grid data and the second intermediate grid data into target grid data, and generate a target output image based on the target grid data, wherein the pixel arrangement of the target output image has a Bayer array structure.

[0008] To achieve the above-mentioned purpose, the present application also provides an electronic device on the other hand, which includes a memory and a processor, and the memory is used to store a computer program. When the computer program is executed by the processor, the above-mentioned image processing method is implemented.

[0009] To achieve the above-mentioned purpose, the present application further provides a computer-readable storage medium on the other hand, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned image processing method is implemented.

[0010] It can be seen that the technical solution provided by the present application is that when the user issues an image magnification instruction, the four-Bayer array image sensor can use its own structural characteristics to merge multiple adjacent pixels into a large pixel, and use the merged large pixel to generate an enlarged low-resolution original output image. In order to improve the resolution of the original output image, the original output image can be processed to obtain the corresponding initial Bayer grid data, and then the above initial Bayer grid data is reconstructed twice using different reconstruction algorithms to obtain the first intermediate grid data and the second intermediate grid data respectively. Different reconstruction algorithms have different focuses. Compared with the initial Bayer grid number, the first intermediate grid data and the second intermediate grid data can contain more feature information and color information. Afterwards, the first intermediate grid data and the second intermediate grid data are fused, so that the above two groups of grid data are integrated into a group of target grid data, and the target output image is generated according to the above target grid data. Since the integrated target grid data contains richer feature information and color information, the target output image generated based on the above target grid data has better image details and color reproduction, and its resolution is higher. Through the above method, the low-resolution original output image can be reconstructed into a high-resolution target output image, thereby solving the problem of image blur in the digital zoom process. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 is a flowchart of an image processing method in an embodiment of the present application;

[0013] Figure 2 is a schematic diagram of functional modules of an image processing device in an embodiment of the present application;

[0014] Figure 3 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0015] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0016] In existing digital cameras or mobile phone products, the zoom function is a very commonly used function. Through the zoom function, users can shoot objects as needed and accurately enlarge or reduce the picture.

[0017] Zoom is generally divided into optical zoom and digital zoom. Optical zoom enlarges and reduces the scene to be photographed by moving the optical lens. The larger the optical zoom ratio, the farther the scene can be photographed without affecting the image quality. However, optical zoom requires the support of hardware equipment such as optical lenses, which is often expensive. Digital zoom is based on algorithms and uses "interpolation" processing to enlarge a part of the pixels on the image sensor. It does not require additional hardware equipment to be added to the product, and the cost is low, so it has been widely used. However, digital zoom will cause image distortion and resolution reduction, which will ultimately affect the clarity and image quality of the image.

[0018] Therefore, when using digital zoom technology to acquire images, how to avoid image distortion and resolution reduction has become a problem that needs to be solved urgently in this field.

[0019] The present application provides an image processing method, device, electronic device and computer-readable storage medium. After obtaining a low-resolution original output image using digital zoom technology, different reconstruction methods can be used to reconstruct the grid data of the original output image, and then the grid data reconstructed by different reconstruction methods are fused to obtain new grid data containing more feature information and color information. Thereafter, the above-mentioned new grid data can be used to generate an image with a higher resolution.

[0020] To facilitate understanding of the contents of the quad-Bayer array image sensor involved in this application, a brief introduction is given below.

[0021] The quad-Bayer array image sensor is also called a Quad Bayer image sensor or a 4cell1 image sensor. The pixel arrangement of a traditional image sensor adopts a Bayer structure, and its pixel arrangement is an RGGB structure. Compared with the traditional image sensor, the quad-Bayer array image sensor has made adjustments to the arrangement of specific pixels. It arranges each color in a combination of four pixels. Four pixels of the same color constitute a pixel set, and the pixel sets of each color are arranged according to the Bayer structure. This application refers to the pixel arrangement of the quad-Bayer array image sensor as a quad-Bayer array structure.

[0022] When the quad-Bayer array image sensor is working, it can synthesize multiple pixels into a large pixel, then use the synthesized large pixel to shoot the object, and then convert the large pixel into four small pixels, thereby restoring the quad-Bayer array to the traditional Bayer structure to achieve high-definition magnification.

[0023] Figure 1This is a flow chart of an image processing method in an embodiment of the present application. The above image processing method is applied to an electronic device having a quad-Bayer array image sensor. The specific form of the electronic device can be a mobile phone, a tablet computer, a digital camera, and other devices.

[0024] S101: Based on an image magnification instruction of a user, controlling a merging mode of the four-Bayer array image sensor to generate an original output image, and performing a preset operation on the original output image to obtain initial Bayer grid data corresponding to the original output image, wherein a pixel arrangement of the original output image has a four-Bayer array structure.

[0025] In this embodiment, when the user needs to use the electronic device to obtain the enlarged image, the user can input an image enlargement instruction to the electronic device. For example, the user can input the desired magnification in the input interface of the electronic device, or directly enlarge the preview image on the touch screen of the electronic device. When the electronic device receives the image enlargement instruction input by the user, the electronic device can control the internal four-Bayer array image sensor to enable the merging mode, thereby merging multiple pixels into a larger pixel. Specifically, the four-Bayer array image sensor can merge adjacent pixels in the same row on the chip pixel array into a larger pixel, or merge pixels in different rows on the chip pixel array into a larger pixel. Afterwards, the four-Bayer array image sensor can use the above-mentioned larger pixels to convert the external optical image into an electronic signal, thereby generating an enlarged original output image.

[0026] In one embodiment, based on the image magnification instruction of the user, controlling the merging mode of the four-Bayer array image sensor to generate the original output image can be achieved in the following manner:

[0027] First, four adjacent same-color pixels in the quad-Bayer array image sensor are merged into one replacement pixel;

[0028] Then, image information is obtained using the replacement pixels, and the original output image is generated based on the image information.

[0029] For example, when the user needs to achieve a 2x magnification function, the above-mentioned four-Bayer array image sensor can merge four adjacent pixels of the same color on the image sensor into one replacement pixel, and then use the replacement pixel to obtain image information such as the color and brightness of the scene, and generate the original output image based on the above image information. Since the area of ​​the replacement pixel is larger than the area of ​​each pixel before merging, the replacement pixel can obtain better photosensitivity and low noise performance.

[0030] It should be pointed out that when the four-Bayer array image sensor enables the merging mode, every four adjacent pixels on the image sensor will be merged into one large pixel, so the pixel arrangement of the original output image generated by the image sensor at this time also conforms to the four-Bayer array structure. At the same time, since multiple pixels are merged into one large pixel, the pixel density on the image sensor will be reduced, and the resolution of the original output image will also decrease.

[0031] Furthermore, in order to improve the resolution of the original output image, a preset operation may be performed on the original output image to obtain initial Bayer grid data corresponding to the original output image, and then the high-resolution image may be reconstructed using the initial Bayer grid data.

[0032] In one embodiment, the original output image may be convolved using convolution kernels, filters, and other techniques to reduce the noise of the original output image, and the image features of each replacement pixel in the original output image may be extracted, thereby obtaining the color information and brightness value corresponding to each replacement pixel. The original output image is then rasterized to rasterize the original output image into initial Bayer grid data. Specifically, a pixel array using different color information may be generated based on different color information, and then the brightness value may be assigned to the corresponding pixel in the pixel array to obtain a set of Bayer grid data (i.e., initial Bayer grid data).

[0033] It should be pointed out that in practical applications, an FPGA (Field Programmable Gate Array) can be used to rasterize the original output image into initial Bayer grid data, or the above-mentioned preset operations can be compiled into a software algorithm and combined with a specific hardware circuit to rasterize the original output image into initial Bayer grid data.

[0034] S102: performing a first reconstruction operation on the initial Bayer grid data to obtain first intermediate grid data, and performing a second reconstruction operation on the initial Bayer grid data to obtain second intermediate grid data.

[0035] In this embodiment, after the original output image is converted into initial Bayer grid data, different reconstruction operations can be used to rearrange the initial Bayer grid data, and the grid data obtained after multiple reconstruction operations can be fused to generate a high-resolution image.

[0036] In one embodiment, a first reconstruction operation may be performed on the initial Bayer grid data to obtain first intermediate grid data. Specifically, for any one of the replacement pixels (i.e., the target pixel), the correlation between the target pixel and its surrounding pixels may be obtained, and then different weight values ​​may be assigned to the surrounding pixels according to the correlation, and then the initial Bayer grid data may be reconstructed into RGB grid data having a Bayer array structure using an interpolation algorithm. The present application refers to the RGB grid data as first intermediate grid data.

[0037] In practical applications, the initial Bayer grid data can be segmented with the target pixel as the center position, so that the initial Bayer grid data is divided into multiple block areas, and the block area contains multiple pixels. For the block area where the target pixel is located, the color information of each pixel in the block area can be further obtained, and then different weight values ​​are assigned to the above pixels based on the distance between the above pixels and the target pixel. For example, the closer a pixel in the block area is to the target pixel, the greater the weight value of the pixel. After assigning corresponding weight values ​​to each pixel in the block area, the RGB value of the target pixel can be calculated by combining the above weight values ​​and the color information of each pixel using interpolation algorithms such as the nearest neighbor interpolation method and the bilinear interpolation method.

[0038] After calculating the RGB value of the target pixel, the RGB value of the target pixel may be adjusted, and then the initial Bayer grid data may be rearranged into RGB grid data according to the adjusted RGB value of the target pixel, thereby minimizing image loss caused by pixel recombination and interpolation operations.

[0039] In one embodiment, the RGB value of the target pixel may be adjusted as follows:

[0040] First, the original output image is subjected to noise reduction processing, and a grayscale image corresponding to the original output image is generated;

[0041] Then, a reference color is selected from the original output image after the noise reduction process, and the channel information of the reference color is filled into the grayscale image to obtain the RGB value of the adjusted target pixel.

[0042] In practical applications, since the signal collected by the image sensor under weak light conditions is weak and easily affected by noise, which ultimately affects the quality of the original output image, the original output image can be subjected to noise reduction processing using convolution kernels and filters, thereby enhancing the clarity and contrast of the original output image and improving the color balance and purity of the original output image. After the original output image is subjected to noise reduction processing, the RGB information of the original output image can be further separated to generate a grayscale image corresponding to the original output image. Afterwards, the color information of the original output image after noise reduction processing is extracted, and a color information is selected from the above color information as the reference color. After determining the reference color, the channel information of the above reference color can be filled into the above grayscale image to generate a new color image. Furthermore, for the missing color information, interpolation or edge filling algorithms can be used for completion operations. At this time, the RGB value of the pixel corresponding to the target pixel in the newly generated color image can be used as the RGB value of the adjusted target pixel.

[0043] After performing the above operation on each replacement pixel in the initial Bayer grid data, the RGB value of each replacement pixel can be obtained, and the initial Bayer grid data can be rearranged into full-color RGB grid data (ie, first intermediate grid data) using the above RGB value.

[0044] In one embodiment, a neural network model may be used to perform a second reconstruction operation on the initial Bayer grid data to obtain second intermediate grid data. Specifically, the initial Bayer grid data may first be normalized to convert the initial Bayer grid data into target tensor data that can be used by the neural network model, and then the target tensor data is input into a pre-established convolutional neural network (i.e., a target neural network model) for processing. After the target neural network model performs encoding and decoding operations on the target tensor data, the initial Bayer grid data can be reconstructed into RGB grid data (i.e., second intermediate grid data) having a Bayer array structure, and the second intermediate grid data may be used to characterize an RGB image.

[0045] Inputting the target tensor data into the target neural network model to reconstruct the initial Bayer grid data into the second intermediate grid data can be achieved as follows:

[0046] First, convolution and pooling operations are performed on the target tensor data to obtain the feature map of the original output image, and the initial image is generated based on the feature map;

[0047] Then, a deconvolution operation and an upsampling operation are performed on the initial image to generate second intermediate grid data.

[0048] In practical applications, after the target tensor data is input into the target neural network model, the target neural network model can use the convolution layer in the model to perform convolution operations on the target tensor data to obtain local features of the original output image. In particular, multiple convolution layers can be set in the target neural network model according to actual needs to improve the accuracy of the local features output by the target neural network model. For example, a shallow convolution layer can be used to extract basic features such as texture and direction of the original output image, and then a deep convolution layer can be used to extract high-order features of the original output image. After the convolution layer obtains the local features of the original output image, the target neural network model can use the pooling layer in the model to perform pooling operations on the above local features to merge features with similar meanings into the same features, and finally generate a feature map corresponding to the original output image. After obtaining the feature map of the original output image, the target neural network model can generate a low-resolution initial image based on the above feature map.

[0049] Furthermore, the target neural network model can perform deconvolution and upsampling operations on the initial image, thereby fusing the low-resolution initial image with the feature information obtained during the encoding process to generate a high-resolution image. Afterwards, the high-resolution image is rasterized to obtain the second intermediate raster data.

[0050] It should be pointed out that when training the above-mentioned target neural network model, a large-scale labeled image data set can be obtained from ImageNet and other channels, or a private data set can be obtained by using a large number of shots with a four-Bayer array image sensor, and combined with the data obtained by the optical zoom lens, and divided into training sets, validation sets, and test sets after evaluation. At the same time, in the process of training the model, the quality of the generated image can be evaluated by using indicators such as root mean square error, structural similarity index, and peak signal-to-noise ratio, and the loss function can be defined to calculate the error between the generated image and the real image, and weight adjustment can be added to control the overfitting and underfitting of the model. The above-mentioned target neural network model can be integrated in the form of a hardware circuit to reduce the computing consumption of the CPU and achieve faster data processing.

[0051] S103: Integrate the first intermediate grid data and the second intermediate grid data into target grid data, and generate a target output image based on the target grid data, wherein a pixel arrangement of the target output image has a Bayer array structure.

[0052] In this embodiment, after obtaining the first intermediate raster data and the second intermediate raster data, a weighted average strategy can be used to fuse the two, and then a high-resolution target output image is generated based on the fused raster data (i.e., the target raster data) to achieve better image reconstruction effect and color restoration.

[0053] In one implementation, integrating the first intermediate raster data and the second intermediate raster data into the target raster data may be achieved in the following manner:

[0054] Firstly, respective weight coefficients are assigned to the first intermediate raster data and the second intermediate raster data, and a weighted average operation is performed on the first intermediate raster data and the second intermediate raster data according to the assigned weight coefficients to generate transition raster data;

[0055] Then, the transition grid data is demosaiced to obtain the target grid data.

[0056] In practical applications, the first intermediate raster data and the second intermediate raster data may be assigned respective weight coefficients, the sum of the two weight coefficients may be 1, and the RGB values ​​of each pixel in the first intermediate raster data and the second intermediate raster data may be weighted averaged according to the assigned weight coefficients according to the following formula:

[0057] P c =a·P1+(1-a)P2

[0058] Among them, P c represents the RGB value of the pixel at position c in the transition raster data, a represents the weight coefficient of the first intermediate raster data, P1 represents the RGB value of the pixel at position c in the first intermediate raster data, P2 represents the RGB value of the pixel at position c in the second intermediate raster data, and position c can be any position.

[0059] After performing a weighted average operation on all the pixels in the first intermediate raster data and the second intermediate raster data, a set of new raster data can be obtained, and the raster data is the transition raster data.

[0060] In the actual reconstruction process, due to unreasonable interpolation or missing information, the RGB values ​​of some pixels in the transition raster data may be discontinuous, which will eventually appear as mosaic or sawtooth phenomenon. To solve the above problems, the transition raster data can be demosaiced, that is, each pixel is averaged to eliminate the mosaic phenomenon, and then the target raster data is generated based on the demosaiced transition raster data.

[0061] In one implementation, de-mosaicing of transitional raster data may be achieved in the following manner:

[0062] Firstly, the cutting block size is set based on the image resolution and application scenario, and the transition grid data is divided into multiple image blocks according to the cutting block size;

[0063] Then, for any image block, the average grayscale value of all pixels in the image block is obtained, and the grayscale value of each pixel in the image block is set to the average grayscale value.

[0064] In practical applications, the above transition grid data can be firstly segmented. Specifically, the size of each block of the image to be processed can be determined according to the image resolution and the application scenario. Since the transition grid data can represent the image to be processed, determining the size of each block of the image to be processed can be regarded as dividing the transition grid data into image blocks of a certain size. In practical operation, the cutting block size can be set based on the image resolution and the application scenario, and then the transition grid data can be segmented into a plurality of image blocks based on the above cutting block size, wherein the size of each image block meets the above cutting block size.

[0065] After the transition grid data is divided into multiple image blocks, the average grayscale value of all pixels in the image block can be calculated. Specifically, for any image block, the total number of pixels in the image block can be counted first, and then the grayscale value of each pixel in the image block can be obtained, and the grayscale value of each pixel can be added up, and then divided by the total number of pixels to obtain the average grayscale value of all pixels in the image block. After obtaining the above average grayscale value, the average grayscale value can be assigned to all pixels in the image block, so that all pixels in the image block have the same grayscale value. By assigning the same grayscale value to all pixels in an image block, the smoothing of the pixels in the block can be achieved, thereby eliminating the mosaic phenomenon in the image block.

[0066] Furthermore, after the grayscale value of each pixel in the image block is set to the average grayscale value, different image blocks can be smoothed to eliminate the mosaic phenomenon between different image blocks. Specifically, since the pixels between adjacent blocks are in different environments, there may be discontinuity between the pixels between adjacent blocks. Therefore, it is possible to determine whether any two adjacent image blocks are continuous. If not, the two image blocks are averaged so that the two image blocks can be smoothly connected.

[0067] In one embodiment, after eliminating the mosaic phenomenon within the image block and the mosaic phenomenon between different image blocks, the image blocks after de-mosaicing can be spliced ​​to obtain raster data with uniformly distributed average grayscale values, which is the target raster data. Then, the image information contained in the above target raster data is used to convert it into a high-resolution full-color image, which is the target output image.

[0068] It can be seen that the technical solution provided by the present application is that when the user issues an image zoom instruction, the four-Bayer array image sensor can use its own structural characteristics to merge multiple adjacent pixels into a large pixel, and use the merged large pixel to generate an enlarged low-resolution original output image. In order to improve the resolution of the original output image, the original output image can be processed to obtain the corresponding initial Bayer grid data, and then the above initial Bayer grid data is reconstructed twice using different reconstruction algorithms to obtain the first intermediate grid data and the second intermediate grid data respectively. After that, the first intermediate grid data and the second intermediate grid data are fused, so that the above two groups of grid data are integrated into a group of target grid data, and the target output image is generated according to the above target grid data. Since the integrated target grid data contains richer feature information and color information, the target output image generated based on the above target grid data has better image details and color reproduction, and its resolution is higher. Through the above method, the low-resolution original output image can be reconstructed into a high-resolution target output image, thereby solving the problem of image blur during digital zoom.

[0069] At the same time, the technical solution of the present application uses a convolutional neural network to build a model, and fuses the RGB high-definition image generated by the model with the image reconstructed by the interpolation algorithm using a weighted average strategy, which can achieve better image reconstruction effect and color restoration.

[0070] See also Figure 2 The present application also provides an image processing device, the device having a quad-Bayer array image sensor, the device comprising:

[0071] an image preprocessing module, configured to control a merging mode of the four-Bayer array image sensor based on an image magnification instruction of a user to generate an original output image, and perform a preset operation on the original output image to obtain initial Bayer grid data corresponding to the original output image, wherein a pixel arrangement of the original output image has a four-Bayer array structure;

[0072] A data reconstruction module, configured to perform a first reconstruction operation on the initial Bayer grid data to obtain first intermediate grid data, and perform a second reconstruction operation on the initial Bayer grid data to obtain second intermediate grid data;

[0073] An image output module is used to integrate the first intermediate grid data and the second intermediate grid data into target grid data, and generate a target output image based on the target grid data, wherein a pixel arrangement of the target output image has a Bayer array structure.

[0074] In one embodiment, based on an image magnification instruction of a user, controlling the merging mode of the quad-Bayer array image sensor to generate an original output image includes:

[0075] Merging four adjacent pixels of the same color in the quad-Bayer array image sensor into one replacement pixel;

[0076] Image information is obtained using the replacement pixels, and the original output image is generated based on the image information.

[0077] In one embodiment, performing a preset operation on the original output image to obtain initial Bayer grid data corresponding to the original output image includes:

[0078] Performing a convolution operation on the original output image to obtain a brightness value of the original output image, and obtaining color information corresponding to each of the replacement pixels in the original output image;

[0079] A pixel array including the brightness value and the color information is generated to rasterize the original output image into the initial Bayer grid data.

[0080] In one embodiment, performing a first reconstruction operation on the initial Bayer grid data to obtain first intermediate grid data includes:

[0081] For any target pixel among the replacement pixels, the initial Bayer grid data is reconstructed into the first intermediate grid data according to the association between the target pixel and surrounding pixels, wherein the first intermediate grid data is RGB grid data with a Bayer array structure.

[0082] In one embodiment, reconstructing the initial Bayer grid data into the first intermediate grid data according to the association between the target pixel and surrounding pixels includes:

[0083] Taking the target pixel as the center position, performing region segmentation on the initial Bayer grid data;

[0084] Obtaining color information of each pixel in the area where the target pixel is located, and calculating the RGB value of the target pixel based on the distance between each pixel and the target pixel;

[0085] The RGB value of the target pixel is adjusted, and the initial Bayer grid data is rearranged into RGB grid data according to the adjusted RGB value of the target pixel.

[0086] In one embodiment, adjusting the RGB value of the target pixel includes:

[0087] Performing noise reduction processing on the original output image, and generating a grayscale image corresponding to the original output image;

[0088] A reference color is selected from the original output image after the noise reduction process, and the channel information of the reference color is filled into the grayscale image to obtain the adjusted RGB value of the target pixel.

[0089] In one embodiment, performing a second reconstruction operation on the initial Bayer grid data to obtain second intermediate grid data includes:

[0090] Normalizing the initial Bayer grid data to convert the initial Bayer grid data into target tensor data;

[0091] The target tensor data is input into a target neural network model to reconstruct the initial Bayer grid data into the second intermediate grid data, wherein the second intermediate grid data is RGB grid data having a Bayer array structure.

[0092] In one embodiment, inputting the target tensor data into a target neural network model to reconstruct the initial Bayer grid data into the second intermediate grid data includes:

[0093] Performing convolution and pooling operations on the target tensor data to obtain a feature map of the original output image, and generating an initial image according to the feature map;

[0094] A deconvolution operation and an upsampling operation are performed on the initial image to generate the second intermediate grid data.

[0095] In one embodiment, integrating the first intermediate raster data and the second intermediate raster data into target raster data includes:

[0096] Assigning respective weight coefficients to the first intermediate raster data and the second intermediate raster data, and performing a weighted average operation on the first intermediate raster data and the second intermediate raster data according to the assigned weight coefficients to generate transition raster data;

[0097] De-mosaicing is performed on the transition grid data to obtain the target grid data.

[0098] In one embodiment, performing demosaicing on the transition grid data includes:

[0099] Based on the image resolution and the application scenario, a cutting block size is set, and the transition grid data is cut into a plurality of image blocks according to the cutting block size;

[0100] For any one of the image blocks, an average grayscale value of all pixels in the image block is obtained, and the grayscale value of each pixel in the image block is set as the average grayscale value.

[0101] In one embodiment, after setting the grayscale value of each pixel in the image block to the average grayscale value, the method further includes:

[0102] It is determined whether any two adjacent image blocks are continuous. If not, the two image blocks are averaged to make the two image blocks smoothly connected.

[0103] In one embodiment, generating a target output image based on the target raster data includes:

[0104] Splicing the image blocks after demosaicing to generate the target raster data;

[0105] The target raster data is converted into the target output image.

[0106] See also Figure 3 The present application also provides an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program. When the computer program is executed by the processor, the image processing method described above can be implemented. Specifically, at the hardware level, the device may include a processor, an internal bus, and a memory. The memory may include a memory and a non-volatile memory. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it. A person skilled in the art can understand that Figure 3 The structure shown is only for illustration and does not limit the structure of the above device. Figure 3 More or fewer components shown in the figure may also include other processing hardware, such as a GPU (Graphics Processing Unit), or an external communication port, etc. Of course, in addition to the software implementation, this application does not exclude other implementations, such as logic devices or a combination of software and hardware, etc.

[0107] In this embodiment, the processor may include a central processing unit (CPU) or a graphics processing unit (GPU), and of course may also include other single-chip microcomputers, logic gate circuits, integrated circuits, etc. with logic processing capabilities, or appropriate combinations thereof. The memory described in this embodiment may be a memory device for storing information. In a digital system, a device that can store binary data may be a memory; in an integrated circuit, a circuit with a storage function without a physical form may also be a memory, such as a RAM, a FIFO, etc.; in a system, a storage device with a physical form may also be called a memory, etc. When implemented, the memory may also be implemented in the form of a cloud storage, and the specific implementation method is not limited in this specification.

[0108] It should be noted that the specific implementation of the electronic device in this specification can refer to the description of the method implementation method, and will not be described in detail here.

[0109] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned image processing method can be implemented.

[0110] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0111] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image processing method, characterized in that: The method is applied to a device having a quad-Bayer array image sensor, and the method comprises: Based on an image magnification instruction of a user, controlling a merging mode of the four-Bayer array image sensor to generate an original output image, and performing a preset operation on the original output image to obtain initial Bayer grid data corresponding to the original output image, wherein a pixel arrangement of the original output image has a four-Bayer array structure; Performing a first reconstruction operation on the initial Bayer grid data to obtain first intermediate grid data, and performing a second reconstruction operation on the initial Bayer grid data to obtain second intermediate grid data; The first intermediate grid data and the second intermediate grid data are integrated into target grid data, and a target output image is generated based on the target grid data, wherein a pixel arrangement of the target output image has a Bayer array structure.

2. The method according to claim 1, characterized in that Based on the image magnification instruction of the user, controlling the merging mode of the quad-Bayer array image sensor to generate an original output image includes: Merging four adjacent pixels of the same color in the quad-Bayer array image sensor into one replacement pixel; Image information is obtained using the replacement pixels, and the original output image is generated based on the image information.

3. The method according to claim 2, characterized in that Performing a preset operation on the original output image to obtain initial Bayer grid data corresponding to the original output image includes: Performing a convolution operation on the original output image to obtain a brightness value of the original output image, and obtaining color information corresponding to each of the replacement pixels in the original output image; A pixel array including the brightness value and the color information is generated to rasterize the original output image into the initial Bayer grid data.

4. The method according to claim 3, characterized in that Performing a first reconstruction operation on the initial Bayer grid data to obtain first intermediate grid data includes: For any target pixel among the replacement pixels, the initial Bayer grid data is reconstructed into the first intermediate grid data according to the association between the target pixel and surrounding pixels, wherein the first intermediate grid data is RGB grid data with a Bayer array structure.

5. The method according to claim 4, characterized in that Reconstructing the initial Bayer grid data into the first intermediate grid data according to the association between the target pixel and the surrounding pixels includes: Taking the target pixel as the center position, performing region segmentation on the initial Bayer grid data; Obtaining color information of each pixel in the area where the target pixel is located, and calculating the RGB value of the target pixel based on the distance between each pixel and the target pixel; The RGB value of the target pixel is adjusted, and the initial Bayer grid data is rearranged into RGB grid data according to the adjusted RGB value of the target pixel.

6. The method according to claim 5, characterized in that Adjusting the RGB value of the target pixel includes: Performing noise reduction processing on the original output image, and generating a grayscale image corresponding to the original output image; A reference color is selected from the original output image after the noise reduction process, and the channel information of the reference color is filled into the grayscale image to obtain the adjusted RGB value of the target pixel.

7. The method according to claim 4, characterized in that Performing a second reconstruction operation on the initial Bayer grid data to obtain second intermediate grid data includes: Normalizing the initial Bayer grid data to convert the initial Bayer grid data into target tensor data; The target tensor data is input into a target neural network model to reconstruct the initial Bayer grid data into the second intermediate grid data, wherein the second intermediate grid data is RGB grid data having a Bayer array structure.

8. The method according to claim 7, characterized in that Inputting the target tensor data into a target neural network model to reconstruct the initial Bayer grid data into the second intermediate grid data includes: Performing convolution and pooling operations on the target tensor data to obtain a feature map of the original output image, and generating an initial image according to the feature map; A deconvolution operation and an upsampling operation are performed on the initial image to generate the second intermediate grid data.

9. The method according to claim 7, characterized in that: Integrating the first intermediate raster data and the second intermediate raster data into target raster data includes: Assigning respective weight coefficients to the first intermediate raster data and the second intermediate raster data, and performing a weighted average operation on the first intermediate raster data and the second intermediate raster data according to the assigned weight coefficients to generate transition raster data; De-mosaicing is performed on the transition grid data to obtain the target grid data.

10. The method according to claim 9, characterized in that The demosaicing process of the transition grid data includes: Based on the image resolution and the application scenario, a cutting block size is set, and the transition grid data is cut into a plurality of image blocks according to the cutting block size; For any one of the image blocks, an average grayscale value of all pixels in the image block is obtained, and the grayscale value of each pixel in the image block is set as the average grayscale value.

11. The method according to claim 10, characterized in that After setting the grayscale value of each pixel in the image block to the average grayscale value, the method further includes: It is determined whether any two adjacent image blocks are continuous. If not, the two image blocks are averaged to make the two image blocks smoothly connected.

12. The method according to claim 11, characterized in that Generating a target output image based on the target raster data includes: Splicing the image blocks after demosaicing to generate the target raster data; The target raster data is converted into the target output image.

13. An image processing device, characterized in that: The device has a quad-Bayer array image sensor, and the device includes: an image preprocessing module, configured to control a merging mode of the four-Bayer array image sensor based on an image magnification instruction of a user to generate an original output image, and perform a preset operation on the original output image to obtain initial Bayer grid data corresponding to the original output image, wherein a pixel arrangement of the original output image has a four-Bayer array structure; A data reconstruction module, configured to perform a first reconstruction operation on the initial Bayer grid data to obtain first intermediate grid data, and perform a second reconstruction operation on the initial Bayer grid data to obtain second intermediate grid data; An image output module is used to integrate the first intermediate grid data and the second intermediate grid data into target grid data, and generate a target output image based on the target grid data, wherein a pixel arrangement of the target output image has a Bayer array structure.

14. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images