Image processing method, processing apparatus, electronic device, and storage medium

Abstract

An image processing method is disclosed. The image processing method includes the following steps: (S11) acquiring image data from an image sensor according to a preset image readout mode to obtain multiple frames of first pixel images and multiple frames of second pixel images; (S12) aligning and fusing the multiple frames of first pixel images to obtain a first processed pixel image, an alignment model, and fusion parameters; (S13) processing the multiple frames of second pixel images according to the alignment model and fusion parameters to obtain a second processed pixel image; and (S14) synthesizing the first processed pixel image and the second processed pixel image to obtain a target image. An image processing apparatus 10, an electronic device 100, and a computer-readable storage medium are also disclosed.

Description

Technical Field

[0001] This application relates to image processing technology, and in particular to an image processing method, an image processing apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Mobile devices such as smartphones are often equipped with image sensors to enable photography. In related technologies, multi-channel readout of image sensor data can be used to better process the image data, thereby improving the image quality. However, existing image sensor readout designs are overly complex, and the fusion of multi-channel image data into a single image can cause image artifacts, resulting in poor image quality. Summary of the Invention

[0003] In view of this, this application aims to at least partially solve one of the problems in the related art. Therefore, the object of this application is to provide an image processing method, an image processing apparatus, an electronic device, and a computer-readable storage medium.

[0004] The image processing method of this application includes:

[0005] Image data from the image sensor is acquired according to a preset image readout mode to obtain multiple frames of first pixel images and multiple frames of second pixel images;

[0006] Align and fuse multiple frames of the first pixel image to obtain a first processed pixel image, an alignment model, and fusion parameters;

[0007] The second processed pixel image is obtained by processing multiple frames of the second pixel image according to the alignment model and the fusion parameters respectively; and

[0008] The target image is obtained by combining the first processed pixel image and the second processed pixel image.

[0009] The image processing apparatus according to embodiments of this application includes:

[0010] The image preprocessing module is used to acquire image data from the image sensor according to a preset image readout mode to obtain multiple frames of first pixel images and multiple frames of second pixel images;

[0011] The first multi-frame processing module is used to align and fuse multiple frames of the first pixel image to obtain a first processed pixel image, an alignment model, and fusion parameters.

[0012] The second multi-frame processing module is used to process multiple frames of the second pixel image according to the alignment model and the fusion parameters to obtain a second processed pixel image; and

[0013] The synthesis module is used to synthesize the first processed pixel image and the second processed pixel image to obtain the target image.

[0014] The electronic device according to embodiments of this application includes an image sensor, a processor, and a memory; and

[0015] One or more programs, wherein the one or more programs are stored in the memory and executed by the processor, the programs including instructions for performing the image processing method. The image processing method includes: acquiring image data from an image sensor according to a preset image readout mode to obtain multiple frames of first pixel images and multiple frames of second pixel images; aligning and fusing the multiple frames of first pixel images to obtain a first processed pixel image, an alignment model, and fusion parameters; processing the multiple frames of second pixel images according to the alignment model and the fusion parameters respectively to obtain a second processed pixel image; and synthesizing the first processed pixel image and the second processed pixel image to obtain a target image.

[0016] The computer-readable storage medium of this application includes a computer program that, when executed by one or more processors, causes the processors to perform the image processing method. The image processing method includes: acquiring image data from an image sensor according to a preset image readout mode to obtain multiple frames of first pixel images and multiple frames of second pixel images; aligning and fusing the multiple frames of first pixel images to obtain a first processed pixel image, an alignment model, and fusion parameters; processing the multiple frames of second pixel images according to the alignment model and the fusion parameters to obtain a second processed pixel image; and synthesizing the first processed pixel image and the second processed pixel image to obtain a target image. Attached Figure Description

[0017] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0018] Figure 1 This is a schematic flowchart of an image processing method according to certain embodiments of this application;

[0019] Figure 2 This is a schematic diagram of a module of an image processing apparatus according to certain embodiments of this application;

[0020] Figure 3 This is a schematic diagram of a module of an electronic device according to certain embodiments of this application;

[0021] Figure 4 This is a schematic diagram of a module of an image sensor according to certain embodiments of this application;

[0022] Figure 5This is a scene diagram of an image processing method according to certain embodiments of this application;

[0023] Figure 6-10 This is a flowchart illustrating an image processing method according to certain embodiments of this application.

[0024] Figure 11 This is another schematic diagram of an electronic device according to certain embodiments of this application;

[0025] Figure 12 This is a schematic diagram showing the connection between a processor and a computer-readable storage medium in some embodiments of this application.

[0026] Explanation of key component symbols:

[0027] Electronic device 100, image processing device 10, preprocessing module 11, first multi-frame processing module 12, alignment unit 122, first fusion unit 124, second multi-frame processing module 13, second fusion unit 132, and compositing module 14.

[0028] Processor 20;

[0029] Image sensor 30, pixel array 301, vertical drive unit 302, control unit 303, column processing unit 304, and horizontal drive unit 305.

[0030] Memory 40, program 42, computer-readable storage medium 50. Detailed Implementation

[0031] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0032] Please see Figure 1 This application provides an image processing method, which includes the following steps:

[0033] S11, acquire image data from the image sensor according to the preset image readout mode to obtain multiple frames of first pixel images and multiple frames of second pixel images;

[0034] S12, Align and fuse multiple frames of first pixel images to obtain a first processed pixel image, alignment model and fusion parameters;

[0035] S13, the second processed pixel image is obtained by processing multiple frames of the second pixel image according to the alignment model and fusion parameters; and

[0036] S14, synthesize the first processed pixel image and the second processed pixel image to obtain the target image.

[0037] Please combine Figure 2 This application also provides an image processing apparatus 10 for processing the above-described image processing method. The image processing apparatus 10 includes a preprocessing module 11, a first multi-frame processing module 12, a second multi-frame processing module 13, and a synthesis module 14.

[0038] Step S11 can be implemented by the preprocessing module 11, step S12 can be implemented by the first multi-frame processing module 12, step S13 can be implemented by the second multi-frame processing module 13, and step S14 can be implemented by the synthesis module 14.

[0039] Alternatively, the preprocessing module 11 can be used to acquire image data from the image sensor according to a preset image readout mode to obtain multiple frames of first pixel images and multiple frames of second pixel images.

[0040] The first multi-frame processing module 12 can be used to align and fuse multiple first-pixel images to obtain a first processed pixel image, an alignment model, and fusion parameters.

[0041] The second multi-frame processing module 13 can be used to process the second pixel images of multiple frames according to the alignment model and fusion parameters to obtain the second processed pixel image.

[0042] The synthesis module 14 can be used to synthesize the first processed pixel image and the second processed pixel image to obtain the target image.

[0043] Please combine Figure 3 This application provides an electronic device 100, and the image processing method of this application can be performed by the electronic device 100. The electronic device 100 includes a processor 20 and an image sensor 30.

[0044] The processor 20 can be used to acquire image data from the image sensor 30 according to a preset image readout mode to obtain multiple frames of first pixel images and multiple frames of second pixel images. The processor 20 can be used to align and fuse the multiple frames of first pixel images to obtain a first processed pixel image, an alignment model, and fusion parameters; the processor 20 can also be used to process the multiple frames of second pixel images according to the alignment model and fusion parameters to obtain a second processed pixel image, and to synthesize the first processed pixel image and the second processed pixel image to obtain a target image.

[0045] In the image processing method, image processing apparatus 10, and electronic device 100 of this application, image data is divided into multiple frames of first pixel images and second pixel images in different channels by using a preset image readout mode. First, the multiple frames of first pixel images of the first pixel image are aligned and fused to obtain a first processed pixel image, an alignment model, and fusion parameters. Then, the multiple frames of second pixel images of the second pixel image are aligned according to the alignment model obtained from the first pixel image, and the aligned second pixel images are fused according to the fusion parameters obtained from the first pixel image to obtain a second processed pixel image. In this way, the quality of the first processed pixel image and the second processed pixel image can be synchronized, avoiding artifacts when the first processed pixel image and the second pixel image are synthesized into a target image, thus improving the image quality.

[0046] Electronic device 100 can be a mobile phone, tablet computer, laptop computer, smart wearable device (smartwatch, smart bracelet, smart helmet, smart glasses, etc.), virtual reality device, etc.

[0047] This embodiment uses a mobile phone as an example to illustrate electronic device 100; that is, the image processing method and image processing apparatus 10 are applied to, but not limited to, mobile phones. Image processing apparatus 10 can be hardware or software pre-installed on the mobile phone and can execute the image processing method when the mobile phone is started and run. For example, image processing apparatus 10 can be a low-level software code segment of the mobile phone or part of the operating system.

[0048] The image sensor 30 can be a camera component, and can be a complementary metal-oxide-semiconductor (CMOS) photosensitive element or a charge-coupled device (CCD) photosensitive element.

[0049] Please see Figure 4 The image sensor 30 may include a pixel array 301, a vertical drive unit 302, a control unit 303, a column processing unit 304, and a horizontal drive unit 305.

[0050] Image sensor 30 can generate image data after exposure via pixel array 301. Pixel array 301 can be a color filter array (CFA), comprising multiple photosensitive pixels arranged in a two-dimensional array (i.e., a two-dimensional matrix). Each photosensitive pixel includes an absorption region with different spectral absorber characteristics, and each photosensitive pixel includes a photoelectric conversion element. Each photosensitive pixel converts absorbed light into electrical charge based on the intensity of the incident light, enabling each photosensitive pixel to generate multiple pixel data with different color channels, thereby ultimately generating image data.

[0051] The vertical drive unit 302 includes a shift register and an address decoder. The vertical drive unit 302 includes readout scan and reset scan functions. Readout scan refers to sequentially scanning each photosensitive pixel row by row, reading signals from these photosensitive pixels row by row. The signal output by each photosensitive pixel in the selected and scanned row of photosensitive pixels is transmitted to the column processing unit 304. Reset scan is used to reset the charge; the photocharge of the photoelectric conversion element is discarded, allowing the accumulation of new photocharge to begin. The signal processing performed by the column processing unit 304 is correlated double sampling (CDS) processing. In CDS processing, the reset level and signal level output from each photosensitive pixel in the selected row are extracted, and the level difference is calculated. Thus, the signal of the photosensitive pixels in a row is obtained. The column processing unit 304 may have an analog-to-digital (A / D) conversion function for converting analog pixel signals to digital format.

[0052] The horizontal drive unit 305 includes a shift register and an address decoder. The horizontal drive unit 305 sequentially scans the pixel array 301 column by column. Through the selection scan operation performed by the horizontal drive unit 305, each column of photosensitive pixels is sequentially processed by the column processing unit 304 and sequentially output.

[0053] The control unit 303 configures timing signals according to the operating mode and uses various timing signals to control the vertical drive unit 302, column processing unit 304 and horizontal drive unit 305 to work together.

[0054] The processor 20 can be connected to the pixel array 301 of the image sensor 30. After the pixel array 301 is exposed to generate image data, the data can be transmitted to the processor 20. The processor 20 can be configured with a preset image readout mode. In the preset image readout mode, it can read the image data generated from the image sensor 30 and separate the pixel data of the image data into multiple frames of first pixel images and multiple frames of second pixel images in different channels. Each frame of first pixel image corresponds to one frame of second pixel image, and the exposure time of different frames of first pixel images or different frames of second pixel images is different. For example, in some embodiments, the first pixel image includes a long exposure pixel image, a medium exposure pixel image, and a short exposure pixel image, and the exposure time corresponding to the long exposure pixel image, medium exposure pixel image, and short exposure pixel image decreases sequentially.

[0055] Each frame's first or second pixel image includes multiple pixels arranged in an array. For example, in this application, the first pixel image includes multiple R pixels, G pixels, and B pixels arranged in a Bayer array, and the second pixel image includes multiple W pixels arranged in an array. That is, the first pixel image includes color information from three color channels: R (red), G (green), and B (blue), and the second processed pixel image has full-color information, which can also be called luminance information. It is understood that in some other embodiments, the generated first and second pixel images may differ due to different preset image readout modes.

[0056] Furthermore, after the processor 20 generates multiple frames of first pixel images and second pixel images from the image data according to the preset image readout mode, the processor 20 can perform alignment processing on the multiple frames of first pixel images and multiple frames of second images, and after alignment, merge the multiple frames of first pixel images or multiple frames of second pixel images into a single frame of first processed pixel image or second processed pixel image.

[0057] It is understandable that different exposure times may cause different pixel image positions to change, resulting in pixel deviation after multiple first pixel images are merged into a first processed pixel image. Therefore, it is necessary to align multiple first pixel images or multiple second pixel images to reduce the pixel deviation of the first processed pixel image or the second processed pixel image.

[0058] Specifically, the processor 20 can first perform alignment calculations on multiple frames of first pixel images to establish an alignment model, and then perform alignment processing on multiple frames of first pixel images and multiple frames of second pixel images according to the alignment model. Then, the aligned multiple frames of first pixel images are fused to generate a first pixel image and obtain fusion parameters. At the same time, the second pixel image is fused according to the fusion parameters generated from the first pixel image to generate a second processed pixel image.

[0059] Furthermore, after generating the first processed pixel image and the second processed pixel image, the processor 20 can synthesize the first processed pixel image and the second processed pixel image to obtain the target image.

[0060] Thus, by aligning the first and second pixel images with the alignment models calculated from multiple frames of first pixel images, the fused first processed pixel image can be synchronized with the second processed pixel image, avoiding misalignment and ensuring improved image quality of the synthesized target image. Furthermore, since only the alignment model of the first pixel image needs to be calculated during the alignment process, computational load is reduced, improving efficiency. Additionally, the participation of fusion parameters generated from the first pixel image during the fusion of the multi-frame aligned second pixel image reduces color cast between the first and second processed pixel images, further enhancing the image quality of the target image.

[0061] Please combine Figure 5 and Figure 6 In some embodiments, the image data includes a plurality of minimum repeating units A1, each minimum repeating unit A1 including a plurality of pixel units a1, each pixel unit a1 including a plurality of color pixels and panchromatic pixels, the color pixels being disposed in a first diagonal direction, and the panchromatic pixels being disposed in a second diagonal direction, the first diagonal direction being different from the second diagonal direction, and step S11 including sub-steps:

[0062] S112, Obtain the color pixels along the first diagonal direction to generate a first pixel image;

[0063] S114, Obtain the full-color pixels in the second diagonal direction to generate the second pixel image.

[0064] Please refer to further information. Figure 2 In some implementations, sub-steps S112 and S114 can be implemented by the preprocessing module 11, or in other words.

[0065] The preprocessing module 11 can be used to acquire colored pixels along the first diagonal direction to generate a first pixel image. The preprocessing module 11 can also be used to acquire full-color pixels along the second diagonal direction to generate a second pixel image.

[0066] In some implementations, processor 20 can be used to acquire color pixels along a first diagonal direction to generate a first pixel image. Processor 20 can also be used to acquire panchromatic pixels along a second diagonal direction to generate a second pixel image.

[0067] Specifically, when the processor 20 reads the image data acquired by the image sensor 30 using a preset image readout mode, the preset image readout mode can be a Binning mode. That is, the processor 20 can read the image data using the Binning mode to generate the first pixel image and the second pixel image. It should be noted that the Binning algorithm adds the charges corresponding to adjacent pixels of the same color within the same pixel unit a1 together and reads them out as a single pixel.

[0068] Furthermore, when reading in Binning mode, the colored pixels in the first diagonal direction of each pixel unit a1 are read, and the full-color pixels in the second diagonal direction of each pixel unit a1 are read. Then, all the read colored pixels are arranged in an array to form a first pixel image, and all the read full-color pixels are arranged in a display to generate a second pixel image.

[0069] Please see Figure 7 In some implementations, step S12 includes the following sub-steps:

[0070] S122, find matching pixels in the first pixel images of multiple frames to calculate the alignment model of the first pixel images of multiple frames;

[0071] S124, Align the first pixel images of multiple frames according to the alignment model;

[0072] S126, fuse the aligned first pixel images of multiple frames to obtain the first processed pixel image.

[0073] In some implementations, the first multi-frame processing module 12 includes an alignment unit 122 and a first fusion unit 124. Steps S122 and S124 can be implemented by the alignment unit 122, and step S126 can be implemented by the first fusion unit 124.

[0074] Alternatively, the alignment unit 122 can be used to find matching pixels in multiple frames of first pixel images to calculate the alignment model of the multiple frames of first pixel images. The alignment unit 122 can also be used to align the multiple frames of first pixel images according to the alignment model.

[0075] The first fusion unit 124 can be used to fuse the aligned first pixel images of multiple frames to obtain the first processed pixel image.

[0076] In some implementations, the processor 20 can be used to find matching pixels in multiple frames of first pixel images to calculate an alignment model for the multiple frames of first pixel images. The processor 20 can also be used to align the multiple frames of first pixel images and the fused aligned multiple frames of first pixel images according to the alignment model to obtain a first processed pixel image.

[0077] It should be noted that since the motion between the matching pixels reflects the motion between multiple frames, the alignment model calculated based on the matching pixels can eliminate the motion relationship between multiple frames, so that the first pixel images of multiple frames can be fused together with high quality.

[0078] The processor 20 can use scale-invariant feature transform (sift), speed-up robust features (surf) feature point matching algorithm or optical flow field algorithm to find matching pixels between the first pixel images of multiple frames.

[0079] For those familiar with computer science, the SIFT algorithm is an algorithm used in computer vision to detect and describe local features in images. It is invariant to rotation, scale changes, and brightness variations, and also maintains a certain degree of stability against viewpoint changes, affine transformations, and noise. SIFT feature detection consists of four steps: 1. Scale-space extremum detection: Searching the image across all scale spaces, using a Gaussian differential function to identify potential scale- and selection-invariant points of interest. 2. Feature point localization: At each candidate location, a fine-fit model is used to determine the location scale; keypoints are selected based on their stability. 3. Feature orientation assignment: Based on the local gradient direction of the image, one or more orientations are assigned to each keypoint location. All subsequent operations transform the orientation, scale, and location of the keypoints, thus providing invariance to these features. 4. Feature point description: Within the neighborhood of each feature point, the local gradient of the image is measured at a selected scale. These gradients are transformed into a representation that allows for relatively large local shape deformations and illumination changes.

[0080] The SURF algorithm is a robust image recognition and description algorithm that can be used for computer vision tasks. The concept and steps of the SURF algorithm are built upon SIFT, but the detailed process differs slightly. The SURF algorithm consists of the following three steps: feature point detection, feature proximity description, and descriptor pairing.

[0081] The optical flow field algorithm is a point-based matching algorithm that uses the changes of pixels in the image sequence over time and the correlation between adjacent frames to find the correspondence between the previous frame and the current frame, thereby calculating the motion information of objects between adjacent frames.

[0082] Furthermore, the alignment model that can be selected in this embodiment can be an affine transformation model or a perspective transformation model. That is, an affine transformation model or a perspective transformation model can be calculated based on the mutually matched pixels, and then the alignment process of multiple frames of first pixel images can be performed based on the affine transformation model or the perspective transformation model to obtain aligned multiple frames of first pixel images.

[0083] Please see Figure 8 In some implementations, step S122 includes the following sub-steps:

[0084] S1222, calculate the scaling and rotation parameters and displacement parameters reflected by the matching pixels;

[0085] S1224, establish an alignment model based on scaling, rotation, and displacement parameters.

[0086] In some implementations, sub-S1222 and sub-step S1224 can be implemented by alignment unit 122. That is, alignment unit 122 can be used to calculate the scaling and rotation parameters and displacement parameters reflected by the matching pixels. Alignment unit 122 can also be used to establish an alignment model based on the scaling and rotation parameters and displacement parameters.

[0087] In some implementations, the processor 20 is used to calculate the scaling and rotation parameters and displacement parameters reflected by the matching pixels and to establish an alignment model based on the scaling and rotation parameters and displacement parameters.

[0088] In this embodiment, the transformation model can be an affine transformation model. A reflection transformation model refers to the fact that any parallelogram in a plane can be mapped to another parallelogram through an affine transformation. The image mapping operation is performed within the same spatial plane, and different types of parallelograms are obtained by varying the transformation parameters. When using an affine transformation matrix model, the scaling and rotation of the image are controlled based on scaling and rotation parameters, while the displacement of the image is controlled based on position parameters.

[0089] Specifically, the coordinates of pixels in the first frame of a multi-frame first pixel image are obtained, and these coordinates are substituted into a preset affine transformation formula. Scaling, rotation, and displacement parameters are then calculated by substituting the coordinates of pixels in the matched second frame first pixel image. Based on these scaling, rotation, and displacement parameters, an affine transformation model can be constructed. In this way, multi-frame image alignment can be performed without relying on equipment such as a mounting frame, reducing pixel deviations after multi-frame image synthesis.

[0090] Please see Figure 9 In some implementations, step S13 includes the following sub-steps:

[0091] S132, Alignment processing of multiple frames of second pixel images according to the alignment model;

[0092] S134, fuse the aligned multi-frame second pixel images according to the fusion parameters to obtain the second processed pixel image.

[0093] In some embodiments, the second multi-frame processing module 13 may include a second fusion unit 132. Sub-step S132 may be implemented by the alignment unit 122, and sub-step S134 may be implemented by the second fusion unit 132. In other words, the alignment unit 122 is also used to align the multi-frame second pixel images according to the alignment model. The second fusion unit 132 is used to fuse the aligned multi-frame second pixel images according to the fusion parameters to obtain the second processed pixel image.

[0094] In some implementations, the processor 20 can be used to perform alignment processing on multiple frames of second pixel images according to an alignment model and to fuse the aligned multiple frames of second pixel images according to fusion parameters to obtain a second processed pixel image.

[0095] Thus, by aligning multiple frames of second pixel images using an alignment model, the pixel deviation of the fused second processed pixel image is reduced, and the second processed pixel image and the first processed pixel image can be aligned synchronously. The aligned multiple frames of second pixel images are then fused according to the fusion parameters to generate the second processed pixel image, reducing the color shift between the second processed pixel image and the first processed pixel image.

[0096] Please see Figure 10 In some implementations, step S14 includes:

[0097] S142, synthesize the first processed pixel image and the second processed pixel image based on the median filtering algorithm to generate the target image.

[0098] In some implementations, step S142 can be implemented by the synthesis module 14, or the synthesis module 14 can be used to synthesize the first processed pixel image and the second processed pixel image based on the median filtering algorithm to generate the target image.

[0099] In some implementations, the processor 20 can be used to synthesize a first processed pixel image and a second processed pixel image based on a median filtering algorithm to generate a target image.

[0100] Specifically, during the synthesis process, the processor 20 can first perform median filtering on the first processed pixel image and the second processed pixel image, and then synthesize the first processed pixel image and the second processed pixel image after median filtering; or, it can first synthesize the first processed pixel image and the second processed pixel image into a new image, and then perform median filtering on that image to generate the final image. In this way, median filtering is more flexible, and the signal-to-noise ratio of the image after median filtering is significantly improved.

[0101] It should be noted that median filtering is a nonlinear signal processing technique based on sorting statistics theory that can effectively suppress noise. The basic principle of median filtering is to replace the value of a point in a digital image or digital sequence with the median value of all points in its neighborhood, so that the surrounding pixel values ​​are close to the true value, thereby eliminating isolated noise points.

[0102] Please see Figure 11 This application provides an electronic device 100, including a processor 20, a memory 30, and one or more programs 32, wherein the one or more programs 32 are stored in the memory 30 and executed by the processor 20, and the processor 20 executes the instructions of the above-described image processing method for the programs 32.

[0103] Please combine Figure 12 This application provides a non-volatile computer-readable storage medium 40 containing a computer program, which, when executed by one or more processors 20, causes the processors 20 to perform the image processing method described above.

[0104] In the description of this specification, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with the described embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0105] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0106] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

[0107] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, include: Image data from the image sensor is acquired according to a preset image readout mode to obtain multiple frames of first pixel images and multiple frames of second pixel images. The multiple frames of first pixel images and multiple frames of second pixel images are in different channels, and the exposure time corresponding to the multiple frames of first pixel images is different or the exposure time of the multiple frames of second pixel images is different. Align and fuse multiple frames of the first pixel image to obtain a first processed pixel image, an alignment model, and fusion parameters; The second processed pixel image is obtained by processing multiple frames of the second pixel image according to the alignment model and the fusion parameters of the first pixel image; and The target image is obtained by combining the first processed pixel image and the second processed pixel image. The preset image readout mode includes a Binning mode. The image data includes multiple minimal repeating units, each minimal repeating unit includes multiple pixel units, and each pixel unit includes multiple color pixels and panchromatic pixels. The color pixels are located in a first diagonal direction, and the panchromatic pixels are located in a second diagonal direction. The first diagonal direction is different from the second diagonal direction. The step of acquiring image data from the image sensor according to the preset image readout mode to obtain a first pixel image and a second pixel image includes: The colored pixels along the first diagonal direction are obtained to generate the first pixel image; The full-color pixels along the second diagonal direction are obtained to generate the second pixel image.

2. The image processing method according to claim 1, characterized in that, The step of aligning and fusing multiple frames of the first pixel image to obtain a first processed pixel image, an alignment model, and fusion parameters includes: Find matching pixels in multiple frames of the first pixel image to calculate the alignment model of the multiple frames of the first pixel image; Align multiple frames of the first pixel image according to the alignment model; The first pixel image of multiple frames after fusion and alignment is obtained to obtain a first processed pixel image and fusion parameters.

3. The image processing method according to claim 2, characterized in that, The step of finding matching pixels in multiple frames of the first pixel image to calculate the alignment model of the multiple frames of the first pixel image includes: Calculate the scaling and rotation parameters and displacement parameters reflected by the mutually matched pixels; The alignment model is obtained based on the scaling and rotation parameters and the displacement parameters.

4. The image processing method according to claim 1, characterized in that, The step of processing the multiple frames of the second pixel image according to the alignment model and the fusion parameters to obtain the second processed pixel image includes: The second pixel image of multiple frames is aligned according to the alignment model; The second pixel image is obtained by fusing and aligning multiple frames of the second pixel image according to the fusion parameters.

5. The image processing method according to claim 1, characterized in that, The process of synthesizing the first processed pixel image and the second processed pixel image to obtain the target image includes: The first processed pixel image and the second processed pixel image are synthesized based on the median filtering algorithm to generate the target image.

6. The image processing method according to claim 1, characterized in that, The first pixel image includes R pixels, G pixels, and B pixels arranged in a Bayer array, and the second pixel image includes W pixels arranged in an array.

7. An image processing apparatus, characterized in that, include: The preprocessing module is used to acquire image data from the image sensor according to a preset image readout mode to obtain multiple frames of first pixel images and multiple frames of second pixel images. The multiple frames of first pixel images and multiple frames of second pixel images are in different channels, and the exposure time corresponding to the multiple frames of first pixel images is different or the exposure time of the multiple frames of second pixel images is different. The preset image readout mode includes a Binning mode. The image data includes multiple minimum repeating units, each minimum repeating unit includes multiple pixel units, and each pixel unit includes multiple color pixels and panchromatic pixels. The color pixels are located in a first diagonal direction, and the panchromatic pixels are located in a second diagonal direction. The first diagonal direction and the second diagonal direction are different. The step of obtaining the image data of the image sensor according to the preset image readout mode to obtain a first pixel image and a second pixel image includes: obtaining the color pixels in the first diagonal direction to generate the first pixel image; and obtaining the panchromatic pixels in the second diagonal direction to generate the second pixel image. The first multi-frame processing module is used to align and fuse multiple frames of the first pixel image to obtain a first processed pixel image, an alignment model, and fusion parameters. The second multi-frame processing module is used to process the multiple frames of the second pixel image according to the alignment model of the first pixel image and the fusion parameters to obtain a second processed pixel image; and The synthesis module is used to synthesize the first processed pixel image and the second processed pixel image to obtain the target image.

8. An electronic device, characterized in that, Includes image sensors, processors, and memory; and One or more programs, wherein the one or more programs are stored in the memory and executed by the processor, the programs comprising instructions for performing the image processing method according to any one of claims 1-6.

9. A non-volatile computer-readable storage medium containing a computer program, characterized in that, When the computer program is executed by one or more processors, the processors perform the image processing method according to any one of claims 1-6.

Images