Image processing method, electronic device, storage medium and program product

Abstract

The embodiments of the present application provide an image processing method, an electronic device, a storage medium and a program product, which are applied in the field of electronic technology. The method obtains a first image captured by a camera when an optical image stabilization module is started, and determines a sensitivity calibration matrix corresponding to the first image according to a first timestamp when the capture of the first image starts, a second timestamp when the capture of the first image ends, a first coordinate of the position of the optical image stabilization module after each movement during the capture of the first image, and a plurality of sensitivity calibration matrices, so as to calibrate the first image. During the calibration of the plurality of sensitivity calibration matrices, the relative positions of the optical center of the lens and the optical center of the image sensor with N pixels in one are different. Therefore, the embodiments of the present application can improve the poor image quality caused by the uneven light received by N pixels of the same color when the optical image stabilization module moves, and improve the image quality of the second image obtained after calibration.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to an image processing method, electronic equipment, storage medium and program product. Background Art

[0002] With the continuous development of electronic technology, electronic devices such as mobile phones and tablet computers have become common tools in people's daily life and work. At present, some electronic devices are provided with cameras to provide users with photo taking or video recording functions based on the cameras.

[0003] In cameras of some electronic devices, an optical image stabilization (OIS) module and an N-pixel-in-one image sensor may be provided, and in a pixel array of the N-pixel-in-one image sensor, N adjacent pixels of the same color may share the same microlens.

[0004] However, when the optical image stabilization module of the camera is activated for image stabilization, the light received by N pixels of the same color will be uneven, resulting in poor image quality of the image captured by the camera. Summary of the invention

[0005] The embodiments of the present application provide an image processing method, an electronic device, a storage medium and a program product, so that when the optical image stabilization module of the camera is activated for image stabilization, a sensitivity calibration matrix is ​​used to calibrate a first image captured by the camera to improve the image quality.

[0006] In a first aspect, an embodiment of the present application proposes an image processing method, which is applied to an electronic device, the electronic device includes a camera, the camera includes an optical image stabilization module, a lens, and an N-pixel-in-one image sensor, the N-pixel-in-one image sensor includes a microlens array and a pixel array, the microlens array includes a plurality of microlenses, each microlens covers N adjacent same-color pixels in the pixel array, and N is an integer greater than 1. The image processing method includes: the electronic device acquires a first image acquired by the camera when the optical image stabilization module is started; the electronic device acquires a first timestamp when the acquisition of the first image starts, and a second timestamp when the acquisition of the first image ends; the electronic device acquires the first coordinates of the position of the optical image stabilization module after each movement during the acquisition process of the first image; the electronic device determines the sensitivity calibration matrix corresponding to the first image according to the first timestamp, the second timestamp, each first coordinate, and a plurality of sensitivity calibration matrices, wherein the relative positions of the optical center of the lens and the optical center of the N-pixel-in-one image sensor are different during the calibration of the plurality of sensitivity calibration matrices; the electronic device calibrates the first image using the sensitivity calibration matrix to obtain the second image.

[0007] In this way, when the camera activates the optical image stabilization module for image stabilization, the sensitivity calibration matrix is ​​calculated according to the first timestamp, the second timestamp, the first coordinate of the position of the optical image stabilization module after each movement, and the multiple sensitivity calibration matrices, so as to calibrate the first image captured by the camera. Therefore, the problem of failure of calibration of the first image using the first sensitivity calibration matrix under a single optical displacement can be overcome, and the poor image quality caused by uneven light received by N pixels of the same color when the optical image stabilization module moves can be improved, so that the sensitivity of the N pixels of the same color in the second image obtained after calibration is almost consistent, thereby improving the image quality of the second image obtained after calibration.

[0008] In a possible implementation, the electronic device determines the sensitivity calibration matrix corresponding to the first image according to the first timestamp, the second timestamp, each first coordinate, and multiple sensitivity calibration matrices, including: the electronic device determines the number of times the optical image stabilization module moves during the acquisition of the first image according to the first timestamp and the second timestamp; the electronic device determines the second coordinate of the centroid position corresponding to each position where the optical image stabilization module moves according to the number of movements and each first coordinate; the electronic device determines the sensitivity calibration matrix corresponding to the first image according to the second coordinate and multiple sensitivity calibration matrices. In this way, since the centroid position corresponding to each position where the optical image stabilization module moves during the acquisition of the first image can represent the average movement position of the lens during the acquisition of the first image to a certain extent, when the second coordinate corresponding to the centroid position is used to determine the sensitivity calibration matrix to calibrate the first image, the calibration effect is better, so as to further improve the image quality of the second image obtained after calibration.

[0009] In a possible implementation, the electronic device determines the number of times the optical image stabilization module moves during the acquisition of the first image based on the first timestamp and the second timestamp, including: the electronic device calculates the time interval between the second timestamp and the first timestamp; the electronic device determines the ratio between the time interval and the movement frequency of the optical image stabilization module as the number of times the optical image stabilization module moves during the acquisition of the first image. In this way, the number of times the optical image stabilization module moves during the acquisition of the first image is determined based on the ratio between the time interval and the movement frequency of the optical image stabilization module, making the calculation method of the number of movements relatively simple.

[0010] In a possible implementation, the electronic device determines, according to the number of movements and each first coordinate, the second coordinate of the center of mass position corresponding to each position where the optical image stabilization module is moved, including: the electronic device calculates the second coordinate by the following formula:

[0011]

[0012] Among them, Sn is the second coordinate, n is the number of moves, P i is the first coordinate of the position of the optical image stabilization module after the i-th movement. In this way, after summing the first coordinates of the position of the optical image stabilization module after n-times of movement, the sum is divided by the number of movements to obtain the second coordinates of the centroid position corresponding to each position of the optical image stabilization module after movement, so that the calculation method of the second coordinate of the centroid position is relatively simple.

[0013] In a possible implementation, the electronic device determines the sensitivity calibration matrix corresponding to the first image according to the second coordinate and multiple sensitivity calibration matrices, including: the electronic device decomposes the second coordinate according to the movable direction of the optical image stabilization module to obtain the first coordinate component in the first moving direction and the second coordinate component in the second moving direction; the electronic device determines the sensitivity calibration matrix corresponding to the first image according to the first coordinate component, the second coordinate component and multiple sensitivity calibration matrices; wherein the first moving direction and the second moving direction are perpendicular to each other, and the first moving direction and the second moving direction are both perpendicular to the optical axis direction of the camera. In this way, the first coordinate component, the second coordinate component and multiple sensitivity calibration matrices can be used to determine the sensitivity calibration matrix, so that the calculated sensitivity calibration matrix is ​​more accurate, so as to further improve the image quality of the second image obtained after calibration.

[0014] In a possible implementation, the multiple sensitivity calibration matrices include a first sensitivity calibration matrix, a second sensitivity calibration matrix, a third sensitivity calibration matrix, a fourth sensitivity calibration matrix, and a fifth sensitivity calibration matrix; the first moving direction includes a first direction and a second direction that are opposite to each other, and the second moving direction includes a third direction and a fourth direction that are opposite to each other. When the first sensitivity calibration matrix is ​​calibrated, the optical center of the lens coincides with the optical center of the N-pixel-in-one image sensor along the optical axis direction of the camera. When the second sensitivity calibration matrix is ​​calibrated, the optical center of the lens is offset by a first distance along the first direction and a second distance along the third direction relative to the optical center of the N-pixel-in-one image sensor; the first distance is the maximum distance that the lens can move along the first direction, and the second distance is the maximum distance that the lens can move along the third direction. When the third sensitivity calibration matrix is ​​calibrated, the optical center of the lens is offset by a first distance along the first direction and a third distance along the fourth direction relative to the optical center of the N-pixel-in-one image sensor; the third distance is the maximum distance that the lens can move along the fourth direction. When the fourth sensitivity calibration matrix is ​​calibrated, the optical center of the lens is offset by a fourth distance along the second direction and a third distance along the fourth direction relative to the optical center of the N-pixel-in-one image sensor; the fourth distance is the maximum distance that the lens can move along the second direction. When the fifth sensitivity calibration matrix is ​​calibrated, the optical center of the lens is offset by a fourth distance along the second direction and a second distance along the third direction relative to the optical center of the N-pixel-in-one image sensor. In this way, multiple sensitivity calibration matrices can be used to calculate the sensitivity calibration matrix, which can overcome the problem of failure of calibration of the first image using the first sensitivity calibration matrix under a single optical displacement, so that when the optical image stabilization module is activated and moved to any position, the first image captured by the camera can be accurately calibrated.

[0015] In a possible implementation, the electronic device determines the sensitivity calibration matrix corresponding to the first image according to the first coordinate component, the second coordinate component and the plurality of sensitivity calibration matrices, including: when the first coordinate component is greater than or equal to 0 and the second coordinate component is greater than or equal to 0, the electronic device calculates the sensitivity calibration matrix by the following formula:

[0016]

[0017] Among them, Q n is the sensitivity calibration matrix, Q0 is the first sensitivity calibration matrix, Q1 is the second sensitivity calibration matrix, Q2 is the third sensitivity calibration matrix, and Q4 is the fifth sensitivity calibration matrix; S nx is the first coordinate component, S ny is the second coordinate component, S xmax1 is the first distance, S xmax1In this way, it is possible to calculate the sensitivity calibration matrix when the optical center of the lens is offset toward the first direction and / or the third direction relative to the optical center of the N-pixel-in-one image sensor.

[0018] In a possible implementation, the electronic device determines the sensitivity calibration matrix corresponding to the first image according to the first coordinate component, the second coordinate component and the plurality of sensitivity calibration matrices, including: when the first coordinate component is greater than 0 and the second coordinate component is less than 0, the electronic device calculates the sensitivity calibration matrix by the following formula:

[0019]

[0020] Among them, Q n is the sensitivity calibration matrix, Q0 is the first sensitivity calibration matrix, Q1 is the second sensitivity calibration matrix, Q2 is the third sensitivity calibration matrix, and Q3 is the fourth sensitivity calibration matrix; S nx is the first coordinate component, S ny is the second coordinate component, S xmax1 is the first distance, S ymax2 In this way, when the optical center of the lens is offset toward the first direction and the fourth direction relative to the optical center of the N-pixel-in-one image sensor, the sensitivity calibration matrix can be calculated.

[0021] In a possible implementation, the electronic device determines the sensitivity calibration matrix corresponding to the first image according to the first coordinate component, the second coordinate component and the plurality of sensitivity calibration matrices, including: when the first coordinate component is less than or equal to 0 and the second coordinate component is less than or equal to 0, the electronic device calculates the sensitivity calibration matrix by the following formula:

[0022]

[0023] Among them, Q n is the sensitivity calibration matrix, Q0 is the first sensitivity calibration matrix, Q2 is the third sensitivity calibration matrix, Q3 is the fourth sensitivity calibration matrix, and Q4 is the fifth sensitivity calibration matrix; S nx is the first coordinate component, S ny is the second coordinate component, S xmax2 is the fourth distance, S ymax2 In this way, it is possible to calculate the sensitivity calibration matrix when the optical center of the lens is offset toward the second direction and / or the fourth direction relative to the optical center of the N-pixel-in-one image sensor.

[0024] In a possible implementation, the electronic device determines the sensitivity calibration matrix corresponding to the first image according to the first coordinate component, the second coordinate component and the plurality of sensitivity calibration matrices, including: when the first coordinate component is less than 0 and the second coordinate component is greater than 0, the electronic device calculates the sensitivity calibration matrix by the following formula:

[0025]

[0026] Among them, Q n is the sensitivity calibration matrix, Q0 is the first sensitivity calibration matrix, Q1 is the second sensitivity calibration matrix, Q3 is the fourth sensitivity calibration matrix, and Q4 is the fifth sensitivity calibration matrix; S nx is the first coordinate component, S ny is the second coordinate component, S xmax2 is the fourth distance, S ymax1 In this way, when the optical center of the lens is offset toward the second direction and the third direction relative to the optical center of the N-pixel-in-one image sensor, the sensitivity calibration matrix can be calculated.

[0027] In a possible implementation, the pixel array includes a plurality of pixel sets, each pixel set includes pixel units corresponding to M color channels, each pixel unit includes pixels corresponding to N color sub-channels, and the pixels corresponding to the N color sub-channels are adjacent N pixels of the same color, and M is an integer greater than 1. The sensitivity calibration matrix includes a first sensitivity calibration sub-matrix corresponding to H color sub-channels, and H is equal to the product of M and N.

[0028] In a possible implementation, an electronic device uses a sensitivity calibration matrix to calibrate a first image to obtain a second image, including: the electronic device splits the first image according to H color sub-channels to obtain H first single-channel images; the electronic device adjusts the sizes of the first sensitivity calibration sub-matrices corresponding to the H color sub-channels to obtain second sensitivity calibration sub-matrices corresponding to the H color sub-channels, and the size of each second sensitivity calibration sub-matrix is ​​equal to the size of the first single-channel image; the electronic device uses the second sensitivity calibration sub-matrices corresponding to the H color sub-channels to calibrate the first single-channel images corresponding to them respectively to obtain H second single-channel images; the electronic device merges the H second single-channel images to obtain the second image. In this way, since the pixels corresponding to different color sub-channels receive uneven light when the camera captures the first image, the second sensitivity calibration sub-matrices corresponding to different color sub-channels are used to calibrate the first single-channel images corresponding to them respectively, so as to further improve the image quality of the second image obtained after calibration.

[0029] In a possible implementation, the electronic device uses the second sensitivity calibration submatrix corresponding to the H color subchannels to calibrate the first single-channel images corresponding to them respectively to obtain H second single-channel images, including: for the same color subchannel, the electronic device uses each calibration parameter in the second sensitivity calibration submatrix, multiplying the pixel value at the corresponding position in the first single-channel image to obtain the second single-channel image. In this way, the product of the calibration parameter and the pixel value can be used to implement the calibration of the single-channel image, and the calibration method is relatively simple.

[0030] In a possible implementation, before the electronic device determines the sensitivity calibration matrix corresponding to the first image according to the first timestamp, the second timestamp, each first coordinate and multiple sensitivity calibration matrices, it also includes: the electronic device obtains multiple pre-calibrated sensitivity calibration matrices; each sensitivity calibration matrix includes sensitivity calibration sub-matrices corresponding to H color sub-channels. Wherein, each calibration parameter in the sensitivity calibration sub-matrix is ​​calculated based on the mean of each pixel in the single-channel mean image and the pixel value at the corresponding position in each single-channel test image corresponding to the single-channel mean image; the mean of each pixel in the single-channel mean image is the average value of the pixel values ​​at the same position in the N single-channel test images belonging to the same color channel in the H single-channel test images; the H single-channel test images are obtained after the size of the test image is reduced and split according to the H color sub-channels; the test image is an image captured when the focus position of the lens is a preset focus position, and the relative position between the optical center of the lens and the optical center of the image sensor with N pixels in one is a preset position. In this way, a plurality of sensitivity calibration matrices can be pre-calibrated to quickly calculate the sensitivity calibration matrix corresponding to the first image, thereby improving the speed of image calibration.

[0031] In a possible implementation, each calibration parameter in the sensitivity calibration submatrix is ​​the ratio of the mean value of each pixel in the single-channel mean image to the pixel value at the corresponding position in each single-channel test image corresponding to the single-channel mean image. In this way, the calculation method of the sensitivity calibration submatrix is ​​relatively simple.

[0032] In a second aspect, an embodiment of the present application proposes an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call the computer program to execute the above-mentioned image processing method.

[0033] In a third aspect, an embodiment of the present application proposes a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed, the above-mentioned image processing method is implemented.

[0034] In a fourth aspect, an embodiment of the present application proposes a computer program product, including a computer program, which enables a computer to execute the above-mentioned image processing method when the computer program is executed.

[0035] The possible implementation methods of the second to fourth aspects have effects similar to those of the first aspect and the possible designs of the first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the structure of a camera provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of the structure of an N-pixel-in-one image sensor in a camera provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of a pixel array in an N-pixel-in-one image sensor provided in an embodiment of the present application;

[0039] Figure 4 A schematic diagram of the optical displacement of the lens and the N-pixel-in-one image sensor in the camera provided in an embodiment of the present application when the optical image stabilization module is turned off and when the optical image stabilization module is turned on;

[0040] Figure 5 A schematic diagram of the hardware system structure of an electronic device provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of the software system structure of an electronic device provided in an embodiment of the present application;

[0042] Figure 7 A schematic diagram of an interface for an application scenario provided in an embodiment of the present application;

[0043] Figure 8 A flowchart of an image processing method provided in an embodiment of the present application;

[0044] Fig. 9 A schematic diagram of a first timestamp and a second timestamp of a first image provided in an embodiment of the present application;

[0045] Fig.10 A schematic diagram of the trajectory of the position of the optical image stabilization module after each movement provided by an embodiment of the present application;

[0046] Fig.11 A schematic diagram of a situation in which the optical center of a lens is offset relative to the optical center of an N-pixel-in-one image sensor during the calibration process of multiple sensitivity calibration matrices provided in an embodiment of the present application;

[0047] Fig.12A flowchart of a calibration process of a first sensitivity calibration matrix provided in an embodiment of the present application;

[0048] Fig.13 A schematic diagram of multiple single-channel test images provided in an embodiment of the present application;

[0049] Fig.14 A schematic diagram of multiple single-channel mean images provided in an embodiment of the present application;

[0050] Fig.15 A schematic diagram of multiple sensitivity calibration sub-matrices provided in an embodiment of the present application;

[0051] Fig.16 A flowchart of a calibration process of multiple sensitivity calibration matrices provided in an embodiment of the present application;

[0052] Fig.17 A schematic diagram of the principle of calculating the sensitivity calibration matrix provided in an embodiment of the present application;

[0053] Fig.18 A schematic diagram of module interaction of an image processing method provided in an embodiment of the present application;

[0054] Fig.19 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0055] Fig. 20 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and their order is not limited. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit them to be different.

[0057] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0058] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0059] Currently, some electronic devices are provided with cameras. When a user uses the electronic device, the camera in the electronic device can be used to take photos or videos.

[0060] When a user holds an electronic device to take a photo, the electronic device may shake to a certain extent, resulting in blurry images captured by the camera. In order to enhance the stability of the images captured by the camera, optical image stabilization technology is introduced into the camera, that is, an optical image stabilization module is set in the camera.

[0061] like Figure 1 As shown, the camera may include a lens 10 , an image sensor 20 , an optical image stabilization module, a bracket 40 and a circuit board 50 .

[0062] The lens 10 and the image sensor 20 are sequentially arranged along the optical axis direction of the camera. The lens 10 may include one or more optical lenses, which may be convex lenses or concave lenses; when the lens 10 includes multiple optical lenses, the multiple optical lenses may be sequentially stacked along the optical axis direction of the camera. The image sensor 20 may also be called a camera sensor (i.e., a camera sensor), which is fixed on the circuit board 50 and electrically connected to the circuit board 50.

[0063] The optical image stabilization module may include a drive motor 30, and the drive motor 30 may be disposed on a bracket 40. The drive motor 30 may be a voice coil motor (VCM), a shape memory alloy (SMA) motor, a stepping motor, a piezoelectric motor, etc., which is not limited in the embodiment of the present application.

[0064] The driving motor 30 may be electrically connected to a driving chip, and the driving chip is used to control the motion state of the driving motor 30 after being powered on.

[0065] Optical image stabilization technology corrects "optical axis offset" through the floating lens of the lens. The principle is to detect tiny jitters through the gyroscope sensor or acceleration sensor in the electronic device, and then send the detected jitter data to the microprocessor, such as the drive chip electrically connected to the drive motor 30. The microprocessor calculates the displacement that needs to be compensated based on the jitter data, and then drives the drive motor 30 in the optical image stabilization module to move through the displacement, thereby driving the lens 10 to move along the first moving direction and / or the second moving direction to compensate for the jitter direction and jitter displacement of the lens 10, thereby effectively improving the image blur problem caused by the jitter of the electronic device. Among them, the first moving direction and the second moving direction are perpendicular to each other, and the first moving direction and the second moving direction are both perpendicular to the optical axis direction of the camera.

[0066] like Figure 2 As shown, the image sensor 20 can be an N-pixel-in-one image sensor, which can include a microlens array 21, a filter 22 and a photosensitive element 23. The microlens array 21 and the filter 22 are both located between the lens 10 and the photosensitive element 23, and the filter 22 is located between the microlens array 21 and the photosensitive element 23.

[0067] The microlens array 21 includes a plurality of microlenses 210 distributed in an array. The microlenses 210 have a light-gathering function and may be OCLs (on chip micro lenses). The light reflected by the object passes through the lens 10 in sequence and is incident on the microlenses 210. The microlenses 210 focus the incident light, so that the focused light is projected onto the photosensitive element 23 after passing through the filter 22. The photosensitive element 23 converts the light signal into an electrical signal for imaging.

[0068] The filter 22 may include a filter unit array, which may include multiple filter unit sets, each filter unit set may include M filter units, and each filter unit allows the same color of light to pass through. The photosensitive element 23 may include a photosensitive unit array, which may include multiple photosensitive unit sets, each photosensitive unit set may include M photosensitive units, and each photosensitive unit includes N photosensitive pixels. Moreover, each filter unit in the filter 22 corresponds to each photosensitive unit in the photosensitive element 23 one by one, and the N photosensitive pixels in each photosensitive unit are used to receive light filtered by its corresponding filter unit. Wherein, M and N are both integers greater than 1.

[0069] In this way, the filter unit array and the photosensitive unit array covered by the filter unit array together constitute the pixel array of the N-pixel-in-one image sensor. In addition, the filter unit and the photosensitive unit covered by the filter unit together constitute the pixel unit in the pixel array of the N-pixel-in-one image sensor.

[0070] In an embodiment of the present application, the pixel array of the N-pixel-in-one image sensor includes multiple pixel sets, each pixel set includes pixel units corresponding to M color channels, each pixel unit includes pixels corresponding to N color sub-channels, and the pixels corresponding to the N color sub-channels are adjacent N pixels of the same color.

[0071] As electronic devices have higher and higher requirements for camera performance, one of the trends is the multi-pixelization of cameras, that is, by reducing the size of a single pixel and arranging more pixels to improve the resolution of the image. However, as the pixels of the camera become smaller, the sensitivity will decrease. Therefore, a four-pixel-in-one image sensor can be used to take into account both the high sensitivity and high resolution of the camera. The pixel array of a four-pixel-in-one image sensor can also be called a quadbayer array. In the pixel array of a four-pixel-in-one image sensor, four adjacent pixels of the same color are arranged together to form a pixel that is four times larger than the original pixel area, so as to take into account the high sensitivity and high resolution of the camera. For example, a camera with a four-pixel-in-one image sensor can improve the decrease in image resolution and reduce image noise when capturing images in low-light environments such as night scenes, thereby improving image quality.

[0072] Take the N-pixel-in-one image sensor as a four-pixel-in-one image sensor as an example, that is, N is 4, such as Figure 3 As shown, the pixel array of the four-pixel-in-one image sensor may include multiple pixel sets 24, each pixel set 24 includes pixel units corresponding to four color channels, such as a first pixel unit 241 corresponding to a red channel, a second pixel unit 242 corresponding to a first green channel, a third pixel unit 243 corresponding to a second green channel, and a fourth pixel unit 244 corresponding to a blue channel.

[0073] Each pixel unit includes pixels corresponding to four color sub-channels, and the pixels corresponding to the four color sub-channels are four adjacent pixels of the same color. For example, the first pixel unit 241 includes pixels corresponding to the first red subchannel (R0 subchannel), pixels corresponding to the second red subchannel (R1 subchannel), pixels corresponding to the third red subchannel (R2 subchannel), and pixels corresponding to the fourth red subchannel (R3 subchannel); the second pixel unit 242 includes pixels corresponding to the first green subchannel (Gr0 subchannel), pixels corresponding to the second green subchannel (Gr1 subchannel), pixels corresponding to the third green subchannel (Gr2 subchannel), and pixels corresponding to the fourth green subchannel (Gr3 subchannel); the third pixel unit 243 includes pixels corresponding to the fifth green subchannel (Gb0 subchannel), pixels corresponding to the sixth green subchannel (Gb1 subchannel), pixels corresponding to the seventh green subchannel (Gb2 subchannel), and pixels corresponding to the eighth green subchannel (Gb3 subchannel); the fourth pixel unit 244 includes pixels corresponding to the first blue subchannel (B0 subchannel), pixels corresponding to the second blue subchannel (B1 subchannel), pixels corresponding to the third blue subchannel (B2 subchannel), and pixels corresponding to the fourth blue subchannel (B3 subchannel).

[0074] It is understandable that the pixel array of the N-pixel-in-one image sensor in the embodiment of the present application adopts Figure 3 In addition to the four-Bayer array shown, a nine-Bayer array or a sixteen-Bayer array may also be used. The embodiment of the present application does not limit the specific form of the pixel array of the image sensor.

[0075] When the pixel array of the N-pixel-in-one image sensor is a nine-Bayer array, the pixel array may include multiple pixel sets 24, each pixel set 24 includes pixel units corresponding to four color channels, each pixel unit includes pixels corresponding to nine color sub-channels, and the pixels corresponding to the nine color sub-channels are nine adjacent pixels of the same color. When the pixel array of the N-pixel-in-one image sensor is a sixteen-Bayer array, the pixel array may include multiple pixel sets 24, each pixel set 24 includes pixel units corresponding to four color channels, each pixel unit includes pixels corresponding to sixteen color sub-channels, and the pixels corresponding to the sixteen color sub-channels are sixteen adjacent pixels of the same color.

[0076] In an embodiment of the present application, in order to achieve a higher amount of light entering, in an N-pixel-in-one image sensor, each microlens 210 covers N adjacent pixels of the same color in the pixel array, that is, N adjacent pixels of the same color in each pixel unit share the same lens 210.

[0077] Taking a four-pixel-in-one image sensor as an example, each microlens 210 covers four adjacent pixels of the same color in the pixel array. For example, one microlens 210 covers pixels corresponding to the R0 subchannel, the R1 subchannel, the R2 subchannel, and the R3 subchannel in the first pixel unit 241, and another microlens 210 covers pixels corresponding to the Gr0 subchannel, the Gr1 subchannel, the Gr2 subchannel, and the Gr3 subchannel in the second pixel unit 242.

[0078] The above-mentioned camera has an optical image stabilization module and an N-pixel-in-one image sensor, and in the pixel array of the N-pixel-in-one image sensor, adjacent N pixels of the same color can share the same microlens.

[0079] However, when the camera starts the optical image stabilization module for image stabilization, the optical image stabilization module will move to drive the lens 10 to move, causing the optical center of the lens 10 to shift from the optical center of the N-pixel-in-one image sensor, and the positions of the N same-color pixels relative to the optical center of the lens 10 are inconsistent, so that after the light passing through the lens 10 is focused by the microlens 210, the N same-color pixels receive uneven light, such as a part of the N same-color pixels receive more light, while another part of the N same-color pixels receive less light. This will result in poor image quality of the image captured by the camera, that is, the image quality is degraded.

[0080] In the related art, for a camera without an optical image stabilization module or with the optical image stabilization module turned off, and in a pixel array of an N-pixel-in-one image sensor, in which N adjacent same-color pixels share the same microlens, in order to minimize the impact of the matching relationship between the lens 10 and the N-pixel-in-one image sensor on the first sensitivity calibration matrix obtained after calibration, as shown in FIG. Figure 4 As shown in (a), when calibrating the first sensitivity calibration matrix, the optical center of the lens 10 and the optical center of the N-pixel image sensor can be set to coincide along the optical axis direction of the camera, that is, the optical displacement between the optical center of the lens 10 and the optical center of the N-pixel image sensor is 0, so as to calibrate the first sensitivity calibration matrix, and subsequently the first sensitivity calibration matrix can be used to calibrate the first image captured by the camera.

[0081] It should be noted that due to the manufacturing process of the N-pixel-in-one image sensor and the influence of factors such as the inconsistency of the microlenses 210 set at various positions, the light received by the N same-color pixels will also be uneven. Therefore, for a camera without an optical image stabilization module or with the optical image stabilization module turned off, it is also necessary to use the first sensitivity calibration matrix to calibrate the first image captured by the camera to improve the poor image quality caused by the uneven light received by the N same-color pixels, so that the sensitivity of the N same-color pixels in the calibrated image is almost consistent.

[0082] However, the related technology uses a first sensitivity calibration matrix to calibrate the first image captured by the camera, which can only calibrate the uneven light received by N pixels of the same color in a scenario where the optical center of the lens 10 coincides with the optical center of the N-pixel image sensor along the optical axis of the camera.

[0083] like Figure 4 As shown in (b), when the camera starts the optical image stabilization module for image stabilization, the optical center of the lens 10 is offset from the optical center of the N-pixel-in-one image sensor, that is, the optical displacement between the optical center of the lens 10 and the optical center of the N-pixel-in-one image sensor can be d, and d is not equal to 0. In this case, the first sensitivity calibration matrix is ​​invalid. Even if the first sensitivity calibration matrix is ​​used to calibrate the first image captured by the camera, it is impossible to effectively improve the poor image quality caused by the uneven light received by the N same-color pixels, that is, it is impossible to calibrate the sensitivity difference of the N same-color pixels, resulting in the degradation of the image quality due to insufficient calibration.

[0084] Based on this, an embodiment of the present application provides an image processing method, which obtains a first image captured by the camera when the optical image stabilization module is started, obtains a first timestamp when the capture of the first image starts, and a second timestamp when the capture of the first image ends, and obtains the first coordinates of the position of the optical image stabilization module after each movement during the capture process of the first image. According to the first timestamp, the second timestamp, each first coordinate and multiple sensitivity calibration matrices, a sensitivity calibration matrix corresponding to the first image is determined. When calibrating the multiple sensitivity calibration matrices, the relative positions of the optical center of the lens and the optical center of the N-pixel image sensor are different. The sensitivity calibration matrix is ​​used to calibrate the first image to obtain the second image.

[0085] Therefore, the embodiment of the present application can calculate the sensitivity calibration matrix according to the first timestamp, the second timestamp, the first coordinate of the position of the optical image stabilization module after each movement, and multiple sensitivity calibration matrices to calibrate the first image captured by the camera when the optical image stabilization module is activated for image stabilization. In this way, the problem of failure to calibrate the first image using the first sensitivity calibration matrix under a single optical displacement (i.e., the optical displacement between the optical center of the lens 10 and the optical center of the image sensor with N pixels in one is 0) can be overcome, and the poor image quality caused by the uneven light received by N pixels of the same color when the optical image stabilization module moves can be improved, so that the sensitivity of the N pixels of the same color in the second image obtained after calibration is almost the same, thereby improving the image quality of the second image obtained after calibration.

[0086] The image processing method provided in the embodiment of the present application can be applied to an electronic device with a camera. The electronic device includes a terminal device, which can also be called a terminal, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The electronic device can be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with a wireless transceiver function, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiment of the present application does not limit the specific technology and specific device form adopted by the electronic device.

[0087] In order to better understand the embodiments of the present application, the structure of the electronic device according to the embodiments of the present application is introduced below.

[0088] Figure 5The schematic diagram of the hardware system structure of the electronic device 500 is shown. The electronic device 500 may include a processor 510, an external memory interface 520, an internal memory 521, a universal serial bus (USB) interface 530, a charging management module 540, a power management module 541, a battery 542, an antenna 1, an antenna 2, a mobile communication module 550, a wireless communication module 560, an audio module 570, a speaker 570A, a receiver 570B, a microphone 570C, an earphone interface 570D, a sensor module 580, a button 590, a motor 591, an indicator 592, a camera 593, a display screen 594, and a subscriber identification module (SIM) card interface 595. Among them, the sensor module 580 may include a gyroscope sensor 580A and an acceleration sensor 580B.

[0089] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 500. In other embodiments of the present application, the electronic device 500 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0090] The processor 510 may include one or more processing units, for example, the processor 510 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0091] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0092] The processor 510 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 510 is a cache memory. The memory may store instructions or data that the processor 510 has just used or cyclically used. If the processor 510 needs to use the instruction or data again, it may be called from the memory. This avoids repeated access, reduces the waiting time of the processor 510, and thus improves the efficiency of the system.

[0093] The charging management module 540 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 540 may receive charging input from a wired charger through the USB interface 530. In some wireless charging embodiments, the charging management module 540 may receive wireless charging input through a wireless charging coil of the electronic device 500. While the charging management module 540 is charging the battery 542, it may also power the electronic device through the power management module 541.

[0094] The power management module 541 is used to connect the battery 542, the charging management module 540 and the processor 510. The power management module 541 receives input from the battery 542 and / or the charging management module 540, and supplies power to the processor 510, the internal memory 521, the display screen 594, the camera 593, and the wireless communication module 560. The power management module 541 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 541 can also be set in the processor 510. In other embodiments, the power management module 541 and the charging management module 540 can also be set in the same device.

[0095] The wireless communication function of the electronic device 500 can be implemented through the antenna 1, the antenna 2, the mobile communication module 550, the wireless communication module 560, the modem processor and the baseband processor.

[0096] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 550 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to electronic device 500. Mobile communication module 550 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc.

[0097] The wireless communication module 560 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 500. The wireless communication module 560 can be one or more devices integrating at least one communication processing module. The wireless communication module 560 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 510. The wireless communication module 560 can also receive the signal to be sent from the processor 510, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0098] In some embodiments, antenna 1 of electronic device 500 is coupled to mobile communication module 550, and antenna 2 is coupled to wireless communication module 560, so that electronic device 500 can communicate with the network and other devices through wireless communication technology.

[0099] The electronic device 500 implements the display function through a GPU, a display screen 594, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 594 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 510 may include one or more GPUs, which execute program instructions to generate or change display information.

[0100] The display screen 594 is used to display images, display videos, and receive sliding operations, etc. The display screen 594 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 500 may include one or more display screens 594.

[0101] The electronic device 500 can realize the shooting function through ISP, camera 593, video codec, GPU, display screen 594 and application processor.

[0102] The ISP is used to process the data fed back by the camera 593. For example, when taking a photo, the shutter is opened, and the light is transmitted to the photosensitive element of the camera through the lens. The light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 593.

[0103] The camera 593 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 500 may include one or more cameras 593.

[0104] The external memory interface 520 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 500. The external memory card communicates with the processor 510 through the external memory interface 520 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0105] The internal memory 521 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 521 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 500 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 521 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 510 executes various functional applications and data processing of the electronic device 500 by running instructions stored in the internal memory 521, and / or instructions stored in a memory provided in the processor.

[0106] The electronic device 500 can implement audio functions such as music playing and recording through the audio module 570, the speaker 570A, the receiver 570B, the microphone 570C, the headphone jack 570D, and the application processor.

[0107] The gyro sensor 580A can be used to determine the motion posture of the electronic device 500. In some embodiments, the angular velocity of the electronic device 500 around three axes (i.e., the x-axis, the y-axis, and the z-axis) can be determined by the gyro sensor 580A. The gyro sensor 580A can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 580A detects the angle of the electronic device 500 shaking, calculates the distance that the lens needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 500 through reverse movement to achieve anti-shake. The gyro sensor 580A can also be used in scenes such as navigation and somatosensory games.

[0108] The acceleration sensor 580B can detect the magnitude of the acceleration of the electronic device 500 in all directions (generally three axes). When the electronic device 500 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0109] The key 590 includes a power key, a volume key, etc. The key 590 may be a mechanical key or a touch key. The electronic device 500 may receive key input and generate key signal input related to user settings and function control of the electronic device 500.

[0110] Motor 591 can generate vibration prompts. Motor 591 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. Indicator 592 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0111] The SIM card interface 595 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 500 by inserting the SIM card interface 595 or removing the SIM card from the SIM card interface 595 .

[0112] The software system of the electronic device 500 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture, etc. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 500.

[0113] Figure 6 Schematic diagram of the software system structure of the electronic device 500 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, the hardware abstraction layer and the kernel layer.

[0114] The application layer can include a series of application packages. Figure 6 As shown, the application package may include applications such as camera, settings, and calendar.

[0115] The camera application is an application with the functions of photographing and recording, and the electronic device can respond to the user's operation of opening the camera application to take photos or record videos. It is understandable that the camera application's photo taking and video recording functions can also be called by other applications.

[0116] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0117] Among them, Figure 6As shown, the application framework layer may also include a camera service, which may be called by a camera application to implement functions such as taking photos or recording videos.

[0118] In addition, if Figure 6 As shown, the application framework layer may also include a window manager, a content provider, a resource manager, a view system, and the like.

[0119] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0120] Content providers are used to store and retrieve data and make it accessible to applications. This data can include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0121] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0122] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0123] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0124] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0125] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0126] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0127] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0128] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG2, H.262, MP3, AAC, AMR, JPG, PNG, etc.

[0129] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.

[0130] The hardware abstraction layer is an abstract structure between the kernel layer and the Android runtime. The hardware abstraction layer can be a package of the hardware driver of the kernel layer, providing a calling interface for the application framework layer.

[0131] In an embodiment of the present application, the hardware abstraction layer may include a camera hardware abstraction module (camera hardware abstraction layer, camera HAL). In some embodiments, when the electronic device is executing the image processing method of the embodiment of the present application, the camera hardware abstraction module may be used to obtain the first image captured by the camera when the optical image stabilization module is started, the first timestamp when the capture of the first image starts, the second timestamp when the capture of the first image ends, and the first coordinates of the position of the optical image stabilization module after each movement during the capture of the first image, determine the sensitivity calibration matrix corresponding to the first image according to the first timestamp, the second timestamp, each first coordinate, and multiple sensitivity calibration matrices, and use the sensitivity calibration matrix to calibrate the first image to obtain the second image.

[0132] The kernel layer is a layer between hardware and software. The kernel layer includes at least a camera driver, a sensor driver, and a display driver. In some embodiments, the camera driver is used to control the operation of the camera, the sensor driver is used to control the operation of the sensor, and the display driver is used to control the display screen to display images.

[0133] The hardware may be a camera, a sensor, a display screen, etc. In the embodiment of the present application, the camera may be a front camera or a rear camera.

[0134] It should be noted that although the embodiments of the present application are described using the Android system, the principles of the image processing method are also applicable to electronic devices with operating systems such as iOS or Windows.

[0135] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be implemented independently or in combination with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0136] For ease of understanding, the embodiment of the present application takes a mobile phone as an electronic device, and first illustrates the application scenarios of the image processing method in combination with some user interfaces shown in the embodiment of the present application.

[0137] When the user turns on the screen of the electronic device and controls the electronic device to be in an unlocked state, the electronic device may display the following Figure 7 The first interface 701 shown in (a) of FIG. The first interface 701 may be the desktop of the electronic device, on which icons of multiple installed applications are displayed, such as a file management application icon, an email application icon, a weather application icon, a calculator application icon, a clock application icon, a recorder application icon, a music application icon, a setting application icon, an address book application icon, a phone application icon, a message application icon, and a camera application icon 7011.

[0138] The user can perform a touch operation on the camera application icon 7011, and the touch operation can be a click operation, a long press operation, etc., so that the electronic device receives the user's touch operation on the camera application icon 7011, and the electronic device starts the camera application in response to the touch operation.

[0139] After the camera application is started, the electronic device may display the following Figure 7 The second interface 702 shown in (b) of FIG. The second interface 702 may be a preview interface provided by a camera application for implementing a shooting function, which includes a preview box 7021, a shooting control 7022, and function controls corresponding to a plurality of shooting modes.

[0140] The preview box 7021 can be used to display a second image obtained by calibrating the first image using a sensitivity calibration matrix. The shooting control 7022 is used to trigger the shooting operation of the electronic device. The functional controls corresponding to the various shooting modes may include night scene mode controls, portrait mode controls, photo mode controls, video mode controls, professional mode controls, and more controls for opening more functions in the camera application.

[0141] Therefore, the embodiment of the present application can start the camera application by performing a touch operation on the camera application icon 7011. After the camera application is started, the electronic device can execute the process corresponding to the image processing method provided in the embodiment of the present application.

[0142] Understandably, Figure 7 (a) and Figure 7 The interface shown in (b) is merely an example of a user interface in the process of launching a camera application by touching the camera application icon of an electronic device, and does not constitute a limitation on the embodiments of the present application.

[0143] In another scenario, the user can also call the corresponding interface through a third-party application installed on the electronic device to access the camera application of the electronic device to start the camera application. After the camera application is started, the electronic device can execute the process corresponding to the image processing method provided in the embodiment of the present application.

[0144] In addition, when the electronic device is in the lock screen state, the user can instruct the electronic device to open the camera application by sliding right on the display screen of the electronic device. Alternatively, when the electronic device is in the lock screen state, the lock screen interface includes an icon of the camera application, and the user instructs the electronic device to open the camera application by clicking the icon of the camera application.

[0145] It should be understood that the above is an example of the operation of opening the camera application; the camera application can also be opened by instructing the electronic device through voice instructions or other operations, and this application does not impose any limitation on this.

[0146] For example, Figure 8 A flowchart of an image processing method provided in an embodiment of the present application can be applied to an electronic device. The electronic device may include a camera, the camera includes an optical image stabilization module, a lens 10, and an N-pixel-in-one image sensor, the N-pixel-in-one image sensor includes a microlens array 21 and a pixel array, the microlens array 21 includes a plurality of microlenses 210, each microlens 210 covers N adjacent pixels of the same color in the pixel array, and N is an integer greater than 1.

[0147] Reference Figure 8 As shown, the image processing method may specifically include the following steps:

[0148] S801, the electronic device obtains a first image captured by a camera when an optical image stabilization module is started.

[0149] In some embodiments, when the camera activates the optical image stabilization module to perform image stabilization, the camera can capture a first image, where the first image is an image in a RAW format.

[0150] S802: The electronic device obtains a first timestamp when the acquisition of the first image starts, and a second timestamp when the acquisition of the first image ends.

[0151] When the camera captures the first image, it will control the photosensitive unit array in the photosensitive element to start exposure from the exposure start time corresponding to the first image, and will start reading the image data generated by the photosensitive unit array in sequence from the exposure end time corresponding to the first image to obtain the first image. Therefore, in the process of capturing the first image, the capture time includes the exposure time of the first image and the reading time of the first image.

[0152] The start point of the first image exposure is defined as the start of frame (SOF) point, and the end point of the first image is defined as the end of frame (EOF) point. The first timestamp at the SOF point is defined as T1, and the second timestamp at the EOF point is defined as T2.

[0153] Therefore, if Fig. 9 As shown, during the acquisition process of the first image, the acquisition duration of the first image is the time interval between the second timestamp T2 and the first timestamp T1. The first timestamp T1 represents the timestamp of the exposure start time of the first image, and is also the timestamp of the time when the camera starts to acquire the first image; the second timestamp T2 represents the timestamp of the reading end time of the first image, and is also the timestamp of the time when the camera ends acquiring the first image.

[0154] S803, the electronic device obtains the first coordinates of the position of the optical image stabilization module after each movement during the process of acquiring the first image.

[0155] When the camera starts the optical image stabilization module to capture the first image, the optical image stabilization module can move to drive the lens 10 to move along the first moving direction and / or the second moving direction to compensate for the impact of the shake and achieve the purpose of anti-shake.

[0156] In this way, during the process of capturing the first image, that is, in the time interval between the second timestamp T2 and the first timestamp T1, the first coordinate of the position of the optical image stabilization module after each movement can be counted.

[0157] like Fig.10 As shown, during the acquisition process of the first image, the optical image stabilization module can move along the first moving direction (ie Fig.10 The X direction in the second moving direction (i.e. Fig.10 The optical image stabilization module moves in the Y direction (in the Y direction), and moves a total of n times. The first coordinates of the position of the optical image stabilization module after the n times of movement are P1 to P n .

[0158] During the acquisition of the first image, the first coordinate of the position of the optical image stabilization module after the i-th movement is P i The first coordinate may include a first coordinate value in a first moving direction and a second coordinate value in a second moving direction.

[0159] It can be understood that the first coordinate of the position of the optical image stabilization module after each movement actually represents the relative position between the optical image stabilization module and the optical center of the N-pixel integrated image sensor after the movement of the optical image stabilization module. The relative position between the optical center of the N-pixel integrated image sensor after the movement of the optical image stabilization module can also be used to represent the relative position between the optical center of the lens 10 and the optical center of the N-pixel integrated image sensor after the optical image stabilization module moves to drive the lens 10 to move.

[0160] For example, when the optical image stabilization module is not activated in the camera, the position coordinates of the optical image stabilization module may be (0, 0), which may indicate that the optical center of the lens 10 coincides with the optical center of the N-pixel image sensor along the optical axis of the camera.

[0161] S804: The electronic device determines, according to the first timestamp and the second timestamp, the number of times the optical image stabilization module moves during the acquisition process of the first image.

[0162] After acquiring the first timestamp when the acquisition of the first image starts and the second timestamp when the acquisition of the first image ends, the electronic device can determine the number of times the optical image stabilization module moves during the acquisition of the first image according to the first timestamp and the second timestamp.

[0163] In some embodiments, the electronic device calculates the time interval between the second timestamp and the first timestamp; the electronic device determines the ratio between the time interval and the movement frequency of the optical image stabilization module as the number of times the optical image stabilization module moves during the acquisition of the first image. Therefore:

[0164]

[0165] Wherein, n is the number of times the optical image stabilization module moves during the acquisition process of the first image, T2 is the second timestamp, T1 is the first timestamp, and f0 is the movement frequency of the optical image stabilization module.

[0166] The number of times the optical image stabilization module moves during the acquisition of the first image may also be referred to as the number of refreshes of the optical image stabilization module during the acquisition of the first image; the movement frequency of the optical image stabilization module may also be referred to as the refresh frequency of the optical image stabilization module.

[0167] For example, the time interval between the second timestamp and the first timestamp is 20 ms (milliseconds), and the movement frequency of the optical image stabilization module may be 1000 Hz, that is, the optical image stabilization module moves once every 1 ms. Therefore, the optical image stabilization module may move 20 times during the capture process of the first image.

[0168] S805: The electronic device determines, according to the number of movements and each first coordinate, a second coordinate of the centroid position corresponding to each position where the optical image stabilization module is located after movement.

[0169] After acquiring the first coordinate of the position of the optical image stabilization module after each movement and the number of movements of the optical image stabilization module during the acquisition of the first image, the electronic device can determine the second coordinate of the center of mass position corresponding to each position of the optical image stabilization module after the movement according to the number of movements and each first coordinate.

[0170] The centroid positions corresponding to the positions where the optical image stabilization module is located after the movement are the centroids of all the positions where the optical image stabilization module is located after the movement.

[0171] In some embodiments, the electronic device calculates the second coordinate using the following formula:

[0172]

[0173] Among them, S n is the second coordinate, n is the number of moves, P i is the first coordinate of the position of the optical image stabilization module after the i-th movement. The second coordinate may also include a first coordinate component in the first moving direction and a second coordinate component in the second moving direction.

[0174] S806, the electronic device determines a sensitivity calibration matrix corresponding to the first image according to the second coordinates and multiple sensitivity calibration matrices; when calibrating the multiple sensitivity calibration matrices, the relative positions of the optical center of the lens and the optical center of the N-pixel integrated image sensor are different.

[0175] Before using multiple sensitivity calibration matrices to determine the sensitivity calibration matrix corresponding to the first image, the electronic device may also obtain multiple pre-calibrated sensitivity calibration matrices. The multiple sensitivity calibration matrices include a first sensitivity calibration matrix, a second sensitivity calibration matrix, a third sensitivity calibration matrix, a fourth sensitivity calibration matrix, and a fifth sensitivity calibration matrix; and when calibrating the multiple sensitivity calibration matrices, the relative positions of the optical center of the lens 10 and the optical center of the N-pixel-in-one image sensor are different.

[0176] The optical image stabilization module can drive the lens 10 to move along a first moving direction and a second moving direction. The first moving direction includes a first direction and a second direction that are opposite to each other, and the second moving direction includes a third direction and a fourth direction that are opposite to each other. Fig.11 As shown, the first direction may be an X1 direction, the second direction may be an X2 direction, the third direction may be a Y1 direction, and the fourth direction may be a Y2 direction.

[0177] like Fig.11 As shown in (a) to (e) in FIG. 1 , it represents the relative position between the optical center of the lens 10 and the optical center of the N-pixel-in-one image sensor during the calibration process of each sensitivity calibration matrix. The intersection of the two dotted lines represents the optical center of the N-pixel-in-one image sensor.

[0178] like Fig.11 As shown in (a) of FIG. 1 , when the first sensitivity calibration matrix is ​​calibrated, the optical center of the lens 10 coincides with the optical center of the N-pixel-in-one image sensor along the optical axis direction of the camera. In this case, the relative position of the optical center of the lens 10 relative to the optical center of the N-pixel-in-one image sensor is G0, and the position coordinates of the G0 position are (0, 0).

[0179] like Fig.11 As shown in (b) of FIG. 1 , during calibration of the second sensitivity calibration matrix, the optical center of the lens 10 is offset by a first distance in the first direction and a second distance in the third direction relative to the optical center of the N-pixel-in-one image sensor; the first distance is the maximum distance that the lens 10 can move along the first direction, and the second distance is the maximum distance that the lens 10 can move along the third direction. In this case, the relative position of the optical center of the lens 10 relative to the optical center of the N-pixel-in-one image sensor is G1, and the position coordinates of the position of G1 are (S xmax1 , S ymax1 ), S xmax1 is the first distance, S ymax1 is the second distance.

[0180] like Fig.11 As shown in (c) of FIG. 1 , when the third sensitivity calibration matrix is ​​calibrated, the optical center of the lens 10 is offset by a first distance in the first direction and a third distance in the fourth direction relative to the optical center of the N-pixel-in-one image sensor; the third distance is the maximum distance that the lens 10 can move along the fourth direction. In this case, the relative position of the optical center of the lens 10 relative to the optical center of the N-pixel-in-one image sensor is G2, and the position coordinates of the position of G2 are (S xmax1 , -S ymax2 ), S ymax2 The third distance.

[0181] like Fig.11 As shown in (d) of FIG. 1 , when the fourth sensitivity calibration matrix is ​​calibrated, the optical center of the lens 10 is offset by a fourth distance along the second direction and a third distance along the fourth direction relative to the optical center of the N-pixel-in-one image sensor; the fourth distance is the maximum distance that the lens 10 can move along the second direction. In this case, the relative position of the optical center of the lens 10 relative to the optical center of the N-pixel-in-one image sensor is G3, and the position coordinates of the position of G3 are (-Sxmax2 , -S ymax2 ), S xmax2 The fourth distance.

[0182] like Fig.11 As shown in (e) of FIG. 1 , when the fifth sensitivity calibration matrix is ​​calibrated, the optical center of the lens 10 is offset by a fourth distance along the second direction and a second distance along the third direction relative to the optical center of the N-pixel-in-one image sensor. In this case, the relative position of the optical center of the lens 10 relative to the optical center of the N-pixel-in-one image sensor is G4, and the position coordinate of the G4 position is (-S xmax2 , S ymax1 ).

[0183] In some embodiments, the first distance and the fourth distance may be equal or unequal; the second distance and the third distance may be equal or unequal, which is not limited in the embodiments of the present application. In addition, the first distance, the second distance, the third distance and the fourth distance are all positive numbers.

[0184] In some embodiments, each sensitivity calibration matrix includes sensitivity calibration sub-matrices corresponding to H color sub-channels, where H is equal to the product of M and N. Wherein, each calibration parameter in the sensitivity calibration sub-matrix is ​​calculated based on the mean of each pixel in the single-channel mean image and the pixel value at the corresponding position in each single-channel test image corresponding to the single-channel mean image; the mean of each pixel in the single-channel mean image is the average value of the pixel values ​​at the same position in the N single-channel test images belonging to the same color channel in the H single-channel test images; the H single-channel test images are obtained by reducing the size of the test image and splitting it according to the H color sub-channels; the test image is an image captured when the focus position of the lens is a preset focus position, and the relative position between the optical center of the lens and the optical center of the image sensor with N pixels in one is a preset position.

[0185] In one implementation, each calibration parameter in the sensitivity calibration submatrix is ​​the ratio of the mean value of each pixel in the single-channel mean image to the pixel value at the corresponding position in each single-channel test image corresponding to the single-channel mean image.

[0186] Taking the calibration process of the first sensitivity calibration matrix as an example, Fig.12 This is a flow chart of the calibration process of the first sensitivity calibration matrix provided in the embodiment of the present application. Fig.12 As shown, the calibration process may specifically include the following steps:

[0187] S1201, when the focus position of the lens is a preset focus position and the optical center of the lens coincides with the optical center of the N-pixel-in-one image sensor along the optical axis direction of the camera, use the camera to capture a test image.

[0188] The focus position of the lens at infinity is fixed as Z1, and the focus position of the lens at the minimum focus distance is Z2, then the focus position of the lens in the camera is set to (Z1+Z2) / 2, that is, the preset focus position is 1 / 2 of the optical travel of the lens; and the optical center of the lens is set to coincide with the optical center of the N-pixel image sensor along the optical axis direction of the camera, that is, the optical displacement between the optical center of the lens 10 and the optical center of the N-pixel image sensor is 0.

[0189] When the focus position of the lens is set at a preset focus position, and the optical center of the lens and the optical center of the N-pixel-in-one image sensor are set to coincide along the optical axis direction of the camera, the camera is used to shoot the uniform light panel.

[0190] When shooting a uniform light plate, the object distance can be 10 mm (millimeter). A uniform light plate refers to a plane light source with uniform illumination and consistent color temperature within the effective area. The test image captured by the camera can be a RAW format image with full pixel resolution.

[0191] S1202: Reduce the size of the test image.

[0192] The size of the test image can be reduced by bilinear interpolation. For example, the original size of the test image is a pixels * b pixels, a represents the width of the test image, b represents the height of the test image, and a and b are both positive integers; the size of the reduced test image can be 64 pixels * 48 pixels.

[0193] It can be understood that the size of the reduced-size test image, 64 pixels*48 pixels, is only an example, and the size of the reduced-size test image may also be other sizes, which is not limited in the embodiments of the present application.

[0194] After the size of the test image is reduced, the amount of data in the subsequent calculation of the first sensitivity calibration matrix can be reduced, so as to reduce the calculation complexity of the first sensitivity calibration matrix.

[0195] S1203 , splitting the reduced-size test image according to H color sub-channels to obtain H single-channel test images.

[0196] When the camera includes an N-pixel-in-one image sensor, and the pixel array of the N-pixel-in-one image sensor includes pixel units corresponding to M color channels, and each pixel unit includes pixels corresponding to N color sub-channels, the test image includes M color channels, and each color channel includes N color sub-channels, that is, the test image may include H color sub-channels, where H is equal to the product of M and N. The test image after size reduction may also include H color sub-channels.

[0197] The reduced-size test image may be split according to H color sub-channels to obtain H single-channel test images, each of which corresponds to one color sub-channel.

[0198] The image sensor with N pixels in one is Figure 3 Taking the four-pixel-in-one image sensor shown in FIG. 1 as an example, the test image and the reduced-size test image may both include four color channels, namely a red channel (R channel), a first green channel (Gr channel), a second green channel (Gb channel), and a blue channel (B channel). Furthermore, each color channel includes four color sub-channels. For example, the red channel includes a first red sub-channel (R0 sub-channel), a second red sub-channel (R1 sub-channel), a third red sub-channel (R2 sub-channel) and a fourth red sub-channel (R3 sub-channel); the first green channel includes a first green sub-channel (Gr0 sub-channel), a second green sub-channel (Gr1 sub-channel), a third green sub-channel (Gr2 sub-channel) and a fourth green sub-channel (Gr3 sub-channel); the second green channel includes a fifth green sub-channel (Gb0 sub-channel), a sixth green sub-channel (Gb1 sub-channel), a seventh green sub-channel (Gb2 sub-channel) and an eighth green sub-channel (Gb3 sub-channel); the blue channel includes a first blue sub-channel (B0 sub-channel), a second blue sub-channel (B1 sub-channel), a third blue sub-channel (B2 sub-channel) and a fourth blue sub-channel (B3 sub-channel).

[0199] Therefore, in the N-pixel-in-one image sensor, Figure 3 In the case of the four-pixel-in-one image sensor shown, the test image and the reduced-size test image both include sixteen color subchannels, namely R0 subchannel, R1 subchannel, R2 subchannel, R3 subchannel, Gr0 subchannel, Gr1 subchannel, Gr2 subchannel, Gr3 subchannel, Gb0 subchannel, Gb1 subchannel, Gb2 subchannel, Gb3 subchannel, B0 subchannel, B1 subchannel, B2 subchannel and B3 subchannel.

[0200] Thus, after the reduced-size test image is split according to the color sub-channels, sixteen single-channel test images can be obtained, including: Fig.13The single-channel test image corresponding to the R0 subchannel shown in (a) is as follows: Fig.13 The single-channel test image corresponding to the R1 subchannel shown in (b) is as follows: Fig.13 The single-channel test image corresponding to the R2 subchannel shown in (c) is as follows: Fig.13 The single-channel test image corresponding to the R3 subchannel shown in (d) is as follows: Fig.13 The single-channel test image corresponding to the Gr0 subchannel shown in (e) is as follows: Fig.13 The single-channel test image corresponding to the Gr1 subchannel shown in (f) is as follows: Fig.13 The single-channel test image corresponding to the Gr2 subchannel shown in (g) is as follows: Fig.13 The single-channel test image corresponding to the Gr3 subchannel shown in (h) is as follows: Fig.13 The single-channel test image corresponding to the Gb0 subchannel shown in (i) is as follows: Fig.13 The single-channel test image corresponding to the Gb1 subchannel shown in (j) is as follows: Fig.13 The single-channel test image corresponding to the Gb2 subchannel shown in (k) is as follows: Fig.13 The single-channel test image corresponding to the Gb3 subchannel shown in (l) is as follows: Fig.13 The single-channel test image corresponding to the B0 subchannel shown in (m) is as follows: Fig.13 The single-channel test image corresponding to the B1 subchannel shown in (n) is as follows: Fig.13 The single-channel test image corresponding to the B2 subchannel shown in (o) and Fig.13 The single-channel test image corresponding to the B3 sub-channel shown in (p) in FIG.

[0201] For example, the size of the reduced-size test image is 64 pixels*48 pixels, and the reduced-size test image includes sixteen color sub-channels. Then, among the sixteen single-channel test images obtained after splitting, the size of each single-channel test image is 16 pixels*12 pixels.

[0202] S1204, calculating the average value of pixel values ​​at the same position in N single-channel test images belonging to the same color channel in the H single-channel test images, and obtaining the average value of each pixel in the single-channel average image.

[0203] For N single-channel test images of the same color channel, the average values ​​of the pixel values ​​at the same position in the N single-channel test images are calculated respectively to obtain the mean value of each pixel in the single-channel mean image.

[0204] For example, the average value of the pixel values ​​in the kth row and jth column in N single-channel test images of the same color channel is the average value of the pixels in the kth row and jth column in the corresponding single-channel mean image, where k and j are both positive integers.

[0205] For example, for Fig.13 The single-channel test image corresponding to the R0 subchannel shown in (a) is as follows: Fig.13 The single-channel test image corresponding to the R1 subchannel shown in (b) is as follows: Fig.13 The single-channel test image corresponding to the R2 subchannel shown in (c) and Fig.13 The single-channel test image corresponding to the R3 subchannel shown in (d) in FIG. 1 is calculated by averaging the pixel values ​​at the same position, and the following can be obtained: Fig.14 The mean value of each pixel in the single-channel mean image corresponding to the R channel shown in (a).

[0206] For example Fig.13 The single-channel test image corresponding to the Gr0 subchannel shown in (e) is as follows: Fig.13 The single-channel test image corresponding to the Gr1 subchannel shown in (f) is as follows: Fig.13 The single-channel test image corresponding to the Gr2 subchannel shown in (g) and Fig.13 The single-channel test image corresponding to the Gr3 subchannel shown in (h) in FIG. 1 is calculated by averaging the pixel values ​​at the same position, and the following can be obtained: Fig.14 The mean value of each pixel in the single-channel mean image corresponding to the Gr channel shown in (b).

[0207] For example Fig.13 The single-channel test image corresponding to the Gb0 subchannel shown in (i) is as follows: Fig.13 The single-channel test image corresponding to the Gb1 subchannel shown in (j) is as follows: Fig.13 The single-channel test image corresponding to the Gb2 subchannel shown in (k) and Fig.13 The single-channel test image corresponding to the Gb3 subchannel shown in (l) in FIG. 1 is calculated by averaging the pixel values ​​at the same position, and the following can be obtained: Fig.14 The mean value of each pixel in the single-channel mean image corresponding to the Gb channel shown in (c).

[0208] For example Fig.13 The single-channel test image corresponding to the B0 subchannel shown in (m) is as follows: Fig.13 The single-channel test image corresponding to the B1 subchannel shown in (n) is as follows: Fig.13 The single-channel test image corresponding to the B2 subchannel shown in (o) and Fig.13The single-channel test image corresponding to the B3 subchannel shown in (p) in FIG. 1 is calculated by averaging the pixel values ​​at the same position, and the following can be obtained: Fig.14 The mean value of each pixel in the single-channel mean image corresponding to the B channel shown in (d).

[0209] S1205, calculating the ratio of the mean value of each pixel in the single-channel mean image to the pixel value at the corresponding position in each single-channel test image corresponding to the single-channel mean image, and obtaining a sensitivity calibration submatrix.

[0210] Among them, the calibration parameter of the kth row and jth column in the sensitivity calibration submatrix is ​​the ratio of the pixel mean of the kth row and jth column in the single-channel mean image to the pixel value of the kth row and jth column in the single-channel test image corresponding to the single-channel mean image.

[0211] When there are H single-channel test images, there are also H sensitivity calibration sub-matrices, that is, the first sensitivity calibration matrix includes sensitivity calibration sub-matrices corresponding to the H color sub-channels, and each sensitivity calibration sub-matrix corresponds to one color sub-channel.

[0212] For example, the calculation is as follows Fig.14 The mean value of each pixel in the single channel mean image corresponding to the R channel shown in (a) is the same as Fig.13 The ratio of the pixel values ​​at the corresponding positions in the single-channel test image corresponding to the R0 subchannel shown in (a) is obtained as follows Fig.15 The sensitivity calibration submatrix corresponding to the R0 subchannel shown in (a) in FIG.

[0213] Calculate as Fig.14 The mean value of each pixel in the single channel mean image corresponding to the R channel shown in (a) is the same as Fig.13 The ratio of the pixel values ​​at the corresponding positions in the single-channel test image corresponding to the R1 subchannel shown in (b) is obtained as follows Fig.15 The sensitivity calibration submatrix corresponding to the R1 subchannel shown in (b).

[0214] Calculate as Fig.14 The mean value of each pixel in the single channel mean image corresponding to the R channel shown in (a) is the same as Fig.13 The ratio of the pixel values ​​at the corresponding positions in the single-channel test image corresponding to the R2 subchannel shown in (c) is obtained as follows Fig.15 The sensitivity calibration submatrix corresponding to the R2 subchannel shown in (c) in FIG.

[0215] Calculate as Fig.14 The mean value of each pixel in the single channel mean image corresponding to the R channel shown in (a) is the same as Fig.13The ratio of the pixel values ​​at the corresponding positions in the single-channel test image corresponding to the R3 subchannel shown in (d) is obtained as follows Fig.15 The sensitivity calibration submatrix corresponding to the R3 subchannel shown in (d) in FIG.

[0216] Calculate as Fig.14 The mean value of each pixel in the single channel mean image corresponding to the Gr channel shown in (b) is the same as Fig.13 The ratio of the pixel values ​​at the corresponding position in the single-channel test image corresponding to the Gr0 subchannel shown in (e) is obtained as follows Fig.15 The sensitivity calibration submatrix corresponding to the Gr0 subchannel shown in (e) in FIG.

[0217] Calculate as Fig.14 The mean value of each pixel in the single channel mean image corresponding to the Gr channel shown in (b) is the same as Fig.13 The ratio of the pixel values ​​at the corresponding positions in the single-channel test image corresponding to the Gr1 subchannel shown in (f) is obtained as follows Fig.15 The sensitivity calibration submatrix corresponding to the Gr1 subchannel shown in (f) in FIG.

[0218] Calculate as Fig.14 The mean value of each pixel in the single channel mean image corresponding to the Gr channel shown in (b) is the same as Fig.13 The ratio of the pixel values ​​at the corresponding positions in the single-channel test image corresponding to the Gr2 subchannel shown in (g) is obtained as follows Fig.15 The sensitivity calibration submatrix corresponding to the Gr2 subchannel shown in (g) in FIG.

[0219] Calculate as Fig.14 The mean value of each pixel in the single channel mean image corresponding to the Gr channel shown in (b) is the same as Fig.13 The ratio of the pixel values ​​at the corresponding positions in the single-channel test image corresponding to the Gr3 subchannel shown in (h) is obtained as follows Fig.15 The sensitivity calibration submatrix corresponding to the Gr3 subchannel shown in (h).

[0220] Calculate as Fig.14 The mean value of each pixel in the single channel mean image corresponding to the Gb channel shown in (c) is the same as Fig.13 The ratio of the pixel values ​​at the corresponding positions in the single-channel test image corresponding to the Gb0 subchannel shown in (i) is obtained as follows Fig.15 The sensitivity calibration submatrix corresponding to the Gb0 subchannel shown in (i) in FIG.

[0221] Calculate as Fig.14 The mean value of each pixel in the single channel mean image corresponding to the Gb channel shown in (c) is the same as Fig.13 The ratio of the pixel values ​​at the corresponding positions in the single-channel test image corresponding to the Gb1 subchannel shown in (j) is obtained as follows Fig.15 The sensitivity calibration submatrix corresponding to the Gb1 subchannel shown in (j) in FIG.

[0222] Calculate as Fig.14 The mean value of each pixel in the single channel mean image corresponding to the Gb channel shown in (c) is the same as Fig.13 The ratio of the pixel values ​​at the corresponding positions in the single-channel test image corresponding to the Gb2 subchannel shown in (k) is obtained as follows: Fig.15 The sensitivity calibration submatrix corresponding to the Gb2 subchannel shown in (k) in FIG.

[0223] Calculate as Fig.14 The mean value of each pixel in the single channel mean image corresponding to the Gb channel shown in (c) is the same as Fig.13 The ratio of the pixel values ​​at the corresponding positions in the single-channel test image corresponding to the Gb3 subchannel shown in (l) is obtained as follows: Fig.15 The sensitivity calibration submatrix corresponding to the Gb3 subchannel shown in (l) in FIG.

[0224] Calculate as Fig.14 The mean value of each pixel in the single channel mean image corresponding to the B channel shown in (d) is the same as Fig.13 The ratio of the pixel values ​​at the corresponding position in the single-channel test image corresponding to the B0 subchannel shown in (m) is obtained as follows Fig.15 The sensitivity calibration submatrix corresponding to the B0 subchannel shown in (m) in FIG.

[0225] Calculate as Fig.14 The mean value of each pixel in the single channel mean image corresponding to the B channel shown in (d) is the same as Fig.13 The ratio of the pixel values ​​at the corresponding positions in the single-channel test image corresponding to the B1 subchannel shown in (n) is obtained as follows: Fig.15 The sensitivity calibration submatrix corresponding to the B1 subchannel shown in (n) in FIG.

[0226] Calculate as Fig.14 The mean value of each pixel in the single channel mean image corresponding to the B channel shown in (d) is the same as Fig.13 The ratio of the pixel values ​​at the corresponding positions in the single-channel test image corresponding to the B2 subchannel shown in (o) is obtained as follows: Fig.15 The sensitivity calibration submatrix corresponding to the B2 subchannel shown in (o) in FIG.

[0227] Calculate as Fig.14 The mean value of each pixel in the single channel mean image corresponding to the B channel shown in (d) is the same as Fig.13 The ratio of the pixel values ​​at the corresponding positions in the single-channel test image corresponding to the B3 subchannel shown in (p) is obtained as follows Fig.15 The sensitivity calibration submatrix corresponding to the B3 subchannel shown in (p) in FIG.

[0228] Therefore, in the N-pixel-in-one image sensor, Figure 3 In the case of the four-pixel-in-one image sensor shown in FIG. 1 , the first sensitivity calibration matrix includes: Fig.15 The sensitivity calibration submatrix corresponding to the sixteen color subchannels shown, that is, H is equal to 16.

[0229] For example, the size of each single-channel test image is 16 pixels * 12 pixels, the size of each single-channel mean image is also 16 pixels * 12 pixels, and the size of each sensitivity calibration submatrix is ​​also 16 pixels * 12 pixels.

[0230] In summary, according to the above steps S1201 to S1205, the first sensitivity calibration matrix can be obtained. In actual application, if the camera is not provided with an optical image stabilization module or the optical image stabilization module is turned off, the first sensitivity calibration matrix can be used to calibrate the first image captured by the camera, so that the responses of the second image obtained after calibration in each color sub-channel under the same color channel are basically consistent, so as to reduce the pixel difference degree of the H color sub-channels in the second image.

[0231] Fig.16 A flowchart of the calibration process of multiple sensitivity calibration matrices provided in an embodiment of the present application. Fig.16 As shown, the calibration process may specifically include the following steps:

[0232] S1601, when the focus position of the lens is a preset focus position and the optical center of the lens and the optical center of the N-pixel-in-one image sensor are in different relative positions in sequence, a camera is used to collect a plurality of test images.

[0233] S1602: Calculate and obtain multiple sensitivity calibration matrices based on multiple test images.

[0234] After the optical image stabilization module is powered on, it enters the command mode, which is a code name used to start the movement of the optical image stabilization module to drive the lens to a fixed optical displacement and keep it stable.

[0235] For example, the optical image stabilization module can sequentially drive the lens to move to the following positions: Fig.11 The G0 position shown in (a) is as follows: Fig.11 The G1 position shown in (b) is as follows Fig.11 The G2 position shown in (c) is as follows: Fig.11 The G3 position shown in (d) and Fig.11 The G4 position shown in (e) in FIG.

[0236] When the focus position of the lens is set at a preset focus position, and the optical center of the lens and the optical center of the N-pixel image sensor are set to coincide along the optical axis direction of the camera, a uniform light plate is photographed by the camera to obtain a test image, and the above steps S1202 to S1205 are followed to obtain a first sensitivity calibration matrix.

[0237] Correspondingly, when the focus position of the lens is set at a preset focus position, and the optical center of the lens and the optical center of the image sensor with N pixels are offset by a first distance in a first direction and by a second distance in a third direction, a camera is used to shoot a uniform light plate to obtain a test image, and the above steps S1202 to S1205 are followed to obtain a second sensitivity calibration matrix.

[0238] When the focus position of the lens is set at a preset focus position and the optical center of the lens is offset by a first distance in a first direction and a third distance in a fourth direction relative to the optical center of the N-pixel image sensor, a camera is used to shoot a uniform light panel to obtain a test image, and the above steps S1202 to S1205 are followed to obtain a third sensitivity calibration matrix.

[0239] When the focus position of the lens is set at a preset focus position, and the optical center of the lens is offset by a fourth distance along the second direction and a third distance along the fourth direction relative to the optical center of the N-pixel image sensor, a camera is used to shoot a uniform light plate to obtain a test image, and the above steps S1202 to S1205 are followed to obtain a fourth sensitivity calibration matrix.

[0240] When the focus position of the lens is set at a preset focus position, and the optical center of the lens is offset by a fourth distance along the second direction and a second distance along the third direction relative to the optical center of the N-pixel image sensor, a camera is used to shoot a uniform light plate to obtain a test image, and the above steps S1202 to S1205 are followed to obtain a fifth sensitivity calibration matrix.

[0241] It should be noted that the camera used in the calibration process of the sensitivity calibration matrix of the embodiment of the present application may be the same camera as the camera used to collect the first image during the execution of the image processing method, or may not be the same camera. In the case where the camera used in the calibration process of the sensitivity calibration matrix is ​​not the same camera as the camera used to collect the first image during the execution of the image processing method, the camera used in the calibration process of the sensitivity calibration matrix may be of the same type as the camera used to collect the first image during the execution of the image processing method. For example, the structural composition of the camera used in the calibration process of the sensitivity calibration matrix may be the same as the structural composition of the camera used to collect the first image during the execution of the image processing method.

[0242] After pre-calibrating and obtaining a plurality of sensitivity calibration matrices in the above manner, the electronic device may determine the sensitivity calibration matrix corresponding to the first image according to the second coordinates and the plurality of sensitivity calibration matrices.

[0243] For example, Fig.17 As shown, Q0 is the first sensitivity calibration matrix, Q1 is the second sensitivity calibration matrix, Q2 is the third sensitivity calibration matrix, Q3 is the fourth sensitivity calibration matrix, and Q4 is the fifth sensitivity calibration matrix. n is the second coordinate, Q n is the sensitivity calibration matrix corresponding to the first image. Therefore, the sensitivity calibration matrix corresponding to the first image, in addition to being related to the second coordinate, is also related to multiple sensitivity calibration matrices in the first sensitivity calibration matrix, the second sensitivity calibration matrix, the third sensitivity calibration matrix, the fourth sensitivity calibration matrix, and the fifth sensitivity calibration matrix.

[0244] In some embodiments, the electronic device decomposes the second coordinate according to the movable direction of the optical image stabilization module to obtain a first coordinate component in the first moving direction and a second coordinate component in the second moving direction; the electronic device determines the sensitivity calibration matrix corresponding to the first image according to the first coordinate component, the second coordinate component and a plurality of sensitivity calibration matrices. The first moving direction and the second moving direction are perpendicular to each other, and both the first moving direction and the second moving direction are perpendicular to the optical axis direction of the camera.

[0245] The electronic device decomposes the second coordinate into the first moving direction to obtain a first coordinate component S in the first moving direction. nx ; The electronic device decomposes the second coordinate into the second moving direction to obtain the second coordinate component S in the second moving direction ny Then, the sensitivity calibration matrix corresponding to the first image is calculated by linear interpolation using the first coordinate component, the second coordinate component and a plurality of sensitivity calibration matrices.

[0246] Since the first moving direction includes a first direction and a second direction which are opposite to each other, and the second moving direction includes a third direction and a fourth direction which are opposite to each other, therefore, when the second coordinate is offset toward the first direction relative to the optical center of the N-pixel-in-one image sensor, the first coordinate component is a value greater than 0; and when the second coordinate is offset toward the second direction relative to the optical center of the N-pixel-in-one image sensor, the first coordinate component is a value less than 0. Correspondingly, when the second coordinate is offset toward the third direction relative to the optical center of the N-pixel-in-one image sensor, the second coordinate component is a value greater than 0; and when the second coordinate is offset toward the fourth direction relative to the optical center of the N-pixel-in-one image sensor, the second coordinate component is a value less than 0.

[0247] Therefore, based on the magnitude relationship between the first coordinate component and 0, and the magnitude relationship between the second coordinate component and 0, the following four situations may exist.

[0248] In the first case, when the first coordinate component is greater than or equal to 0 and the second coordinate component is greater than or equal to 0, the electronic device calculates the sensitivity calibration matrix by the following formula:

[0249]

[0250] Among them, Q n is the sensitivity calibration matrix, Q0 is the first sensitivity calibration matrix, Q1 is the second sensitivity calibration matrix, Q2 is the third sensitivity calibration matrix, and Q4 is the fifth sensitivity calibration matrix; S nx is the first coordinate component, S ny is the second coordinate component, S xmax1 is the first distance, S ymax1 is the second distance.

[0251] Therefore, when the first coordinate component is greater than or equal to 0 and the second coordinate component is greater than or equal to 0, the electronic device can determine the sensitivity calibration matrix corresponding to the first image based on the first coordinate component, the second coordinate component, the first sensitivity calibration matrix, the second sensitivity calibration matrix, the third sensitivity calibration matrix and the fifth sensitivity calibration matrix.

[0252] Since the first sensitivity calibration matrix, the second sensitivity calibration matrix, the third sensitivity calibration matrix and the fifth sensitivity calibration matrix all include sensitivity calibration sub-matrices corresponding to the H color sub-channels, and the sensitivity calibration matrix also includes the first sensitivity calibration sub-matrix corresponding to the H color sub-channels.

[0253] Therefore, in the above formula, for each color sub-channel, the first sensitivity calibration matrix, the second sensitivity calibration matrix, the third sensitivity calibration matrix and the fifth sensitivity calibration matrix are used respectively, including each calibration parameter in the sensitivity calibration sub-matrix corresponding to the color sub-channel, to calculate the calibration parameters at the corresponding positions in the first sensitivity calibration sub-matrix corresponding to the color sub-channel included in the sensitivity calibration matrix, so as to obtain the sensitivity calibration matrix.

[0254] For example, for the R0 sub-channel, the above formula is used to respectively utilize the first sensitivity calibration matrix, the second sensitivity calibration matrix, the third sensitivity calibration matrix and the fifth sensitivity calibration matrix, including the calibration parameters of the kth row and jth column in the sensitivity calibration sub-matrix corresponding to the R0 sub-channel, to calculate the calibration parameters of the kth row and jth column in the first sensitivity calibration sub-matrix corresponding to the R0 sub-channel.

[0255] When the size of each sensitivity calibration submatrix is ​​16 pixels*12 pixels, the size of the first sensitivity calibration submatrix is ​​also 16 pixels*12 pixels.

[0256] In the second case, when the first coordinate component is greater than 0 and the second coordinate component is less than 0, the electronic device calculates the sensitivity calibration matrix by the following formula:

[0257]

[0258] Among them, Q n is the sensitivity calibration matrix, Q0 is the first sensitivity calibration matrix, Q1 is the second sensitivity calibration matrix, Q2 is the third sensitivity calibration matrix, and Q3 is the fourth sensitivity calibration matrix; S nx is the first coordinate component, S ny is the second coordinate component, S xmax1 is the first distance, S ymax2 The third distance.

[0259] Therefore, when the first coordinate component is greater than 0 and the second coordinate component is less than 0, the electronic device can determine the sensitivity calibration matrix corresponding to the first image based on the first coordinate component, the second coordinate component, the first sensitivity calibration matrix, the second sensitivity calibration matrix, the third sensitivity calibration matrix and the fourth sensitivity calibration matrix.

[0260] Since the first sensitivity calibration matrix, the second sensitivity calibration matrix, the third sensitivity calibration matrix and the fourth sensitivity calibration matrix all include sensitivity calibration sub-matrices corresponding to the H color sub-channels, and the sensitivity calibration matrix also includes the first sensitivity calibration sub-matrix corresponding to the H color sub-channels.

[0261] Therefore, in the above formula, for each color sub-channel, the first sensitivity calibration matrix, the second sensitivity calibration matrix, the third sensitivity calibration matrix and the fourth sensitivity calibration matrix are used respectively, including each calibration parameter in the sensitivity calibration sub-matrix corresponding to the color sub-channel, to calculate the calibration parameters at the corresponding positions in the first sensitivity calibration sub-matrix corresponding to the color sub-channel included in the sensitivity calibration matrix, so as to obtain the sensitivity calibration matrix.

[0262] In the third case, when the first coordinate component is less than or equal to 0 and the second coordinate component is less than or equal to 0, the electronic device calculates the sensitivity calibration matrix by the following formula:

[0263]

[0264] Among them, Q n is the sensitivity calibration matrix, Q0 is the first sensitivity calibration matrix, Q2 is the third sensitivity calibration matrix, Q3 is the fourth sensitivity calibration matrix, and Q4 is the fifth sensitivity calibration matrix; S nx is the first coordinate component, S ny is the second coordinate component, S xmax2 is the fourth distance, S ymax2 The third distance.

[0265] Therefore, when the first coordinate component is less than or equal to 0 and the second coordinate component is less than or equal to 0, the electronic device can determine the sensitivity calibration matrix corresponding to the first image based on the first coordinate component, the second coordinate component, the first sensitivity calibration matrix, the third sensitivity calibration matrix, the fourth sensitivity calibration matrix and the fifth sensitivity calibration matrix.

[0266] Since the first sensitivity calibration matrix, the third sensitivity calibration matrix, the fourth sensitivity calibration matrix and the fifth sensitivity calibration matrix all include sensitivity calibration sub-matrices corresponding to the H color sub-channels, and the sensitivity calibration matrix also includes the first sensitivity calibration sub-matrix corresponding to the H color sub-channels.

[0267] Therefore, in the above formula, for each color sub-channel, the first sensitivity calibration matrix, the third sensitivity calibration matrix, the fourth sensitivity calibration matrix and the fifth sensitivity calibration matrix are used respectively, including each calibration parameter in the sensitivity calibration sub-matrix corresponding to the color sub-channel, to calculate the calibration parameters at the corresponding positions in the first sensitivity calibration sub-matrix corresponding to the color sub-channel included in the sensitivity calibration matrix, so as to obtain the sensitivity calibration matrix.

[0268] In the fourth case, when the first coordinate component is less than 0 and the second coordinate component is greater than 0, the electronic device calculates the sensitivity calibration matrix by the following formula:

[0269]

[0270] Among them, Q n is the sensitivity calibration matrix, Q0 is the first sensitivity calibration matrix, Q1 is the second sensitivity calibration matrix, Q3 is the fourth sensitivity calibration matrix, and Q4 is the fifth sensitivity calibration matrix; S nx is the first coordinate component, S ny is the second coordinate component, S xmax2 is the fourth distance, S ymax1 is the second distance.

[0271] Therefore, when the first coordinate component is less than 0 and the second coordinate component is greater than 0, the electronic device can determine the sensitivity calibration matrix corresponding to the first image based on the first coordinate component, the second coordinate component, the first sensitivity calibration matrix, the second sensitivity calibration matrix, the fourth sensitivity calibration matrix and the fifth sensitivity calibration matrix.

[0272] Since the first sensitivity calibration matrix, the second sensitivity calibration matrix, the fourth sensitivity calibration matrix and the fifth sensitivity calibration matrix all include sensitivity calibration sub-matrices corresponding to the H color sub-channels, and the sensitivity calibration matrix also includes the first sensitivity calibration sub-matrix corresponding to the H color sub-channels.

[0273] Therefore, in the above formula, for each color sub-channel, the first sensitivity calibration matrix, the second sensitivity calibration matrix, the fourth sensitivity calibration matrix and the fifth sensitivity calibration matrix are used respectively, including each calibration parameter in the sensitivity calibration sub-matrix corresponding to the color sub-channel, to calculate the calibration parameters at the corresponding positions in the first sensitivity calibration sub-matrix corresponding to the color sub-channel included in the sensitivity calibration matrix, so as to obtain the sensitivity calibration matrix.

[0274] In summary, according to the above steps S804 to S806, the electronic device determines the sensitivity calibration matrix corresponding to the first image according to the first timestamp, the second timestamp, each first coordinate and multiple sensitivity calibration matrices.

[0275] S807: The electronic device calibrates the first image using a sensitivity calibration matrix to obtain a second image.

[0276] After calculating the sensitivity calibration matrix corresponding to the first image, the electronic device can use the sensitivity calibration matrix corresponding to the first image to calibrate the first image to obtain a second image, so as to improve the poor image quality caused by the uneven light received by N pixels of the same color when the optical image stabilization module moves, so that the sensitivities of the N pixels of the same color in the second image obtained after calibration are almost consistent, thereby improving the image quality of the second image obtained after calibration.

[0277] In some embodiments, the electronic device splits the first image according to H color sub-channels to obtain H first single-channel images; the electronic device adjusts the sizes of the first sensitivity calibration sub-matrices corresponding to the H color sub-channels to obtain second sensitivity calibration sub-matrices corresponding to the H color sub-channels, and the size of each second sensitivity calibration sub-matrix is ​​equal to the size of the first single-channel image; the electronic device uses the second sensitivity calibration sub-matrices corresponding to the H color sub-channels to calibrate the corresponding first single-channel images to obtain H second single-channel images; the electronic device merges the H second single-channel images to obtain a second image.

[0278] In one possible manner, for the same color sub-channel, the electronic device uses each calibration parameter in the second sensitivity calibration sub-matrix and multiplies it by the pixel value at the corresponding position in the first single-channel image to obtain the second single-channel image.

[0279] The electronic device may split the first image according to H color sub-channels to obtain H first single-channel images, each of which corresponds to one color sub-channel.

[0280] Since the size of the first single-channel image may be inconsistent with the size of the first sensitivity calibration submatrix, the electronic device can use bilinear interpolation to adjust the size of the first sensitivity calibration submatrix corresponding to the H color subchannels included in the sensitivity calibration matrix to obtain the second sensitivity calibration submatrix corresponding to the H color subchannels.

[0281] For example, the size of each first single-channel image is 1200 pixels * 900 pixels, and the size of the first sensitivity calibration submatrix is ​​16 pixels * 12 pixels. Therefore, it is necessary to adjust the size of the first sensitivity calibration submatrix from 16 pixels * 12 pixels to 1200 pixels * 900 pixels to obtain the second sensitivity calibration submatrix, that is, the size of the second sensitivity calibration submatrix is ​​1200 pixels * 900 pixels.

[0282] For the same color sub-channel, the electronic device uses each calibration parameter in the second sensitivity calibration sub-matrix, multiplying the pixel value at the corresponding position in the first single-channel image, to obtain the second single-channel image. For example, for the same color sub-channel, the electronic device uses the calibration parameter in the kth row and jth column in the second sensitivity calibration sub-matrix, multiplying the pixel value in the kth row and jth column in the first single-channel image, to obtain the pixel value in the kth row and jth column in the second single-channel image.

[0283] According to the above method, the corresponding second single-channel images are calculated for the H color sub-channels respectively to obtain H second single-channel images, each of which corresponds to a color sub-channel. Finally, the electronic device merges the H second single-channel images to obtain a second image. The second image can also be an image in RAW format.

[0284] In summary, the embodiment of the present application can calculate the sensitivity calibration matrix according to the first timestamp, the second timestamp, the first coordinate of the position of the optical image stabilization module after each movement, and multiple sensitivity calibration matrices when the camera starts the optical image stabilization module to perform image stabilization, so as to calibrate the first image captured by the camera. In this way, the problem of failure of calibration of the first image using the first sensitivity calibration matrix under a single optical displacement can be overcome, and the poor image quality caused by uneven light received by N pixels of the same color when the optical image stabilization module moves can be improved, so that the sensitivity of the N pixels of the same color in the second image obtained after calibration is almost consistent, thereby improving the image quality of the second image obtained after calibration.

[0285] For ease of understanding, the following Fig.18 The interaction process between the various modules involved in the image processing method provided in the embodiment of the present application is explained.

[0286] like Fig.18 As shown, the electronic device may include a camera application, a camera service, a camera hardware abstraction module, a camera driver and a camera, the camera includes an optical image stabilization module, a lens 10 and an N-pixel-in-one image sensor, the N-pixel-in-one image sensor includes a microlens array 21 and a pixel array, the microlens array 21 includes a plurality of microlenses 210, each microlens 210 covers N adjacent pixels of the same color in the pixel array, and N is an integer greater than 1. Fig.18 As shown, the image processing method may specifically include the following steps:

[0287] S1801: The camera application receives a touch operation on a camera application icon by a user.

[0288] S1802: In response to a touch operation on a camera application icon, the camera application sends an image preview request to a camera service.

[0289] S1803, the camera service sends an image preview request to the camera hardware abstraction module.

[0290] S1804, the camera hardware abstraction module sends an image preview request to the camera driver.

[0291] S1805: Based on the image preview request, the camera driver drives the camera to capture a first image when the optical image stabilization module is started, and obtains a first timestamp when the capture of the first image starts, a second timestamp when the capture of the first image ends, and a first coordinate of the position of the optical image stabilization module after each movement during the capture of the first image.

[0292] For example, a camera application icon may be displayed on the desktop of the electronic device. When a user wants to use the camera application, the user may perform a touch operation on the camera application icon, such as a click operation, and the camera application may receive the user's touch operation on the camera application icon.

[0293] The camera application responds to the touch operation on the camera application icon, starts the camera application, and starts running the camera application on the electronic device. After the camera application is started, the camera application can send an image preview request to the camera service by calling the camera access interface in the application framework layer. The image preview request is used to request to obtain the first image captured by the camera when the optical image stabilization module is started. In addition, the image preview request may also include but is not limited to: a first timestamp when the capture of the first image starts, a second timestamp when the capture of the first image ends, and the first coordinates of the position of the optical image stabilization module after each movement during the capture of the first image, etc.

[0294] After receiving the image preview request, the camera service sends the image preview request to the camera driver through the camera hardware abstraction module, so that the camera driver drives the camera to capture the first image when the optical image stabilization module is started based on the image preview request, and obtains a first timestamp when the capture of the first image starts, a second timestamp when the capture of the first image ends, and a first coordinate of the position of the optical image stabilization module after each movement during the capture of the first image.

[0295] S1806, the camera sends the first image, the first timestamp, the second timestamp, and each first coordinate to the camera driver.

[0296] In some embodiments, a Hall sensor may be provided in the camera, and the Hall sensor may be electrically connected to a driver chip in the camera. The Hall sensor is used to collect Hall data, and the Hall data may be used to indicate the first coordinate of the position of the optical image stabilization module after each movement, that is, the Hall data may be used to indicate the current position information of the optical image stabilization module. The Hall sensor may send the Hall data to the driver chip in the camera. Therefore, the camera may obtain the first coordinate of the position of the optical image stabilization module after each movement.

[0297] After the camera acquires the first image, the first timestamp, the second timestamp, and each first coordinate, the camera may send the first image, the first timestamp, the second timestamp, and each first coordinate to a camera driver.

[0298] S1807: The camera driver sends the first image, the first timestamp, the second timestamp, and each first coordinate to the camera hardware abstraction module.

[0299] S1808, the camera hardware abstraction module determines a sensitivity calibration matrix corresponding to the first image according to the first timestamp, the second timestamp, each first coordinate, and a plurality of sensitivity calibration matrices.

[0300] Therefore, after receiving the first timestamp, the second timestamp, and each first coordinate, the camera hardware abstraction module can follow the above steps S804 to S806 to enable the camera hardware abstraction module to determine the sensitivity calibration matrix corresponding to the first image based on the first timestamp, the second timestamp, each first coordinate and multiple sensitivity calibration matrices.

[0301] S1809, the camera hardware abstraction module uses a sensitivity calibration matrix to calibrate the first image to obtain a second image.

[0302] S1810. The camera hardware abstraction module sends a second image to the camera service.

[0303] S1811: The camera service sends the second image to the camera application.

[0304] S1812, the camera application displays the second image.

[0305] After the camera hardware abstraction module calibrates the first image using the sensitivity calibration matrix to obtain the second image, the second image may be sent to the camera application through the camera service. After receiving the second image, the camera application may display the second image in the preview interface.

[0306] The image processing method provided in the embodiment of the present application can be applied not only to the preview scenario of the above-mentioned camera application, but also to the shooting scenario of the camera application.

[0307] Exemplarily, the camera application receives a user's touch operation on the shooting control; in response to the touch operation on the shooting control, the camera application sends an image shooting request to the camera service, the image shooting request can be used to request to obtain the first image captured by the camera when the optical image stabilization module is started. In addition, the image preview request may also include but is not limited to: the first timestamp when the first image is started to be captured, the second timestamp when the first image is finished to be captured, and the first coordinates of the position of the optical image stabilization module after each movement during the capture of the first image. The camera service sends the image shooting request to the camera hardware abstract module; the camera hardware abstract module sends the image shooting request to the camera driver; based on the image shooting request, the camera driver drives the camera to capture the first image when the optical image stabilization module is started, and obtains the first timestamp when the first image is started to be captured, the second timestamp when the first image is finished to be captured, and the first coordinates of the position of the optical image stabilization module after each movement during the capture of the first image. The camera sends the first image, the first timestamp, the second timestamp, and each first coordinate to the camera driver; the camera driver sends the first image, the first timestamp, the second timestamp, and each first coordinate to the camera hardware abstract module. The camera hardware abstraction module determines a sensitivity calibration matrix corresponding to the first image according to the first timestamp, the second timestamp, each first coordinate, and the plurality of sensitivity calibration matrices; the camera hardware abstraction module calibrates the first image using the sensitivity calibration matrix to obtain a second image. The camera hardware abstraction module sends the second image to the camera service; the camera service sends the second image to the camera application; and the camera application stores the second image.

[0308] The image processing method provided in the embodiments of the present application can be applied to various scenarios such as portrait capture, pet capture, motion capture, sports capture, security detection, and medical health detection.

[0309] Combined with the above Figures 8 to 18 , the image processing method provided in the embodiment of the present application is described, and the device for executing the above method provided in the embodiment of the present application is described below. Fig.19 As shown, Fig.19 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device may be an electronic device in an embodiment of the present application, or a chip or chip system in an electronic device.

[0310] like Fig.19 As shown, the image processing apparatus 1900 may include a processing unit 1901. The processing unit 1901 is used to support the image processing apparatus 1900 to perform the above processing steps.

[0311] Specifically, the processing unit 1901 is used to obtain a first image captured by the camera when the optical image stabilization module is started; the processing unit 1901 is used to obtain a first timestamp when the capture of the first image starts, and a second timestamp when the capture of the first image ends; the processing unit 1901 is used to obtain the first coordinates of the position of the optical image stabilization module after each movement during the capture of the first image; the processing unit 1901 is used to determine the sensitivity calibration matrix corresponding to the first image according to the first timestamp, the second timestamp, each first coordinate and multiple sensitivity calibration matrices, and the relative positions of the optical center of the lens and the optical center of the N-pixel image sensor are different during calibration; the processing unit 1901 is used to calibrate the first image using the sensitivity calibration matrix to obtain the second image.

[0312] In a possible implementation, the image processing device 1900 further includes a storage unit 1902. The storage unit 1902 is connected to the processing unit 1901 via a line. The storage unit 1902 may include one or more memories, and the memory may be a device used to store programs or data in one or more devices or circuits. The storage unit 1902 may exist independently and be connected to the processing unit 1901 via a communication bus. The storage unit 1902 may also be integrated with the processing unit 1901.

[0313] The storage unit 1902 may store computer-executable instructions of the method in the electronic device so that the processing unit 1901 executes the method in the above embodiment. The storage unit 1902 may be a register, a cache, or a random access memory (RAM), etc. The storage unit 1902 may be integrated with the processing unit 1901. The storage unit 1902 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, and the storage unit 1902 may be independent of the processing unit 1901.

[0314] Fig. 20 This is a schematic diagram of the structure of a chip provided in an embodiment of the present application. Fig. 20 As shown, the chip 2000 includes one or more (including two) processors 2001 , a communication line 2002 and a communication interface 2003 . Optionally, the chip 2000 also includes a memory 2004 .

[0315] In some implementations, the memory 2004 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0316] The method described in the above embodiment of the present application can be applied to the processor 2001, or implemented by the processor 2001. The processor 2001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware or software instructions in the processor 2001. The above processor 2001 can be a general processor (for example, a microprocessor or a conventional processor), a digital signal processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 2001 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application.

[0317] The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. Among them, the software module can be located in a mature storage medium in the field such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read only memory (EEPROM). The storage medium is located in the memory 2004, and the processor 2001 reads the information in the memory 2004 and completes the steps of the above method in combination with its hardware.

[0318] The processor 2001 , the memory 2004 , and the communication interface 2003 may communicate with each other via the communication line 2002 .

[0319] In the above embodiments, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory, or may be downloaded and installed in the memory in the form of software.

[0320] The present application embodiment also provides a computer program product, which includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server, a data center, etc. that includes one or more available media integrated. For example, the available medium can include a magnetic medium (e.g., a floppy disk, a hard disk or a tape), an optical medium (e.g., a digital versatile disc (digital versatiledisc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0321] An embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above-mentioned image processing method.

[0322] The embodiment of the present application provides a chip. The chip includes a processor, and the processor is used to call a computer program in a memory to execute the technical solution in the above embodiment. Its implementation principle and technical effect are similar to those of the above related embodiments, and will not be repeated here.

[0323] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program or instruction. The above method is implemented when the computer program or instruction is executed by the processor. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0324] As a possible design, the computer-readable medium may include a compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disc storage; the computer-readable medium may include a magnetic disk storage or other magnetic disk storage device. Moreover, any connecting line may also be appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of the medium. Disks and optical discs as used herein include compact discs (CDs), laser discs, optical discs, DVDs, floppy disks and Blu-ray discs, where disks generally reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer-readable media.

[0325] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0326] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above are only specific implementation methods of the present application and are not used to limit the protection scope of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The invention is applied to an electronic device, wherein the electronic device comprises a camera, the camera comprises an optical image stabilization module, a lens and an image sensor, the image sensor comprises a microlens array and a pixel array, the pixel array comprises a plurality of pixel sets, each of the pixel sets comprises pixel units corresponding to M color channels, each of the pixel units comprises pixels corresponding to N color sub-channels, and the pixels corresponding to the N color sub-channels are adjacent N pixels of the same color; M is an integer greater than 1; the microlens array comprises a plurality of microlenses, each of the microlenses covers N adjacent pixels of the same color in the pixel array, and N is an integer greater than 1; the method comprises: The electronic device obtains a first image captured by the camera when the optical image stabilization module is activated; The electronic device acquires a first timestamp when the acquisition of the first image starts, and a second timestamp when the acquisition of the first image ends; During the process of acquiring the first image by the electronic device, the first coordinates of the position of the optical image stabilization module after each movement; The electronic device determines, according to the first timestamp and the second timestamp, the number of times the optical image stabilization module moves during the acquisition process of the first image; The electronic device determines, according to the number of movements and each of the first coordinates, a second coordinate of a center of mass position corresponding to each position where the optical image stabilization module is moved; The electronic device determines a sensitivity calibration matrix corresponding to the first image according to the second coordinates and a plurality of sensitivity calibration matrices; when calibrating the plurality of sensitivity calibration matrices, relative positions of the optical center of the lens and the optical center of the image sensor are different; The electronic device calibrates the first image using the sensitivity calibration matrix to obtain a second image.

2. The method according to claim 1, characterized in that The electronic device determines, according to the first timestamp and the second timestamp, the number of times the optical image stabilization module moves during the acquisition process of the first image, including: The electronic device calculates the time interval between the second timestamp and the first timestamp; The electronic device determines the ratio between the time interval and the movement frequency of the optical image stabilization module as the number of times the optical image stabilization module moves during the acquisition process of the first image.

3. The method according to claim 1, characterized in that The electronic device determines, according to the movement times and each of the first coordinates, a second coordinate of a center of mass position corresponding to each position where the optical image stabilization module is moved, including: The electronic device calculates the second coordinate using the following formula: Among them, S n is the second coordinate, n is the number of moves, P i is the first coordinate of the position of the optical image stabilization module after the i-th movement.

4. The method according to claim 1, characterized in that: The electronic device determines, according to the second coordinates and the plurality of sensitivity calibration matrices, a sensitivity calibration matrix corresponding to the first image, including: The electronic device decomposes the second coordinate according to the movable direction of the optical image stabilization module to obtain a first coordinate component in the first moving direction and a second coordinate component in the second moving direction; The electronic device determines a sensitivity calibration matrix corresponding to the first image according to the first coordinate component, the second coordinate component and the plurality of sensitivity calibration matrices; The first moving direction and the second moving direction are perpendicular to each other, and both the first moving direction and the second moving direction are perpendicular to the optical axis direction of the camera.

5. The method according to claim 4, characterized in that The multiple sensitivity calibration matrices include a first sensitivity calibration matrix, a second sensitivity calibration matrix, a third sensitivity calibration matrix, a fourth sensitivity calibration matrix, and a fifth sensitivity calibration matrix; The first moving direction includes a first direction and a second direction which are opposite to each other, and the second moving direction includes a third direction and a fourth direction which are opposite to each other; Wherein, when the first sensitivity calibration matrix is ​​calibrated, the optical center of the lens coincides with the optical center of the image sensor along the optical axis direction of the camera; When calibrating the second sensitivity calibration matrix, the optical center of the lens is offset by a first distance along the first direction and a second distance along the third direction relative to the optical center of the image sensor; the first distance is a maximum distance that the lens can move along the first direction, and the second distance is a maximum distance that the lens can move along the third direction; When calibrating the third sensitivity calibration matrix, the optical center of the lens is offset by the first distance along the first direction and by a third distance along the fourth direction relative to the optical center of the image sensor; the third distance is the maximum distance that the lens can move along the fourth direction; When calibrating the fourth sensitivity calibration matrix, the optical center of the lens is offset by a fourth distance along the second direction and by the third distance along the fourth direction relative to the optical center of the image sensor; the fourth distance is the maximum distance that the lens can move along the second direction; When calibrating the fifth sensitivity calibration matrix, the optical center of the lens is offset by the fourth distance along the second direction and by the second distance along the third direction relative to the optical center of the image sensor.

6. The method according to claim 5, characterized in that The electronic device determines, according to the first coordinate component, the second coordinate component and the plurality of sensitivity calibration matrices, a sensitivity calibration matrix corresponding to the first image, including: When the first coordinate component is greater than or equal to 0, and the second coordinate component is greater than or equal to 0, the electronic device calculates the sensitivity calibration matrix by the following formula: Among them, Q n is the sensitivity calibration matrix, Q0 is the first sensitivity calibration matrix, Q1 is the second sensitivity calibration matrix, Q2 is the third sensitivity calibration matrix, and Q4 is the fifth sensitivity calibration matrix; S nx is the first coordinate component, S ny is the second coordinate component, S xmax1 is the first distance, S xmax1 is the second distance.

7. The method according to claim 5, characterized in that The electronic device determines, according to the first coordinate component, the second coordinate component and the plurality of sensitivity calibration matrices, a sensitivity calibration matrix corresponding to the first image, including: When the first coordinate component is greater than 0 and the second coordinate component is less than 0, the electronic device calculates the sensitivity calibration matrix by the following formula: Among them, Q n is the sensitivity calibration matrix, Q0 is the first sensitivity calibration matrix, Q1 is the second sensitivity calibration matrix, Q2 is the third sensitivity calibration matrix, and Q3 is the fourth sensitivity calibration matrix; S nx is the first coordinate component, S ny is the second coordinate component, S xmax1 is the first distance, S ymax2 is the third distance.

8. The method according to claim 5, characterized in that The electronic device determines, according to the first coordinate component, the second coordinate component and the plurality of sensitivity calibration matrices, a sensitivity calibration matrix corresponding to the first image, including: When the first coordinate component is less than or equal to 0, and the second coordinate component is less than or equal to 0, the electronic device calculates the sensitivity calibration matrix by the following formula: Among them, Q n is the sensitivity calibration matrix, Q0 is the first sensitivity calibration matrix, Q2 is the third sensitivity calibration matrix, Q3 is the fourth sensitivity calibration matrix, and Q4 is the fifth sensitivity calibration matrix; S nx is the first coordinate component, S ny is the second coordinate component, S xmax2 is the fourth distance, S ymax2 is the third distance.

9. The method according to claim 5, characterized in that The electronic device determines, according to the first coordinate component, the second coordinate component and the plurality of sensitivity calibration matrices, a sensitivity calibration matrix corresponding to the first image, including: When the first coordinate component is less than 0 and the second coordinate component is greater than 0, the electronic device calculates the sensitivity calibration matrix by the following formula: Among them, Q n is the sensitivity calibration matrix, Q0 is the first sensitivity calibration matrix, Q1 is the second sensitivity calibration matrix, Q3 is the fourth sensitivity calibration matrix, and Q4 is the fifth sensitivity calibration matrix; S nx is the first coordinate component, S ny is the second coordinate component, S xmax2 is the fourth distance, S ymax1 is the second distance.

10. The method according to any one of claims 1 to 9, characterized in that The sensitivity calibration matrix includes a first sensitivity calibration sub-matrix corresponding to H color sub-channels; the H is equal to the product of the M and the N.

11. The method according to claim 10, characterized in that The electronic device calibrates the first image using the sensitivity calibration matrix to obtain a second image, including: The electronic device splits the first image according to H color sub-channels to obtain H first single-channel images; The electronic device adjusts the sizes of the first sensitivity calibration submatrices corresponding to the H color subchannels respectively to obtain second sensitivity calibration submatrices corresponding to the H color subchannels; the size of each of the second sensitivity calibration submatrices is equal to the size of the first single-channel image; The electronic device uses the second sensitivity calibration sub-matrices corresponding to the H color sub-channels to calibrate the first single-channel images corresponding to the H color sub-channels, respectively, to obtain H second single-channel images; The electronic device combines the H second single-channel images to obtain the second image.

12. The method according to claim 11, characterized in that The electronic device uses the second sensitivity calibration sub-matrices corresponding to the H color sub-channels to calibrate the first single-channel images corresponding to the H color sub-channels respectively to obtain H second single-channel images, including: For the same color sub-channel, the electronic device uses each calibration parameter in the second sensitivity calibration sub-matrix and multiplies it by the pixel value at the corresponding position in the first single-channel image to obtain the second single-channel image.

13. The method according to claim 10, characterized in that Before the electronic device determines the sensitivity calibration matrix corresponding to the first image according to the first timestamp, the second timestamp, each of the first coordinates, and a plurality of sensitivity calibration matrices, the electronic device further includes: The electronic device acquires the plurality of pre-calibrated sensitivity calibration matrices; each of the sensitivity calibration matrices includes sensitivity calibration sub-matrices corresponding to H color sub-channels; Wherein, each calibration parameter in the sensitivity calibration submatrix is ​​calculated based on the mean value of each pixel in the single-channel mean image and the pixel value at the corresponding position in each single-channel test image corresponding to the single-channel mean image; each pixel mean in the single-channel mean image is the average value of the pixel values ​​at the same position in the N single-channel test images belonging to the same color channel in the H single-channel test images; The H single-channel test images are obtained by reducing the size of the test image and splitting it according to H color sub-channels; the test image is an image captured when the focus position of the lens is a preset focus position and the relative position between the optical center of the lens and the optical center of the image sensor is a preset position.

14. The method according to claim 13, characterized in that Each calibration parameter in the sensitivity calibration submatrix is ​​the ratio of each pixel mean in the single-channel mean image to the pixel value at the corresponding position in each single-channel test image corresponding to the single-channel mean image.

15. An electronic device, characterized in that: The device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call the computer program to execute the image processing method according to any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or instruction, and when the computer program or instruction is executed, the image processing method according to any one of claims 1 to 14 is implemented.

17. A computer program product, characterized in that The invention comprises a computer program, which, when being executed, enables a computer to execute the image processing method according to any one of claims 1 to 14.

Images