Image processing method, electronic device, and non-volatile computer-readable storage medium
By selecting the merge mode according to the camera usage status in image processing, the problem of difficulty in taking into account power consumption and clarity in the prior art is solved, and flexible adjustments are achieved in different scenarios to meet user needs.
Patent Information
- Application Number
- CN202110750609.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-01
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-07-01
AI Technical Summary
The existing image processing technology is difficult to meet the user's demand for image clarity when reducing power consumption, and the power consumption and clarity requirements in different usage scenarios vary greatly, and lacks flexible processing modes.
Provide an image processing method, select different image processing modes according to the usage status of the camera application (preview status, photo status and video status), including the first-level merge mode and the second-level merge mode, and adjust the image processing by merging pixel values. Power consumption and clarity meet the needs of different scenarios.
Reduce clarity in high-power scenarios to save power consumption. On the contrary, providing higher-definition images in low-power scenarios enables flexible adjustments in different usage states to meet users' needs for clarity and power consumption. .
Smart Images

Figure CN113556519B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing, and in particular to an image processing method, an electronic device and a non-volatile computer-readable storage medium. Background Art
[0002] As the electronic equipment market becomes more popular and the usage increases, image processing technology becomes more and more important for electronic equipment. The current image processing technology consumes a lot of power. If the image is compressed to save power, the clarity is difficult to meet the needs of users. Therefore, an image processing technology is needed to solve the above problems. Summary of the invention
[0003] The present application provides an image processing method, an electronic device and a non-volatile computer-readable storage medium to reduce the power consumption of image processing while meeting the user's clarity requirements.
[0004] The present application provides an image processing method, which is applied to electronic equipment.
[0005] The electronic device includes an image sensor. The image sensor includes a pixel array, and the pixel array includes a minimum repeating unit that is repeatedly arranged. The minimum repeating unit includes at least two color pixel groups and at least two full-color pixel groups. The color pixel group includes at least four color pixels, and the full-color pixel group includes at least four full-color pixels. The color pixel includes a color filter, and the full-color pixel includes a full-color filter.
[0006] The image processing method includes adopting an image processing mode according to the use state of the camera application, the use state of the camera application includes a preview state, a photo state and a video state, and the image processing mode includes:
[0007] A primary merging mode includes reading pixel values of a pixel array, merging pixel values of all color pixels in a color pixel group as the pixel value of the color pixel group, merging pixel values of all panchromatic pixels in a panchromatic pixel group as the pixel value of the panchromatic pixel group, forming a primary pixel array according to the pixel values of the color pixel group and the pixel values of the panchromatic pixel group, and converting the primary pixel array into a Bayer array;
[0008] The secondary merging mode includes reading pixel values of a pixel array, merging pixel values of all color pixels within a minimum repeating unit as the color pixel value of the minimum repeating unit, merging pixel values of all full-color pixels within the minimum repeating unit as the full-color pixel value of the minimum repeating unit, forming a secondary pixel array based on the color pixel value of the minimum repeating unit and the full-color pixel value of the minimum repeating unit, and converting the secondary pixel array into a Bayer array.
[0009] The present application also provides an electronic device, which includes an image sensor and a processor.
[0010] The image sensor includes a pixel array, the pixel array includes a minimum repeating unit that is repeatedly arranged, the minimum repeating unit includes at least two color pixel groups and at least two full-color pixel groups, the color pixel group includes at least four color pixels, the full-color pixel group includes at least four full-color pixels, the color pixels include color filters, and the full-color pixels include full-color filters.
[0011] The processor is used to execute the above image processing method.
[0012] The present application also provides another electronic device, including an image acquisition module and an image processing module.
[0013] The image acquisition module includes a pixel array, the pixel array includes a minimum repeating unit that is repeatedly arranged, the minimum repeating unit includes at least two color pixel groups and at least two full-color pixel groups, the color pixel group includes at least four color pixels, the full-color pixel group includes at least four full-color pixels, the color pixels include color filters, and the full-color pixels include full-color filters.
[0014] The image processing module is used to execute the above image processing method.
[0015] The present application also provides a non-volatile computer-readable storage medium, which, when the computer program is executed by one or more processors, enables the processors to execute the above-mentioned image processing method.
[0016] The above-mentioned image processing method, electronic device and non-volatile computer-readable storage medium can select different image processing modes according to different usage states of the camera application. When using the primary merging mode, power consumption is reduced and the clarity is still high. When using the secondary merging mode, power consumption is greatly reduced and the clarity is reduced. Under different usage states of the camera application, users have different requirements for clarity and power consumption. Therefore, in a high-power camera application state with low clarity requirements, the secondary merging mode can be used to significantly save power consumption, and in a low-power camera application state with high clarity requirements, the primary merging mode can be used to provide a clearer image. This not only meets the user's demand for clarity, but also greatly saves power consumption. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the implementation methods will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0018] Figure 1 is a schematic diagram of the internal structure of an electronic device provided by the present application;
[0019] Figure 2 is a schematic diagram of a pixel array provided by the present application;
[0020] Figure 3 is another pixel array schematic diagram provided by the present application;
[0021] Figure 4 is a schematic diagram of the minimum repeating unit in the pixel array provided in the present application;
[0022] Figure 5 is a flow chart of a full-resolution mode provided by this application;
[0023] Figure 6 It is a flow chart of a first-level merging mode provided by this application;
[0024] Figure 7 It is a flow chart of a secondary merger mode provided by this application;
[0025] Figure 8 It is a schematic diagram of the internal structure of another electronic device provided by the present application;
[0026] Fig. 9 is a block diagram of an image processing method provided by the present application;
[0027] Fig.10 is a block diagram of another image processing method provided by the present application;
[0028] Fig.11 is a block diagram of another image processing method provided by the present application;
[0029] Fig.12 is a block diagram of another image processing method provided by the present application;
[0030] Fig.13 is a block diagram of another image processing method provided by the present application;
[0031] Fig.14 is a block diagram of another image processing method provided by the present application;
[0032] Fig.15 is a block diagram of another image processing method provided by the present application;
[0033] Fig.16 is a block diagram of another image processing method provided by the present application;
[0034] Fig.17 is a block diagram of another image processing method provided by the present application;
[0035] Fig.18It is a block diagram of another image processing method provided by the present application. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0037] At present, the image processing modes on electronic devices are usually relatively simple. If the image processing mode chooses to output high-definition images, the power consumption of the image processing mode will be high. If the image processing mode chooses to output low-definition images to save power consumption, the low-definition images output by the image processing mode cannot meet the needs of users. In actual use, users' requirements for clarity and power consumption are not constant. For example, when previewing, very high-definition images are not required. At the same time, due to the real-time display of the preview screen, a low-power image processing mode is more needed. If you are shooting an image with more details (such as grass), the user needs a high-definition image at this time, and the low-power consumption requirement of image processing is relatively secondary at this time. Therefore, there is a need for an image processing technology that can adjust different image processing modes according to different user scenarios, so as to meet the user's requirements for image clarity and save the power consumption of image processing.
[0038] In view of the above problems, the inventors have proposed the image processing method, electronic device and non-volatile computer-readable storage medium provided in the embodiments of the present application, which can realize the use of different image processing modes for different usage scenarios, thereby satisfying the user's demand for image clarity and saving power consumption of image processing. The specific image processing method will be described in detail in the subsequent embodiments.
[0039] Please refer to Figure 1 , which shows an electronic device 10 provided by the present application, the electronic device 10 includes an image sensor 100 and a processor 200. Further, the electronic device 10 may also include a color temperature sensor 300 and a light sensor 400.
[0040] The electronic device 10 includes any electronic device such as a mobile terminal, a wearable device, a tablet computer, a smart TV, a desktop computer, a smart phone, a smart car, a camera, etc., and this application does not limit this.
[0041] See also Figure 2 and Figure 3, the image sensor 100 includes a pixel array 110. The pixel array 110 includes a minimum repeating unit 1166 that is repeatedly arranged. The minimum repeating unit 1166 includes at least two color pixel groups 1164 and at least two panchromatic pixel groups 1162. The color pixel group 1166 includes at least four color pixels, and the panchromatic pixel group 1162 includes at least four panchromatic pixels. The color pixels (R, G, B) include color filters, and the panchromatic pixels include panchromatic filters (W). The color filters and the panchromatic filters together constitute the filter 112, and the color pixels and the panchromatic pixels together constitute the pixel layer 114.
[0042] The image sensor 100 is a device that converts an optical signal into an electronic signal, including a charge coupled device (CCD) image sensor and a complementary metal-oxide semiconductor (CMOS) image sensor. The image sensor 100 includes a pixel array 110 .
[0043] The pixel array 110 refers to pixels arranged in an array on the image sensor 100. The array can have various shapes, such as a rectangular array, a diamond array, a triangular array, etc. The pixel array 110 can receive light signals and convert the light signals into electrical signals. The pixel array 110 includes color pixels and full-color pixels.
[0044] The Bayer array is a common pixel array arrangement. The odd columns (or even columns) of the Bayer array contain staggered red and green pixels, while the even columns (or odd columns) contain staggered green and blue pixels. Since the human eye is more sensitive to green light, the number of green filters is twice that of red or blue filters, so it is also called RGBG, GRGB or RGGB. Since it is the standard processing format for the platform image processing chip 240 to perform image processing, it is necessary to convert the pixel values of the pixel array 110 arranged in a non-Bayer array into a Bayer array, that is, convert the pixel value of each pixel into a data value in the Bayer array format for processing by the platform image processing chip 240.
[0045] Color pixels refer to pixels that are used to receive colored light (e.g., red light, green light, blue light) and convert optical signals into electrical signals, including colored pixels such as red pixels, green pixels, and blue pixels. Color pixels include color filters, that is, colored filters such as red filters, green filters, and blue filters. Color filters can separate light and only allow corresponding colored light to pass through, so that color pixels only receive light of the color corresponding to the color filter. For example, a red pixel including a red filter will only receive red light signals. Among them, red (R) can correspond to colors in the wavelength range of 600nm to 750nm (including 600nm and 750nm), green (G) can correspond to colors in the wavelength range of 495nm to 570nm (including 495nm and 570nm), and blue (B) can correspond to colors in the wavelength range of 450nm to 495nm (including 450nm and 495nm).
[0046] A full-color pixel refers to a pixel point that is used to receive white light and convert light signals into electrical signals. A full-color pixel can also be called a white pixel. A full-color pixel includes a full-color filter, which can also be called a white filter, a transparent layer or a transparent filter. A full-color filter allows visible light to pass through, or in other words, a full-color filter does not separate the color of light, so that a full-color pixel can receive any visible light. Among them, the range of visible light can be light with a wavelength of 400nm to 800nm. At the same time, compared with color pixels, full-color pixels do not filter light, so full-color pixels usually have a higher amount of light input than color pixels.
[0047] See also Figure 3 , a color pixel group 1164 includes at least four color pixels. At least four refers to four or more positive integers. At least four color pixels include four color pixels arranged adjacent to each other in a matrix form, and of course other arrangements are also included, which are not limited in this application. The color pixel group 1164 is composed of color pixels, and the pixel value of the color pixel group 1164 is equal to the sum of the pixel values of the color pixels in the color pixel group 1164. For example, Figure 3 The red pixel group 1164 in the dotted box consists of four adjacent red pixels, and the pixel value R group of the red pixel group 1164 is equal to the sum of the pixel values R of the four adjacent red pixels, that is, R group = R1 + R2 + R3 + R4.
[0048] A full-color pixel group 1162 includes at least four full-color pixels. At least four refers to four or more positive integers. At least four full-color pixels include four full-color pixels arranged adjacent to each other in a matrix form, and of course other arrangements are also included, which are not limited in this application. Full-color pixel groups 1162 are composed of full-color pixels, and the pixel value of full-color pixel group 1162 is equal to the sum of the pixel values of the full-color pixels in full-color pixel group 1162. For example, Figure 3 The full-color pixel group 1162 in the dotted frame consists of four adjacent full-color pixels, and the pixel value W group of the full-color pixel group 1162 is equal to the sum of the pixel values W of the four adjacent full-color pixels, that is, W group = W1 + W2 + W3 + W4.
[0049] See also Figure 3 and Figure 4 , the minimum repeating unit 1166 includes at least two color pixel groups 1164 and at least two full-color pixel groups 1162. The minimum repeating unit 1166 is the minimum repeatable pixel unit of the pixel array 110. However, it should be noted that the minimum repeatable arrangement and the minimum repeating unit 1166 here regard all color pixels of different colors as completely identical pixels, for example, red pixels and blue pixels are regarded as identical pixels. The number of color pixel groups 1164 and full-color pixel groups 1162 included in the minimum repeating unit 1166 is a positive integer greater than or equal to 2.
[0050] Further, four adjacent color pixels may together constitute a color pixel group 1164, wherein the four adjacent color pixels are arranged in a matrix. Four adjacent full-color pixels may together constitute a full-color pixel group 1162, wherein the four adjacent full-color pixels are arranged in a matrix.
[0051] For further information, please refer to Figure 4 The minimum repeating unit 1166 may also include two color pixel groups 1164 and two full-color pixel groups 1162, wherein the two color pixel groups 1164 and the two full-color pixel groups 1162 are arranged in a matrix, and the two color pixel groups 1164 are located in a first diagonal direction, and the two full-color pixel groups 1162 are located in a second diagonal direction, and the first diagonal direction and the second diagonal direction are different. The present application does not specifically limit the directions of the two diagonals.
[0052] For further information, please refer to Figure 4 In some embodiments, the minimum repeating unit 1166 may also be four rows and four columns, with a total of sixteen pixels, arranged as follows:
[0053]
[0054] Among them, W represents full-color pixel and C represents color pixel.
[0055] It can be understood that the minimum repeating unit 1166 can be four rows and four columns, with a total of sixteen pixels. Two full-color pixel groups 1162, that is, eight full-color pixels (W), are located on the diagonal from the upper left corner to the lower right corner of the matrix. Two color pixel groups 1164, that is, eight color pixels (C), are located on the diagonal from the lower left corner to the upper right corner of the matrix. Among them, the color pixel (C) can be any red pixel (R), green pixel (G) or blue pixel (B). For example:
[0056]
[0057]
[0058] For further information, please refer to Figure 3 and Figure 4 In some embodiments, the present application may further include:
[0059] The color pixel includes a first color pixel, a second color pixel and a third color pixel, wherein the color filters of the first color pixel, the second color pixel and the third color pixel have different colors.
[0060] Four adjacent minimum repeating units 1166 constitute a color repeating unit 1168. The color repeating unit 1168 has eight rows and eight columns, with a total of sixty-four pixels, and is arranged as follows:
[0061]
[0062] Wherein, W represents a full-color pixel, X represents a first color pixel, Y represents a second color pixel, and Z represents a third color pixel.
[0063] Please refer to Figure 3 and Figure 4 It can be understood that the color pixel may include a first color pixel (X), a second color pixel (Y) and a third color pixel (Z), and the colors of the color filters of the first color pixel, the second color pixel and the third color pixel are different. For example, the first color pixel (X) is a green pixel (G), the second color pixel (Y) is a red pixel (R), and the third color pixel (Z) is a blue pixel (B). Four adjacent minimum repeating units 1166 can constitute a color repeating unit 1168, and the color repeating unit 1168 regards pixels of different colors as non-repeating, for example, red pixels and green pixels are not repeated pixels.
[0064] The color repeating unit 1168 may be eight rows and eight columns, with a total of sixty-four pixels. Among them, the four minimum repeating units 1166 may be arranged in a matrix, and there may be two minimum repeating units 1166 on the diagonal line from the upper left corner to the lower right corner of the matrix, and the colors of the color pixel groups 1164 included in the two minimum repeating units 1166 are different. There may be two minimum repeating units 1166 on the diagonal line from the lower left corner to the upper right corner of the matrix, and the colors of the color pixel groups 1164 included in the two minimum repeating units 1166 are the same. At the same time, the colors of the color pixel groups 1164 in the minimum repeating units 1166 on the two diagonals are different. Taking the first color pixel (X) as a green pixel (G), the second color pixel (Y) as a red pixel (R), and the third color pixel (Z) as a blue pixel (B) as an example, the arrangement method may be:
[0065]
[0066] Furthermore, in some embodiments, the minimum repeating unit 1166 of the present application may also be four rows and four columns, with a total of sixteen pixels, arranged as follows:
[0067]
[0068] Among them, W represents full-color pixel and C represents color pixel.
[0069] It can be understood that the minimum repeating unit 1166 can be four rows and four columns, with a total of sixteen pixels. Two color pixel groups 1164, that is, eight color pixels are located on the diagonal from the upper left corner to the lower right corner of the matrix, and two full-color pixel groups 1162, that is, eight full-color pixels are located on the diagonal from the lower left corner to the upper right corner of the matrix. Among them, the color pixel can be any red pixel, green pixel or blue pixel. For example:
[0070]
[0071] Furthermore, in certain embodiments, the present application may also include:
[0072] The color pixels include a first color pixel, a second color pixel and a third color pixel, wherein the color filters of the first color pixel, the second color pixel and the third color pixel have different colors.
[0073] Four adjacent minimum repeating units 1166 constitute a color repeating unit 1168. The color repeating unit 1168 has eight rows and eight columns, with a total of sixty-four pixels, and is arranged as follows:
[0074]
[0075]
[0076] Wherein, W represents a full-color pixel, X represents a first color pixel, Y represents a second color pixel, and Z represents a third color pixel.
[0077] It can be understood that the color pixel may include a first color pixel (X), a second color pixel (Y), and a third color pixel (Z), and the colors of the color filters of the first color pixel, the second color pixel, and the third color pixel are different. For example, the first color pixel (X) is a green pixel (G), the second color pixel (Y) is a red pixel (R), and the third color pixel (Z) is a blue pixel (B). Four adjacent minimum repeating units 1166 can constitute a color repeating unit 1168, and the color repeating unit 1168 regards color pixels of different colors as non-repeating, for example, red pixels and green pixels are not repeated pixels.
[0078] The color repeating unit 1168 may be eight rows and eight columns, with a total of sixty-four pixels. Among them, the four minimum repeating units 1166 may be arranged in a matrix, and there may be two minimum repeating units 1166 on the diagonal line from the upper left corner to the lower right corner of the matrix, and the colors of the color pixel groups 1164 included in these two minimum repeating units 1166 are the same. There may be two minimum repeating units 1166 on the diagonal line from the lower left corner to the upper right corner of the matrix, and the colors of the color pixel groups 1164 included in these two minimum repeating units 1166 are different. At the same time, the colors of the color pixel groups 1164 in the minimum repeating units 1166 on the two diagonals are different. Taking the first color pixel (X) as a green pixel (G), the second color pixel (Y) as a red pixel (R), and the third color pixel (Z) as a blue pixel (B) as an example, the arrangement method may be:
[0079]
[0080]
[0081] Furthermore, in some embodiments, the electronic device 10 may also include a color temperature sensor 300 and a light sensor 400. The color temperature sensor 300 may be used to obtain color temperature information of the current environment, including ambient color temperature (CCT), infrared light information (IR, including light in the 800nm band), and flicker detection information, and the flicker detection information is correlated with AC power energy. The light sensor 400 may be used to obtain ambient light intensity information of the current environment.
[0082] The processor 200 may include one or more processing units, for example, the processor 200 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processor (NPU). Different processing units may be independent devices or integrated into one or more processors. In addition, a memory may be provided in the processor 200 for storing instructions and data.
[0083] The processor 200 can be used to adopt an image processing mode according to the usage status of the camera application in the electronic device 10. The image processing mode includes a primary merging mode and a secondary merging mode. Further, the image processing mode also includes a full resolution mode.
[0084] Camera applications include camera applications (Application, APP) or camera-related software programs. Taking the camera APP as an example, the usage status of the camera application includes the usage status when the camera APP is used alone (such as using the camera APP to preview photos or videos, take photos, and record videos) and the usage status when the camera APP is called (such as instant messaging software calling the camera APP for video calls). Of course, the usage status of camera-related software programs also includes the above-mentioned usage status when used alone and when called.
[0085] It should be noted that the camera application in this application is a software program and cannot be regarded as a limitation on the camera hardware. In this application, the camera hardware can be located inside the electronic device 10, such as a front camera, a rear camera, a side camera, or an under-screen camera, or it can be located outside the electronic device 10, such as camera hardware connected by a data cable, wireless Bluetooth, a wireless network (WIFI), a wired network, etc. Of course, camera hardware with different parameters and functions (such as fixed-focus cameras and zoom cameras) used in this application also fall within the scope of protection of this application.
[0086] The usage status of the camera application may include a preview status, a photo taking status, and a video status. According to the usage status of the camera application, an image processing mode may be adopted. It is understandable that in some cases, two or more image processing modes may need to be performed simultaneously, for example, the image processing mode of the image taken by the front camera may be different from the image processing mode of the image taken by the rear camera. Therefore, the present application may adopt an image processing mode.
[0087] The image processing mode includes a primary merging mode and a secondary merging mode. Furthermore, the image processing mode also includes a full resolution mode and a dual output mode.
[0088] See also Figure 5 In the full-resolution mode (Fullsize), the processor 200 reads all pixel values or part of the pixel values in the pixel array 110, and then interpolates the pixel values of the pixel array 110 into a Bayer array according to the read pixel values. Taking the first color pixel (X) as a green pixel (G), the second color pixel (Y) as a red pixel (R), and the third color pixel (Z) as a blue pixel (B) as an example, the conversion process of the pixel array 110 into the Bayer array can be seen in the following schematic diagram and Figure 5 .
[0089] The pixel values of the pixel array 110 are arranged as follows:
[0090]
[0091] The converted Bayer array is arranged as follows:
[0092]
[0093] The present application does not limit the conversion method. For example, the processor 200 first divides the judgment window with the color pixel as the central pixel point, the judgment window size can be 9×9 pixels, and the color pixel is located at the center of the judgment window, and then obtains the texture direction of all pixels in the judgment window, and judges the texture direction of each pixel point (for example, horizontal direction, vertical direction, diagonal direction, anti-diagonal direction or flat area). Taking the central pixel point as the horizontal direction as an example, the interpolation window can be divided with the color pixel as the central pixel point, the interpolation window size can be 5×5 pixels, and the color pixel is located at the center of the interpolation window, and the pixel values of the panchromatic pixels in the interpolation window that are all horizontal directions are obtained, and the panchromatic pixel value of the color pixel point is interpolated according to the weighted average of the distance relationship, for example, W=aW1+bW2+cW3. Select each color pixel in each pixel array 110 as the central pixel point in turn, repeat the above interpolation operation, and finally interpolate the corresponding panchromatic pixel array. It can be understood that for a pixel point with a corresponding color in the Bayer array (such as the pixel point in the lower left corner of the list above, where both the pixel array and the Bayer array are G pixels), the pixel value of the pixel point can be directly used as the pixel value of the corresponding pixel point in the Bayer array without interpolation.
[0094] Then, according to the correspondence between the pixel array 110 and the full-color pixel array 110, the pixel value of the color pixel at a specific position is interpolated through bilateral filtering. Figure 5 In the array, the full-color pixel in the upper left corner is interpolated as a red pixel. First, the pixel value W of the full-color pixel is obtained, and the pixel value R is calculated by weighted averaging based on the red pixels near the full-color pixel. For example, R=aR1+bR2+cR3. In the same way, at the pixel point corresponding to the full-color pixel in the full-color pixel array, the pixel value W` is calculated by weighted averaging, such as W`=aW1+bW2+cW3. Finally, according to the corresponding relationship between W and W`, such as W / W`=R / R`, R` is finally obtained. R` is used as the pixel value of the point. Then, all pixel points in the pixel array 110 are converted into a Bayer array through the above interpolation method.
[0095] It is understandable that the main power consumption of image processing comes from the power consumption of analog-to-digital converter (ADC) and the image processing process, which is directly related to the image resolution. It can be considered that the higher the image resolution, the higher the power consumption of ADC and the image processing process. Therefore, in full-resolution mode, the image clarity is the highest and the power consumption is also the highest, which is usually used in scenes with more details or textures (such as grass). At the same time, in terms of signal-to-noise ratio, the signal-to-noise ratio is proportional to the amount of light entering, that is, the higher the amount of light entering a single pixel, the higher the signal-to-noise ratio of the image. Since the full-resolution mode does not perform any pixel merging operations, the amount of light entering a single pixel is low, which ultimately leads to a low signal-to-noise ratio in the full-resolution mode. Therefore, the full-resolution mode is not suitable for use in dark light or night scenes.
[0096] Please refer to Figure 6 In the first-level merging mode, the processor 200 reads the pixel values of the pixel array 110, bins the pixel values of all color pixels in the color pixel group 1164 as the pixel value of the color pixel group 1164, bins the pixel values of all panchromatic pixels in the panchromatic pixel group 1162 as the pixel value of the panchromatic pixel group 1162, forms a first-level pixel array 120 according to the pixel values of the color pixel group 1164 and the pixel values of the panchromatic pixel group 1162, and converts the first-level pixel array 120 into a Bayer array.
[0097] Binning includes physical methods and computational methods. Physical methods include outputting the pixel values of different pixels through an ADC by controlling the switch of the circuit. Computational methods include outputting the pixel values of different pixels separately and then merging the pixel values of different pixels through interpolation calculation.
[0098] Taking the case where the first color pixel (X) is a green pixel (G), the second color pixel (Y) is a red pixel (R), and the third color pixel (Z) is a blue pixel (B) as an example, the conversion process of the pixel array 110 into the primary pixel array 120 and then into the Bayer array can be seen in the following schematic diagram and Figure 6 .
[0099]
[0100] After being processed in the primary merging mode, the primary pixel array 120 is arranged as follows:
[0101]
[0102] Finally, it is converted into a Bayer array, and the arrangement of the Bayer array is:
[0103]
[0104] For example, a red pixel group 1164 includes four red pixels and a full-color pixel group 1162 includes four full-color pixels. The pixel values of the red pixel group 1164, R group = R1 + R2 + R3 + R4, and the pixel values of the full-color pixel group 1162, W group = W1 + W2 + W3 + W4. At this time, the primary pixel array 120 is formed according to the pixel values of the color pixel group 1164 and the pixel values of the full-color pixel group 1162, and then the primary pixel array 120 is converted into a Bayer array.
[0105] The present application does not limit the method of converting the primary pixel array 120 into a Bayer array. For example, the processor 200 first divides the judgment window with the color pixel group 1164 as the central pixel group, the judgment window size can be 9×9 pixel groups, and the color pixel group 1164 is located at the center of the judgment window, obtains the texture direction of all pixel groups in the judgment window, and judges the texture direction of each pixel group (for example, horizontal direction, vertical direction, diagonal direction, anti-diagonal direction or flat area). Taking the central pixel group as the horizontal direction as an example, the interpolation window can be divided with the color pixel group 1164 as the central pixel group, the interpolation window size can be 5×5 pixel groups, and the color pixel group 1164 is located at the center of the interpolation window, and the pixel values of the panchromatic pixel group 1162 in the horizontal direction in the interpolation window are obtained, and the panchromatic pixel value of the color pixel group 1164 is interpolated according to the weighted average of the distance relationship. Each color pixel group 1164 in the primary pixel array 120 is selected in turn, and the above interpolation operation is repeated, and finally the corresponding primary panchromatic pixel group array is interpolated. At the same time, since there are fewer W pixels after the first-level merging, you can also choose to interpolate the first-level green (G) pixel array, and the interpolation method is similar to the interpolation of W pixels. The specific calculation method can be found in the corresponding description of the full resolution mode, which will not be repeated here.
[0106] According to the correspondence between the primary pixel array 120 and the primary full-color pixel group array, the pixel value of the color pixel at a specific position is interpolated by bilateral filtering. Figure 6 Taking the interpolation of the full-color pixel group 1162 in the upper left corner to the red pixel group 164 as an example, first obtain the pixel value W group of the full-color pixel group 1162, and then calculate the pixel value R group by weighted average based on the red pixel group 1164 near the full-color pixel group 1162. In the same way, the W' group is calculated by weighted average on the pixel group corresponding to the full-color pixel group 1162 in the first-level full-color pixel group 1162 array. According to the corresponding relationship between the W group and the W' group, such as W group / W' group=R group / R' group, the R' group is finally obtained. The R' group is used as the pixel value of the pixel group. Then, through the above interpolation method, the pixel values of all pixel groups in the pixel array 110 are converted into the Bayer array.
[0107] It can be understood that the first-level merging mode merges the pixel values of the pixels once, and the merging operation can connect the signals through the analog terminal, so that the analog charge signal is directly collected into a transistor. Therefore, the power consumption of the merging operation is low and can be almost ignored. After the merging operation, the number of pixels that the ADC and the algorithm need to process is reduced, thereby reducing power consumption. At the same time, in terms of signal-to-noise ratio, since multiple pixels are merged into a pixel group, the amount of light entering a single pixel is greatly increased, so its signal-to-noise ratio is higher. For example, compared with the full-resolution mode, if the pixel values of four pixels are merged into a pixel group, only 25% of the ADC readout is required, and the power consumption of image processing is also approximately 25% of the full-resolution mode, the amount of light entering a single pixel becomes 400%, and the image clarity becomes 25% of the full-resolution mode. Therefore, the power consumption of the first-level merging mode will be greatly reduced, the signal-to-noise ratio will be greatly improved, and the clarity will be reduced. It can be applied to 4K2K video recording, normal scene photography, high-definition video live broadcast, etc. In addition, since only 25% of the ADC readout is required, its frame rate can be four times that of the full-resolution mode.
[0108] See also Figure 7 In the secondary merging mode, the processor 200 reads the pixel values of the pixel array 110, merges the pixel values of all color pixels in the minimum repeating unit 1166 as the color pixel value of the minimum repeating unit 1166, merges the pixel values of all panchromatic pixels in the minimum repeating unit 1166 as the panchromatic pixel value of the minimum repeating unit 1166, forms a secondary pixel array 130 according to the color pixel value of the minimum repeating unit 1166 and the panchromatic pixel value of the minimum repeating unit 1166, and converts the secondary pixel array 130 into a Bayer array. Taking the first color pixel (X) as a green pixel (G), the second color pixel (Y) as a red pixel (R), and the third color pixel (Z) as a blue pixel (B) as an example, the pixel array 110 is processed to form a secondary pixel array 130, and then converted into a Bayer array, as shown in the following schematic diagram and Figure 7 :
[0109]
[0110] After the secondary merging mode processing, the secondary pixel array 130 is arranged as follows:
[0111] WWG
[0112] Wlq
[0113] Finally, it is converted into a Bayer array, and the arrangement of the Bayer array is:
[0114] RG
[0115] GB
[0116] Please continue to see Figure 7 For example, a minimum repeating unit 1166 may include two red pixel groups 1164 and two full-color pixel groups 1162. A red pixel group 1164 may include four red pixels, and a full-color pixel group 1162 may include four full-color pixels. The two red pixel groups 1164 and the two full-color pixel groups 1162 are respectively on two diagonals and arranged in a rectangular shape. The pixel value of the red pixel group 1164, R group = R1 + R2 + R3 + R4, and the pixel value of the full-color pixel group 1162, W group = W1 + W2 + W3 + W4. The red pixel value R unit of the minimum repeating unit 1166 = R group a + R group b = R1a + R2a + R3a + R4a + R1b + R2b + R3b + R4b. The full-color pixel value W unit of the minimum repeating unit 1166 is W group a + W group b = W1a + W2a + W3a + W4a + W1b + W2b + W3b + W4b.
[0117] At this time, a secondary pixel array 130 is formed according to the red pixel value R unit of the minimum repeating unit 1166 and the full-color pixel value W unit of the minimum repeating unit 1166, and then the secondary pixel array 130 is converted into a Bayer array. The present application does not limit the method of converting the secondary pixel array 130 into a Bayer array. For example, the processor 200 first divides the judgment window with the red pixel of the minimum repeating unit 1166 as the central pixel group. The judgment window size can be 9×9 pixels, and the red pixel of the minimum repeating unit 1166 is located at the center of the judgment window. The texture direction of all pixels in the judgment window is obtained, and the texture direction of each pixel (for example, horizontal direction, vertical direction, diagonal direction, anti-diagonal direction or flat area) is determined. Taking the central pixel as the horizontal direction as an example, the red pixel of the minimum repeating unit 1166 can be used as the central pixel to divide the interpolation window. The interpolation window size can be 5×5 pixels, and the red pixel of the minimum repeating unit 1166 is located at the center of the interpolation window. The pixel values of the panchromatic pixels in the horizontal direction in the interpolation window are obtained, and the panchromatic pixel value of the red pixel of the minimum repeating unit 1166 is interpolated according to the weighted average of the distance relationship. The color pixels of each minimum repeating unit 1166 in the secondary pixel array 130 are selected in turn, and the above interpolation operation is repeated to finally interpolate the corresponding secondary panchromatic pixel array.
[0118] Then, according to the correspondence between the secondary pixel array 130 and the secondary full-color pixel array, the pixel value of the color pixel at a specific position is interpolated through bilateral filtering. Figure 7Taking the interpolation of the full-color pixel W unit of the minimum repeating unit 1166 in the upper left corner as a red pixel as an example, first obtain the pixel value W unit of the full-color pixel of the minimum repeating unit 1166, and then calculate the pixel value R unit by weighted average based on the red pixels near the full-color pixel value W unit of the minimum repeating unit 1166. In the same way, the W' unit is calculated by weighted average on the pixel group corresponding to the full-color pixel array of the secondary minimum repeating unit 1166 of the full-color pixel of the minimum repeating unit 1166, and finally the R' unit is obtained according to the corresponding relationship between the W unit and the W' unit, such as W unit / W' unit=R unit / R' unit. The R' unit is used as the pixel value of the pixel unit. Then, the pixel values of all pixel units in the secondary pixel array 130 are converted into the Bayer array by the above interpolation method.
[0119] It is understandable that, since the secondary pixel array 130 already includes an RGGB pixel arrangement, the RGGB pixels can be extracted in alternate rows and directly converted into a Bayer array, which can save computing power and power consumption, but the signal-to-noise ratio is low and the clarity is poor. In addition, the RGGB pixels and W pixels can be extracted first, and the RGGB pixels and the W pixels are superimposed and summed to obtain a pixel array of (R+W)(G+W)(G+W)(B+W), and then the pixel array and the RGGB pixels in the secondary pixel array 130 are bilaterally filtered to obtain a Bayer array with a high signal-to-noise ratio. The specific calculation method is similar to the above description and will not be repeated here.
[0120] Understandably, the secondary merging mode merges a greater number of pixel values. After the merging operation, the number of pixels that the ADC and algorithm need to process is further reduced, thereby reducing power consumption. At the same time, in terms of signal-to-noise ratio, since more pixels are merged into a pixel group, the amount of light entering a single pixel is greatly increased, so its signal-to-noise ratio is very high. For example, if the pixel values of eight pixels are merged into a pixel group, compared to the full-resolution mode, only 6.25% of the ADC readout is required, and the power consumption of image processing is also approximately 6.25% of the full-resolution mode, the amount of light entering a single pixel becomes 800%, and the image clarity becomes 6.25% of the full-resolution mode. Therefore, the power consumption of the secondary merging mode is extremely low, the signal-to-noise ratio is extremely high, the clarity is poor, 1080p video recording, night scene photography, preview video, video call, video live broadcast, etc. In addition, since only 6.25% of the ADC readout is required, its frame rate can be eight times that of the full-resolution mode.
[0121] Furthermore, when the dual output mode is adopted, the pixel values of the color pixels of the pixel array are read respectively and the pixel values of the color pixels are converted into the Bayer array; the pixel values of the panchromatic pixels of the pixel array are read and the pixel values of the panchromatic pixels are converted into the black and white array.
[0122] The method for reading the pixel values of color pixels and full-color pixels can use the reading method used in the full-resolution mode, the primary merging mode, and the secondary merging mode. That is, the pixel value of each pixel point can be directly read in the full-resolution mode. It is also possible to merge the pixel values in the color pixel group 1164 and the pixel values in the full-color pixel group 1162 in the primary merging mode before reading. It is also possible to merge the pixel values of the color pixels in the minimum repeating unit 1166 and the pixel values of the full-color pixels in the secondary merging mode before reading. When converting the Bayer array, it is also possible to use the conversion methods of the full-resolution mode, the primary merging mode, and the secondary merging mode, or directly use only the pixel values of the color pixels to interpolate to form the Bayer array. The specific reading and conversion process can be referred to the previous text, which will not be repeated here.
[0123] The black and white array is similar to the Bayer array, that is, the full-color pixels are repeatedly arranged in a matrix form. Converting to a black and white array means converting the pixel value of each pixel into a data value arranged in a black and white array format. When converting the black and white array, the pixel values of the full-color pixels can be subjected to texture direction judgment, bilateral filtering and conversion in a manner similar to converting the Bayer array, or only the pixel values of the full-color pixels can be interpolated to form a black and white array. It can be understood that the above-mentioned reading and conversion method cannot be regarded as a limitation of the present application. As long as the Bayer array and the black and white array are obtained by reading and conversion, they are all within the protection scope of the present application.
[0124] It is understandable that since the black and white array does not filter the light, its signal-to-noise ratio is relatively high. The black and white array and the color array can be matched to enhance the image quality or achieve image special effects. For example, a black and white image and a color image can be displayed simultaneously on the electronic device 10. The black and white image can be obtained by processing the black and white array, and the color image can be obtained by processing the Bayer array. Displaying black and white images and color images at the same time helps users to carefully observe the imaging details and enhance the interest of shooting. For another example, the black and white image and the color image can be partially superimposed and displayed, that is, in the same picture, some parts display black and white pictures, and some parts display color pictures, so as to achieve superimposed display, highlight the focus of the picture, and increase the beauty of the picture and the interest of shooting. The black and white picture can be obtained by processing the black and white array, and the color picture can be obtained by processing the Bayer array. This application does not limit the use of the black and white array and the Bayer array.
[0125] For further information, see Figure 8 In some embodiments, the electronic device 10 may further include a first chip 210, a second chip 220, a platform image processing chip 240 (platform ISP) and a memory 230 (DDR).
[0126] The first chip 210 is a built-in chip (image sensor ISP) of the image sensor 100, which is used to control the imaging of the image sensor 100. The second chip 220 is an external chip (external chip, IP) of the image sensor 100, which is used to process the data after the image sensor 100 is imaged. The platform image processing chip 240 (platform ISP) is used to receive the image data transmitted by the first chip 210, the second chip 220 and the memory 230, and process the image data, including color correction, white balance processing and other operations. The memory 230 (DDR) is a data storage area of the electronic device 10, which is used to store relevant data of the electronic device 10.
[0127] See also Fig. 9 , which shows an image processing method provided by the present application, comprising:
[0128] 010Get the usage status of the camera application;
[0129] This application can obtain the usage status of the camera application by retrieving camera application data, detecting the camera application display screen, etc. This application does not limit the method of obtaining the usage status of the camera application. The status of the camera application can refer to the description above and will not be repeated here.
[0130] 020 Adopt an image processing mode according to the usage status of the camera application.
[0131] It is understandable that the present application can use different image processing modes by the processor 200 according to the different usage states of the camera application. Specifically, it can be executed by the central processing unit (CPU). For example, when the camera application is in the preview state, the secondary merging mode can be used to reduce power consumption while meeting user needs.
[0132] The usage status of the camera application may include preview status, photo status, and video status. The preview status may include photo preview and video preview. The video status may include video recording status, video live broadcast status, and video call status.
[0133] The preview state means that the camera application is previewing the image at this moment, including preview before taking a photo and preview before video.
[0134] The photo state refers to the camera application taking photos at this time, including taking photos alone and being called to take photos, etc. This application does not limit the photo mode, photo parameters, etc.
[0135] The video state refers to whether the camera application is recording video, live streaming video, or making video calls, including using video alone and being called video, etc. This application does not limit the video mode, video parameters, etc.
[0136] As mentioned above, the current image processing method has high power consumption, and when the power consumption is reduced, it may not meet the user's requirements for clarity. Therefore, a technology is needed to solve the contradiction between power consumption and clarity. The present application provides three image processing modes. Compared with the three modes, the full-resolution mode has the highest clarity, the highest power consumption, and the lowest signal-to-noise ratio; the first-level merging mode has medium clarity, medium power consumption, and medium signal-to-noise ratio; the second-level merging mode has the lowest clarity, the lowest power consumption, and the highest signal-to-noise ratio. Therefore, different image processing modes can be selected according to the different needs of users, so as to reduce power consumption while meeting the user's requirements for clarity. For example, the full-resolution mode is selected in scenes with sufficient light and more details to meet the clarity requirements. When taking pictures in normal scenes and recording 4K2K videos, select the first-level merging mode to balance power consumption, clarity, and signal-to-noise ratio. At night or in scenes with low light, select the second-level merging mode to greatly improve the signal-to-noise ratio while reducing power consumption.
[0137] Please refer to Fig.18 Further, in some embodiments, the image processing method provided by the present application, after adopting an image processing mode according to the usage status of the camera application, may also include:
[0138] 080 When a preset switching instruction is received, the currently used image processing mode is switched to another image processing mode.
[0139] The preset switching instruction refers to an instruction that can switch the image processing mode, including various touch instructions such as single click, double click, slide, gesture, voice control, etc., and can also be pre-set by the user, and preset conditions can also be set. This application does not limit the specific form of the preset switching instruction. It can be understood that the preset switching instruction can include preview zoom instructions, touch instructions for improving video resolution, and preset conditions can include the relationship between ambient light intensity and light intensity threshold, the relationship between the texture ratio of the target image and the texture ratio threshold, etc.
[0140] It is understandable that the preset switching instruction may also include an instruction to switch the camera application state. For example, when an instruction to switch the camera application state is received and the camera is switched from the preview state to the photo taking state, the image processing mode is switched from the secondary merging mode to the primary merging mode.
[0141] The image processing mode currently used may be one or more of a full resolution mode, a primary merge mode, a secondary merge mode, and a dual output mode. Another image processing mode includes a full resolution mode, a primary merge mode, a secondary merge mode, and a dual output mode, which is different from the image processing mode currently used. It should be noted that as long as any image processing mode in the current image processing mode is switched to another image processing mode, such as switching the primary merge mode to the secondary merge mode, it falls within the protection scope of the present application.
[0142] See also Fig.10 Furthermore, in some embodiments, the image processing method provided by the present application may also include:
[0143] 030 When the camera application is in preview mode, the secondary merging mode is used.
[0144] It is understandable that when previewing an image, the camera hardware will always be turned on, and the image sensor 100 will convert the optical signal into an electrical signal in real time, and finally display it to the user after image processing. Therefore, when previewing an image, the power consumption is very high. And since it is only a preview image, the user does not need a particularly clear image. Therefore, in the preview state, a secondary merging mode can be used for image processing, which not only meets the user's usage needs but also saves power consumption. Of course, if the user needs, such as when using a microscope lens, the user may want to observe the image carefully in the preview state, and can also be adjusted to full resolution mode or primary merging mode in the preview state. Using the secondary merging mode can be switching other image processing modes to the secondary merging mode.
[0145] See also Fig.11 Furthermore, in some embodiments, the image processing method provided by the present application may also include:
[0146] 031 When receiving a preview zoom instruction, switch the secondary merge mode to the full resolution mode.
[0147] If a preview zoom instruction is received during preview, for example, the preview image needs to be zoomed in five times, in order to avoid the enlarged preview image being too blurry, the full resolution mode can be used, and the image processed in the full resolution mode can be cropped or downsampled to obtain better clarity to meet the user's need to enlarge the preview image.
[0148] See also Fig.12 Furthermore, in some embodiments, the image processing method provided by the present application may also include:
[0149] 040 When the camera application is in the photo-taking state, the first-level merging mode is adopted.
[0150] When the camera application is in the state of taking pictures, the primary merging mode can be used. As mentioned above, the primary merging mode consumes less power than the full-resolution mode, improves the signal-to-noise ratio, and has better clarity than the secondary merging mode, so it is suitable for ordinary photography. The primary merging mode can be used to switch other image processing modes to the primary merging mode.
[0151] See also Fig.13Furthermore, in some embodiments, the image processing method provided by the present application may also include:
[0152] 041 When the camera application is in the photographing state, the color temperature information of the target image is obtained.
[0153] The color temperature information of the target image can be obtained to determine the current shooting environment state. The present application does not limit the specific method of obtaining the color temperature information. For example, the color temperature information can be obtained by the color temperature sensor 300, and the color temperature information can include ambient color temperature (CCT), infrared light information (IR, including light in the 800nm band) and flicker detection information, and the flicker detection information is correlated with the AC power energy.
[0154] 042Determine whether the electronic device is indoors or outdoors based on the color temperature information.
[0155] Generally speaking, when IR>1.0, AC power energy is less than 5%, and CCT>4000K, it can be considered that the current location is outdoors. Otherwise, it can be considered that the current location is indoors. At the same time, the present application determines whether the electronic device 10 is indoors or outdoors, including determining whether the external camera hardware is indoors or outdoors. When the electronic device 10 and the external camera hardware are not located indoors or outdoors at the same time, the location of the external camera hardware is used as the execution result of this judgment.
[0156] 043 When the electronic device is indoors, obtain the ambient light intensity.
[0157] The present application does not limit the specific method of obtaining the ambient light intensity. For example, the present application can obtain ambient light intensity information through the light sensor 400 to determine the ambient light intensity of the environment in which the electronic device is currently located. Of course, the ambient light intensity includes the ambient light intensity of the environment in which the electronic device is located and the ambient light intensity of the environment in which the external camera hardware is located.
[0158] 044 When the ambient light intensity is greater than or equal to the first light intensity threshold, the first-level merging mode is adopted.
[0159] When you are indoors and the ambient light intensity is greater than or equal to the first light intensity threshold, the first-level merging mode may be used by default. As explained above, the clarity, signal-to-noise ratio, and power consumption of the first-level merging mode are moderate, meeting the general requirements of indoor shooting.
[0160] 045 When the ambient light intensity is less than the first light intensity threshold, the secondary merging mode is adopted.
[0161] When the ambient light intensity is less than the first light intensity threshold, it can be considered that the current environment is dark, so a higher signal-to-noise ratio image processing mode is required. Therefore, the secondary merging mode can be used at this time to maximize the signal-to-noise ratio of the image to meet the needs of the dark environment. The present application does not limit the specific value of the first light intensity threshold, for example, it can be 50 lux. It can be understood that the use of the secondary merging mode can be to switch the primary merging mode to the secondary merging mode.
[0162] See also Fig.14 Furthermore, in some embodiments, the image processing method provided by the present application may also include:
[0163] 046 When the electronic device is outdoors, obtain the ambient light intensity.
[0164] The method for obtaining the ambient light intensity is the same as above and will not be repeated here.
[0165] 047 When the ambient light intensity is less than or equal to the second light intensity threshold and greater than or equal to the third light intensity threshold, the first-level merging mode is adopted.
[0166] When you are outdoors and the ambient light intensity is less than or equal to the second light intensity threshold and greater than or equal to the third light intensity threshold, the first-level merging mode can be used by default. The clarity, signal-to-noise ratio, and power consumption of the first-level merging mode are moderate, meeting the general needs of outdoor shooting.
[0167] 048 When the ambient light intensity is greater than the second light intensity threshold, the full resolution mode is adopted.
[0168] When the ambient light intensity is greater than the second light intensity threshold, it can be considered that the current environment is bright light. At this time, the signal-to-noise ratio is relatively high, so the full resolution mode can be used to maximize the clarity of the image to improve the display effect of the image in the bright light environment. It can be understood that the full resolution mode can be used to switch the primary merging mode to the full resolution mode.
[0169] 049 When the ambient light intensity is less than the third light intensity threshold, the secondary merging mode is adopted.
[0170] When the ambient light intensity is less than the third light intensity threshold, it can be considered that it is currently at night. At this time, the signal-to-noise ratio itself is low, so the secondary merging mode can be used to maximize the signal-to-noise ratio of the image to meet the shooting needs in the night environment. The present application does not limit the specific values of the second light intensity threshold and the third light intensity threshold. For example, the second light intensity threshold may be 3000 lux, and the third light intensity threshold may be 50 lux. Of course, if the ambient light intensity is greater than the second light intensity threshold indoors, the full-resolution mode can also be used for image processing. It can be understood that the use of the secondary merging mode can be to switch the primary merging mode to the secondary merging mode.
[0171] Please refer to Fig.15 Furthermore, in some embodiments, the image processing method provided by the present application may also include:
[0172] 050Get the texture ratio of the target image.
[0173] Texture is a visual feature that reflects homogeneous phenomena in an image. It reflects the slowly changing or periodically changing surface structure organization and arrangement properties of the surface of an object. At the same time, texture also has a direction, and the texture direction can be used to indicate the interpolation direction of the pixel. Among them, the texture direction includes horizontal direction, vertical direction, diagonal direction, anti-diagonal direction, and plane direction. The present application does not limit the method of obtaining the texture ratio of the target image. For example, it can be obtained directly by reading the pixel value of the pixel array, or it can be obtained by artificial intelligence (AI) scene recognition.
[0174] 051 When the texture ratio of the target image exceeds the texture ratio threshold, the full resolution mode is adopted.
[0175] When the texture ratio of the target image exceeds the texture ratio threshold, it can be considered that the target image is a high-texture (high frequency, dense stripes) and high-detail image, such as grass, text images, etc., so the full-resolution mode can be used to process the image with maximum clarity and retain image details. This application does not limit the texture ratio threshold, for example, the texture ratio threshold can be 30%. Using the full-resolution mode can be switching other image processing modes to the full-resolution mode.
[0176] Please refer to Fig.16 Furthermore, in some embodiments, the image processing method provided by the present application may also include:
[0177] 060When the camera application is in video state, the secondary merging mode is used.
[0178] It is understandable that when the camera application is in the video state, the camera hardware will be turned on all the time, and the image sensor 100 will convert the optical signal into an electrical signal in real time, and finally display or store it after image processing. Therefore, in the video state, the power consumption is very high, and the secondary merging mode can be used by default to save power consumption. The secondary merging mode can be used to switch other image processing modes to the secondary merging mode.
[0179] Please continue to refer to Fig.16 Furthermore, in some embodiments, the image processing method provided by the present application may also include:
[0180] 061 When a touch command for increasing the video resolution is received, the secondary merging mode is switched to the primary merging mode.
[0181] Since the secondary merge mode may not meet the user's demand for clarity, the secondary merge mode can be switched to the primary merge mode by receiving a touch command to increase the video resolution, thereby improving the clarity of the video. Touch commands include single click, double click, slide, gesture, voice control and other commands, which are not limited in this application.
[0182] Please refer to Fig.17 In some implementations, the above embodiments may include:
[0183] 0601 When the camera application is in video recording state, a secondary merging mode is adopted, wherein when a touch command to increase the video recording resolution is received, the secondary merging mode is switched to the primary merging mode.
[0184] As before, the video status may include video recording status, video live broadcast status and video call status. The video recording status includes directly using the camera application to record a video and calling the camera application to record a video, the video live broadcast status includes calling the camera application for video live broadcast, and the video call status includes directly using the camera application for calling and calling the camera application for video calling.
[0185] When the camera application is in video recording state, 1080P video recording is usually performed by default, so its clarity requirement is not high, and the secondary merge mode can be used. If a touch command to increase the video recording resolution is received, such as a 4K or 2K video shooting command, the primary merge mode can be used to increase the resolution. Of course, if a touch command to further increase the video recording resolution is received, the full resolution mode can also be used. Touch commands include single-click, double-click, slide, gesture, voice control and other types of commands, which are not limited in this application.
[0186] 0602 When the camera application is in the live video state, the secondary merging mode is adopted. When a touch command to increase the resolution of the live video is received, the secondary merging mode is switched to the primary merging mode.
[0187] When the camera application is in the live video state, it usually defaults to 1080P live video, so its clarity requirement is not high, and the secondary merge mode can be used. If a touch command to increase the resolution of the live video is received, such as a 4K or 2K live video command, the primary merge mode can be used. Of course, if a touch command to further increase the resolution of the live video is received, the full resolution mode can also be used. Touch commands include single-click, double-click, slide, gesture, voice control and other commands, which are not limited in this application.
[0188] 0603 When the camera application is in video call state, the secondary merging mode is used.
[0189] When the camera application is in the video call state, it usually defaults to 720P video call, so the clarity requirement is not high, and the secondary merge mode can be used. Of course, if a touch command to increase the video call resolution is received, the primary merge mode or full resolution mode can also be used, and this application will not go into details.
[0190] Please continue to refer to Fig.17 Furthermore, in some embodiments, the image processing method provided by the present application may also include:
[0191] 070 Adjust the video frame rate according to the frame rate control instruction.
[0192] When the camera application is in video mode, including video recording, live streaming, and video calling, it can receive frame rate control instructions, such as 30fps, 60fps, etc., and adjust the video frame rate according to the frame rate control instructions. As mentioned above, the frame rate of the first-level merge mode is four times the frame rate of the full-resolution mode, and the frame rate of the second-level merge mode is eight times the frame rate of the full-resolution mode. Therefore, the frame rate can be dynamically adjusted according to user needs to improve the shooting effect.
[0193] Furthermore, in some embodiments, the image processing method of the present application may also include: executing an image processing mode in the first chip 210, and transmitting the Bayer array and / or the black and white array to the platform image processing chip 240, wherein the first chip 210 is a built-in chip of the image sensor 100.
[0194] The first chip 210 is a built-in chip (image sensor ISP) of the image sensor 100. An image processing mode, including a full resolution mode, a primary merging mode or a secondary merging mode, can be executed in the first chip 210, and the Bayer array and / or the black-and-white array obtained after the image processing mode is transmitted to the platform image processing chip 240 (platform ISP). The platform image processing chip 240 can perform operations such as color correction and white balance processing on the image, and can encode it into an image format and resolution supported by the electronic device 10. Among them, the platform image processing chip 240 can have a camera interface module (CAMIF). After the first chip 210 executes the image processing mode, the Bayer array and / or the black-and-white array can be transmitted to the camera interface module of the platform image processing chip 240, and then the platform image processing chip 240 performs operations such as color correction and white balance processing on the Bayer array and / or the black-and-white array.
[0195] The image processing mode can be executed in the first chip 210 to improve the efficiency of image processing. For example, modes with low power consumption and small amount of calculation, such as the two-level merging mode, are more suitable for being directly processed in the first chip 210.
[0196] The image processing method of the present application may further include: executing an image processing mode in the second chip 220 , and transmitting the Bayer array and / or the black-and-white array to the platform image processing chip 240 , where the second chip 220 is an external chip of the image sensor 100 .
[0197] The second chip 220 is an external chip (IP) of the image sensor 100. The first chip 210 can transmit image data to the second chip 220, and the image processing mode, including the full resolution mode, the primary merging mode or the secondary merging mode, is executed in the second chip 220, and the Bayer array and / or the black-and-white array obtained after the image processing mode is transmitted to the platform image processing chip 240 (platform ISP). The platform image processing chip 240 can perform color correction, white balance processing and other operations on the image, and can encode it into an image format and resolution supported by the electronic device 10. Among them, the platform image processing chip 240 can have a camera interface module (CAMIF). After the second chip 220 completes the image processing mode, the Bayer array and / or the black-and-white array can be transmitted to the camera interface module of the platform image processing chip 240, and then the platform image processing chip 240 performs color correction, white balance processing and other operations on the Bayer array and / or the black-and-white array.
[0198] It is understandable that since the second chip 220 is an external chip, its chip capacity and processing power are usually higher than the first chip. Using the second chip 220 to execute the image processing mode can greatly improve the image processing capability of the electronic device 10 and avoid full-load calculation. For example, a high-power consumption mode such as the full-resolution mode is more suitable for execution in the second chip 220.
[0199] The image processing method of the present application may further include: dumping the data of the pixel array into the memory 230 of the electronic device 10 , executing the image processing mode, and dumping the data of the Bayer array and / or the black and white array to the platform image processing chip 240 .
[0200] When the software algorithm is used to execute the image processing mode, the data of the pixel array can be dumped into the memory 230 (DDR) of the electronic device 10, and then the processor 200 converts the data of the pixel array in the memory 230 into a Bayer array and / or a black-and-white array, and finally dumps the data of the Bayer array and / or the black-and-white array to the platform image processing chip 240. The platform image processing chip 240 can perform color correction, white balance processing and other operations on the image, and can encode it into an image format and resolution supported by the electronic device 10. Among them, the platform image processing chip 240 can also have a black level correction module (Pedestal Correction), which is used to deduct part of the black level. After the processor 200 completes the image processing mode, the data of the Bayer array and / or the black-and-white array can be dumped into the black level correction module before the black level correction, and then the platform image processing chip 240 performs color correction, white balance processing and other operations on the Bayer array and / or the black-and-white array.
[0201] It is understandable that dumping the data of the pixel array into the memory 230 of the electronic device 10 can reduce the load of the first chip 210 or the second chip 220, and the memory 230 usually has a large space and is suitable for storing and processing data.
[0202] In addition, in the image processing method provided by the present application, the above steps can be combined arbitrarily without contradiction. The specific implementation details have been described in detail in the previous text and will not be repeated here.
[0203] The present application also provides an electronic device 10 , which includes an image sensor 100 and a processor 200 .
[0204] The image sensor 100 includes a pixel array 110, and the pixel array 110 includes a minimum repeating unit 1166 that is repeatedly arranged. The minimum repeating unit 1166 includes at least two color pixel groups 1164 and at least two panchromatic pixel groups 1162. The color pixel group 1166 includes at least four color pixels, and the panchromatic pixel group 1162 includes at least four panchromatic pixels. The color pixels (R, G, B) include color filters, and the panchromatic pixels include panchromatic filters (W). The color filters and the panchromatic filters together constitute the filter 112, and the color pixels and the panchromatic pixels together constitute the pixel layer 114.
[0205] The processor 200 is used to adopt an image processing mode according to the usage status of the camera application in the electronic device 10. The image processing mode includes:
[0206] The first-level merging mode includes reading pixel values of the pixel array 110, merging the pixel values of all color pixels in the color pixel group 1164 as the pixel value of the color pixel group 1164, merging the pixel values of all panchromatic pixels in the panchromatic pixel group 1162 as the pixel value of the panchromatic pixel group 1162, forming a first-level pixel array 110 according to the pixel values of the color pixel group 1164 and the pixel values of the panchromatic pixel group 1162, and converting the first-level pixel array 110 into a Bayer array.
[0207] The secondary merging mode includes reading pixel values of the pixel array 110, merging the pixel values of all color pixels within the minimum repeating unit 1166 as the color pixel value of the minimum repeating unit 1166, merging the pixel values of all full-color pixels within the minimum repeating unit 1166 as the full-color pixel value of the minimum repeating unit 1166, forming a secondary pixel array 110 based on the color pixel values of the minimum repeating unit 1166 and the full-color pixel values of the minimum repeating unit 1166, and converting the secondary pixel array 110 into a Bayer array.
[0208] The processor 200 of the electronic device 10 provided in the present application can also execute all the steps in the foregoing, and the image processing mode can also include the full resolution mode and dual output mode mentioned above, which will not be repeated here.
[0209] The present application also provides another electronic device 10, which includes an image acquisition module and an image processing module.
[0210] The image acquisition module includes a pixel array 110, and the pixel array 110 includes color pixels and panchromatic pixels. The color pixels include color filters, and the panchromatic pixels include panchromatic filters. A plurality of adjacent color pixels together constitute a color pixel group 1164, and a plurality of adjacent panchromatic pixels together constitute a panchromatic pixel group 1162. At least one color pixel group 1164 and at least one panchromatic pixel group 1162 together constitute a minimum repeating unit 1166 of the pixel array 110.
[0211] The image processing module is used to adopt an image processing mode according to the usage status of the camera application in the electronic device 10. The image processing mode includes:
[0212] The first-level merging mode includes reading pixel values of the pixel array 110, merging the pixel values of all color pixels in the color pixel group 1164 as the pixel value of the color pixel group 1164, merging the pixel values of all panchromatic pixels in the panchromatic pixel group 1162 as the pixel value of the panchromatic pixel group 1162, forming a first-level pixel array 110 according to the pixel values of the color pixel group 1164 and the pixel values of the panchromatic pixel group 1162, and converting the first-level pixel array 110 into a Bayer array.
[0213] The secondary merging mode includes reading pixel values of the pixel array 110, merging the pixel values of all color pixels within the minimum repeating unit 1166 as the color pixel value of the minimum repeating unit 1166, merging the pixel values of all full-color pixels within the minimum repeating unit 1166 as the full-color pixel value of the minimum repeating unit 1166, forming a secondary pixel array 110 based on the color pixel values of the minimum repeating unit 1166 and the full-color pixel values of the minimum repeating unit 1166, and converting the secondary pixel array 110 into a Bayer array.
[0214] The image processing module of the electronic device 10 provided in the present application can also execute all the steps in the foregoing text, and the image processing mode can also include the full resolution mode and dual output mode mentioned in the foregoing text, which will not be repeated here.
[0215] The present application also provides a non-volatile computer-readable storage medium, which, when the computer program is executed by one or more processors 200, enables the processor 200 to perform the image processing method.
[0216] The image processing method is applied to an electronic device 10, which includes an image sensor 100, and the image sensor 100 includes a pixel array 110, and the pixel array 110 includes color pixels and panchromatic pixels. The color pixels include color filters, and the panchromatic pixels include panchromatic filters. A plurality of adjacent color pixels together constitute a color pixel group 1164, and a plurality of adjacent panchromatic pixels together constitute a panchromatic pixel group 1162. At least one color pixel group 1164 and at least one panchromatic pixel group 1162 together constitute a minimum repeating unit 1166 of the pixel array 110.
[0217] The image processing method includes, according to the use state of the camera application in the electronic device 10, adopting an image processing mode, the image processing mode including:
[0218] The first-level merging mode includes reading pixel values of the pixel array 110, merging the pixel values of all color pixels in the color pixel group 1164 as the pixel value of the color pixel group 1164, merging the pixel values of all panchromatic pixels in the panchromatic pixel group 1162 as the pixel value of the panchromatic pixel group 1162, forming a first-level pixel array 110 according to the pixel values of the color pixel group 1164 and the pixel values of the panchromatic pixel group 1162, and converting the first-level pixel array 110 into a Bayer array.
[0219] The secondary merging mode includes reading pixel values of the pixel array 110, merging the pixel values of all color pixels within the minimum repeating unit 1166 as the color pixel value of the minimum repeating unit 1166, merging the pixel values of all full-color pixels within the minimum repeating unit 1166 as the full-color pixel value of the minimum repeating unit 1166, forming a secondary pixel array 110 based on the color pixel values of the minimum repeating unit 1166 and the full-color pixel values of the minimum repeating unit 1166, and converting the secondary pixel array 110 into a Bayer array.
[0220] The processor 200 provided in the present application can also execute all the steps described in the foregoing text, and the image processing mode can also include the full-resolution mode and dual-output mode mentioned in the foregoing text, which will not be elaborated here.
[0221] The above is an implementation method of the embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present application. These improvements and modifications are also considered to be within the scope of protection of the present application.
Claims
1. An image processing method, applied to an electronic device, characterized in that: The electronic device comprises an image sensor, the image sensor comprises a pixel array, the pixel array comprises a minimum repeating unit of repeated arrangement, the minimum repeating unit comprises at least two color pixel groups and at least two panchromatic pixel groups, the color pixel group comprises at least four color pixels, the panchromatic pixel group comprises at least four panchromatic pixels, the color pixel comprises a color filter, the panchromatic pixel comprises a panchromatic filter, and the image processing method comprises: According to the use state of the camera application, an image processing mode is adopted, the use state of the camera application includes preview state, photo state and video state, and the image processing mode includes: A primary merging mode, comprising: reading pixel values of the pixel array, merging pixel values of all the color pixels in the color pixel group as the pixel value of the color pixel group, merging pixel values of all the panchromatic pixels in the panchromatic pixel group as the pixel value of the panchromatic pixel group, forming a primary pixel array according to the pixel values of the color pixel group and the pixel values of the panchromatic pixel group, and converting the primary pixel array into a Bayer array; A secondary merging mode, comprising: reading pixel values of the pixel array, merging pixel values of all the color pixels in the minimum repeating unit as the color pixel value of the minimum repeating unit, merging pixel values of all the panchromatic pixels in the minimum repeating unit as the panchromatic pixel value of the minimum repeating unit, forming a secondary pixel array according to the color pixel values of the minimum repeating unit and the panchromatic pixel values of the minimum repeating unit, and converting the secondary pixel array into a Bayer array; When the camera application is in the preview state, using the secondary merging mode; When the camera application is in the photographing state, the primary merging mode is adopted, or when the camera application is in the photographing state, color temperature information of the target image is obtained, and it is determined whether the electronic device is indoors or outdoors according to the color temperature information; when the electronic device is indoors, the ambient light intensity is obtained, and when the ambient light intensity is greater than or equal to a first light intensity threshold, the primary merging mode is adopted, and when the ambient light intensity is less than the first light intensity threshold, the secondary merging mode is adopted; When the camera application is in the video state, the secondary merging mode is adopted, and when a touch instruction to increase the video resolution is received, the secondary merging mode is switched to the primary merging mode.
2. An image processing method, applied to an electronic device, characterized in that: The electronic device comprises an image sensor, the image sensor comprises a pixel array, the pixel array comprises a minimum repeating unit of repeated arrangement, the minimum repeating unit comprises at least two color pixel groups and at least two panchromatic pixel groups, the color pixel group comprises at least four color pixels, the panchromatic pixel group comprises at least four panchromatic pixels, the color pixel comprises a color filter, the panchromatic pixel comprises a panchromatic filter, and the image processing method comprises: According to the use state of the camera application, an image processing mode is adopted, the use state of the camera application includes preview state, photo state and video state, and the image processing mode includes: A primary merging mode, comprising: reading pixel values of the pixel array, merging pixel values of all the color pixels in the color pixel group as the pixel value of the color pixel group, merging pixel values of all the panchromatic pixels in the panchromatic pixel group as the pixel value of the panchromatic pixel group, forming a primary pixel array according to the pixel values of the color pixel group and the pixel values of the panchromatic pixel group, and converting the primary pixel array into a Bayer array; A secondary merging mode, comprising: reading pixel values of the pixel array, merging pixel values of all the color pixels in the minimum repeating unit as the color pixel value of the minimum repeating unit, merging pixel values of all the panchromatic pixels in the minimum repeating unit as the panchromatic pixel value of the minimum repeating unit, forming a secondary pixel array according to the color pixel values of the minimum repeating unit and the panchromatic pixel values of the minimum repeating unit, and converting the secondary pixel array into a Bayer array; A full resolution mode includes reading pixel values of the pixel array and converting the pixel values of the pixel array into a Bayer array; When the camera application is in the preview state, using the secondary merging mode; When the camera application is in the photographing state, color temperature information of the target image is obtained, and the electronic device is judged to be indoors or outdoors according to the color temperature information; when the electronic device is outdoors, the ambient light intensity is obtained; when the ambient light intensity is less than or equal to the second light intensity threshold and greater than or equal to the third light intensity threshold, the primary merging mode is adopted; when the ambient light intensity is greater than the second light intensity threshold, the full resolution mode is adopted; and when the ambient light intensity is less than the third light intensity threshold, the secondary merging mode is adopted; When the camera application is in the video state, the secondary merging mode is adopted, and when a touch instruction to increase the video resolution is received, the secondary merging mode is switched to the primary merging mode.
3. The image processing method according to claim 1, characterized in that: The image processing mode also includes: The full resolution mode includes reading pixel values of the pixel array and converting the pixel values of the pixel array into a Bayer array.
4. The image processing method according to claim 1 or 2, characterized in that: The image processing mode also includes: Dual output models include, Reading pixel values of color pixels of the pixel array, and converting the pixel values of the color pixels into a Bayer array; The pixel values of the panchromatic pixels of the pixel array are read, and the pixel values of the panchromatic pixels are converted into a black and white array.
5. The image processing method according to any one of claims 1 to 3, characterized in that: According to the usage status of the camera application, after adopting one or more of the image processing modes, the method further includes: When a preset switching instruction is received, the currently used image processing mode is switched to another image processing mode.
6. The image processing method according to claim 2 or 3, characterized in that: When a preview zoom-in instruction is received, the secondary merging mode is switched to the full-resolution mode.
7. The image processing method according to claim 2 or 3, characterized in that: Also includes: Get the texture ratio of the target image; When the texture ratio of the target image exceeds a texture ratio threshold, the full resolution mode is adopted.
8. The image processing method according to claim 1, characterized in that: Also includes: When the camera application is in video state, it receives a frame rate control instruction and adjusts the video frame rate according to the frame rate control instruction.
9. The image processing method according to claim 4, characterized in that: Also includes: Executing the image processing mode in a first chip, and transmitting the Bayer array and / or the black-and-white array to a platform image processing chip, wherein the first chip is a built-in chip of the image sensor; or Executing the image processing mode in a second chip, and transmitting the Bayer array and / or the black-and-white array to the platform image processing chip, wherein the second chip is an external chip of the image sensor; or The data of the pixel array is dumped into the memory of the electronic device, the image processing mode is executed, and the Bayer array and / or the black-and-white array is dumped to the platform image processing chip.
10. The image processing method according to any one of claims 1 to 3, characterized in that: Four adjacent color pixels together constitute a color pixel group, wherein the four adjacent color pixels are arranged in a matrix; Four adjacent full-color pixels together constitute a full-color pixel group, wherein the four adjacent full-color pixels are arranged in a matrix.
11. The image processing method according to claim 10, characterized in that: The minimum repeating unit includes two color pixel groups and two full-color pixel groups, wherein the two color pixel groups and the two full-color pixel groups are arranged in a matrix, and the two color pixel groups are located in a first diagonal direction of the matrix, and the two full-color pixel groups are located in a second diagonal direction of the matrix, and the first diagonal direction and the second diagonal direction are different.
12. The image processing method according to claim 11, characterized in that: The minimum repeating unit is four rows and four columns, a total of sixteen pixels, arranged as follows: Among them, W represents full-color pixel and C represents color pixel.
13. The image processing method according to claim 12, characterized in that: The color pixels include a first color pixel, a second color pixel and a third color pixel, wherein the colors of the color filters of the first color pixel, the second color pixel and the third color pixel are different; Four adjacent minimum repeating units constitute a color repeating unit. The color repeating unit has eight rows and eight columns, with a total of sixty-four pixels, and the arrangement is as follows: Wherein, W represents a full-color pixel, X represents a first color pixel, Y represents a second color pixel, and Z represents a third color pixel.
14. The image processing method according to claim 11, characterized in that: The minimum repeating unit is four rows and four columns, a total of sixteen pixels, arranged as follows: Among them, W represents full-color pixel and C represents color pixel.
15. The image processing method according to claim 14, characterized in that: The color pixels include a first color pixel, a second color pixel and a third color pixel, wherein the colors of the color filters of the first color pixel, the second color pixel and the third color pixel are different; Four adjacent minimum repeating units constitute a color repeating unit. The color repeating unit has eight rows and eight columns, with a total of sixty-four pixels, and the arrangement is as follows: Wherein, W represents a full-color pixel, X represents a first color pixel, Y represents a second color pixel, and Z represents a third color pixel.
16. An electronic device, characterized in that: include, An image sensor, wherein the image sensor comprises a pixel array, the pixel array comprises a minimum repeating unit of repeated arrangement, the minimum repeating unit comprises at least two color pixel groups and at least two panchromatic pixel groups, the color pixel group comprises at least four color pixels, the panchromatic pixel group comprises at least four panchromatic pixels, the color pixels comprise color filters, and the panchromatic pixels comprise panchromatic filters; A processor, wherein the processor is configured to execute the method according to any one of claims 1 to 15.
17. An electronic device, characterized in that: include, An image acquisition module, the image acquisition module comprising a pixel array, the pixel array comprising a minimum repeating unit of repeated arrangement, the minimum repeating unit comprising at least two color pixel groups and at least two panchromatic pixel groups, the color pixel group comprising at least four color pixels, the panchromatic pixel group comprising at least four panchromatic pixels, the color pixels comprising color filters, and the panchromatic pixels comprising panchromatic filters; An image processing module, wherein the image processing module is used to adopt an image processing mode according to the use state of the camera application, the use state of the camera application includes a preview state, a photo state and a video state, and the image processing mode includes: A primary merging mode, comprising: reading pixel values of the pixel array, merging pixel values of all the color pixels in the color pixel group as the pixel value of the color pixel group, merging pixel values of all the panchromatic pixels in the panchromatic pixel group as the pixel value of the panchromatic pixel group, forming a primary pixel array according to the pixel values of the color pixel group and the pixel values of the panchromatic pixel group, and converting the primary pixel array into a Bayer array; A secondary merging mode, comprising: reading pixel values of the pixel array, merging pixel values of all the color pixels in the minimum repeating unit as the color pixel value of the minimum repeating unit, merging pixel values of all the panchromatic pixels in the minimum repeating unit as the panchromatic pixel value of the minimum repeating unit, forming a secondary pixel array according to the color pixel values of the minimum repeating unit and the panchromatic pixel values of the minimum repeating unit, and converting the secondary pixel array into a Bayer array; When the camera application is in the preview state, using the secondary merging mode; When the camera application is in the photographing state, the primary merging mode is adopted, or when the camera application is in the photographing state, color temperature information of the target image is obtained, and it is determined whether the electronic device is indoors or outdoors according to the color temperature information; when the electronic device is indoors, the ambient light intensity is obtained, and when the ambient light intensity is greater than or equal to a first light intensity threshold, the primary merging mode is adopted, and when the ambient light intensity is less than the first light intensity threshold, the secondary merging mode is adopted; When the camera application is in the video state, the secondary merging mode is adopted, and when a touch instruction to increase the video resolution is received, the secondary merging mode is switched to the primary merging mode.
18. An electronic device according to claim 17, characterized in that: The image processing mode also includes: The full resolution mode includes reading pixel values of the pixel array and converting the pixel values of the pixel array into a Bayer array.
19. An electronic device, characterized in that: include, An image acquisition module, the image acquisition module comprising a pixel array, the pixel array comprising a minimum repeating unit of repeated arrangement, the minimum repeating unit comprising at least two color pixel groups and at least two panchromatic pixel groups, the color pixel group comprising at least four color pixels, the panchromatic pixel group comprising at least four panchromatic pixels, the color pixels comprising color filters, and the panchromatic pixels comprising panchromatic filters; An image processing module, wherein the image processing module is used to adopt an image processing mode according to the use state of the camera application, the use state of the camera application includes a preview state, a photo state and a video state, and the image processing mode includes: A primary merging mode, comprising: reading pixel values of the pixel array, merging pixel values of all the color pixels in the color pixel group as the pixel value of the color pixel group, merging pixel values of all the panchromatic pixels in the panchromatic pixel group as the pixel value of the panchromatic pixel group, forming a primary pixel array according to the pixel values of the color pixel group and the pixel values of the panchromatic pixel group, and converting the primary pixel array into a Bayer array; A secondary merging mode, comprising: reading pixel values of the pixel array, merging pixel values of all the color pixels in the minimum repeating unit as the color pixel value of the minimum repeating unit, merging pixel values of all the panchromatic pixels in the minimum repeating unit as the panchromatic pixel value of the minimum repeating unit, forming a secondary pixel array according to the color pixel values of the minimum repeating unit and the panchromatic pixel values of the minimum repeating unit, and converting the secondary pixel array into a Bayer array; A full resolution mode includes reading pixel values of the pixel array and converting the pixel values of the pixel array into a Bayer array; When the camera application is in the preview state, using the secondary merging mode; When the camera application is in the photographing state, color temperature information of the target image is obtained, and the electronic device is judged to be indoors or outdoors according to the color temperature information; when the electronic device is outdoors, the ambient light intensity is obtained; when the ambient light intensity is less than or equal to the second light intensity threshold and greater than or equal to the third light intensity threshold, the primary merging mode is adopted; when the ambient light intensity is greater than the second light intensity threshold, the full resolution mode is adopted; and when the ambient light intensity is less than the third light intensity threshold, the secondary merging mode is adopted; When the camera application is in the video state, the secondary merging mode is adopted, and when a touch instruction to increase the video resolution is received, the secondary merging mode is switched to the primary merging mode.
20. A non-volatile computer-readable storage medium containing a computer program, characterized in that: When the computer program is executed by one or more processors, the processors are caused to perform the image processing method according to any one of claims 1 to 15.
Citation Information
Patent Citations
Image previewing method and device
CN106657780A
Photographing method, photographing device and terminal
CN107079083A
Image acquisition method and device, terminal and computer readable storage medium
CN112118378A
