A way to improve underwater photography

Abstract

A method for improving underwater photography, which relates to the field of data processing technology, includes: S1, turning on the camera of a mobile phone in water, and the image sensor starts working; S2, obtaining a small pixel image and an edge large pixel image after taking a photo, and the small pixel image and the edge large pixel image constitute an original image; S3, correcting the signal value of the color channel in the small pixel image according to the signal value of the color channel in the edge large pixel image to obtain a corrected new image; S4, outputting a format image after ISP processing of the corrected new image. The present invention improves the pixel's ability to collect light signals by adding a filter pixel array with different transmittance and spectrum at the edge of a conventional image sensor, and using an independent readout circuit, and using different color filter large pixels to calibrate conventional small pixels under different water qualities and water depths, which can reduce image noise, effectively solve the problem of color cast in underwater shooting, and further improve the underwater photography ability of mobile phones.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for improving underwater photography. Background Art

[0002] With the rapid development of mobile Internet technology and social platforms, mobile phones are not only a communication tool, but also an indispensable camera device. In daily life, the camera function of mobile phones can be used to record daily life, scan QR codes, live broadcast, etc. At the same time, the camera function of mobile phones has gradually become popular among directors and photographers. Mobile phones have a variety of shooting lenses, including ultra-wide angle, portrait and periscope, which can meet the needs of different occasions. For example, the ultra-wide angle can record wider scenes, the super anti-shake and main camera can obtain clearer and sharper images, and the portrait mode can take more beautiful portraits.

[0003] In addition, as mobile phone photography becomes more professional, the waterproof function of mobile phones has been improved day by day, and it can already meet the needs of shooting in the sky or on the ground. However, because water absorbs red light, underwater photography is often affected by light, resulting in image distortion and a bluish appearance.

[0004] Because when shooting underwater, as the depth changes, light of different wavelengths will gradually attenuate, and long-wave light (R) will be absorbed first. When the red light decreases, if it is restored according to conventional AWB, the overall image will be blue.

[0005] The current solution is to superimpose filters of different colors in front of the camera to restore the color of the seabed. However, this method will affect the transmittance and clarity of the camera, and it is very inconvenient to carry filters at all times.

[0006] In pursuit of being thin and light, mobile phones are equipped with sensors with small apertures and small pixel sizes, and their photosensitivity is far inferior to that of cameras. In addition, the light transmittance in water is relatively low. If a layer of filter is added in front of the lens, the light transmittance will be further reduced, increasing the image noise and affecting the clarity.

[0007] The patent document of Chinese patent application publication number CN110620861A discloses an image sensor, a camera module and a terminal. The image sensor includes a pixel layer, the pixel layer includes a plurality of pixel units, the pixel layer includes a central area located at the center of the pixel layer and at least one edge area located around the central area, and the area of ​​the pixel unit in the edge area is larger than the area of ​​the pixel unit in the central area. In the image sensor, the camera module and the terminal, the pixel unit of the pixel area closer to the center of the pixel layer is smaller, so that the pixel unit of the central area with more light received is smaller, and more pixels can be distributed to reduce the amount of light received by each pixel unit in the central area; while the pixel of the edge area with less light received is larger, and fewer pixels can be distributed to increase the amount of light received by each pixel unit in the edge area, so that the amount of light obtained by each pixel unit of the entire pixel layer is basically the same, thereby reducing the vignetting effect and facilitating the improvement of imaging quality.

[0008] It can be seen that the mutual interference of pixel units of different sizes in the image sensor and the shadow effect produced by the lens affect the imaging quality, and it is also impossible to provide different calibration signals in different water qualities and water depths.

[0009] Therefore, how to solve the interference between pixel units of different sizes and the color cast problem of mobile phone underwater photography and improve the quality of underwater photography images is an urgent problem to be solved today. Summary of the invention

[0010] In order to overcome the deficiencies in the background technology, the present invention discloses a method for improving underwater photography.

[0011] In order to achieve the above-mentioned invention object, the present invention adopts the following technical scheme:

[0012] An image sensor for improving underwater photography, the image sensor comprising regular pixels and large edge pixels, the large edge pixels surrounding the regular pixels; the area of ​​the regular pixels is smaller than the area of ​​the large edge pixels.

[0013] Specifically, the pixel array of the image sensor is arranged such that the center is a regular pixel for normal imaging, and surrounding the outer edges of the regular pixels are large edge pixels with special filters.

[0014] Specifically, the pixel array of the edge large pixels has an independent circuit design and works in parallel with the pixel array of the regular pixels.

[0015] Specifically, the colors of the special filter include red, orange, magenta and purple.

[0016] Specifically, the edge large pixels are one column or multiple columns.

[0017] Specifically, the image sensor can select and enable large pixel arrays with different color filters according to different water qualities or water depths.

[0018] The present invention also discloses a method for improving underwater photography, which uses the above-mentioned image sensor for improving underwater photography, comprising:

[0019] S1. The phone turns on the camera in the water and the image sensor starts working;

[0020] S2, after taking a photo, a small pixel image and an edge large pixel image are obtained, wherein the small pixel image and the edge large pixel image constitute an original image, wherein the small pixel image is an image obtained by the conventional pixels; and the edge large pixel image is an image obtained by the edge large pixels;

[0021] S3, correcting the signal value of the color channel in the small pixel image according to the signal value of the color channel in the edge large pixel image to obtain a corrected new image;

[0022] Correction is performed by counting the difference between the color channel in the edge large pixel image and the R channel signal in the small pixel image, specifically including the following steps:

[0023] S31, counting the average value R1 of the color channel signal value in the edge large pixel image;

[0024] S32, segmenting the small pixel blocks adjacent to the edge large pixel image in the small pixel image to obtain an edge small pixel image, and calculating an average value R2 of the R channel signal value in the edge small pixel image;

[0025] S33, calculate the R channel signal correction proportional coefficient m, m=R1 / R2;

[0026] S34, multiplying the signal value of each R pixel in the small-pixel image in the original image by the R channel signal correction ratio coefficient m to obtain a new R pixel signal, and then generating a new image according to the new R pixel signal to replace the original image.

[0027] S4, performing ISP processing on the corrected new image and outputting a format image.

[0028] Specifically, in step S1, when the mobile phone turns on the camera in the water, the water entry detection is turned on, and the sensor starts to work after the water entry detection passes. The water entry detection includes the following steps:

[0029] S11, detecting the charging port to determine whether the mobile phone has been submerged in water;

[0030] S12, judging whether the mobile phone has entered water by the change in resistance of the touch screen;

[0031] S13. When the water entry conditions of the charging port and the touch screen are both met, the barometer inside the mobile phone is further used to detect changes in air pressure to ultimately determine whether the mobile phone has been submerged in water.

[0032] Specifically, the small pixel image and the edge large pixel image in step S2 form the original image as follows: the original image consists of a small pixel image in the central part and an edge large pixel image surrounding the outer edge of the conventional small pixel image.

[0033] Specifically, step S3 can also calculate the pixel value of the small pixel image by performing bilinear interpolation on the signal values ​​of different color channels of the large edge pixels to obtain a corrected new image; the bilinear interpolation formula is as follows:

[0034] S C =((C R *S L +C L *S R ) / (C L +C R )+(R T *S B +R B *S T ) / (R T +R B )) / 2

[0035] Among them, C L For S C The large pixel S on the left L For pixel S C The distance, C R For S C The large pixel S on the right R For pixel S C The distance, R T For S C Upper large pixel S T For pixel S C The distance, R B For S C Lower large pixel S B For pixel S C The distance between each edge large pixel and pixel S C The distance to pixel S C The weight of .

[0036] Specifically, step S4 is as follows: after calibrating the R channel back to a normal signal, a new image is formed, and then the new image is output as a format image after AWB, gamma, and Demosaic.

[0037] Specifically, the image sensor can select and enable large pixel arrays with different color filters according to different water qualities or water depths.

[0038] The present invention discloses a method for improving underwater photography, comprising the following steps: S1, turning on the camera of a mobile phone in water, and the image sensor starts working; S2, obtaining a small pixel image and an edge large pixel image after taking the photo, wherein the small pixel image and the edge large pixel image constitute the original image; S3, correcting the signal value of the color channel in the small pixel image according to the signal value of the color channel in the edge large pixel image to obtain a corrected new image; S4, performing ISP processing on the corrected new image and outputting a format image. The present invention uses different color filter large pixels to calibrate the center conventional small pixels under different water qualities and water depths, can effectively detect color temperature and perform color correction, ensure that the image maintains accurate color restoration under different lighting conditions, can reduce image noise, effectively solve the color cast problem of underwater shooting, and further enhance the underwater photography capability of mobile phones.

[0039] The present invention discloses an image sensor for improving underwater photography. By adding filter pixel arrays with different transmittances and spectra at the edge of a conventional image sensor, the pixel array of large pixels at the edge has an independent circuit design and works in parallel with the pixel array of conventional pixels, thereby improving the pixel's ability to collect light signals, effectively eliminating consistency problems, and having higher sensitivity, being able to detect signals of different spectra, and providing different calibration signals at different water qualities and water depths, thereby achieving better photosensitivity and signal management.

[0040] Furthermore, the underwater image sensor disclosed in this embodiment does not need to add a layer of filter on the module lens, which effectively reduces the impact on the photosensitivity of the central conventional pixel. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 is a schematic diagram of the structure of an image sensor for improving underwater photography provided according to an embodiment of the present invention;

[0043] Figure 2 is a schematic diagram of a variant of an image sensor structure for improving underwater photography provided according to an embodiment of the present invention;

[0044] Figure 3is a schematic diagram of a circuit structure of an image sensor for improving underwater photography provided according to an embodiment of the present invention;

[0045] Figure 4 is a schematic flow chart of a method for improving underwater photography provided according to an embodiment of the present invention;

[0046] Figure 5 A method for improving underwater photography provided by an embodiment of the present invention includes small pixel images and edge large pixel images; wherein Figure 5 A is a small pixel image; Figure 5 B is the edge large pixel image;

[0047] Figure 6 is a schematic diagram of an original image in a method for improving underwater photography provided by an embodiment of the present invention;

[0048] Figure 7 is a schematic diagram of an image with small edge pixels in a method for improving underwater photography provided by an embodiment of the present invention;

[0049] Figure 8 is a schematic diagram of R pixel correction in a method for improving underwater photography provided according to an embodiment of the present invention; wherein Figure 8 A is the original image of R pixels, Figure 8 B is the R pixel correction image;

[0050] Fig. 9 It is a schematic diagram of a concise bilinear interpolation method provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The present invention can be explained in detail through the following embodiments. The purpose of disclosing the present invention is to protect all technical improvements within the scope of the present invention. In the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "front", "back", "left", "right", etc. indicating directions or positional relationships, they only correspond to the drawings of the present application for the convenience of describing the present invention, and do not indicate or imply that the device or element referred to must have a specific direction.

[0052] Some terms involved in this application are explained below.

[0053] (1)Sensor

[0054] An image sensor is a semiconductor chip that contains hundreds of thousands to millions of photodiodes on its surface. When a photodiode is exposed to light, it generates an electric charge. Each photodiode is a photosensitivity point. What we usually call 300,000 pixels or 1.3 million pixels means that there are 300,000 or 1.3 million photosensitivity points, and one photosensitivity point corresponds to one pixel.

[0055] The basis of the sensor's photosensitivity principle is the photoelectric effect. The sensor's photosensitive unit, also known as the pixel unit, will generate a certain electrical signal after receiving external light stimulation, and after processing, it can display external image information.

[0056] (2) Color filter array

[0057] Color filter array, also known as Color Filter Array or Color Filter Mosaic, abbreviated as CFA or CFM, is a mosaic layer above the pixel sensor, used to collect the color information of the image. General photoelectric sensors can only sense the intensity of light, but cannot distinguish the wavelength (color) of light, so image sensors need to use color filtering to obtain the color information of pixels.

[0058] (3) Color Filter

[0059] A color filter is an optical filter that expresses color. It can accurately select a small range of light waves to pass through, and reflect other bands that are not desired to pass through. For example, the most common Bayer Filter, also known as the RGGB Filter, the image sensor obtains the intensity information of the red (Red), green (Green) and blue (Blue) light on the pixel through the Bayer Filter, and then uses the Demosaicing Algorithm to infer the color value of the pixel. The spectral filtering characteristics and color restoration algorithm of CFA determine the color acquisition capability of CFA. Usually, the photon passband of the image sensor and the spectral response range of CFA are larger than the visible light spectrum range, thus ensuring that the image sensor can capture all color information in the visible light range.

[0060] (4) AWB calibration

[0061] When processing Raw images into normal color images, image white balance calibration (WhiteBalance) is required first. Image white balance is to adjust the colors in the image so that the white objects in the image appear to be truly white visually, without being affected by the color of the light source. This is because different light sources (such as sunlight, incandescent lamps or fluorescent lamps, etc.) have their own specific color temperatures, that is, color temperatures, and these light sources can make the objects appear to have different color biases. Therefore, when processing images, it is necessary to first restore the original gray or white objects to true gray and white.

[0062] AWB calibration methods include the gray world hypothesis and the white area statistics method;

[0063] Gray world assumption: It assumes that the average reflectivity of the scene is gray, and the algorithm makes the average color of the image close to gray to achieve the effect of white balance.

[0064] White area statistics method: This method assumes that there is at least one area in the image that is white or gray. The algorithm will search for these areas in the image and calculate their RGB ratio to adjust the white balance of the entire image.

[0065] Embodiment 1

[0066] refer to Figure 1 , this embodiment discloses an image sensor for improving underwater photography, the image sensor comprising regular pixels and large edge pixels, the large edge pixels surrounding the regular pixels; the area of ​​the regular pixels is smaller than the area of ​​the large edge pixels;

[0067] Specifically, the pixel array of the image sensor is arranged such that the center is a regular pixel for normal imaging, and surrounding the outer edge of the regular pixel is a row of large edge pixels with special filters;

[0068] The large edge pixel with a special filter is obtained by adding a color filter with higher transmittance to the edge of a conventional image sensor. After conversion through the color filter, the corresponding pixel is the large edge pixel described in this embodiment.

[0069] In one embodiment, the area of ​​the large edge pixel is 4 times that of the regular pixel. The pixel area of ​​the image sensor of this embodiment is measured in um. In one embodiment, for example, the regular pixel is an RGGB pixel, and its size is 0.8um*0.8um; the large edge pixel is a Red filter, and its pixel area is 1.6um*1.6um. Furthermore, the large edge pixel of this embodiment only includes a single color filter, for example, it may include one of Red, Orange, Magenta, and Purple.

[0070] That is, the conventional pixel of the image sensor for improving underwater photography in this embodiment is a pixel unit including multiple color filters, such as an RGGB structure. The edge pixel unit is a pixel unit including only one color filter, such as only a Red filter.

[0071] Specifically, the colors of the special filter include red, orange, magenta and purple.

[0072] Conventional pixels generally use an RGGB structure. Preferably, the conventional pixels in this embodiment are a four-in-one RGGB structure.

[0073] The structure of the regular pixel in the middle of the image sensor is a regular four-in-one RGGB structure, while the large edge pixels surrounding the regular pixels are monochrome, and the colors of the special filters of the large edge pixels include but are not limited to Red, Orange, Magenta, and Purple. The specific colors of the special filters are not limited here, and can be designed and expanded to multiple colors according to different needs to facilitate the acquisition of more comprehensive spectral information.

[0074] Specifically, the edge large pixels are one column or multiple columns.

[0075] refer to Figure 2 In another embodiment, the edge large pixels surrounding the outer edge of the conventional pixels may be multiple columns, and the specific number of columns is not limited. Different numbers of columns may be added according to requirements, and each column of edge large pixels may use different colors. In order to better illustrate the content of the present invention, this embodiment preferably uses one column of edge large pixels.

[0076] Specifically, the pixel array of the edge large pixels has an independent circuit design and works in parallel with the pixel array of the regular pixels.

[0077] refer to Figure 3 The pixel array of the large edge pixels has an independent circuit design and works in parallel with the pixel array of the conventional pixels. Due to the use of a separate circuit structure, it can achieve better photosensitivity and signal management.

[0078] Specifically, regular pixels and edge large pixels adopt separate circuit designs:

[0079] The regular pixels and large edge pixels of the image sensor use separate circuit designs, and each group of pixels is equipped with an independent ADC (analog-to-digital converter).

[0080] Conventional pixels and edge large pixels are designed with different ADC slopes. Different ADC slopes can be configured according to the required photoelectric conversion efficiency:

[0081] The ADC slope of the large edge pixels can be designed to be steeper to improve sensitivity in low-light conditions.

[0082] The ADC slope of conventional pixels can be designed to be gentler to maintain image quality under normal lighting.

[0083] Large edge pixels capture more light signals and provide additional spectral information for image correction.

[0084] By combining different ADC slopes, the dynamic range and signal-to-noise ratio of the overall image can be improved, especially in complex lighting environments such as underwater shooting, which can effectively optimize image quality.

[0085] This embodiment provides an implementation method of conventional pixel and edge large pixel circuit design:

[0086] Edge large pixel circuit design: Use high-gain ADC to improve signal capture capabilities in low-light environments; configure a steeper slope to improve photoelectric conversion efficiency.

[0087] Conventional pixel circuit design: Use low-gain ADC to maintain image quality under normal lighting; configure a gentler slope to avoid noise increase caused by over-amplification.

[0088] The separation design of the conventional pixel and the edge large pixel circuit in this embodiment has the following advantages:

[0089] Flexibility: Ability to dynamically adjust photoelectric conversion efficiency according to different environments (such as different depths underwater);

[0090] High efficiency: Improved signal capture capability in low-light environments and reduced image noise;

[0091] Image quality: Achieve higher dynamic range and better signal-to-noise ratio by optimizing ADC slope combination.

[0092] The central pixel array corresponding to the regular pixels follows the standard RGB color order, and the edge large pixel Cx color filter array is located in the peripheral array.

[0093] Specifically, the image sensor can select and enable large pixel arrays with different color filters according to different water qualities or water depths.

[0094] For example, when taking photos in blue sea water, the large red pixel array at the edge of the image is enabled, and when taking photos in the deep sea, the large purple filter pixel array at the edge is enabled. At different depths, the signal amount in the long-wave band is increased to achieve color correction.

[0095] The image sensor in this embodiment is a multi-spectral sensor, and the inherent characteristics of the filter itself can provide different calibration signals for different water qualities and water depths.

[0096] This embodiment provides a specific implementation method for selecting and enabling large pixels of different color filters according to different water qualities and water depths:

[0097] 1) Establish a database of water quality and depth:

[0098] Collect and record spectral characteristic data of different water qualities (water transparency, suspended matter content, etc.) and water depths (shallow water, medium water, deep water).

[0099] Determine the absorption and scattering characteristics of light in water under different conditions, especially the effects on different wavelengths of light.

[0100] 2) Sensor design and configuration:

[0101] Design an image sensor with large pixel arrays at the edge that contain multiple color filters (e.g., red, orange, magenta, violet).

[0102] An independent circuit design is configured so that the large pixels of each color filter can be enabled independently.

[0103] 3) Environmental testing and analysis:

[0104] Use sensors to detect the current underwater environment, including water depth and water quality.

[0105] Get real-time data of the current underwater environment through built-in environmental sensors (such as pressure sensor, turbidity sensor).

[0106] 4) Select filter configuration:

[0107] Determine the best filter configuration based on detected water depth and water quality data by searching a pre-built database.

[0108] Enables filter large pixels of corresponding color.

[0109] This embodiment also provides a specific implementation method of using large pixels with different color filters at different water depths:

[0110] A. Shallow water:

[0111] In shallow water areas (0-10 meters), the water is clear and the light is relatively sufficient, but the red light attenuation is obvious.

[0112] Large pixels with red and orange filters are enabled to compensate for the lack of red light and enhance the color reproduction of images.

[0113] B. Recycled water:

[0114] At moderate water depths (10-30 meters), light begins to attenuate significantly, especially long wavelength light.

[0115] Enables large pixels with a magenta filter to capture more long-wavelength light signals and maintain image quality.

[0116] C. Deep Water:

[0117] In deep water (above 30 meters), the light is extremely weak and the red light almost completely disappears.

[0118] Enables large pixels with a purple filter to maximize capture of available light, improving image brightness and detail.

[0119] Purple filter large pixel signal capture:

[0120] In deep sea environments, a large pixel array with a purple filter is enabled. The spectral distribution of each area in the image is analyzed through the signals captured by the large purple filter pixels. The purple filter is able to capture some near-infrared spectral signals, which still exist in underwater environments. These large purple pixels will capture spectral information between longer wavelengths (near infrared) and shorter wavelengths (blue-violet light).

[0121] 5) Real-time adjustment:

[0122] The sensor monitors environmental changes in real time and dynamically adjusts the large-pixel configuration of enabled filters to respond to changes in water quality and depth.

[0123] The real-time data is analyzed by algorithms and the filter configuration is adjusted to optimize image quality.

[0124] An underwater image sensor disclosed in this embodiment adds a filter pixel array with different transmittance and spectrum at the edge of a conventional image sensor, and the pixel array of the large edge pixels has an independent circuit design and works in parallel with the pixel array of the conventional pixels, thereby improving the pixel's ability to collect light signals, effectively eliminating consistency problems, and having higher sensitivity, capable of detecting signals of different spectra, and providing different calibration signals for different water qualities and water depths, thereby achieving better photosensitivity and signal management.

[0125] Furthermore, the underwater image sensor disclosed in this embodiment does not need to add a layer of filter on the module lens, which effectively reduces the impact on the photosensitivity of the central conventional pixel.

[0126] Embodiment 2

[0127] refer to Figure 4 This embodiment discloses a method for improving underwater photography using the image sensor of the first embodiment, comprising:

[0128] S1. The mobile phone turns on the camera in the water and the underwater image sensor starts working;

[0129] The phone can detect environmental and altitude information through the charging port, touch screen and barometer to determine whether the phone has entered water.

[0130] Specifically, the method for detecting water entry of a mobile phone proposed in this embodiment includes the following method:

[0131] 1. Check the charging port to determine whether the phone has been submerged in water;

[0132] When the contacts in the charging port come into contact with water, the instantaneous current surges, exceeding the normal operating current range of the device, and it is preliminarily determined that the phone has been submerged in water;

[0133] In this embodiment, the type of mobile phone charging port is not limited, and all charging ports that can achieve the above functions are protected. For details, please refer to the disclosure of Chinese patent application publication number CN110797721A, a USB-based multifunctional underwater detection equipment dedicated integrated interface, which will not be repeated here.

[0134] 2. Determine whether the phone has been submerged in water by the change in resistance of the touch screen;

[0135] Since water is conductive, a capacitive touch screen can be used to perform a secondary detection to see if the phone is exposed to water. The capacitive touch screen has a network of tiny electrodes that continuously generate an electric field. When a finger or other conductive object approaches or touches the screen, it causes a local capacitance change. The controller detects these changes and determines the location. If water completely covers the entire screen, it will cause a significant increase in the resistance of the entire screen. Compared with conventional touch (such as finger contact), the resistance change caused by water is more significant, and further judgment can be made based on this.

[0136] 3. When the conditions for the charging port and the touch screen to be immersed in water are met, the barometer inside the phone is further used to detect changes in air pressure to ultimately determine whether the phone has been immersed in water.

[0137] For details, please refer to the contents disclosed in Chinese patent application publication number: CN115148004A, which are methods, devices, wearable devices and storage media for water entry detection, and will not be repeated here.

[0138] In another embodiment, a microphone and a speaker can be used to determine whether the mobile phone is in water: ultrasonic waves are emitted and received by the microphone and the speaker, and then the returned signals are analyzed. The sound propagation characteristics in water are different from those in air, so by analyzing the returned signals, it can be determined whether the mobile phone is in water.

[0139] If the mobile phone is equipped with a humidity sensor, then when it comes into contact with water, the humidity sensor detects a sharp increase in humidity, which can also be used as a basis for judging whether the mobile phone is in water.

[0140] When the phone is submerged in water and the camera is turned on, the regular small pixels in the center of the underwater image sensor and the large edge pixels surrounding the outer edges of the regular small pixels start working and output signals.

[0141] S2, after taking a photo, a small pixel image and an edge large pixel image are obtained, wherein the small pixel image and the edge large pixel image constitute an original image, wherein the small pixel image is an image obtained by the conventional pixels; and the edge large pixel image is an image obtained by the edge large pixels;

[0142] Specifically, the small pixel image and the edge large pixel image in step S2 form the original image as follows: the original image consists of a small pixel image in the central part and an edge large pixel image surrounding the outer edge of the conventional small pixel image.

[0143] like Figure 5 As shown, after taking a photo, a small pixel image and an edge large pixel image are obtained through the image sensor. The small pixel image is a regular pixel image in the center of the sensor, such as Figure 5 A,edge large pixel images are obtained by special filters at the outer edge of the image sensor, e.g. Figure 5 B.

[0144] The small pixel image and the edge large pixel image are combined into an original image, such as Figure 6 shown.

[0145] The original raw data output by the sensor is colorless, and the signal value of each pixel is represented by a number.

[0146] There are many formats for representing signal values, and 8bit and 10bit are commonly used. In the 8bit format, the signal value range is 0 to 255, where 0 means completely black without any signal, and 255 means the highest brightness, and the image is completely white; in the 10bit format, the signal value range is 0 to 1023, where 0 means completely black without any signal, and 1023 means the highest brightness, and the image is completely white. The signals of each pixel are superimposed together to form a color image that the human eye sees. If the three-channel signals are the same, it will be completely black, completely white, or completely gray.

[0147] Preferably, the pixel signal value in this embodiment uses a 10-bit format, referring to Figure 5 As shown in B.

[0148] The underwater shooting method currently used in the market is to add a filter in front of the camera. For example, after adding a red filter in front of the entire sensor, the signals of the B channel and the G channel are reduced to varying degrees. Although reducing the signal of the blue channel can reduce the problem of the image being blue, it will also affect the transmittance of the G channel. In some low-light conditions, a higher digital signal needs to be superimposed to increase the image brightness and white balance calibration. Using too high a digital gain will make the image noise worse. After using the filter, the AWB ratio and target luma will be greatly affected, so the image needs to be AWB calibrated after using the filter.

[0149] S3, correcting the signal value of the color channel in the small pixel image according to the signal value of the color channel in the edge large pixel image to obtain a corrected new image;

[0150] Because the edge special filter in Example 1 has high sensitivity and can detect signals of different spectra, and provide different calibration signals in different water qualities and water depths, thus achieving better photosensitivity and signal management, the color of the edge large pixel image corresponding to the edge special filter can be closer to the color of the real image, and the color of the central small pixel image can be calibrated using the color of the edge large pixel.

[0151] Given the difference between the signals of edge pixels and regular pixels, the center R channel can refer to the signals of large edge pixels as a reference to align the signals of the center R channel with the edge image signals.

[0152] However, due to the influence of lens shadows and the presence of different objects in real scenes, the R pixel signals on the entire image are not the same, so the normal pixel signals in the central area cannot be directly used for average calculation. The image needs to be divided into two parts to count the signal values ​​at different positions. This processing can more accurately reflect the signal differences between different parts of the image.

[0153] Specifically, step S3 can be corrected by counting the difference between the color channel in the edge large pixel image and the R channel signal in the small pixel image, and specifically includes the following steps:

[0154] S31, counting the average value R1 of the color channel signal value in the edge large pixel image;

[0155] S32, segmenting the small pixel blocks adjacent to the edge large pixel image in the small pixel image to obtain an edge small pixel image, and calculating an average value R2 of the R channel signal value in the edge small pixel image;

[0156] S33, calculate the R channel signal correction proportional coefficient m, m=R1 / R2;

[0157] S34, multiplying the signal value of each R pixel in the small-pixel image in the original image by the R channel signal correction ratio coefficient m to obtain a new R pixel signal, and then generating a new image according to the new R pixel signal to replace the original image.

[0158] For example, in shallow water, the large edge pixels enabled by the image sensor are red, referring to Figure 5 B edge large pixel image, the average value of the R channel signal value in the edge large pixel image is 660LSB, reference Figure 7 For edge small pixel images, the average value of the R channel signal value in the edge small pixels is 600LSB, and the R channel signal correction coefficient m is calculated, m=660 / 600=1.1.

[0159] After understanding the signal ratio of large edge pixels and regular small pixels, take the following steps to enhance the red channel of the image to improve the overall color balance:

[0160] For example, reference Figure 8 , use the calculated R channel signal correction ratio m (e.g. 1.1x) to enlarge the original image as Figure 8 The R signal value of the conventional small pixel image in A is multiplied by the coefficient for each R pixel value of the small pixel image; a new image is generated as Figure 8 B; then replace the original image with the new image.

[0161] In this way, the red signal in the new image is enhanced, thereby offsetting the blue cast common in underwater images and making the colors closer to normal visual perception.

[0162] For example, in a deep sea environment, i.e., deep water, a large pixel array with a purple filter is enabled, and the red channel in the image is calibrated using the purple large pixel signal. The specific steps are as follows:

[0163] 1) Count the average signal value of the purple large pixels:

[0164] Calculates the average signal value of the purple filter channel in the purple large pixel image (R1).

[0165] 2) Segmentation of small pixel images on the edge:

[0166] The small pixel blocks in the small pixel image that are adjacent to the purple large pixel image are segmented to obtain an edge small pixel image.

[0167] 3) Calculate the average signal value of edge small pixel images:

[0168] Count the average R channel signal value (R2) in the edge small pixel image.

[0169] 4) Calculate the correction ratio coefficient:

[0170] Calculate the R channel signal correction proportional coefficient m, the formula is: m=R1 / R2.

[0171] 5) Calibrate the R channel signal:

[0172] Each R pixel signal value of the small pixel image in the original image is multiplied by the correction scale factor m to obtain a new R pixel signal.

[0173] 6) Generate a new calibrated image:

[0174] Generate a new image based on the calibrated R pixel signal and replace the R channel signal in the original image.

[0175] This embodiment provides a specific implementation method of calibrating a conventional small pixel image using a purple edge large pixel:

[0176] 1. Count the signal values ​​of large purple pixels:

[0177] Assume that the average signal value of the purple channel in the purple large pixel image is R1=450LSB.

[0178] 2. Segment edge small pixel images:

[0179] Small pixel blocks adjacent to the purple large pixel are extracted from the small pixel image to obtain the edge small pixel image.

[0180] 3. Count the R channel signal values ​​of edge small pixel images:

[0181] Assume that the average value of the R channel signal value in the edge small pixel image is R2=300LSB.

[0182] 4. Calculate the correction scale factor:

[0183] Correction proportional coefficient m=R1 / R2=450 / 300=1.5.

[0184] 5. Calibrate the R channel signal:

[0185] Multiply each R pixel signal value of the small pixel image in the original image by the correction ratio coefficient m. For example, if a certain R pixel signal value is 200LSB, the calibrated signal value is 200*1.5=300LSB.

[0186] 6. Generate a new calibrated image:

[0187] According to the calibrated R pixel signal, a calibrated new image is generated to replace the R channel signal in the original image.

[0188] Calibrating regular small-pixel images with large edge pixels can effectively detect color temperature and perform color correction, ensuring that the image maintains accurate color reproduction under different lighting conditions.

[0189] Specifically, step S3 may also calculate the pixel values ​​of the small pixel image by performing bilinear interpolation on the signal values ​​of different color channels of the large edge pixels to obtain a corrected new image.

[0190] The bilinear interpolation algorithm is an interpolation method for two-dimensional space. It estimates the value of any point by taking a weighted average of the four nearest pixel points. It is used for scaling and other occasions where the value of the new pixel position needs to be re-estimated.

[0191] This example refers to the idea of ​​the bilinear interpolation algorithm. In a more advanced color restoration process, bilinear interpolation is used to predict the central spectral information using the different color channel signals of the edge large pixels, and accurately compensate the color information of the central small pixels.

[0192] refer to Fig. 9 , suppose we want to calculate the pixel S in the image C The signal values ​​are as follows:

[0193] By aligning the large pixels around the edge (S L , S R , S T , S B ) according to their distance (C R , C L , R T , R B ) for weighted averaging, the final pixel S C The bilinear interpolation formula in this embodiment is as follows:

[0194] S C =((C R *S L +C L *S R ) / (C L +C R )+(R T *S B +R B *S T ) / (R T +R B )) / 2

[0195] Among them, C L For S C The large pixel S on the left L For pixel S C The distance, C R For S C The large pixel S on the right R For pixel S C The distance, R T For S C Upper large pixel S T For pixel S C The distance, R B For S C Lower large pixel S B For pixel S C The distance between each edge large pixel and pixel S C The distance to pixel S C The weight of .

[0196] S4, performing ISP processing on the corrected new image and outputting a format image.

[0197] Specifically, after calibrating the R channel back to a normal signal, a new image is formed, and then the new image is output as a format image after AWB, gamma, and Demosaic. The format of the format image is not limited here and is selected according to actual needs. In this embodiment, the JPG format is preferably used.

[0198] The present embodiment discloses a method for improving underwater photography, comprising the following steps: S1, turning on the camera of a mobile phone in water, and the image sensor starts working; S2, obtaining a small pixel image and an edge large pixel image after taking a photo, wherein the small pixel image and the edge large pixel image constitute an original image; S3, correcting the signal value of the color channel in the small pixel image according to the signal value of the color channel in the edge large pixel image to obtain a corrected new image; S4, outputting a format image after ISP processing of the corrected new image. The present embodiment uses different color filter large pixels to calibrate the center conventional small pixels under different water qualities and water depths, which can effectively detect color temperature and perform color correction, ensure that the image maintains accurate color reproduction under different lighting conditions, reduce image noise, effectively solve the color cast problem of underwater shooting, and further improve the underwater photography capability of mobile phones.

[0199] The parts of the present invention that are not described in detail are prior art. It is obvious to those skilled in the art that the present invention is not limited to the details of the above-mentioned exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention; therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and it is intended that all changes that fall within the meaning and scope of equivalent elements are included in the present invention.

Claims

1. A method for improving underwater photography, the method being applied to an image sensor, the image sensor comprising regular pixels and large edge pixels, the large edge pixels surrounding the regular pixels; The area of ​​the regular pixel is smaller than the area of ​​the edge large pixel, and is characterized by comprising: S1. The phone turns on the camera in the water and the image sensor starts working; S2, after taking a photo, a small pixel image and an edge large pixel image are obtained, wherein the small pixel image and the edge large pixel image constitute an original image, wherein the small pixel image is an image obtained by the conventional pixels; and the edge large pixel image is an image obtained by the edge large pixels; S3, correcting the signal value of the color channel in the small pixel image according to the signal value of the color channel in the edge large pixel image to obtain a corrected new image; Correction is performed by counting the difference between the color channel in the edge large pixel image and the R channel signal in the small pixel image, specifically including the following steps: S31, counting the average value R1 of the color channel signal value in the edge large pixel image; S32, segmenting the small pixel blocks adjacent to the edge large pixel image in the small pixel image to obtain an edge small pixel image, and calculating an average value R2 of the R channel signal value in the edge small pixel image; S33, calculate the R channel signal correction proportional coefficient m, m=R1 / R2; S34, multiplying the signal value of each R pixel in the small-pixel image in the original image by the R channel signal correction ratio coefficient m to obtain a new R pixel signal, and then generating a new image according to the new R pixel signal to replace the original image; S4, performing ISP processing on the corrected new image and outputting a format image.

2. The method according to claim 1, characterized in that In step S1, when the mobile phone turns on the camera in the water, water entry detection is turned on. After the water entry detection is passed, the sensor starts to work. The water entry detection includes the following steps: S11, detecting the charging port to determine whether the mobile phone has been submerged in water; S12, judging whether the mobile phone has entered water by the change in resistance of the touch screen; S13. When the water entry conditions of the charging port and the touch screen are both met, the barometer inside the mobile phone is further used to detect changes in air pressure to ultimately determine whether the mobile phone has been submerged in water.

3. The method according to claim 1, characterized in that The small pixel image and the edge large pixel image in step S2 form the original image specifically as follows: the original image consists of a small pixel image in the central part and an edge large pixel image surrounding the outer edge of the conventional small pixel image.

4. The method according to claim 1, characterized in that Step S3 can also calculate the pixel value of the small pixel image by performing bilinear interpolation on the signal values ​​of different color channels of the large edge pixels to obtain a corrected new image; the bilinear interpolation formula is as follows: S C =((C R *S L +C L *S R ) / (C L +C R )+(R T *S B +R B *S T ) / (R T +R B )) / 2 Among them, C L For S C The large pixel S on the left L For pixel S C The distance, C R For S C The large pixel S on the right R For pixel S C The distance, R T For S C Upper large pixel S T For pixel S C The distance, R B For S C Lower large pixel S B For pixel S C The distance between each edge large pixel and pixel S C The distance to pixel S C The weight of .

5. The method according to claim 1, characterized in that Step S4 is specifically as follows: after calibrating the R channel back to a normal signal, a new image is formed, and then the new image is output as a format image after AWB, gamma, and Demosaic.

6. The method according to claim 1, characterized in that The pixel array of the image sensor is arranged such that the center is a regular pixel for normal imaging, and surrounding the outer edges of the regular pixels are large edge pixels with special filters.

7. The method according to claim 6, characterized in that The pixel array of the edge large pixels has an independent circuit design and works in parallel with the pixel array of the regular pixels.

8. The method according to claim 6, characterized in that The colors of the special filters include red, orange, magenta and violet.

9. The method according to claim 6, characterized in that The edge large pixels are one or more columns.

10. The method according to claim 8, characterized in that The image sensor can select and enable large pixel arrays with different color filters according to different water qualities or water depths.

Images