Image sensor, camera module, electronic equipment and image processing method

Abstract

The invention relates to the technical field of image processing, and discloses an image sensor, a camera module, electronic equipment and an image processing method. The image sensor includes an array of pixels. The pixel array comprises a plurality of color pixel units and a plurality of gray pixel units, and the distribution density of the color pixel units in the image sensor is smaller than or equal to the distribution density of the gray pixel units in the image sensor. Moreover, the distribution density of the color pixel units in the central area of the image sensor is larger than or equal to the distribution density of the gray pixel units in the central area, and the distribution density of the color pixel units in the edge area of the image sensor is smaller than the distribution density of the gray pixel units in the edge area. According to the image sensor, video acquisition with high frame rate, high resolution and high dark light sensitivity can be realized at the same time under the condition of limited bandwidth, and the shooting effect is good.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular, to an image sensor, a camera module, an electronic device, and an image processing method. Background Art

[0002] With the continuous development of photography technology, electronic devices with shooting functions have been widely used, and users' requirements for the shooting performance of electronic devices are also getting higher and higher. For example, users hope that electronic devices can shoot more realistic, smooth, and clear videos, and at the same time, they also hope that electronic devices can shoot videos normally in low-light (or "low illuminance") environments. Among them, high frame rate is beneficial to improving vividness and smoothness, high resolution is beneficial to improving clarity, and high low-light sensitivity is beneficial to improving low-light shooting effects. However, the bandwidth of the image sensors of current electronic devices (that is, the maximum value of the speed of information that can be transmitted per unit time) is limited, making it difficult to simultaneously achieve high frame rate, high resolution, and high low-light sensitivity video acquisition, and the shooting effect needs to be improved. Summary of the Invention

[0003] To solve the above technical problems, this application provides an image sensor, a camera module, an electronic device, and an image processing method. The following introduces this application from multiple aspects, and the implementation manners and beneficial effects of the following multiple aspects can be referred to each other.

[0004] In a first aspect of this application, an image sensor is provided. The image sensor includes a pixel array. The pixel array includes a plurality of color pixel units and a plurality of grayscale pixel units, and the distribution density of the color pixel units in the image sensor is less than or equal to the distribution density of the grayscale pixel units in the image sensor. Moreover, the distribution density of the color pixel units in the central region of the image sensor is greater than or equal to the distribution density of the grayscale pixel units in the central region, and the distribution density of the color pixel units in the edge region of the image sensor is less than the distribution density of the grayscale pixel units in the edge region.

[0005] In the above image sensor, the distribution rules of the color pixel units and the grayscale pixel units are similar to the distribution rules of the cone cells and rod cells of the human eye. Therefore, whether in a well-lit environment or in a low-light environment, a picture close to what the human eye sees can be obtained. Exemplarily, the image sensor can collect high-frame-rate, low-resolution color videos through the color pixel units, and collect low-frame-rate, high-resolution, and high low-light sensitivity grayscale videos through the grayscale pixel units. Based on the color video, colorization processing and frame interpolation processing are performed on the grayscale video, so that high-frame-rate, high-resolution, and high low-light sensitivity video acquisition can be simultaneously achieved under the limited bandwidth of the image sensor, and the shooting effect is good.

[0006] In a possible implementation of the first aspect described above, the response frame rate of the color pixel units is greater than that of the grayscale pixel units.

[0007] Since the number of color pixel units in the image sensor is relatively small, collecting high-frame-rate videos through the color pixel units will not significantly increase the data read / write pressure on the image sensor.

[0008] In a possible implementation of the first aspect described above, from the central region to the edge region, the distribution density of the color pixel units gradually decreases, and the distribution density of the grayscale pixel units gradually increases.

[0009] In this way, the distribution rules of the color pixel units and the grayscale pixel units can be made closer to the distribution rules of cone cells and rod cells in the human eye, so that the video images collected by the image sensor are closer to the high-resolution and high low-light sensitivity color video images seen by the human eye.

[0010] In a possible implementation of the first aspect described above, the ratio of the distribution density of the color pixel units in the image sensor to the distribution density of the grayscale pixel units in the image sensor is 1:20 to 1:1.

[0011] In this way, the distribution rules of the color pixel units and the grayscale pixel units can be made closer to the distribution rules of cone cells and rod cells in the human eye, so that the video images collected by the image sensor are closer to the high-resolution and high low-light sensitivity color video images seen by the human eye.

[0012] In a possible implementation of the first aspect described above, in the central region, the ratio of the distribution density of the color pixel units to the distribution density of the grayscale pixel units is 5:1 to 1:1, and in the edge region, the ratio of the distribution density of the color pixel units to the distribution density of the grayscale pixel units is 1:30 to 1:1.

[0013] In this way, the distribution rules of the color pixel units and the grayscale pixel units in the central region and the edge region can be made closer to the distribution rules of cone cells and rod cells in the human eye, so that the video images collected by the image sensor are closer to the high-resolution and high low-light sensitivity color video images seen by the human eye.

[0014] In a possible implementation of the above first aspect, the image sensor further includes an image processing unit. The image processing unit is electrically connected to the color pixel unit and the grayscale pixel unit, and is configured to obtain a first color video signal with a first frame rate and a first resolution output by the color pixel unit, obtain a first grayscale video signal with a second frame rate and a second resolution output by the grayscale pixel unit, and process the first color video signal and the first grayscale video signal to obtain a target color video signal with the first frame rate and the second resolution. The first frame rate is greater than the second frame rate, and the first resolution is less than the second resolution.

[0015] In this way, a color video signal with high frame rate, high resolution, and high dark light sensitivity can be obtained under the condition of limited bandwidth of the image sensor.

[0016] The second aspect of the present application provides a camera module. The camera module includes a lens module and the image sensor in the above first aspect and any possible implementation of the above first aspect. The lens module is configured to receive incident light and transmit the incident light to the image sensor.

[0017] The third aspect of the present application provides an electronic device. The electronic device includes a housing and the camera module in the above first aspect. The camera module is disposed in the housing.

[0018] The fourth aspect of the present application provides an image processing method, which is applied to an image sensor. The image sensor includes a plurality of color pixel units, a plurality of grayscale pixel units, and an image processing unit. The image processing unit is electrically connected to the color pixel unit and the grayscale pixel unit.

[0019] The image processing method includes: First, the image processing unit obtains a first color video signal with a first frame rate and a first resolution output by the color pixel unit, and obtains a first grayscale video signal with a second frame rate and a second resolution output by the grayscale pixel unit. The first frame rate is greater than the second frame rate, and the first resolution is less than the second resolution. Then, the image processing unit performs coloring processing on the first grayscale video signal based on the first color video signal to obtain a second color video signal, and the second color video signal has the second frame rate and the second resolution. Finally, the image processing unit performs frame interpolation processing on the second color video signal based on the first color video signal to obtain a target color video signal, and the target color video signal has the first frame rate and the second resolution.

[0020] The above image processing method obtains a first color video signal with a high frame rate and low resolution output by a color pixel unit through an image processing unit of an image sensor, obtains a first grayscale video signal with a low frame rate and high resolution output by a grayscale pixel unit, and performs colorization processing and frame interpolation processing on the first grayscale video signal based on the first color video signal, so as to obtain a target color video signal with a high frame rate, high resolution, and high dark light sensitivity under the condition of limited bandwidth of the image sensor.

[0021] In a possible implementation of the above fourth aspect, the colorization processing includes: First, perform position encoding processing on the first grayscale image frame of the first grayscale video signal and the first color image frame of the first color video signal synchronized with it in time, so that the color image blocks at different positions of the first color image frame correspond one by one to the grayscale image blocks at different positions of the first grayscale image frame. Then, based on the color information of the color image blocks, colorize the associated grayscale image blocks to obtain the second color image frame of the second color video signal.

[0022] The above colorization processing first associates each grayscale image block of the first grayscale image frame in the first grayscale video signal with each color image block of the first color image frame synchronized with the first grayscale image frame in the first color video signal through position encoding, and then colorizes each grayscale image block of the first grayscale image frame one by one based on the color image blocks of the first color image frame, with good colorization effect, so that the generated second color video signal has the color details of the first color video signal while retaining the high dark light sensitivity and high resolution of the first grayscale video signal.

[0023] In a possible implementation of the above fourth aspect, the frame interpolation processing includes: First, perform time encoding processing on the first color video signal and the second color video signal to determine multiple color image frames in the first color video signal corresponding to the second color image frame in the second color video signal, and the multiple color image frames include the first color image frame synchronized with the second color image frame in time. Then, perform motion detail processing on the second color image frame based on the first color image frame to obtain the first target color image frame of the target color video signal. Next, perform motion detail processing on the first target color image frame based on the third color image frame adjacent to the first color image frame to obtain the second target color image frame of the target color video signal. Then, perform motion detail processing on the second target color image frame based on the fourth color image frame adjacent to the third color image frame to obtain the third target color image frame of the target color video signal, and the fourth color image frame is different from the first color image frame.

[0024] The above-mentioned frame interpolation processing method first makes the first color video signal and the second color video signal temporally correlated through temporal coding, and then performs motion detail processing on the corresponding color image frames of the second color video signal based on the color image frames of the first color video signal through position coding, so as to obtain target color image frames with clearer motion details. Then, based on other frame color image frames of the first color video signal, motion detail processing is continued on the target color image frames, so as to obtain new target color image frames. By repeating this process, frame interpolation processing of the second color video can be achieved, so as to obtain target color image frames with clearer motion details in more frames, so that the finally generated target color video signal has the high frame rate of the first color video signal while retaining the high dark and light sensitivity, high resolution, color, and motion details of the second color video signal.

[0025] The fifth aspect of the present application provides a computer-readable storage medium, on which instructions are stored, and when the instructions are executed by a computer, the computer implements the image processing method in the above-mentioned fourth aspect and any possible implementation of the above-mentioned fourth aspect.

[0026] The sixth aspect of the present application provides an image sensor, including a memory and one or more processors. Among them, the memory is used to store instructions. When the instructions are executed by one or more processors, the processors execute the image processing method in the above-mentioned fourth aspect and any possible implementation of the above-mentioned fourth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1A Shows a three-dimensional view of a mobile phone in an embodiment of the present application;

[0028] Figure 1B Shows an exploded view of a mobile phone in an embodiment of the present application;

[0029] Figure 2 Shows a schematic distribution diagram of human eye cone cells and rod cells in an embodiment of the present application;

[0030] Figure 3 Shows a schematic pixel distribution diagram of an image sensor in an embodiment of the present application;

[0031] Figure 4 Shows an exemplary distribution manner of color pixel units and gray pixel units in the central region in an embodiment of the present application;

[0032] Figure 5 Shows an exemplary distribution manner of color pixel units and gray pixel units in the edge region in an embodiment of the present application;

[0033] Figure 6Shows an exemplary structure of a color pixel unit in an embodiment of the present application;

[0034] Figure 7 Shows an exemplary structure of a grayscale pixel unit in an embodiment of the present application;

[0035] Figure 8 Shows a structural block diagram of an image sensor in an embodiment of the present application;

[0036] Figure 9 Shows a flowchart of an image processing method in an embodiment of the present application;

[0037] Figure 10 Shows a color image frame of the first color video signal at time t1 in an embodiment of the present application;

[0038] Figure 11 Shows a grayscale image frame of the first grayscale video signal at time t1 in an embodiment of the present application;

[0039] Figure 12 Shows a target color image frame of the target color video signal at time t1 in an embodiment of the present application;

[0040] Figure 13 Shows a flowchart of coloring the first grayscale video signal based on the first color video signal in an embodiment of the present application;

[0041] Figure 14 Shows a schematic diagram of encoding the first color video signal and the first grayscale video signal in an embodiment of the present application;

[0042] Figure 15 Shows a flowchart of frame interpolation processing of the second color video signal based on the first color video signal in an embodiment of the present application;

[0043] Figure 16 Shows a schematic diagram of frame interpolation processing of the second color video signal based on the first color video signal in an embodiment of the present application;

[0044] Figure 17 Is another schematic block diagram of an image sensor provided in an embodiment of the present application;

[0045] Figure 18A Shows a schematic diagram of pixel distribution of several image sensors in some technical solutions;

[0046] Figure 18B Shows the pixel distribution of several image sensors in some technical solutions Figure 2 ;

[0047] Figure 18CSchematic diagrams of pixel distributions of several image sensors in some technical solutions are shown Figure 3 。 Detailed implementation manners

[0048] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] The present application provides an image sensor, a camera module and an electronic device including the image sensor, which can simultaneously achieve video acquisition with high frame rate, high resolution and high low-light sensitivity under the limited bandwidth of the image sensor, and have good shooting quality.

[0050] It can be understood that the electronic device provided by the present application may include, but is not limited to, any one of electronic devices with a camera module such as a mobile phone, a tablet computer, a laptop computer, a camera, an ultra-mobile personal computer (UMPC), a handheld computer, a touch TV, a walkie-talkie, a netbook, a POS machine, a personal digital assistant (PDA), a wearable device, a virtual reality device, an intelligent vehicle, an intelligent robot, an industrial device, etc. The present application does not make any limitation thereto. For the convenience of description, the mobile phone will be taken as an example for illustration below.

[0051] Figure 1A and Figure 1B show an exemplary structure of the mobile phone 1 in an embodiment of the present application. Among them, Figure 1A is a perspective view of the mobile phone 1, Figure 1B is an exploded view of the mobile phone 1. Referring to Figure 1A and Figure 1B , the mobile phone 1 includes a camera module 10, a main board 20, a housing 30 and a display screen 40. Among them, the camera module 10 and the main board 20 are located in the housing 30, and the display screen 40 covers the housing 30. Moreover, the camera module 10 is electrically connected to the display screen 40 through the main board 20 so that the display screen 40 can display the video or image captured by the camera module 10.

[0052] Specifically, the camera module 10 includes an image sensor 100 and a lens module 200. The image sensor 100 is disposed on the light-emitting side of the lens module 200. A light inlet hole 31 is formed in the housing 30 of the mobile phone 1, and the light inlet hole 31 is opposite to the lens module 200 of the camera module 10. The incident light outside the mobile phone 1 can pass through the light inlet hole 31 in the housing 30 and enter the lens module 200 of the camera module 10, and then exit the lens module 200 from the light-emitting side of the lens module 200. The light exiting the lens module 200 can reach the image sensor 100. The image sensor 100 receives the light transmitted by the lens module 200, converts the received light into an electrical signal, and then displays it on the display screen 40 in the form of a video or an image through the main board 20.

[0053] It can be understood that the above Figure 1A and Figure 1B only schematically show the setting manner of the camera module 10 of the mobile phone 1. In other embodiments, the camera module 10 of the mobile phone 1 may also have other setting manners. For example, in some embodiments, the camera module 10 may also be installed at other positions of the housing 10. For example, the camera module 10 may be installed at the upper right corner of the housing 30. Another example is that the camera module 30 may also be detachably installed on the housing 30 through an auxiliary component, and the auxiliary component can rotate or translate relative to the housing 30. Another example is that the camera module 10 may also be a front camera, and the present application does not limit this.

[0054] As described above, users hope that the camera module 10 of the mobile phone 1 can capture more realistic, smooth, and clear videos, and at the same time, they also hope that the camera module 10 of the mobile phone 1 can normally capture videos in low-light environments. Among them, high frame rate is beneficial to improving the fidelity and smoothness, high resolution is beneficial to improving the clarity, and high low-light sensitivity is beneficial to improving the low-light shooting performance of the mobile phone 1.

[0055] However, the bandwidth of current image sensors is limited. To achieve higher frame rate video recording, it is necessary to reduce the video resolution. For example, when collecting a standard dynamic range (SDR) video with a resolution of 4K and a frame rate of 60fps, each frame contains approximately 8M pixels. Each pixel (R pixel, G pixel, B pixel) records 8bit depth, so the amount of information contained in each frame is 192M bit, that is, 24MB. Coupled with the shooting frame rate of 60fps, the actual processing ability of the image sensor needs to reach 1440MB / s. Another example is that when collecting an SDR video with a resolution of 1080P and a frame rate of 480fps, the actual processing ability required by its image sensor also needs to be close to 1440MB / s.

[0056] If high frame rate and high resolution are both taken into account, for example, when collecting a video with a resolution of 4K and a frame rate of 480fps, it is necessary to improve the processing ability of the image sensor. The increase in data read / write pressure will cause the image sensor to heat up severely, resulting in a series of problems such as hardware thermal damage.

[0057] Therefore, it is currently difficult to simultaneously achieve high frame rate, high resolution, and high low-light sensitivity video acquisition under the limited bandwidth of the image sensor.

[0058] To solve the above problems, the present application provides an image sensor. Referring to the distribution of cone cells and rod cells in the human eye, color pixel units and grayscale pixel units are non-uniformly arranged in its pixel array. Color video signals are collected through the color pixel units, grayscale video signals are collected through the grayscale pixel units, and the grayscale video signals are subjected to coloring processing and frame interpolation processing based on the collected color video signals. Thus, under the limited bandwidth of the image sensor, a color video with high frame rate, high resolution, and high low-light sensitivity can be obtained. The following will be introduced in detail with reference to the accompanying drawings.

[0059] Figure 2 FIG. shows the distribution schematic diagram of cone cells and rod cells in the human eye in the embodiment of the present application. Referring to Figure 2 , in a typical human eye structure, the number of photoreceptor cells is approximately 137 million. Photoreceptor cells include cone cells (cones) and rod cells (rods). Among them, cone cells are used to perceive strong light and colors, with a number of approximately 7 million, accounting for 5%. Rod cells are used to perceive weak light and grayscales, with a number of approximately 130 million, accounting for 95%. And there are a high density of cone cells and a small number of rod cells in the center of the retina, and a high density of rod cells and a relatively low density of cone cells in the peripheral region of the retina.

[0060] Figure 3 FIG. shows the pixel distribution schematic diagram of the image sensor 100 in the embodiment of the present application. Referring to Figure 2 and combining with Figure 3 , according to the above distribution characteristics of cone cells and rod cells, the image sensor 100 provided by the present application includes a pixel array F1. Exemplarily, the pixel array F1 can be a 15×15 rectangular array. The pixel array F1 includes a plurality of color pixel units Pc (for example, Figure 3 shown as the dark regions in Figure 3 shown as the white regions in

[0061] Among them, the distribution density of the color pixel units Pc in the pixel array F1 is less than or equal to the distribution density of the grayscale pixel units Pw in the pixel array F1. Moreover, the distribution density of the color pixel units Pc in the central region S1 of the pixel array F1 is greater than or equal to the distribution density of the grayscale pixel units Pw in the central region S1, and the distribution density of the color pixel units Pc in the edge region S2 of the pixel array F1 is less than the distribution density of the grayscale pixel units Pw in the edge region S2. Herein, the central region S1 may be a rectangular region surrounded by the dotted line in the middle of the pixel array F1, and the edge region S2 may be an annular region surrounded by the dotted line in the middle of the pixel array F1 and the dotted line at the edge of the pixel array F1. It can also be understood that the edge region S2 is the other region on the pixel array F1 except the central region S1.

[0062] That is to say, the number of the color pixel units Pc in the pixel array F1 is less than or equal to the number of the grayscale pixel units Pw in the pixel array F1. Moreover, in the central region S1, the number of the color pixel units Pc is greater than the number of the grayscale pixel units Pw, and in the edge region S2, the number of the color pixel units Pc is less than the number of the grayscale pixel units Pw.

[0063] For example, referring to Figure 3 , the 15×15 pixel array F1 includes 28 color pixel units Pc and 197 grayscale pixel units Pw. Among them, the distribution density of the color pixel units Pc in the pixel array F1 is about 0.12, and the distribution density of the grayscale pixel units Pw in the pixel array F1 is about 0.88. Moreover, in the central region S1, the number of the color pixel units Pc is 20, and the distribution density is 0.8; the number of the grayscale pixel units Pw is 5, and the distribution density is 0.2; in the edge region S2, the number of the color pixel units Pc is 8, and the distribution density is 0.04; the number of the grayscale pixel units Pw is 192, and the distribution density is 0.96.

[0064] Since the overall proportion of the color pixel units Pc in the pixel array F1 is relatively low and the overall proportion of the grayscale pixel units Pw in the pixel array F1 is relatively high, therefore, a color video with a relatively low resolution (e.g., 720p) can be captured by the color pixel units Pc, and a grayscale video with a relatively high resolution (e.g., 4K) can be captured by the grayscale pixel units Pw.

[0065] The response frame rate of the color pixel unit Pc is higher than that of the grayscale pixel unit Pw. Herein, the response frame rate refers to the number of images that a pixel unit can update per unit time. The higher the response frame rate of a pixel unit, the higher frame rate of the video signal that can be collected through this pixel unit. That is to say, a high-frame-rate (e.g., 960 fps) color video can be collected through the color pixel unit Pc, and a low-frame-rate (e.g., 60 fps) grayscale video can be collected through the grayscale pixel unit Pw.

[0066] In the above image sensor 100, the distribution rules of the color pixel unit Pc and the grayscale pixel unit Pw are similar to the distribution rules of the cone cells and rod cells of the human eye. Therefore, whether in a well-lit environment or in a low-light environment, a picture close to what the human eye sees can be obtained. The image sensor 100 can collect a high-frame-rate (e.g., 960 fps), low-resolution (e.g., 720p) color video through the color pixel unit Pc, and collect a low-frame-rate (e.g., 60 fps), high-resolution (e.g., 4K), high low-light sensitivity grayscale video through the grayscale pixel unit Pw. Since the number of color pixel units Pc in the pixel array F1 is relatively small, collecting a high-frame-rate video through the color pixel unit Pc will not significantly increase the data read / write pressure on the image sensor 100. Moreover, by performing color processing and frame interpolation on the grayscale video based on the color video, high-frame-rate (e.g., 960 fps), high-resolution (e.g., 4K), and high low-light sensitivity video collection can be simultaneously achieved under the limited bandwidth of the image sensor 100, and the shooting effect is good.

[0067] Taking the example of collecting a video with a resolution of 4K and a frame rate of 240 fps through the image sensor 100, the data volume is only 771 MB / s. That is to say, the processing capacity of the image sensor 100 only needs to reach 771 MB / s. For other mobile phone solutions, only collecting a video with a resolution of 4K and a frame rate of 60 fps, the original data volume is as high as 1490 MB / s. That is to say, the processing capacity of its image sensor needs to reach 1490 MB / s, and the data read / write pressure on the image sensor is relatively large, resulting in serious heat generation of the image sensor, and then a series of problems such as hardware thermal damage. Compared with the current solutions, the solution provided in this application can simultaneously achieve high-frame-rate, high-resolution, and high low-light sensitivity video collection under a lower bandwidth.

[0068] In some embodiments of the present application, the ratio of the distribution density of the color pixel units Pc in the pixel array F1 to the distribution density of the grayscale pixel units Pw in the pixel array F1 is from 1:20 to 1:1. For example, the ratio of the distribution density of the color pixel units Pc in the pixel array F1 to the distribution density of the grayscale pixel units Pw in the pixel array F1 is 1:20, that is to say, among every 21 groups of pixel units, there is 1 group of color pixel units Pc and 20 groups of grayscale pixel units Pw. Another example, the ratio of the distribution density of the color pixel units Pc in the pixel array F1 to the distribution density of the grayscale pixel units Pw in the pixel array F1 is 1:19, that is to say, among every 20 groups of pixel units, there is 1 group of color pixel units Pc and 19 groups of grayscale pixel units Pw.

[0069] In this way, the distribution rules of the color pixel units Pc and the grayscale pixel units Pw in the pixel array F1 can be made closer to the distribution rules of cone cells and rod cells in the human eye, so that the video image captured by the image sensor 100 is closer to the high-resolution, high low-light sensitivity color video image seen by the human eye.

[0070] In some embodiments of the present application, in the central region S1, the ratio of the distribution density of the color pixel units Pc to the distribution density of the grayscale pixel units Pw is from 5:1 to 1:1. For example, Figure 4 shows an exemplary distribution manner of the color pixel units Pc and the grayscale pixel units Pw in the central region S1 in the embodiments of the present application. Refer to Figure 4 , the central region S1 includes 4 groups of pixel units, and each group of pixel units contains 4 pixels. Among them, the number of the color pixel units Pc and the number of the grayscale pixel units Pw are both 2 groups. The 2 groups of color pixel units Pc and the 2 groups of grayscale pixel units Pw are distributed diagonally. In the central region S1, the ratio of the distribution density of the color pixel units Pc to the distribution density of the grayscale pixel units Pw is 1:1. However, the present application is not limited thereto. For example, in the central region S1, the ratio of the distribution density of the color pixel units Pc to the distribution density of the grayscale pixel units Pw can also be 5:1, 4:1, 3:1, etc.

[0071] In this way, the distribution rules of the color pixel units Pc and the grayscale pixel units Pw in the central region S1 can be made closer to the distribution rules of cone cells and rod cells in the human eye, so that the video image captured by the image sensor 100 is closer to the high-resolution, high low-light sensitivity color video image seen by the human eye.

[0072] In some embodiments of the present application, in the edge region S2, the ratio of the distribution density of the color pixel units Pc to the distribution density of the grayscale pixel units Pw is from 1:30 to 1:1. For example, Figure 5An exemplary distribution pattern of the color pixel units Pc and the grayscale pixel units Pw at the edge region S2 in the embodiments of the present application is shown. Refer to Figure 5 , the edge region S2 includes 25 groups of pixel units, and each group of pixel units includes 4 pixels. Among them, the number of color pixel units Pc is 1 group, and the number of grayscale pixel units Pw is 24 groups. The 24 groups of grayscale pixel units Pw surround the 1 group of color pixel units Pc. At the edge region S2, the ratio of the distribution density of the color pixel units Pc to the distribution density of the grayscale pixel units Pw is 1:24. However, the present application is not limited thereto. For example, at the edge region S2, the ratio of the distribution density of the color pixel units Pc to the distribution density of the grayscale pixel units Pw can also be 1:30, 1:29, 1:28, etc.

[0073] In this way, the distribution pattern of the color pixel units Pc and the grayscale pixel units Pw at the edge region S2 can be made closer to the distribution pattern of cone cells and rod cells in the human eye, so that the video image collected by the image sensor 100 is closer to the high-resolution and high low-light sensitivity color video image seen by the human eye.

[0074] In some embodiments of the present application, from the central region S1 to the edge region S2, the distribution density of the color pixel units Pc gradually decreases, and the distribution density of the grayscale pixel units Pw gradually increases. In this way, the distribution pattern of the color pixel units Pc and the grayscale pixel units Pw in the pixel array F1 can be made closer to the distribution pattern of cone cells and rod cells in the human eye, so that the video image collected by the image sensor 100 is closer to the high-resolution and high low-light sensitivity color video image seen by the human eye.

[0075] Figure 6 An exemplary structure of the color pixel unit Pc in the embodiments of the present application is shown. Refer to Figure 6 and in combination with Figure 3 , in some embodiments of the present application, the color pixel unit Pc may include four pixels, namely one red (R) pixel, two green (G) pixels, and one blue (B) pixel, and the four pixels may be a 2×2 rectangular array. That is to say, the color pixel unit Pc is of the RGGB structure.

[0076] Among them, the R pixel may include a first color filter 110R and a first photoelectric converter 120R which are oppositely arranged. The first color filter 110R is used to filter out the light other than the red light Lr in the incident light L (for example, the blue light Lb and the green light Lg). The first photoelectric converter 120R is used to convert the red light Lr passing through the first color filter 110R into an electrical signal, so as to store the image information. Among them, the first photoelectric converter 120R may be a photosensitive device such as a photodiode.

[0077] The G pixel may include a second color filter 110G and a second photoelectric converter 120G which are oppositely arranged. The second color filter 110G is used to filter out the light other than the green light Lg in the incident light L (for example, the blue light Lb and the red light Lr). The second photoelectric converter 120G is used to convert the green light Lg passing through the second color filter 110G into an electrical signal, so as to store the image information.

[0078] The B pixel may include a third color filter 110B and a third photoelectric converter 120B which are oppositely arranged. The third color filter 110B is used to filter out the light other than the blue light Lb in the incident light L (for example, the green light Lg and the red light Lr). The third photoelectric converter 120B is used to convert the blue light Lb passing through the third color filter 110B into an electrical signal, so as to store the image information.

[0079] It can be understood that the above Figure 6 only schematically shows the structure of the color pixel unit Pc. In other embodiments, the color pixel unit Pc may also have other structural forms. For example, the color pixel unit Pc may also be of the RYYB structure, that is to say, the color pixel unit Pc includes four pixels, namely one G, two yellow (Y) pixels and one B pixel. Another example is that the color pixel unit Pc may also be of the RGB structure, that is to say, the color pixel unit Pc includes three pixels, namely one R pixel, one G pixel and one B pixel, and the three pixels may be a 1×3 rectangular array. The present application does not limit this.

[0080] Figure 7 An exemplary structure of the grayscale pixel unit Pw in the embodiment of the present application is shown. Refer to Figure 7 and in combination with Figure 3 , in some embodiments of the present application, the number of pixels included in the grayscale pixel unit Pw may be the same as the number of pixels included in the color pixel unit Pc. For example, Figure 6 the shown color pixel unit Pc includes four pixels, and the four pixels are a 2×2 rectangular array. Correspondingly, Figure 7The grayscale pixel unit Pw shown may also include four white (W) pixels, and the four W pixels also form a 2×2 rectangular array. For another example, in other embodiments, when the color pixel unit Pc includes three pixels (for example, one R pixel, one G pixel, and one B pixel), and the three pixels form a 1×3 rectangular array, the grayscale pixel unit Pw may also include three W pixels, and the three W pixels also form a 1×3 rectangular array.

[0081] Continuing to refer to Figure 7 , each W pixel includes a fourth photoelectric converter 120W. The full-color incident light L is incident on the fourth photoelectric converter 120W without passing through a color filter. The fourth photoelectric converter 120W can convert the incident light L into an electrical signal, thereby realizing the storage of image information. Since there is no need to filter the incident light L through a specific color filter, the loss of the incident light L during the process of being incident on the fourth photoelectric converter 120W is relatively small, so that the light input can be effectively improved, and the grayscale pixel unit Pw has high sensitivity to low light, which is beneficial to imaging in low-light scenarios.

[0082] As described above, the response frame rate of the color pixel unit Pc is higher than that of the grayscale pixel unit Pw. The response frame rate of the pixel unit is related to the response speed of the pixel unit (that is, the time from when the pixel unit receives the optical signal to when it generates an electrical signal). Therefore, in some embodiments of the present application, the response speed of the color pixel unit Pc is greater than that of the grayscale pixel unit Pw. Exemplarily, by adjusting the structure, material of the color pixel unit Pc or using a higher driving voltage, etc., the time of charge transfer and charge accumulation in the color pixel unit Pc can be reduced, thereby improving the response speed of the color unit Pc.

[0083] In some embodiments of the present application, the image sensor 100 may collect each frame of image signal in a rolling shutter or global shutter manner.

[0084] Figure 8 shows a structural block diagram of the image sensor 100 in an embodiment of the present application. Refer to Figure 8, in some embodiments of the present application, the image sensor 100 may further include an image processing unit 200. The image processing unit 200 is electrically connected to the color pixel unit Pc and the grayscale pixel unit Pw, and is configured to: obtain a high-frame-rate (e.g., 960 fps), low-resolution (e.g., 720p) color video signal, obtain a low-frame-rate (e.g., 60 fps), high-resolution (e.g., 4K), high low-light sensitivity grayscale video signal output by the grayscale pixel unit Pw, and perform colorization processing and frame interpolation processing on the grayscale video signal based on the obtained color video signal, so as to obtain a high-frame-rate (e.g., 960 fps), high-resolution (e.g., 4K), and high low-light sensitivity color video signal under the condition of limited bandwidth of the image sensor 100.

[0085] In some embodiments of the present application, the image processing unit 200 may be electrically connected to the color pixel unit Pc through the color pixel circuit 300C and electrically connected to the grayscale pixel unit Pw through the grayscale pixel circuit 300W.

[0086] Among them, the color pixel circuit 300C is configured to output a high-frame-rate (e.g., 960 fps), low-resolution (e.g., 720p) color video signal collected by the color pixel unit Pc to the image processing unit 200, and the grayscale pixel circuit 300W is configured to output a low-frame-rate (e.g., 60 fps), high-resolution (e.g., 4K), high low-light sensitivity grayscale video signal collected by the grayscale pixel unit Pw to the image processing unit 200. Then, the image processing unit 200 performs colorization and frame interpolation processing on the grayscale video signal based on the color video signal, so as to obtain a high-frame-rate (e.g., 960 fps), high-resolution (e.g., 4K), and high low-light sensitivity color video signal under the condition of limited bandwidth of the image sensor 100.

[0087] Based on this, an embodiment of the present application further provides an image processing method, which is applicable to the above-mentioned image sensor 100.

[0088] Figure 9 shows a flowchart of the image processing method in the embodiment of the present application. As Figure 9 shown, the image processing method provided by the present application specifically includes the following steps:

[0089] S110: The image processing unit 200 obtains a first color video signal and a first grayscale video signal, the first color video signal has a first frame rate and a first resolution, the first grayscale video signal has a second frame rate and a second resolution, wherein the first frame rate is greater than the second frame rate, and the first resolution is less than the second resolution.

[0090] It can be understood that the first color video signal with the first frame rate and the first resolution can be the high-frame-rate and low-resolution color video signal described above. The first grayscale video signal with the second frame rate and the second resolution can be the low-frame-rate and high-resolution grayscale video signal described above.

[0091] For ease of description, the following takes the first frame rate of 960 fps, the second frame rate of 60 fps, the first resolution of 720p, and the second resolution of 4K as examples for description. However, it can be understood that in other embodiments, the first frame rate, the first resolution, the second frame rate, and the second resolution can also be other values, and the present application does not make specific limitations thereon.

[0092] Continuing to refer to Figure 8 , in some embodiments of the present application, the image processing unit 200 can obtain the first color video signal with a frame rate of 960 fps and a resolution of 720p collected by the color pixel unit Pc through the color pixel circuit 300C, and obtain the first grayscale video signal with a frame rate of 60 fps and a resolution of 4K collected by the grayscale pixel unit Pw through the grayscale pixel circuit 300W. It can be understood that the first color video signal includes a plurality of color image frames at different times, and the first grayscale video signal includes a plurality of grayscale image frames at different times. For ease of observation, the following takes the image frames of the first color video signal and the first grayscale video signal at the same moment as an example to introduce the characteristics of the first color video signal and the first grayscale video signal.

[0093] Figure 10 shows the color image frame of the first color video signal at time t1 in the embodiment of the present application. Referring to Figure 10 , the first color video signal has color details. It can be understood that although Figure 10 does not show the specific colors, different colors are indicated by text. For example, the clothes and hat are blue, the baseball is white, etc. Secondly, due to the high frame rate of the first color video signal, the movement process of the baseball can be better recorded. For example, as shown in Figure 10 , the position of the baseball can be clearly observed. Finally, due to the low resolution of the first color video signal, the picture is relatively blurred. For example, the human figure outline, facial details, baseball outline and details are all relatively blurred.

[0094] Figure 11 shows the grayscale image frame of the first grayscale video signal at time t1 in the embodiment of the present application. Referring to Figure 11 , the first grayscale video signal has high dark-light sensitivity and good brightness details. Secondly, due to the low frame rate of the first grayscale video signal, the movement process of the baseball is relatively blurred. For example, as shown in Figure 11As shown, there are motion afterimages of the baseball, and the position of the baseball cannot be accurately observed. Finally, since the first grayscale video signal has a high resolution, the picture clarity is good. For example, the outlines and facial details of people are relatively clear.

[0095] S120: The image processing unit 200 performs coloring processing on the first grayscale video signal based on the first color video signal to obtain a second color video signal, and the second color video signal has a second frame rate and a second resolution.

[0096] In some embodiments of the present application, performing coloring processing on a first grayscale video signal with a frame rate of 60fps and a resolution of 4K based on a first color video signal with a frame rate of 960fps and a resolution of 720p can enable the first grayscale video signal to have the color details of the first color video signal, thereby obtaining a second color video signal. Among them, the frame rate of the second color video signal is 60fps and the resolution is 4K. In this way, while the second color video signal retains the high dark-light sensitivity and 4K high resolution of the first grayscale video signal, it has the color details of the first color video signal.

[0097] S130: The image processing unit 200 performs frame interpolation processing on the second color video signal based on the first color video signal to obtain a target color video signal, and the target color video signal has a first frame rate and a second resolution.

[0098] In some embodiments of the present application, performing frame interpolation processing on a second color video signal with a frame rate of 60fps and a resolution of 4K based on a first color video signal with a frame rate of 960fps and a resolution of 720p can enable the second color video signal to have the frame rate of the first color video signal, thereby obtaining a target color video signal. Among them, the frame rate of the target color video signal is 960fps and the resolution is 4K. In this way, while the target color video signal retains the high dark-light sensitivity, 4K high resolution and color details of the second color video signal, it has the 960fps high frame rate of the first color video signal.

[0099] For example, Figure 12 shows the target color image frame of the target color video signal at time t1 in the embodiment of the present application. Refer to Figure 12 and in combination with Figure 10 and Figure 11 , the target color image signal has high dark-light sensitivity and good brightness details. Secondly, the target color image signal has better color details. And because the target color image signal has a high resolution, the picture clarity is relatively high. For example, the outlines and details of people and baseballs are relatively clear. Since the target color image signal has a high frame rate, the motion process of the baseball can be better recorded. For example Figure 12As shown, the position of the baseball can be clearly observed.

[0100] The above image processing method obtains a first color video signal with a high frame rate and low resolution output by the color pixel unit Pc and a first grayscale video signal with a low frame rate and high resolution output by the grayscale pixel unit Pw through the image processing unit 200 of the image sensor 100, and performs coloring processing and frame interpolation processing on the first grayscale video signal based on the first color video signal, so that a target color video signal with a high frame rate, high resolution, and high low-light sensitivity can be obtained under the condition of limited bandwidth of the image sensor 100.

[0101] Figure 13 The flowchart of performing coloring processing on the first grayscale video signal based on the first color video signal in the embodiment of the present application is shown.

[0102] Reference Figure 13 , step S120 may include the following sub-steps:

[0103] S121: Perform position encoding processing on the first grayscale image frame in the first grayscale video signal and the first color image frame in the first color video signal, so that the color image blocks at different positions of the first color image frame correspond one-to-one with the grayscale image blocks at different positions of the first grayscale image frame, where the first grayscale image frame and the first color image frame are time-synchronized.

[0104] Figure 14 The schematic diagram of performing encoding processing on the first color video signal and the first grayscale video signal in the embodiment of the present application is shown. For ease of description, the time-synchronized first grayscale image frame W1 and first color image frame C1 are taken as examples for introduction. It can be understood that two time-synchronized image frames refer to two video image frames aligned on the time axis. For example, the first grayscale image frame W1 is the image frame of the first grayscale video signal at time t1, and the first color image frame C1 is the image frame of the first color video signal at time t1.

[0105] In some embodiments of the present application, the first color image frame C1 and the first grayscale image frame W1 can be position-encoded by a position encoder, so that the color image blocks at different positions of the first color image frame C1 correspond one-to-one with the grayscale image blocks at different positions of the first grayscale image frame W1.

[0106] For example, first, the first grayscale image frame W1 is divided into a plurality of non-overlapping grayscale image blocks, and the first color image frame C1 is respectively divided into a plurality of non-overlapping color image blocks. The number of the plurality of color image blocks is the same as that of the plurality of grayscale image blocks. Then, feature extraction is performed on each grayscale image block and color image block to obtain the position feature representing the image block. Next, based on relationships such as the similarity or position distance of the position features, a corresponding relationship is established between each grayscale image block of the first grayscale image frame W1 and each color image block of the first color image frame C1. For example, the position feature may include the row and column information of the image block. Then, the grayscale image block located in the first row and first column of the first grayscale image frame W1 corresponds to the color image block located in the first row and first column of the first color image frame C1. Another example is that the grayscale image block located in the first row and second column of the first grayscale image frame W1 corresponds to the color image block located in the first row and second column of the first color image frame C1.

[0107] In some embodiments of the present application, the position encoder may be a masked transform position encoder, an absolute position encoder, a relative position encoder, a grid position encoder, or a self-attention position encoder, etc.

[0108] S122: Color the associated grayscale image block based on the color information of the color image block to obtain the second color image frame of the second color video signal.

[0109] According to the above embodiments, the plurality of grayscale image blocks of the first grayscale image frame W1 and the plurality of color image blocks of the first color image frame C1 are in one-to-one correspondence. In this way, the color information of each color image block can be used to color the grayscale image block associated with each color image block. For example, a trained neural network model is used to extract the color features of the color image block, and then the color features of the color image block and the features of the corresponding grayscale image block are fused to generate a new color image block, and then the second color image frame C2 is obtained.

[0110] It can be understood that the first grayscale video signal includes grayscale image frames at multiple different times. The processing method of each grayscale image frame can refer to the processing method of the above first grayscale image frame, which will not be elaborated here. By coloring all the grayscale image frames of the first grayscale video signal, a second color video signal can be obtained.

[0111] For the above coloring process, first, the gray image blocks of the first gray image frame in the first gray video signal are associated with the color image blocks of the first color image frame synchronized with the first gray image frame in time in the first color video signal through position encoding. Then, based on the color image blocks of the first color image frame, the gray image blocks of the first gray image frame are colored one by one, and the coloring effect is good. As a result, the generated second color video signal has the color details of the first color video signal while retaining the high dark and light sensitivity and 4K high resolution of the first gray video signal.

[0112] Figure 15 The flowchart shows the process of performing frame interpolation on the second color video signal based on the first color video signal in an embodiment of the present application. Figure 16 The schematic diagram shows the process of performing frame interpolation on the second color video signal based on the first color video signal in an embodiment of the present application. Refer to Figure 15 and in combination with Figure 16 , step S130 may include the following sub-steps:

[0113] S131: Perform time encoding processing on the first color video signal and the second color video signal to determine multiple color image frames in the first color video signal corresponding to the second color image frame in the second color video signal. The multiple color image frames include the first color image frame synchronized with the second color image frame in time.

[0114] In some embodiments of the present application, the time encoding processing of the first color video signal and the second color video signal can be performed through a time encoder. For example, feature extraction can be performed on the first color video signal and the second color video signal to obtain the time stamps of each color image frame in the first color video signal and the time stamps of each color image frame in the second color video signal. Taking the second color image frame C2 in the second color video signal as an example, based on the time stamp of the second color image frame C2, it is aligned with the first color image frame C1 having the same time stamp in the first color video signal, and then the first color image frame C1 and a preset number of color image frames before and after it (for example, C31, C32, C41, C42) are used as the multiple color image frames in the first color video signal corresponding to the second color image frame C2. It can be understood that C1, C31, C32, C41, C42 are used to implement the frame interpolation processing of C2.

[0115] It can be understood that for each color image frame in the second color video signal, the corresponding multiple color image frames in the first color video signal are determined in the same manner as the above-mentioned second color image frame C2. The multiple color image frames corresponding to different color image frames in the second color video signal in the first color video signal may not overlap. For example, the multiple color image frames corresponding to the color image frame C2 in the second color video signal in the first color video signal are C1, C31, C32, C41, and C42, and the multiple color image frames corresponding to the color image frame C3 in the second color video signal in the first color video signal are C51, C52, C53, C54, and C55. Among them, C51 and C42 are adjacent on the time axis, or there is a certain number of frames between C51 and C42. The multiple color image frames corresponding to different color image frames in the second color video signal in the first color video signal may also partially overlap. For example, the multiple color image frames corresponding to the color image frame C2 in the second color video signal in the first color video signal are C1, C31, C32, C41, and C42, and the multiple color image frames corresponding to the color image frame C3 in the second color video signal in the first color video signal are C42, C51, C52, C53, and C54.

[0116] S132: Perform motion detail processing on the second color image frame based on the first color image frame to obtain the first target color image frame of the target color video signal.

[0117] In some embodiments of the present application, position encoding may be performed on the first color image frame C1 and the second color image frame C2 so that the color image blocks at different positions of the first chromaticity image frame C1 correspond one-to-one to the color image blocks at different positions of the second color image frame C2. In this way, multiple color image blocks of the first chromaticity image frame C1 can be used to supplement the motion details of the color image blocks of the associated second color image frame C2 to obtain the first target color image frame C100 with clearer motion details.

[0118] It can be understood that the specific implementation manners of the above-mentioned position encoding and motion detail supplement of the first color image frame C1 and the second color image frame C2 are substantially the same as the specific implementation manners of the above-mentioned position encoding and coloring of the first grayscale image frame W1 and the first color image frame C1. Specifically, reference can be made to step S121 and step S122, which will not be elaborated here.

[0119] In this way, the problem of image blurring caused by a low frame rate in the second color video signal can be effectively avoided, so that the generated target color video signal can better restore the actual motion details.

[0120] S133: Perform motion detail processing on the first target color image frame based on the third color image frame adjacent to the first color image frame to obtain the second target color image frame of the target color video signal.

[0121] It can be understood that the multiple color image frames corresponding to the second color image frame determined in S131 in the first color video signal include the third color image frame adjacent to the first color image frame.

[0122] According to the above embodiments, the third color image frame may include the color image frame C31 located in the previous frame of the first color image frame C1 and the color image frame C32 located in the next frame of the first color image frame C1.

[0123] Motion detail processing can be performed on the first target color image frame C10 based on the color image frame C31, so as to obtain the color image frame C210 of the target color video signal (as an example of the second target color image frame). Motion detail processing can be performed on the first target color image frame C10 based on the color image frame C32, so as to obtain the color image frame C220 of the target color video signal (as another example of the second target color image frame). The specific implementation manner is substantially the same as the specific implementation manner of performing coloring processing on the first grayscale image frame W1 based on the first color image frame C1. Specifically, reference can be made to step S121 and step S122, which will not be elaborated here.

[0124] In this way, frame interpolation processing of the second color video signal can be realized, so as to obtain the target color image video signal with more frame images. For example, for the second color image frame C2, the target color image video signal obtained after executing step S132 includes three target color image frames (for example, C100, C210, and C220). At the same time, it can also make the target color image video signal have clearer motion details.

[0125] S134: Perform motion detail processing on the second target color image frame based on the fourth color image frame adjacent to the third color image frame to obtain the third target color image frame of the target color video signal.

[0126] It can be understood that the multiple color image frames corresponding to the second color image frame determined in S131 in the first color video signal include the third color image frame adjacent to the first color image frame, and the fourth color image frame adjacent to the third color image frame. That is, the fourth color image frame, the third color image frame, and the first color image frame are adjacent in sequence on the time axis.

[0127] According to the above embodiments, the third color image frame may include a color image frame C31 and a color image frame C32. Among them, the fourth color image frame adjacent to the color image frame C31 may be the color image frame C41 located in the previous frame of the color image frame C31; the fourth color image frame adjacent to the color image frame C32 may be the color image frame C42 located in the next frame of the color image frame C32.

[0128] Then, motion detail processing is performed on the color image frame C21 based on the color image frame C41, so as to obtain the color image frame C310 of the target color video signal (as an example of the third target color image frame). Motion detail processing is performed on the color image frame C22 based on the color image frame C42, so as to obtain the color image frame C320 of the target color video signal (as another example of the third target color image frame). The specific implementation manner is substantially the same as the specific implementation manner of coloring the first grayscale image frame W1 based on the first color image frame C1, and reference may specifically be made to step S121 and step S122, which will not be elaborated here.

[0129] In this way, by looping, frame interpolation processing can be performed on each color image frame in the second color video signal based on the multiple color image frames corresponding to it in the first color video signal, so that the number of image frames of the finally generated target color image video signal is greatly increased compared to the number of image frames of the second color video signal. Moreover, the number of image frames of the target color image video signal can be the same as the number of image frames of the first color video signal. For example Figure 16 As shown, after frame interpolation processing is performed on the second color video signal based on five color image frames (for example, C1, C31, C32, C41, C42) of the first color video signal, the obtained target color image video signal includes five color image frames (for example, C100, C210, C220, C310, C320). At the same time, it can also make the target color image video signal have clearer motion details.

[0130] It can be understood that the frame interpolation processing method for other color image frames in the second color video signal may refer to the above frame interpolation processing method, which will not be elaborated here. By processing all the color image frames in the second color video signal, a target color video signal can be obtained, and the target color video signal has the same frame rate as the first color video signal.

[0131] The above-mentioned frame interpolation processing method first makes the first color video signal and the second color video signal temporally correlated through temporal encoding, and then performs motion detail processing on the corresponding color image frames of the second color video signal based on the color image frames of the first color video signal through position encoding, so as to obtain target color image frames with clearer motion details. Then, based on other frame color image frames of the first color video signal, continue to perform motion detail processing on the target color image frames, so as to obtain new target color image frames. By repeating this process, frame interpolation processing of the second color video can be achieved, so as to obtain target color image frames with clear motion details in more frames, so that the finally generated target color video signal has the high dark-light sensitivity, 4K high resolution, color, and motion details of the second color video signal, while having a high frame rate of 960fps of the first color video signal.

[0132] Figure 17 It is another schematic block diagram of the image sensor 100 provided by the embodiments of the present application. As Figure 17 shown, the image sensor 100 may include at least one processor 101, which can be used to implement the functions of image processing in the above method embodiments. For specific details, please refer to the detailed description in the method examples, and details will not be elaborated here.

[0133] The image sensor 100 may further include a memory 102 for storing program instructions and / or data. The memory 102 is coupled to the processor 101. The coupling in the present application is an indirect coupling or communication connection between devices, units or modules, which can be electrical, mechanical or other forms, and is used for information interaction between devices, units or modules. The processor 101 may cooperate with the memory 102. The processor 101 may execute the program instructions stored in the memory 102. At least one of the at least one memory 102 may be included in the processor 101.

[0134] The image sensor 100 may further include a communication interface 103 for communicating with other devices through a transmission medium, so that the devices in the image sensor 100 can communicate with other devices. The communication interface 103 may be, for example, a transceiver, an interface, a bus, a circuit or a device capable of realizing transceiver functions. The processor 101 may use the communication interface 103 to transmit and receive data and / or information, and is used to implement the image processing method described in the above embodiments.

[0135] In the present application, the specific connection medium between the above-mentioned processor 101, memory 102 and communication interface 103 is not limited. In the present application Figure 17 it is assumed that the processor 101, memory 102 and communication interface 103 are connected through a bus 104. The bus 104 is in Figure 17The connection in the [device] is represented by a thick line. The connection methods between other components are only for illustrative purposes and are not limited thereto. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 17 it is only represented by a thick line in the [device], but it does not mean that there is only one bus or one type of bus.

[0136] In the embodiments of the present application, the processor can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps and logic block diagrams disclosed in the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the present application can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0137] According to the method provided by the present application, the present application also provides a computer-readable storage medium, which stores program codes. When the program codes run on a computer, the computer is enabled to execute the image processing method in the above embodiments.

[0138] The technical solution provided by the present application can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a terminal device or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a server or a data center to another website, a computer, a server or a data center by wire, such as coaxial cable, optical fiber, digital subscriber line (DSL) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, etc. that includes one or more available media integrated. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a digital video disc (DVD)), or a semiconductor medium, etc.

[0139] It should be noted that this embodiment is an exemplary illustration of the technical solution of the present application, and those skilled in the art can make other deformations. For example, in this embodiment, the first color video has a high frame rate, and the first grayscale video has a low frame rate. In other embodiments, the first color video signal may also have a low frame rate, and the first grayscale video signal has a high frame rate, and then the first color video signal and the first grayscale video signal are fused to obtain a target color video with a high frame rate, high resolution, and high low-light sensitivity. The specific implementation method is substantially the same as the specific implementation method of fusing the high-frame-rate first color video signal and the low-frame-rate first grayscale video signal, and specific reference can be made to Figure 9 , Figure 13 and Figure 15 and their related descriptions, which will not be elaborated here.

[0140] The following introduces the comparison between the image sensor and the image processing method provided by the embodiments of the present application and the image sensors and image processing methods in some other technical solutions.

[0141] In some technical solutions, by simultaneously using pixels that can sense color and pixels that sense grayscale in a single image sensor, the light input of the image sensor is increased, thereby improving the low-light sensitivity and further improving the low-light shooting performance.

[0142] Figures 18A to 18C shows a schematic diagram of the pixel distribution of several image sensors 100a in some technical solutions. Refer to Figures 18A to 18C , the image sensor 100a includes evenly arranged R pixels, G pixels, B pixels, and W pixels. That is, the image sensor 100a has an RGBW structure. Among them, the W pixels are used to sense weak light and grayscale, so as to increase the light input of the image sensor, which is beneficial to improving the low-light sensitivity and further improving the low-light shooting performance.

[0143] However, since the electrical properties of the W pixels and the RGB pixels are the same, the frame rate of video acquisition cannot be improved, and only the low-light sensitivity can be improved. In addition, the above solution needs to use a difference algorithm to restore the RGB color of the image. Therefore, the RGBW pixels can only be evenly arranged in a fixed arrangement pattern. When it is necessary to further increase the light input, it is necessary to further increase the proportion of the W pixels, which will cause the image to be unable to be restored and the shooting effect is limited.

[0144] In some other technical solutions, two camera modules are used to synchronously acquire a color image and a grayscale image respectively, and then the color image and the grayscale image are fused. Among them, the camera module for acquiring the color image is used to sense the color, and the camera module for acquiring the grayscale image is used to sense the details of the low-light scene, which is beneficial to improving the low-light sensitivity and further improving the low-light shooting performance.

[0145] However, the above solution can only improve the low-light sensitivity and cannot achieve an increase in the frame rate of video capture. Moreover, simultaneous capture of dual cameras will cause the data volume of the image signal to increase exponentially, making it even more difficult to apply the above solution to the field of video capture. Additionally, there is usually a misalignment between the two images captured by the dual cameras. Therefore, the dual-camera images usually cannot be directly superimposed and require additional algorithms for spatial position compensation, which will to some extent reduce the picture accuracy and result in poor shooting effects.

[0146] In some other technical solutions, a large-sized image sensor is used for shooting. The large-sized sensor can effectively increase the photosensitive area of each pixel, thereby improving the low-light performance. Moreover, it can accommodate more pixels to achieve a higher resolution.

[0147] However, the above solution can only improve the low-light sensitivity and resolution and cannot increase the frame rate of video capture. In addition, a larger-sized image sensor requires a lens module with a larger aperture, which will significantly increase the size of the camera module and is not conducive to the miniaturization development of the camera module.

[0148] In the present application, the distribution pattern of the color pixel units Pc and the gray pixel units Pw of the image sensor 100 is similar to the distribution pattern of the cone cells and rod cells of the human eye. Therefore, whether in a well-lit environment or a low-light environment, a picture close to what the human eye sees can be obtained. The image sensor 100 can capture a high-frame-rate, low-resolution color video through the color pixel units Pc, and capture a low-frame-rate, high-resolution, high low-light sensitivity gray video through the gray pixel units Pw. By performing coloring processing and frame interpolation processing on the color video and the gray video, it is possible to simultaneously achieve high-frame-rate, high-resolution, and high low-light sensitivity video capture under the limited bandwidth of the image sensor 100, with good image restoration and better shooting effects. Moreover, only one camera module 10 including the image sensor 100 is required to achieve high-quality shooting. Compared with the above solutions, there is no problem of image misalignment, and the overall structure is simpler and more compact, which is conducive to the miniaturization development of the camera module.

[0149] The above specific embodiments illustrate the implementation manners of the present application. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. Although the description of the present application will be introduced in combination with some embodiments, this does not mean that the features of this application are limited to this implementation manner. On the contrary, the purpose of introducing the application in combination with the implementation manner is to cover other alternatives or modifications that may be extended based on the claims of the present application. The present application can also be implemented without using these details. In addition, to avoid confusion or obscuring the key points of the present application, some specific details are omitted in the description. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0150] In the description of the present application, it should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "outer", "inner", "circumferential", "radial", "axial", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation to the present application.

[0151] In the description of the present application, it should be noted that unless otherwise clearly specified and defined, the terms "set", "install", "connect", "fit" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific circumstances.

[0152] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these changes and modifications.

Claims

1. An image sensor, characterized in that, It includes a pixel array, and the pixel array includes a plurality of color pixel units and a plurality of grayscale pixel units, and the distribution density of the color pixel units in the image sensor is less than or equal to the distribution density of the grayscale pixel units in the image sensor; Moreover, the distribution density of the color pixel units in the central region of the image sensor is greater than or equal to the distribution density of the grayscale pixel units in the central region, and the distribution density of the color pixel units in the edge region of the image sensor is less than the distribution density of the grayscale pixel units in the edge region.

2. The image sensor according to claim 1, wherein The response frame rate of the color pixel units is greater than the response frame rate of the grayscale pixel units.

3. The image sensor according to claim 1, characterized in that, From the central region to the edge region, the distribution density of the color pixel units gradually decreases, and the distribution density of the grayscale pixel units gradually increases.

4. The image sensor according to claim 1, characterized in that, The ratio of the distribution density of the color pixel units in the image sensor to the distribution density of the grayscale pixel units in the image sensor is 1:20 to 1:

1.

5. The image sensor according to claim 1, wherein In the central region, the ratio of the distribution density of the color pixel units to the distribution density of the grayscale pixel units is 5:1 to 1:1, and in the edge region, the ratio of the distribution density of the color pixel units to the distribution density of the grayscale pixel units is 1:30 to 1:

1.

6. The image sensor according to claim 2, wherein, It further includes an image processing unit, and the image processing unit is electrically connected to the color pixel units and the grayscale pixel units, and is used to obtain a first color video signal with a first frame rate and a first resolution output by the color pixel units, obtain a first grayscale video signal with a second frame rate and a second resolution output by the grayscale pixel units, and is used to process the first color video signal and the first grayscale video signal to obtain a target color video signal with the first frame rate and the second resolution, the first frame rate is greater than the second frame rate, and the first resolution is less than the second resolution.

7. A camera module, characterized in that, It includes: A lens module, and the image sensor according to any one of claims 1 to 6, and the lens module is used to receive incident light and transmit the incident light to the image sensor.

8. An electronic device, characterized in that, It includes a housing, and the camera module according to claim 7, and the camera module is arranged in the housing.

9. An image processing method, characterized in that, Applied to an image sensor, the image sensor includes a plurality of color pixel units, a plurality of grayscale pixel units and an image processing unit, and the image processing unit is electrically connected to the color pixel units and the grayscale pixel units; The method includes: The image processing unit obtains a first color video signal with a first frame rate and a first resolution output by the color pixel units, and obtains a first grayscale video signal with a second frame rate and a second resolution output by the grayscale pixel units, the first frame rate is greater than the second frame rate, and the first resolution is less than the second resolution; The image processing unit performs coloring processing on the first grayscale video signal based on the first color video signal to obtain a second color video signal, and the second color video signal has the second frame rate and the second resolution; The image processing unit performs frame interpolation processing on the second color video signal based on the first color video signal to obtain a target color video signal, and the target color video signal has the first frame rate and the second resolution.

10. The image processing method according to claim 9, wherein, The coloring process includes: Performing position encoding processing on the first grayscale image frame of the first grayscale video signal and the first color image frame of the first color video signal synchronized with it in time, so that the color image blocks at different positions of the first color image frame correspond one-to-one with the grayscale image blocks at different positions of the first grayscale image frame; Based on the color information of the color image blocks, coloring the associated grayscale image blocks to obtain the second color image frame of the second color video signal.

11. The image processing method according to claim 9, wherein The frame interpolation processing includes: Performing time encoding processing on the first color video signal and the second color video signal to determine multiple color image frames in the first color video signal corresponding to the second color image frame in the second color video signal, and the multiple color image frames include the first color image frame synchronized with the second color image frame in time; Performing motion detail processing on the second color image frame based on the first color image frame to obtain the first target color image frame of the target color video signal; Performing motion detail processing on the first target color image frame based on the third color image frame adjacent to the first color image frame to obtain the second target color image frame of the target color video signal; Performing motion detail processing on the second target color image frame based on the fourth color image frame adjacent to the third color image frame to obtain the third target color image frame of the target color video signal, and the fourth color image frame is different from the first color image frame.

12. A computer-readable storage medium, characterized in that, Instructions are stored on the computer-readable storage medium, and when the instructions are executed by a computer, the computer implements the image processing method according to any one of claims 9 to 11.

13. An image sensor, characterized in that, Including: A memory for storing instructions, and One or more processors, when the instructions are executed by the one or more processors, the processors execute the image processing method according to any one of claims 9 to 11.

Images