Image sensor, electronic device, and image generation method

Abstract

The application discloses an image sensor, an electronic device and an image generation method, and belongs to the technical field of image processing. The image sensor comprises a photosensitive module and a pixel circuit module, and the photosensitive module and the pixel circuit module are connected; the photosensitive module comprises an optical lens, at least two RGB pixel elements and at least one EVS pixel element; the EVS pixel element is embedded between the RGB pixel elements; the RGB pixel element is used for capturing a color light signal of an incident light beam of the optical lens within an exposure time length and transmitting the color light signal to the pixel circuit module; the EVS pixel element is used for capturing a light brightness signal of the incident light beam within the exposure time length and transmitting the light brightness signal to the pixel circuit module; and the pixel circuit module is used for converting the color light signal and the light brightness signal into a first electric signal; an RGB image of a moving object and brightness change information of the moving object within the exposure time length are generated according to the first electric signal; and the brightness change information is used for calibrating the RGB image.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image sensor, an electronic device, and an image generation method. Background Art

[0002] Traditional image sensors lack photosensitivity in low-light conditions, often requiring extended exposure times to obtain sufficient light information. However, long exposure times can easily lead to motion blur, especially in fast-moving scenes. EVS (Event Vision Sensor) technology effectively addresses this problem through a different operating principle. In low-light or complex lighting conditions, EVS technology captures critical information about brightness changes, avoiding the motion blur caused by long exposure times. Unlike traditional image sensors, EVS utilizes a more efficient light processing mechanism to provide clear, detailed image data even in low-light conditions, significantly improving sensor performance in low-light environments.

[0003] In the related art, the EVS sensor structure usually adds EVS pixel elements by changing the position of the RGB pixel elements. The conventional RGB pixel structure uses filters of different colors to cover the top of the photodiode. The filter may include one or more of a red filter, a green filter or a blue filter. The RGB pixel structure is usually a pixel array, which is composed of multiple rows and columns of pixel units. Each pixel unit captures the light signal and converts it into an electrical signal. Each row controls the signal output of the entire row of pixels through a row selection signal, and each column processes the signal output of the corresponding column of pixels through a column analog-to-digital converter. In the related art, when improving the conventional RGB pixel structure, the specific approach is to remove some RGB pixel elements and use EVS pixel elements to directly replace the removed RGB pixel elements. Although this EVS sensor structure can solve the motion blur problem caused by long exposure time to a certain extent, the removal of some RGB pixel elements reduces the RGB pixel density, which easily leads to a decrease in accuracy. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide an image sensor, an electronic device, and an image generation method, which can improve the imaging quality of images of moving objects.

[0005] In a first aspect, an embodiment of the present application provides an image sensor, comprising: a photosensitive module and a pixel circuit module; the photosensitive module is connected to the pixel circuit;

[0006] The photosensitive module includes an optical lens, at least two RGB pixel elements and at least one EVS pixel element; the EVS pixel element is embedded between the RGB pixel elements;

[0007] The RGB pixel element is used to capture the color light signal of the incident light beam of the optical lens within the exposure time, and transmit the color light signal to the pixel circuit module;

[0008] The EVS pixel element is used to capture the brightness signal of the incident light beam within the exposure time and transmit the brightness signal to the pixel circuit module;

[0009] The pixel circuit module is used to convert the color light signal and the light brightness signal into a first electrical signal; based on the first electrical signal, generate an RGB image of the moving object and brightness change information of the moving object during the exposure time; the brightness change information is used to calibrate the RGB image.

[0010] In a second aspect, an embodiment of the present application provides an electronic device, including the image sensor and image processor described in the first aspect above;

[0011] The image sensor is configured to capture a color light signal and a brightness signal of a moving object during an exposure time, and convert the color light signal and the brightness signal into a first electrical signal; generate an RGB image of the moving object and brightness change information of the moving object during the exposure time based on the first electrical signal; and transmit the RGB image and the brightness change information to the image processor;

[0012] The image processor is configured to calibrate the RGB image according to the brightness change information to obtain an object image of the moving object.

[0013] In a third aspect, an embodiment of the present application provides an image generation method, which is performed by the electronic device described in the second aspect above, and the method includes:

[0014] The image sensor captures a color light signal and a brightness signal of the moving object within an exposure time, and converts the color light signal and the brightness signal into a first electrical signal;

[0015] The image sensor generates an RGB image of the moving object and brightness change information of the moving object within the exposure time according to the first electrical signal;

[0016] The image processor calibrates the RGB image according to the brightness change information to obtain an object image of the moving object.

[0017] In a fourth aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0019] In a sixth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0020] In a seventh aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0021] In an embodiment of the present application, an image sensor includes a photosensitive module and a pixel circuit module. The photosensitive module is connected to the pixel circuit. The photosensitive module includes an optical lens, at least two RGB pixel elements, and at least one EVS pixel element. The EVS pixel element is embedded between the RGB pixel elements. The RGB pixel element is used to capture the color light signal of the incident light beam of the optical lens during the exposure time and transmit the color light signal to the pixel circuit module; the EVS pixel element is used to capture the brightness signal of the incident light beam during the exposure time and transmit the brightness signal to the pixel circuit module; the pixel circuit module is used to convert the color light signal and the brightness signal into a first electrical signal, and generate an RGB image of the moving object and brightness change information of the moving object during the exposure time based on the first electrical signal. The brightness change information is used to calibrate the RGB image. It can be seen that when generating an image based on an RGB image and brightness change information, there is no need to change the number and position of the RGB pixel elements in the image sensor. It is only necessary to embed the EVS pixel elements between the RGB pixel elements. Therefore, the addition of the EVS pixel elements will not affect the RGB pixel density of the image sensor, thereby reducing the problem of insufficient precision caused by the movement of the pixel elements, and achieving the effect of outputting high-quality images without affecting the RGB pixel density. In addition, this structure of embedding the EVS pixel elements between the RGB pixel elements has a lower limit on the number of EVS pixel elements, that is, more EVS pixel elements can be embedded. The more EVS pixel elements there are, the larger the photosensitive area of ​​the EVS pixel elements. Therefore, when photographing moving objects, more accurate brightness change information can be captured, and richer brightness change information can be captured, thereby providing more accurate and richer information for calibrating the RGB image. Furthermore, the RGB pixel elements and the EVS pixel elements share an optical lens to focus the incident light beam, avoiding the problem of optical crosstalk between multiple optical lenses and improving image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a schematic structural cross-sectional view of an image sensor provided by some embodiments of the present application;

[0023] Figure 2 is a schematic diagram of a method for embedding EVS pixel elements with a horizontal inline structure provided by some embodiments of the present application;

[0024] Figure 3 is a schematic diagram of a method for embedding EVS pixel elements of a vertical inline structure provided by some embodiments of the present application;

[0025] Figure 4 is a schematic top view of the structure of an image sensor provided by some embodiments of the present application;

[0026] Figure 5 is a schematic top view of the structure of an image sensor provided by some embodiments of the present application;

[0027] Figure 6A is a schematic top view of the structure of an image sensor provided by some embodiments of the present application;

[0028] Figure 6B is a schematic top view of the structure of an image sensor provided by some embodiments of the present application;

[0029] Figure 7 is a schematic structural cross-sectional view of an image sensor provided by some embodiments of the present application;

[0030] Figure 8 is a schematic block diagram of a pixel circuit module provided in some embodiments of the present application;

[0031] Figure 9 is a schematic block diagram of a pixel circuit module provided in some embodiments of the present application;

[0032] Figure 10 is a schematic circuit diagram of a pixel circuit module provided in some embodiments of the present application;

[0033] Figure 11 is a schematic graph showing voltage changes over time in a pixel circuit module provided by some embodiments of the present application;

[0034] Figure 12 is a schematic block diagram of an electronic device provided by some embodiments of the present application;

[0035] Figure 13 is a schematic flow chart of an image generation method provided in some embodiments of the present application;

[0036] Figure 14 is a schematic interface diagram of an application scenario of the image generation method provided in some embodiments of the present application;

[0037] Figure 15 is a schematic output interface diagram of an RGB pixel circuit provided in some embodiments of the present application;

[0038] Figure 16 is a schematic output interface diagram of an EVS pixel circuit provided in some embodiments of the present application;

[0039] Figure 17 is a schematic interface diagram of a calibrated image in the image generation method provided in some embodiments of the present application;

[0040] Figure 18 is a schematic structural diagram of an electronic device provided in some embodiments of the present application;

[0041] Figure 19 This is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0043] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0044] The image sensor, electronic device, and image generation method provided in the embodiments of the present application are described in detail below with reference to specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0045] Figure 1 is a schematic structural diagram of an image sensor according to some embodiments of the present application. Figure 1 As shown, the image sensor includes a photosensitive module 10 and a pixel circuit module 20, which are connected to each other. The photosensitive module 10 includes an optical lens 11, at least two RGB pixel elements 12 and at least one EVS pixel element 13, and the EVS pixel element 13 is embedded between the RGB pixel elements 12.

[0046] The RGB pixel element 12 is used to capture the color light signal of the incident light beam of the optical lens within the exposure time, and transmit the color light signal to the pixel circuit module 20.

[0047] The EVS pixel element 13 is used to capture the brightness signal of the incident light beam during the exposure time and transmit the brightness signal to the pixel circuit module 20 .

[0048] The pixel circuit module 20 is used to convert the color light signal and the light brightness signal into a first electrical signal; based on the first electrical signal, an RGB image of the moving object and brightness change information of the moving object during the exposure time are generated, and the brightness change information is used to calibrate the RGB image.

[0049] The RGB pixel element 12 may be a pixel element covered with a color filter, and the color filter may include one or more of a red filter, a green filter, a blue filter, etc. Covering the RGB pixel element 12 on the surface of the pixel circuit module 20 can capture a color image.

[0050] When capturing light brightness signals, the EVS pixel element 13 does not rely on long exposure times, but instead captures light brightness signals at a constant frequency. After transmitting the light brightness signals to the pixel circuit module 20, the pixel circuit module 20 generates event information by detecting changes in pixel brightness in real time. For example, the EVS pixel element 13 captures a light brightness signal once every millisecond and transmits the captured light brightness information to the pixel circuit module 20. The pixel circuit module 20 performs real-time detection on the received light brightness signals to determine brightness change information. The brightness change information captured by the EVS pixel element 13 includes brightness change information for each pixel on the moving object during the exposure time. As can be seen, the synergistic effect of the EVS pixel element 13 and the pixel circuit module 20 allows for instantaneous response to light changes without extending the exposure time to capture sufficient light signals. This event-based capture mechanism significantly reduces the need for long exposure times due to insufficient light sensitivity, such as in low-light environments where insufficient light sensitivity would require longer exposure times. The event-based capture mechanism of this embodiment can avoid image blurring caused by long exposure times.

[0051] In an embodiment of the present application, an image sensor includes a photosensitive module and a pixel circuit module. The photosensitive module is connected to the pixel circuit. The photosensitive module includes an optical lens, at least two RGB pixel elements, and at least one EVS pixel element. The EVS pixel element is embedded between the RGB pixel elements. The RGB pixel element is used to capture the color light signal of the incident light beam of the optical lens during the exposure time and transmit the color light signal to the pixel circuit module; the EVS pixel element is used to capture the brightness signal of the incident light beam during the exposure time and transmit the brightness signal to the pixel circuit module; the pixel circuit module is used to convert the color light signal and the brightness signal into a first electrical signal, and generate an RGB image of the moving object and brightness change information of the moving object during the exposure time based on the first electrical signal. The brightness change information is used to calibrate the RGB image. It can be seen that when generating an image based on an RGB image and brightness change information, there is no need to change the number and position of the RGB pixel elements in the image sensor. It is only necessary to embed the EVS pixel elements between the RGB pixel elements. Therefore, the addition of the EVS pixel elements will not affect the RGB pixel density of the image sensor, thereby reducing the problem of insufficient precision caused by the movement of the pixel elements, and achieving the effect of outputting high-quality images without affecting the RGB pixel density. In addition, this structure of embedding the EVS pixel elements between the RGB pixel elements has a lower limit on the number of EVS pixel elements, that is, more EVS pixel elements can be embedded. The more EVS pixel elements there are, the larger the photosensitive area of ​​the EVS pixel elements. Therefore, when photographing moving objects, more accurate brightness change information can be captured, and richer brightness change information can be captured, thereby providing more accurate and richer information for calibrating the RGB image. Furthermore, the RGB pixel elements and the EVS pixel elements share an optical lens to focus the incident light beam, avoiding the problem of optical crosstalk between multiple optical lenses and improving image quality.

[0052] In some embodiments of the present application, the EVS pixel element 13 has a cross-shaped structure or a straight-line structure. The straight-line structure of the EVS pixel element 13 may include a horizontal straight-line structure or a vertical straight-line structure.

[0053] In the case where the EVS pixel elements 13 are in a horizontally inline structure, the EVS pixel elements 13 in the horizontally inline structure are embedded between every two rows of RGB pixel elements 12 . Figure 2 1 is a schematic structural diagram of the embedding method of the EVS pixel element 13 in a horizontal straight line structure according to some embodiments of the present application. Figure 2As shown, the EVS pixel elements 13 of the horizontal I-shaped structure are embedded between two rows of RGB pixel elements 12. The number of RGB pixel elements 12 in each row of RGB pixel elements 12 is not limited. Moreover, a plurality of EVS pixel elements 13 of the horizontal I-shaped structure can be embedded between every two rows of RGB pixel elements 12. The plurality of EVS pixel elements 13 of the horizontal I-shaped structure are spliced ​​together, and the number of spliced ​​EVS pixel elements 13 depends on the number and length of the RGB pixel elements 12 in each row. Figure 2 The dotted line in the EVS pixel element 13 in the horizontal in-line structure shows that two EVS pixel elements 13 in the horizontal in-line structure are spliced ​​together to form an EVS pixel element 13 in the horizontal in-line structure with a longer length. Figure 2 The structure of some pixel elements is only shown as an example. Figure 2 The structure shown is regarded as a group of pixel elements. In actual applications, multiple groups of pixel elements with such structures can be provided in the photosensitive module 10, and the arrangement of the multiple groups of pixel elements is not limited.

[0054] When the EVS pixel elements 13 are in a vertical in-line structure, the EVS pixel elements 13 in the vertical in-line structure are embedded between every two columns of RGB pixel elements 12 . Figure 3 1 is a schematic structural diagram of the embedding method of the EVS pixel element 13 of the vertical straight line structure according to some embodiments of the present application. Figure 3 As shown, the EVS pixel elements 13 of the vertical straight-line structure are embedded between two columns of RGB pixel elements 12. The number of RGB pixel elements 12 in each column of RGB pixel elements 12 is not limited. Moreover, a plurality of EVS pixel elements 13 of the vertical straight-line structure can be embedded between every two columns of RGB pixel elements 12. The plurality of EVS pixel elements 13 of the vertical straight-line structure are spliced ​​together, and the number of spliced ​​EVS pixel elements 13 depends on the number and length of the RGB pixel elements 12 in each column. Figure 3 The dotted line in the vertical in-line EVS pixel element 13 shows that two vertical in-line EVS pixel elements 13 are spliced ​​together to form a vertical in-line EVS pixel element 13 with a longer length. Figure 3 The structure of some pixel elements is only shown as an example. Figure 3 The structure shown is regarded as a group of pixel elements. In actual applications, multiple groups of pixel elements with such structures can be provided in the photosensitive module 10, and the arrangement of the multiple groups of pixel elements is not limited.

[0055] In the photosensitive module 10 , only the EVS pixel elements 13 with a straight line structure or the EVS pixel elements 13 with a cross structure may be embedded, or both the EVS pixel elements 13 with a straight line structure and the EVS pixel elements 13 with a cross structure may be embedded.

[0056] In the case where all the EVS pixel elements 13 in the photosensitive module 10 are in a straight line structure, an EVS pixel element 13 in a straight line structure is embedded between at least two RGB pixel elements 12. The embedding method of the EVS pixel element 13 in a straight line structure is as follows: Figure 2 and Figure 3 The same photosensitive module 10 can be embedded with both the EVS pixel elements 13 in a horizontal straight line structure and the EVS pixel elements 13 in a vertical straight line structure, or any one of them can be embedded.

[0057] In the case where all the EVS pixel elements 13 in the photosensitive module 10 are of a cross-shaped structure, an EVS pixel element 13 of a cross-shaped structure is embedded between at least every four RGB pixel elements 12 . Figure 4 Schematic top view showing the embedding mode of the cross-shaped EVS pixel element 13 in the RGB pixel element 12. Figure 4 As shown, the dotted circle represents the optical lens 11, which can be a microlens, located above the EVS pixel element 13 and the RGB pixel element 12. Below the optical lens 11, the cross-shaped structure embedded in the center is the EVS pixel element 13, and the RGB pixel elements 12 are located around the EVS pixel element 13. After the incident light beam enters the optical lens 11, the EVS pixel element 13 can capture the brightness signal of the incident light beam, and the RGB pixel element 12 can capture the color light signal of the incident light beam.

[0058] In the case where the photosensitive module 10 includes at least one EVS pixel element 13 with a straight-line structure and at least one EVS pixel element 13 with a cross-shaped structure, an EVS pixel element 13 with a straight-line structure is embedded between every at least two RGB pixel elements 12, and an EVS pixel element 13 with a cross-shaped structure is embedded between every at least four RGB pixel elements 12.

[0059] like Figure 5As shown in the figure, it shows the embedding mode of the two structures of EVS pixel elements in the same image sensor. The minimum pixel unit of the image sensor is an RGB pixel element or an EVS pixel element, wherein the area filled with oblique lines is the EVS pixel element, and the unfilled area is the RGB pixel element 12. The EVS pixel element of the cross structure is indicated by the left oblique line filling method, and the EVS pixel element of the straight structure is indicated by the right oblique line filling method. Considering every four RGB pixel elements as an RGB unit, it can be seen that Figure 5 In the two left columns of RGB units shown, a cross-shaped EVS pixel element is embedded in each RGB unit. Taking the RGB unit in the upper left corner as an example, a cross-shaped EVS pixel element 1301 is embedded between the four RGB pixel elements 12. Figure 5 In the two right columns of RGB units shown, each RGB unit is embedded with an EVS pixel element in a straight line structure. Taking the RGB unit in the upper right corner as an example, an EVS pixel element 1302 in a straight line structure is embedded between the four RGB pixel elements 12. The length of the EVS pixel element 1302 in the straight line structure is the same as the width of the RGB unit. The width of the RGB unit is equal to the sum of the widths of the two RGB pixel elements 12. It should be noted that Figure 5 The structure of some pixel elements is only shown as an example, and the filter type of each RGB pixel element 12 in the structure is not limited. For example, the RGB pixel element 12 can be an R pixel element covering a red filter, a GR pixel element covering a green filter, a GB pixel element covering a green filter, or a B pixel element covering a blue filter. In actual applications, the embedding positions of the EVS pixel element 1302 of the straight structure and the EVS pixel element 1301 of the cross structure in the photosensitive module are not limited. In addition, for the embedding method of the EVS pixel element 1302 of the straight structure, except for Figure 5 In addition to the above, a plurality of EVS pixel elements 1302 in a straight line structure may be embedded in each RGB unit, and the total length of the plurality of EVS pixel elements 1302 in a straight line structure is the same as the width of the RGB unit.

[0060] Figure 6A and Figure 6B : is a schematic top view of the structure of an image sensor according to some embodiments of the present application. The smallest pixel unit of the image sensor is an RGB pixel element or an EVS pixel element, wherein the area filled with oblique lines is an EVS pixel element 13, and the unfilled area is an RGB pixel element. Figure 6A and Figure 6BIn the embodiment, the image sensor includes a plurality of RGB pixel elements and a plurality of cross-shaped EVS pixel elements 13. The number of cross-shaped EVS pixel elements 13 is related to the number of RGB pixel elements. Figure 6A and Figure 6B In the embedding method, the number of RGB pixel elements is four times the number of EVS pixel elements 13. For example, assuming that the image sensor includes 64 RGB pixel elements, 16 EVS pixel elements 13 with a cross structure can be embedded in the image sensor. If every four RGB pixel elements are regarded as an RGB unit, the image sensor includes a total of 16 RGB units. One EVS pixel element 13 can be embedded in each of the 16 RGB units, or EVS pixel elements 13 can be embedded in only some of the RGB units. In other words, the number of EVS pixel elements 13 can be any number between 1 and 16. Each EVS pixel element 13 with a cross structure is embedded in the center of the four RGB pixel elements. The four RGB pixel elements surrounding each EVS pixel element 13 with a cross structure can be RGB pixel elements of any filtering type.

[0061] Optionally, the four RGB pixel elements surrounding the same EVS pixel element 13 have different filter types, such as Figure 6A As shown, in each RGB unit, the four RGB pixel elements include: an R pixel element covered with a red filter, a GR pixel element covered with a green filter, a GB pixel element covered with a green filter, and a B pixel element covered with a blue filter. Optionally, the four RGB pixel elements surrounding the same EVS pixel element 13 have the same filter type, such as Figure 6B As shown, in each RGB unit, the four RGB pixel elements may include one of an R pixel element covered with a red filter, a GR pixel element covered with a green filter, a GB pixel element covered with a green filter, or a B pixel element covered with a blue filter. Figure 6A Taking the RGB unit located in the upper left corner as an example, the RGB unit in the upper left corner includes four RGB pixel elements, which are respectively an R pixel element 1201 located on the upper left side and covered with a red filter, a GR pixel element 1202 located on the upper right side and covered with a green filter, a GB pixel element 1203 located on the lower left side and covered with a green filter, and a B pixel element 1204 located on the lower right side and covered with a blue filter. Figure 6BTaking the RGB unit located in the upper left corner as an example, the RGB unit in the upper left corner includes four RGB pixel elements, namely, an R pixel element 1201 located on the upper left side and covered with a red filter, an R pixel element 1202 located on the upper right side and covered with a red filter, an R pixel element 1203 located on the lower left side and covered with a red filter, and an R pixel element 1204 located on the lower right side and covered with a red filter. It should be noted that Figure 6A and Figure 6B The arrangement of the RGB pixel elements is only an example. In actual applications, there can be many arrangements of RGB pixel elements. Regardless of the arrangement, the method of embedding the EVS pixel element 13 in multiple RGB pixel elements is similar and will not be listed one by one.

[0062] In some embodiments of the present application, since the EVS pixel element 13 can be a straight-line structure or a cross-shaped structure, the embedding position of the EVS pixel element 13 with a straight-line structure or a cross-shaped structure in the photosensitive module 10 is not limited, and there are multiple embedding methods, thereby achieving the effect of flexibly setting the EVS pixel element 13 in the photosensitive module 10, and this embedding method of the EVS pixel element 13 does not affect the number of RGB pixel elements 12 in the photosensitive module 10, thereby not affecting the RGB pixel density of the image sensor, effectively avoiding the problem of insufficient accuracy caused by changes in pixel density, and ensuring the output of high-quality images while capturing the brightness change information of the moving object.

[0063] In some embodiments of the present application, Figure 7 , which schematically shows a cross-sectional view of the structure of the image sensor. The pixel circuit module 20 includes an RGB pixel circuit 21 and an EVS pixel circuit 22. The photosensitive module 10 is connected to the RGB pixel circuit 21 and the EVS pixel circuit 22 respectively. The RGB pixel circuit 21 and the EVS pixel circuit 22 are connected.

[0064] The RGB pixel element 12 is further configured to transmit the color light signal to the RGB pixel circuit 21 .

[0065] The EVS pixel element 13 is also used to transmit the light brightness signal to the RGB pixel circuit 21 .

[0066] The RGB pixel circuit 21 is configured to convert the color light signal and the light brightness signal into a first electrical signal, generate an RGB image of the moving object according to the first electrical signal, and transmit the first electrical signal to the EVS pixel circuit.

[0067] The EVS pixel circuit 22 is configured to generate brightness change information according to the first electrical signal.

[0068] In some embodiments of the present application, an RGB image of a moving object is generated by the RGB pixel circuit 21, and brightness change information of the moving object is generated by the EVS pixel circuit 22. Thus, the collaborative work of the RGB pixel circuit 21 and the EVS pixel circuit 22 can reduce the problem of unclearness in the RGB image caused by the long exposure time, especially in low-light environments where the exposure time is longer and the image clarity is lower. In addition, since the RGB pixel circuit 21 can transmit the first electrical signal obtained after the optical signal is converted to the EVS pixel circuit 22 for use, the EVS pixel circuit 22 can share the photoelectric conversion module of the RGB pixel circuit 21, such as a photodiode, a photodetector, etc., reducing the circuit area of ​​the additional photoelectric conversion module, thereby using more circuit area to configure pixel elements and improving the pixel density of the image sensor. Furthermore, since the RGB pixel circuit 21 and the EVS pixel circuit 22 are designed in layers, that is, both use independent circuit designs, they can ensure the independent operation of each circuit layer, optimizing the overall performance and image generation quality of the image sensor.

[0069] In some embodiments of the present application, Figure 8 As shown, the RGB pixel circuit 21 includes a first photodetection element 211 and a color image generator 212, and the first photodetection element 211 is connected to the color image generator 212. The EVS pixel circuit 22 includes a brightness change detector 221, and the first photodetection element 211 is connected to the brightness change detector 221. The RGB pixel elements 12 and the EVS pixel elements 13 in the photosensitive module 10 are both connected to the first photodetection element 211, so that the captured color light signal and light brightness signal can be transmitted to the first photodetection element 211.

[0070] The first photodetection element 211 is used to convert the color light signal and the light brightness signal into a first electrical signal, and transmit the first electrical signal to the color image generator 212 and the brightness change detector 221 respectively.

[0071] The color image generator 212 is configured to generate an RGB image according to the first electrical signal.

[0072] The brightness change detector 221 is configured to generate brightness change information according to the first electrical signal.

[0073] The first photodetection element 211 may be a photodiode, or other components with a photoelectric conversion function.

[0074] Depend on Figure 8It can be seen that based on the connection relationship between the first photodetection element 211 and the brightness change detector 221, the first photodetection element 211 can transmit the first electrical signal obtained after conversion to the EVS pixel circuit 22 for use, so that the EVS pixel circuit 22 can share the first photodetection element 211 of the RGB pixel circuit 21, reducing the circuit area of ​​the additional photodetection element, so that more circuit area can be used to configure pixel elements, thereby improving the pixel density of the image sensor.

[0075] In some embodiments of the present application, Figure 9 As shown, the EVS pixel circuit 22 further includes a second photodetection element 222 ; the second photodetection element 222 is connected to the EVS pixel element 13 and the brightness change detector 221 , respectively.

[0076] The EVS pixel element 13 is further configured to transmit the light brightness signal to the second photodetection element 222 .

[0077] The second photodetection element 222 is used to convert the light brightness signal into a second electrical signal and transmit the second electrical signal to the brightness change detector 221 .

[0078] The brightness change detector 221 is configured to generate brightness change information according to the first electrical signal and the second electrical signal.

[0079] Optionally, the second photodetection element 222 is further configured to convert the light brightness signal into a second electrical signal when the RGB pixel circuit 21 is in a non-conducting state, and transmit the second electrical signal to the brightness change detector 221. The brightness change detector 221 is further configured to generate brightness change information based on the second electrical signal.

[0080] In some embodiments of the present application, by configuring a second photodetection element 222 in the EVS pixel circuit 22, the EVS pixel circuit 22 can share the first photodetection element 211 in the RGB pixel circuit 21 when working in conjunction with the RGB pixel circuit 21. Sharing the same photodetection element ensures minimal bit error, thereby improving the response accuracy of the EVS pixel circuit 22. In addition, this structure allows the EVS pixel circuit 22 to operate independently without relying on the RGB pixel circuit 21, and can use the second photodetection element 222 to convert the light brightness signal into an electrical signal, thereby obtaining brightness change information.

[0081] In some embodiments of the present application, the brightness change detector 221 includes an electrical signal conversion element and a comparator. The electrical signal conversion element is configured to convert the first electrical signal into an electrical signal reference value. The comparator is configured to compare the electrical signal reference value with a reference threshold value and generate brightness change information based on the comparison result.

[0082] Optionally, the electrical signal conversion element is a capacitor, and the electrical signal reference value is a voltage value.

[0083] Optionally, the brightness change information includes: a first event value for representing an increase in brightness, or a second event value for representing a decrease in brightness. The reference threshold includes an on threshold and an off threshold of the brightness change detector 221. The comparator is used to compare the electrical signal reference value with the on threshold, and output the first event value when the electrical signal reference value is greater than or equal to the on threshold. The comparator is used to compare the electrical signal reference value with the off threshold, and output the second event value when the electrical signal reference value is less than or equal to the off threshold.

[0084] For example, the on-threshold value is 2.7V and the off-threshold value is 1V. When the electrical signal reference value is greater than or equal to the on-threshold value of 2.7V, a first event value of "1" is output. When the electrical signal reference value is less than or equal to the off-threshold value of 1V, a second event value of "0" is output. The values ​​of the on-threshold and off-threshold values ​​are merely examples and are not limited in this embodiment. It suffices that the on-threshold value be greater than the off-threshold value.

[0085] Optionally, each time brightness change information is detected, the EVS pixel circuit 22 outputs an event value, which includes the brightness change information detected at the corresponding moment. The brightness change information can be represented as a frame of binary image, which includes two colors for representing brightness increase or brightness decrease. For example, in an event value output once, for a pixel whose brightness increases to exceed the turn-on threshold, the position corresponding to the pixel is represented as black in the binary image, and the corresponding event value is 0; for a pixel whose brightness decreases to below the turn-off threshold, the position corresponding to the pixel is represented as white in the binary image, and the corresponding event value is 1. In this way, the brightness change information of the moving object at the corresponding moment can be obtained through the color information in the binary image, achieving the effect of outputting brightness change information in a simple and easy-to-understand manner. In addition, the brightness change detection effect of the brightness change detector is realized by a comparator, which is relatively simple in hardware implementation and saves hardware costs.

[0086] In some embodiments of the present application, the RGB pixel circuit 21 and the EVS pixel circuit 22 are connected via a switch device, and the switch device is used to control whether the RGB pixel circuit 21 and the EVS pixel circuit 22 are conductive.

[0087] When the switch device is closed, the RGB pixel circuit 21 and the EVS pixel circuit 22 are electrically connected, and the RGB pixel circuit 21 transmits the first electrical signal to the EVS pixel circuit 22. At this time, the RGB pixel circuit 21 and the EVS pixel circuit 22 can share the same photodetection element, that is, the first photodetection element 211 in the above embodiment.

[0088] When the switch device is off, the RGB pixel circuit 21 and the EVS pixel circuit 22 are disconnected, and the RGB pixel circuit 21 and the EVS pixel circuit 22 can work independently. At this time, the RGB pixel circuit 21 generates an RGB image by processing the color light signal captured by the RGB pixel element 12; and the EVS pixel circuit 22 generates brightness change information by processing the light brightness signal captured by the EVS pixel element 13.

[0089] In some embodiments of the present application, the RGB pixel circuit 21 and the EVS pixel circuit 22 are connected by the switch device, so that the RGB pixel circuit 21 and the EVS pixel circuit 22 can work independently in some scenes and can work cooperatively in some scenes, such as a scene in which an object in motion needs to be photographed. In addition, when the EVS pixel circuit 22 works independently, accurate brightness change information can be obtained.

[0090] Figure 10 is a schematic circuit diagram of a pixel circuit module according to some embodiments of the present application. As shown in Figure 10 In the RGB pixel circuit 21, a first photodiode PD1, a transfer gate MOS_2, a reset transistor MOS_1, a source follower MOS_3, and a row selector MOS_4 are connected in sequence. One end of the row selector MOS_4 is grounded. The source follower MOS_3 is further connected between the reset transistor MOS_1 and the transfer gate MOS_2. The first photodiode PD1 is a first photoelectric detection element 211, which is used to perform photoelectric conversion on the light signal captured by the light sensing module 10 to obtain an electrical signal. The light signal includes a color light signal and a light brightness signal. The electrical signal is stored and used for subsequent signal processing. The reset transistor MOS_1 is used to reset the electrical signal of the floating diffusion region between the reset transistor MOS_1 and the source follower MOS_3 to a fixed voltage, such as Vdd as shown in Figure 10 The transfer gate MOS_2 is used to control the first photodiode PD1 to transmit the electrical signal to the floating diffusion region. When the transfer gate MOS_2 is turned on, the electrical signal is transmitted to the floating diffusion region. When the transfer gate MOS_2 is off, the electrical signal cannot be transmitted. The floating diffusion region is used to store the electrical signal transmitted from the first photodiode PD1. Then, the electrical signal is converted into a voltage signal and amplified. The source follower MOS_3 is used to amplify the voltage signal of the floating diffusion region and output it to the next stage circuit, which has high input impedance and low output impedance. The row selector MOS_4 is used to control the final output path of the signal and determine when to output the amplified voltage signal to the processing circuit.

[0091] During the reset phase of the RGB pixel circuit 21, when the reset transistor MOS_1 turns on, the floating diffusion is reset to a fixed voltage Vdd and the previously stored electrical signal is cleared. The first photodiode PD1 accumulates an electrical signal under illumination. After the transmission gate MOS_2 turns on, the electrical signal is transmitted from the first photodiode PD1 to the floating diffusion. The electrical signal in the floating diffusion is amplified by the source follower MOS_3, generating a corresponding voltage signal. The row selector MOS_4 then controls whether the amplified voltage signal is output to the subsequent processing circuitry.

[0092] The EVS pixel circuit 22 includes a second photodiode PD2, a logic operation device D, a capacitor C, an amplifier A, and two comparators connected in sequence. The second photodiode PD2 is the second photodetection element 222, which is used to perform photoelectric conversion on the brightness signal captured by the photosensitive module 10 to obtain an electrical signal, such as a photocurrent Ip. The photocurrent Ip is proportional to the light intensity, and the dark current Idark is superimposed on the photocurrent Ip. The capacitor C is an electrical signal conversion element, and the photocurrent Ip causes the voltage Vp on the capacitor C to change, that is, causes the electrical signal reference value to change. As the light increases or decreases, the voltage value on the capacitor C will rise or fall accordingly. After the voltage Vp is amplified by the amplifier A, the resulting voltage Vd enters the two comparators. The two comparators are executed in parallel. Figure 10 As shown in is the opening threshold, When the light changes and the voltage value is greater than or equal to the on threshold, the comparator outputs a first event value, such as "ON" or "1," to represent an increase in brightness. When the light changes and the voltage value is less than or equal to the off threshold, the comparator outputs a second event value, such as "OFF" or "0," to represent a decrease in brightness.

[0093] In the pixel circuit module 20, the output end of the first photodiode PD1 is connected between the second photodiode PD2 and the logic operation device D, so that the first photodiode PD1 can transmit the obtained electrical signal to the EVS pixel circuit 22, so that the EVS pixel circuit 22 can perform signal processing based on the electrical signal transmitted by the first photodiode PD1, thereby obtaining brightness change information.

[0094] Figure 11 The schematic diagram shows the curve of voltage Vp changing with time t. When the voltage Vp increases and exceeds the turn-on threshold When the voltage Vp decreases and falls below the off threshold, the EVS pixel circuit 22 outputs the first event value "ON", indicating that the brightness increases. When the light intensity decreases, the EVS pixel circuit 22 outputs the second event value "OFF," indicating a decrease in brightness. It can be seen that the EVS pixel circuit 22 is able to respond to changes in brightness, and its response frequency is proportional to the light intensity. That is, the stronger the light intensity, the faster the EVS pixel circuit 22 responds.

[0095] Figure 12 : is a schematic structural diagram of an electronic device according to some embodiments of the present application. Figure 12 As shown, the electronic device 120 includes an image sensor 121 and an image processor 122. The image sensor 121 can be the image sensor described in any of the above embodiments, and its specific structure will not be repeated.

[0096] Among them, the image sensor 121 is used to capture the color light signal and light brightness signal of the moving object within the exposure time, and convert the color light signal and light brightness signal into a first electrical signal; generate an RGB image of the moving object and brightness change information of the moving object during the exposure time based on the first electrical signal; and transmit the RGB image and brightness change information to the image processor.

[0097] The image processor 122 is configured to calibrate the RGB image according to the brightness change information to obtain an object image of the moving object.

[0098] Alternatively, the electronic device 120 may be any electronic device capable of generating images, such as a camera, a mobile phone, or a tablet computer. For example, when a camera is used to capture a moving object, the image sensor 121 in the camera generates an RGB image and brightness change information of the moving object during the exposure time, and transmits the RGB image and brightness change information to the image processor 122. The image processor 122 calibrates the RGB image based on the brightness change information to obtain an object image of the moving object.

[0099] In some embodiments of the present application, the image sensor 121 captures the color light signal and light brightness signal of the moving object during the exposure time, converts the color light signal and light brightness signal into a first electrical signal, and then generates an RGB image of the moving object and brightness change information of the moving object during the exposure time based on the first electrical signal, and transmits the RGB image and brightness change information to the image processor 122. The image processor 122 calibrates the RGB image based on the brightness change information to obtain an object image of the moving object. Because the EVS pixel elements 13 are distributed between the RGB pixel elements 12 in an embedded manner in the image sensor 121, more accurate brightness change information can be obtained and richer brightness change information can be captured, thereby providing more accurate and richer information for calibrating the RGB image and optimizing the image quality of the object image finally generated.

[0100] In some embodiments of the present application, the execution subject may be an electronic device or a component in the electronic device, such as an integrated circuit or chip. The electronic device may be a terminal or other device other than a terminal. For example, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. It may also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc., and the embodiments of the present application do not specifically limit this.

[0101] The electronic device in some embodiments of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

[0102] Figure 13 is a schematic flow chart of an image generation method according to some embodiments of the present application. Figure 13 As shown, the image generation method is as follows Figure 12 The electronic device shown performs the following steps:

[0103] Step 132 : The image sensor captures the color light signal and the brightness signal of the moving object within the exposure time, and converts the color light signal and the brightness signal into a first electrical signal.

[0104] Among them, the color light signal is captured by the RGB pixel element, and the light brightness signal is captured by the EVS pixel element.

[0105] In step 134 , the image sensor generates an RGB image of the moving object and brightness change information of the moving object during the exposure time according to the first electrical signal.

[0106] The brightness change information includes: a first event value for representing an increase in brightness, or a second event value for representing a decrease in brightness.

[0107] Step 134 can be executed as follows: convert the first electrical signal into an electrical signal reference value. When the electrical signal reference value is greater than or equal to the turn-on threshold, the first event value is output; when the electrical signal reference value is less than or equal to the turn-off threshold, the second event value is output. Optionally, the electrical signal reference value is a voltage value. The representation method of the first event value and the second event value is not limited, for example, it can be a number, a letter, an identification symbol, etc. Taking numbers as an example, outputting the first event value "1" indicates that the brightness increases and is greater than or equal to the turn-on threshold. Outputting the second event value "0" indicates that the brightness decreases and is less than or equal to the turn-off threshold.

[0108] In step 136 , the image processor calibrates the RGB image according to the brightness change information to obtain an object image of the moving object.

[0109] Optionally, step 136 may be performed as the following steps A1-A2:

[0110] Step A1: Determine the motion trajectory data of the moving object within the exposure time based on the brightness change information; the motion trajectory data is used to represent the motion trajectory of each pixel point in the area where the moving object is located in the RGB image within the exposure time.

[0111] Step A22: calibrate the image of the motion blur area in the RGB image according to the motion trajectory data to obtain the object image; the motion blur area is the image area formed by motion blur generated by the motion of the moving object during the exposure time.

[0112] When calibrating the image of the motion blurred area in the RGB image according to the motion trajectory data to obtain the object image, the pixel position coordinates of each pixel point in the motion blurred area within the exposure time can be first determined according to the motion trajectory data, and then the motion end point position coordinates of each pixel point in the motion blurred area within the exposure time can be determined according to the pixel position coordinates of each pixel point in the motion blurred area within the exposure time; based on the motion end point position coordinates, a calibration area image corresponding to the image of the motion blurred area is generated; and then the image of the motion blurred area is replaced with the calibration area image to obtain the target image.

[0113] In some embodiments of the present application, a color light signal and a brightness signal of a target object are captured during an exposure period, and the color light signal and the brightness signal are converted into a first electrical signal. Based on the first electrical signal, an RGB image of the moving object and brightness variation information of the moving object during the exposure period are generated. The RGB image is calibrated based on the brightness variation information to obtain an object image of the moving object. This allows the generation of an object image of the moving object to obtain not only the RGB image but also accurate and rich brightness variation information, providing more accurate and rich information for calibrating the RGB image, thereby improving the quality of the resulting object image.

[0114] The image generation method provided by this application is described below through a specific embodiment.

[0115] In some embodiments of the present application, the electronic device is taken as an example. The camera is pre-configured with the image sensor 121 and image processor 122 described in the above embodiments. Figure 14 Taking the scenario shown as an example, Figure 14 In the example, the shooting scene is a sports scene of throwing a ball, and the ball 100 is the moving object. In this embodiment, while maintaining normal image brightness, an exposure time of 33 milliseconds is required. Assuming that the speed of the ball 100 is 30 pixels per millimeter, when the RGB pixel element 12 completes the 33 millisecond exposure, the ball 100 has moved 990 pixels. Therefore, the RGB pixel element 12 can only capture the result of the ball 100 moving 990 pixels, which will cause obvious motion blur and smearing of the ball 100, for example Figure 15 As shown, the solid dot drawn on sphere 100 represents the center of sphere 100. Based on the center position of sphere 100, sphere 100 moves from point A to point B. That is, the center position of sphere 100 moves from point A to point B. Point A is the position of sphere 100 at the beginning of the exposure time, and point B is the position of sphere 100 at the end of the exposure time. At this point, sphere 100 has moved 990 pixels on the image. The beginning of the exposure time is millisecond 0, and the end of the exposure time is millisecond 33.

[0116] Because the EVS pixel element 13 works by capturing brightness changes in real time, under the same 33-millisecond exposure conditions, the EVS pixel element 13 can capture brightness changes at a frequency of once per millisecond, allowing the EVS pixel circuit 22 to output brightness change information of the sphere 100 at a frequency of per millisecond, ultimately outputting 33 event frames within 33 milliseconds, each event frame containing the event value corresponding to each pixel on the sphere 100. This method not only fully records the movement process of the sphere 100, but also allows these event frame data to be merged with the RGB image data through subsequent image processing algorithms to calculate and restore the true position of the sphere 100 in the image, eliminating motion blur, and ensuring that the final output image is clear and rich in detail.

[0117] When the user turns on the camera and starts the shooting mode, the RGB pixel element 12 and the EVS pixel element 13 in the image sensor 121 are activated and sense light at the same time. The camera program dynamically sets the exposure conditions, including the exposure time, according to the current ambient brightness to ensure the correct exposure of the RGB image. The EVS pixel element 13 continuously tracks the brightness changes of the moving object and outputs the brightness change information of the moving object through the EVS pixel circuit 22, providing key data for the subsequent calibration of the RGB image. It can be understood that the exposure condition setting action can be set in real time according to the current ambient brightness after the camera is turned on, or it can be set in advance. For example, if the exposure conditions of the camera are set in advance, the pre-set exposure conditions will be automatically used each time the camera is turned on.

[0118] The RGB pixel element 12 and the EVS pixel element 13 are simultaneously sensitive to light, wherein the RGB pixel element 12 captures the color light signal of the incident light beam of the optical lens 11. Due to the long exposure time, which is 33 milliseconds in this embodiment, the RGB image output by the RGB pixel circuit 21 may contain a streak or blur of the sphere 100, but the RGB image can provide the overall color information and brightness data of the scene.

[0119] The EVS pixel element 13 captures the brightness signal of the incident light beam. Unlike the RGB pixel element 12, the EVS pixel element 13 has no exposure concept and does not rely on a fixed frequency. When the brightness in the scene changes, the EVS pixel element 13 will respond and capture the brightness signal, and then the EVS pixel circuit 22 processes the brightness signal and outputs brightness change information. During the entire exposure time of 33 milliseconds, the EVS pixel element 13 captures the brightness signal once per millisecond, so that the EVS pixel circuit 22 outputs brightness change information once per millisecond. Whenever there is a significant change in brightness, such as when the brightness change causes an electrical signal reference value, Figure 10 and Figure 11 When the voltage value Vp in increases to exceed the turn-on threshold or decreases to below the turn-off threshold, the EVS pixel circuit 22 outputs an event value, which includes a first event value for representing an increase in brightness or a second event value for representing a decrease in brightness. In this embodiment, each pixel on the sphere 100 corresponds to its own event value. Therefore, at each moment during the exposure time, the EVS pixel circuit 22 outputs an event frame, each event frame containing the event value corresponding to each pixel on the sphere 100 and the position coordinates of the pixel point. If the event value is represented by "0" and "1", the output form of the event frame is a binary image, where the event value "0" is represented by black and the event value "1" is represented by white.

[0120] After the entire exposure time is complete, the EVS pixel circuit 22 outputs 33 event frames. Based on these 33 event frames, the motion trajectory data of the sphere 100 during the entire exposure time can be determined. Because the EVS pixel elements 13 are fewer in number than the RGB pixel elements 12 and are equipped with independent EVS pixel circuits 22, data can be output at a higher frame rate, capturing the brightness change information of the moving object during the entire exposure time. Even in scenes where the sphere 100 is moving rapidly, the EVS pixel circuit 22 can obtain accurate brightness change information, thereby generating precise motion trajectory data.

[0121] It can be understood that the RGB pixel circuit 21 outputs a complete static image, which may contain smears or blurred areas, such as Figure 15 The EVS pixel circuit 22 outputs a series of data streams, which are brightness change information. Based on the data stream, the motion trajectory data of the sphere 100 can be determined, such as Figure 16 As shown. Figure 16 It can be seen that the motion trajectory of the sphere 100 during the entire exposure time is still represented by the solid dot drawn on the sphere 100, and the position of the sphere center is used as the reference. Figure 16 The position of point B in the image is the final position of the movement of the sphere 100 during the exposure time. Here, the movement of the person during the exposure time is ignored in order to highlight the movement trajectory of the sphere. In actual applications, if the person moves during the entire exposure time, the data stream output by the EVS pixel circuit 22 can also determine the person's movement trajectory data.

[0122] Image sensor 121 then transmits the RGB image and brightness change information to image processor 122, which calibrates the RGB image based on the brightness change information. Because RGB pixel elements 12 and EVS pixel elements 13 share the same location and have minimal positional difference, the RGB image can rely on the brightness change information output by EVS pixel circuit 22 to accurately calculate how many pixels the moving object has moved during the exposure time. This allows for pixel position calibration, eliminating image blur caused by motion events in the RGB image and outputting a clear, blur-free image.

[0123] When the image processor 122 calibrates the RGB image based on the brightness change information, it uses the high temporal resolution characteristics of the EVS pixel element 13 to compensate for the motion blur in the RGB image caused by the poor exposure time, thereby obtaining a clear and detailed object image, e.g. Figure 17 As shown in Figure 3, the object image obtained after calibration is clear and has no blurred areas.

[0124] Specifically, the image processor 122 first determines the motion blur area in the RGB image, which is the image area formed by the motion blur generated by the motion of the moving object during the exposure time. Figure 15 As shown, basketball player 300 throws a basketball, and the throwing action causes ball 100 to move, moving from point A to point B within the exposure time. Motion blur region 200 is the image region formed by the motion blur generated by the movement of ball 100 within the exposure time. Then, image processor 122 determines the motion trajectory data of ball 100 within the exposure time based on the brightness change information. The image processor 122 does not limit the order when determining the motion blur region and motion trajectory data. Because the motion trajectory data can represent the motion trajectory of each pixel point in the area where ball 100 is located in the RGB image within the exposure time, the image processor 122 can determine the pixel position coordinates of each pixel point on ball 100 in the RGB image based on the motion trajectory data, and then generate a calibration area image corresponding to the image of the motion blur region based on the pixel position coordinates. The image of the motion blur region is then replaced with the calibration area image to obtain an object image of ball 100. During the image calibration process, the image processor 122 can retain the color and brightness information of the RGB image, and at the same time combine the motion trajectory data of the EVS to ensure the clarity and accuracy of the moving object in the image, and restore the true position and shape of the fast-moving object. Therefore, the image quality of the final object image can be optimized.

[0125] Optionally, the EVS pixel circuit 22 outputs multiple event frames, each of which contains the event value and pixel position coordinates corresponding to each pixel on the sphere 100 at the corresponding moment. Each event frame is used to represent the brightness change information of the sphere 100 at the corresponding moment. When calibrating the RGB image based on multiple brightness change information, the event frame output at the last moment of the exposure duration represents the final position information of the sphere 100's movement, that is, the motion endpoint position coordinates of each pixel in the motion blur region 200 within the exposure duration. Therefore, based on the motion endpoint position coordinates of the sphere 100, a calibration region image corresponding to the image of the motion blur region 200 can be generated, for example Figure 16 The image corresponding to the position of point B to which the sphere 100 finally moves is the calibration area image. The image of the motion blur area 200 is replaced by the calibration area image to obtain the following: Figure 17 An object image of a sphere 100 is shown.

[0126] The image processor 122 can further optimize the image through conventional image processing methods. For example, image processing algorithms such as gamma correction, color matrix adjustment, and interpolation can be used to generate higher-quality images, ensuring accurate colors and clear details. Since image processing algorithms such as gamma correction, color matrix adjustment, and interpolation are conventional image processing methods, they will not be described in detail.

[0127] The object image processed by the image processor 122 is displayed on the preview interface of the camera, ensuring that the user can view a clear and detailed image in real time.

[0128] Alternatively, as Figure 18 As shown, some embodiments of the present application also provide an electronic device 1800, including a processor 1801 and a memory 1802, and the memory 1802 stores a program or instruction that can be run on the processor 1801. When the program or instruction is executed by the processor 1801, the various steps of the above-mentioned image generation method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0129] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0130] Figure 19 A schematic diagram of the hardware structure of an electronic device for implementing some embodiments of the present application.

[0131] The electronic device 2000 includes but is not limited to components such as a radio frequency unit 2001, a network module 2002, an audio output unit 2003, an input unit 2004, a sensor 2005, a display unit 2006, a user input unit 2007, an interface unit 2008, a memory 2009, and a processor 2010.

[0132] Those skilled in the art will understand that the electronic device 2000 may also include a power source (such as a battery) to supply power to each component. The power source may be logically connected to the processor 2010 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 19 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0133] Among them, the processor 2010 is used to capture the color light signal and light brightness signal of the moving object within the exposure time, and convert the color light signal and the light brightness signal into a first electrical signal; generate an RGB image of the moving object and brightness change information of the moving object within the exposure time based on the first electrical signal; and calibrate the RGB image based on the brightness change information to obtain an object image of the moving object.

[0134] Optionally, the brightness change information includes: a first event value for representing an increase in brightness, or a second event value for representing a decrease in brightness;

[0135] Processor 2010 is also used to convert the first electrical signal into an electrical signal reference value; when the electrical signal reference value is greater than or equal to the open threshold, output the first event value; when the electrical signal reference value is less than or equal to the close threshold, output the second event value.

[0136] Optionally, the processor 2010 is further used to determine the motion trajectory data of the moving object within the exposure time based on the brightness change information; the motion trajectory data is used to characterize the motion trajectory of each pixel point in the area where the moving object is located in the RGB image within the exposure time; based on the motion trajectory data, the image of the motion blurred area in the RGB image is calibrated to obtain the object image; the motion blurred area is the image area formed by the motion blur generated by the movement of the moving object within the exposure time.

[0137] Optionally, the processor 2010 is further used to determine the pixel position coordinates of each pixel point in the motion blurred area within the exposure time based on the motion trajectory data; determine the motion endpoint position coordinates of each pixel point in the motion blurred area within the exposure time based on the pixel position coordinates of each pixel point in the motion blurred area within the exposure time; generate a calibration area image corresponding to the image of the motion blurred area based on the motion endpoint position coordinates; and replace the image of the motion blurred area with the calibration area image to obtain an object image.

[0138] In some embodiments of the present application, by capturing the color light signal and the light intensity signal of the moving object within the exposure duration, the color light signal and the light intensity signal are converted into a first electric signal; according to the first electric signal, an RGB image of the moving object and the brightness change information of the moving object within the exposure duration are generated; and the RGB image is calibrated according to the brightness change information to obtain an object image of the moving object. When the object image of the moving object is generated, not only the RGB image can be obtained, but also accurate and rich brightness change information can be obtained, which provides more accurate and rich information for calibrating the RGB image, and optimizes the image quality of the finally generated object image.

[0139] It should be understood that in the embodiments of the present application, the input unit 2004 can include a graphics processor (Graphics Processing Unit, GPU) 20041 and a microphone 20042. The graphics processor 2041 processes image data of a still picture or a video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 2006 can include a display panel 20061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 2007 includes at least one of a touch panel 20071 and other input devices 20072. The touch panel 20071 is also called a touch screen. The touch panel 20071 can include two parts of a touch detection device and a touch controller. The other input devices 20072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), a trackball, a mouse, an operation lever, and the like, which will not be described here.

[0140] Memory 2009 can be used to store software programs and various data. Memory 2009 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function or an image playback function), and the like. Furthermore, memory 2009 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 2009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0141] Processor 2010 may include one or more processing units. Optionally, processor 2010 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 2010.

[0142] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image generation method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0143] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0144] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0145] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0146] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0147] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0148] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of this application.

[0149] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An image sensor, characterized in that: include: A photosensitive module and a pixel circuit module; the photosensitive module is connected to the pixel circuit; The photosensitive module includes an optical lens, at least two RGB pixel elements and at least one EVS pixel element; the EVS pixel element is embedded between the RGB pixel elements; The RGB pixel element is used to capture the color light signal of the incident light beam of the optical lens within the exposure time, and transmit the color light signal to the pixel circuit module; The EVS pixel element is used to capture the brightness signal of the incident light beam within the exposure time and transmit the brightness signal to the pixel circuit module; The pixel circuit module is configured to convert the color light signal and the light brightness signal into a first electrical signal; and generate an RGB image of the moving object and brightness change information of the moving object within the exposure time according to the first electrical signal; The brightness change information is used to calibrate the RGB image.

2. The image sensor according to claim 1, wherein The EVS pixel element has a cross-shaped structure or a straight-shaped structure.

3. The image sensor according to claim 2, wherein: The EVS pixel element is a horizontal straight line structure or a vertical straight line structure; Wherein, when the EVS pixel elements are in a horizontal inline structure, the EVS pixel elements in the horizontal inline structure are embedded between every two rows of RGB pixel elements; In the case where the EVS pixel elements are in a vertical in-line structure, the EVS pixel elements in the vertical in-line structure are embedded between every two columns of RGB pixel elements.

4. The image sensor according to claim 2, wherein: In the case where all the EVS pixel elements in the photosensitive module are in a straight-line structure, an EVS pixel element in a straight-line structure is embedded between at least two RGB pixel elements; In the case where all EVS pixel elements in the photosensitive module are of a cross-shaped structure, an EVS pixel element of a cross-shaped structure is embedded between at least four RGB pixel elements; In the case where the photosensitive module includes at least one EVS pixel element with a straight-line structure and at least one EVS pixel element with a cross-shaped structure, an EVS pixel element with a straight-line structure is embedded between every at least two RGB pixel elements, and an EVS pixel element with a cross-shaped structure is embedded between every at least four RGB pixel elements.

5. The image sensor according to claim 1, wherein The pixel circuit module includes an RGB pixel circuit and an EVS pixel circuit; the photosensitive module is connected to the RGB pixel circuit and the EVS pixel circuit respectively, and the RGB pixel circuit is connected to the EVS pixel circuit; The RGB pixel element is further used to transmit the color light signal to the RGB pixel circuit; The EVS pixel element is further used to transmit the light brightness signal to the RGB pixel circuit; The RGB pixel circuit is configured to convert the color light signal and the light brightness signal into a first electrical signal, generate an RGB image of the moving object based on the first electrical signal, and transmit the first electrical signal to the EVS pixel circuit; The EVS pixel circuit is used to generate brightness change information according to the first electrical signal.

6. The image sensor according to claim 5, wherein: The RGB pixel circuit includes a first photodetection element and a color image generator, and the first photodetection element and the color image generator are connected; the EVS pixel circuit includes a brightness change detector; the first photodetection element is connected to the brightness change detector; The first photoelectric detection element is used to convert the color light signal and the light brightness signal into the first electrical signal; transmitting the first electrical signal to the color image generator and the brightness change detector respectively; The color image generator is configured to generate an RGB image based on the first electrical signal; The brightness change detector is used to generate brightness change information according to the first electrical signal.

7. The image sensor according to claim 6, wherein: The EVS pixel circuit further includes a second photodetection element; the second photodetection element is connected to the EVS pixel element and the brightness change detector respectively; The EVS pixel element is further configured to transmit the light brightness signal to the second photodetection element; The second photodetection element is configured to convert the light brightness signal into a second electrical signal and transmit the second electrical signal to the brightness change detector; The brightness change detector is used to generate brightness change information according to the first electrical signal and the second electrical signal.

8. The image sensor according to claim 7, wherein: The second photodetection element is further configured to convert the light brightness signal into the second electrical signal when the RGB pixel circuit is in a non-conducting state, and transmit the second electrical signal to the brightness change detector; The brightness change detector is further configured to generate brightness change information according to the second electrical signal.

9. The image sensor according to claim 6, wherein: The brightness change detector includes: an electrical signal conversion element and a comparator; The electrical signal conversion element is configured to convert the first electrical signal into an electrical signal reference value; The comparator is used to compare the electrical signal reference value with a reference threshold value, and generate brightness change information according to the comparison result.

10. The image sensor according to claim 9, wherein: The reference threshold value includes an on threshold value and an off threshold value of the brightness change detector; The brightness change information includes: a first event value for representing an increase in brightness, or a second event value for representing a decrease in brightness; The comparator is further used to compare the electrical signal reference value with the turn-on threshold, and output the first event value when the electrical signal reference value is greater than or equal to the turn-on threshold; and to compare the electrical signal reference value with the turn-off threshold, and output the second event value when the electrical signal reference value is less than or equal to the turn-off threshold.

11. The image sensor according to claim 5, wherein The RGB pixel circuit and the EVS pixel circuit are connected via a switch device; the switch device is used to control whether the RGB pixel circuit and the EVS pixel circuit are conductive; Wherein, when the switch device is closed, the RGB pixel circuit and the EVS pixel circuit are connected, and the RGB pixel circuit transmits the first electrical signal to the EVS pixel circuit.

12. An electronic device, characterized in that: comprising the image sensor and image processor according to any one of claims 1 to 11; The image sensor is configured to capture a color light signal and a brightness signal of a moving object during an exposure time, and convert the color light signal and the brightness signal into a first electrical signal; generate an RGB image of the moving object and brightness change information of the moving object during the exposure time based on the first electrical signal; and transmit the RGB image and the brightness change information to the image processor; The image processor is configured to calibrate the RGB image according to the brightness change information to obtain an object image of the moving object.

13. An image generation method, executed by the electronic device according to claim 12, characterized in that: The method comprises: The image sensor captures a color light signal and a brightness signal of the moving object within an exposure time, and converts the color light signal and the brightness signal into a first electrical signal; The image sensor generates an RGB image of the moving object and brightness change information of the moving object within the exposure time according to the first electrical signal; The image processor calibrates the RGB image according to the brightness change information to obtain an object image of the moving object.

14. The method according to claim 13, characterized in that The brightness change information includes: a first event value for representing an increase in brightness, or a second event value for representing a decrease in brightness; Generating, according to the first electrical signal, the RGB image of the moving object and the brightness change information of the moving object during the exposure time includes: converting the first electrical signal into an electrical signal reference value; When the electrical signal reference value is greater than or equal to the turn-on threshold, outputting a first event value; When the electrical signal reference value is less than or equal to the shutdown threshold, a second event value is output.

15. The method according to claim 13, characterized in that The calibrating the RGB image according to the brightness change information to obtain the object image of the moving object includes: Determining motion trajectory data of the moving object within the exposure time according to the brightness change information; the motion trajectory data is used to represent the motion trajectory of each pixel point in the area where the moving object is located in the RGB image within the exposure time; According to the motion trajectory data, an image of a motion blur area in the RGB image is calibrated to obtain an object image; the motion blur area is an image area formed by motion blur generated by the movement of the moving object during the exposure time.

16. The method according to claim 15, characterized in that The step of calibrating an image of a motion blurred area in the RGB image according to the motion trajectory data to obtain an object image includes: Determining pixel position coordinates of each pixel point in the motion blur area within the exposure time according to the motion trajectory data; determining, according to the pixel position coordinates of each pixel point in the motion blurred area within the exposure time, the motion endpoint position coordinates of each pixel point in the motion blurred area within the exposure time; generating a calibration area image corresponding to the image of the motion blurred area according to the motion endpoint position coordinates; The image of the motion blurred area is replaced by the image of the calibration area to obtain an object image.

Images