Image sensor, electronic device, and image generation method

Abstract

The invention discloses an image sensor, electronic equipment and an image generation method, and belongs to the technical field of image processing. The image sensor comprises a photosensitive module and a pixel circuit module, and the photosensitive module is connected with the pixel circuit module; the photosensitive module comprises an optical lens, at least two RGB pixel elements and at least one EVS pixel element; the EVS pixel elements are embedded between the RGB pixel elements; the RGB pixel element is used for capturing a color light signal of an incident light beam of the optical lens within the exposure time and transmitting the color light signal to the pixel circuit module; the EVS pixel element is used for capturing a brightness signal of an incident light beam within the exposure time and transmitting the brightness signal to the pixel circuit module; the pixel circuit module is used for converting the color light signal and the light brightness signal into a first electric signal; according to the first electric signal, generating an RGB image of the moving object and brightness change information of the moving object in the exposure duration; the brightness change information is used for calibrating the RGB image.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image sensor, an electronic device and an image generation method. Background Art

[0002] Traditional image sensors have insufficient photosensitivity in low-light conditions and usually require extended exposure time to obtain sufficient light information. However, long exposures can easily lead to motion blur, especially in fast-moving scenes. EVS (Event Vision Sensor) technology effectively solves this problem through different working principles. In low-light or complex lighting conditions, EVS technology can capture key information about brightness changes and avoid motion blur caused by long exposure times. Unlike traditional image sensors, EVS uses a more efficient light processing mechanism to provide clear, detailed image data even in low-light conditions, significantly improving the performance of the sensor in low-light environments.

[0003] In the related art, the EVS sensor structure usually adds EVS pixel elements by changing the position of RGB pixel elements. The conventional RGB pixel structure uses filters of different colors to cover the top of the photodiode, and the filter may include one or more of a red filter, a green filter or a blue filter. The RGB pixel structure is usually a pixel array, which consists of multiple rows and columns of pixel units. Each pixel unit captures light signals and converts them into electrical signals. Each row controls the signal output of the entire row of pixels through a row selection signal, and each column processes the signal output of the corresponding column of pixels through the column analog-to-digital converter. In the related art, when improving the conventional RGB pixel structure, the specific approach is to remove some RGB pixel elements and use EVS pixel elements to directly replace the removed RGB pixel elements. Although this EVS sensor structure can solve the motion blur problem caused by long exposure time to a certain extent, the RGB pixel density is reduced due to the removal of some RGB pixel elements, which easily leads to a decrease in accuracy. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image sensor, an electronic device, and an image generation method, which can improve the imaging quality of images of moving objects.

[0005] In a first aspect, an embodiment of the present application provides an image sensor, comprising: a photosensitive module and a pixel circuit module; the photosensitive module is connected to the pixel circuit; The photosensitive module includes an optical lens, at least two RGB pixel elements and at least one EVS pixel element; the EVS pixel element is embedded between the RGB pixel elements; The RGB pixel element is used to capture the color light signal of the incident light beam of the optical lens within the exposure time, and transmit the color light signal to the pixel circuit module; The EVS pixel element is used to capture the brightness signal of the incident light beam within the exposure time and transmit the brightness signal to the pixel circuit module; The pixel circuit module is used to convert the color light signal and the light brightness signal into a first electrical signal; based on the first electrical signal, an RGB image of the moving object and the brightness change information of the moving object within the exposure time are generated; the brightness change information is used to calibrate the RGB image.

[0006] In a second aspect, an embodiment of the present application provides an electronic device, including the image sensor and the image processor described in the first aspect above; The image sensor is used to capture a color light signal and a light brightness signal of a moving object within an exposure time, and convert the color light signal and the light brightness signal into a first electrical signal; generate an RGB image of the moving object and brightness change information of the moving object within the exposure time according to the first electrical signal; and transmit the RGB image and the brightness change information to the image processor; The image processor is used to calibrate the RGB image according to the brightness change information to obtain the object image of the moving object.

[0007] In a third aspect, an embodiment of the present application provides an image generation method, which is performed by the electronic device described in the second aspect, and the method includes: The image sensor captures the color light signal and the light brightness signal of the moving object within the exposure time, and converts the color light signal and the light brightness signal into a first electrical signal; The image sensor generates an RGB image of the moving object and brightness change information of the moving object within the exposure time according to the first electrical signal; The image processor calibrates the RGB image according to the brightness change information to obtain an object image of the moving object.

[0008] In a fourth aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a sixth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0011] In a seventh aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0012] In an embodiment of the present application, the image sensor includes a photosensitive module and a pixel circuit module. The photosensitive module is connected to the pixel circuit. The photosensitive module includes an optical lens, at least two RGB pixel elements and at least one EVS pixel element. The EVS pixel element is embedded between the RGB pixel elements. Among them, the RGB pixel element is used to capture the color light signal of the incident light beam of the optical lens during the exposure time, and transmit the color light signal to the pixel circuit module; the EVS pixel element is used to capture the light brightness signal of the incident light beam during the exposure time, and transmit the light brightness signal to the pixel circuit module; the pixel circuit module is used to convert the color light signal and the light brightness signal into a first electrical signal, and generate an RGB image of a moving object and brightness change information of the moving object during the exposure time according to the first electrical signal, and the brightness change information is used to calibrate the RGB image. It can be seen that when generating an image based on an RGB image and brightness change information, there is no need to change the number and position of the RGB pixel elements in the image sensor. It is only necessary to embed the EVS pixel elements between the RGB pixel elements. Therefore, the addition of the EVS pixel elements will not affect the RGB pixel density of the image sensor, thereby reducing the problem of insufficient precision caused by the movement of the pixel elements, and achieving the effect of outputting high-quality images without affecting the RGB pixel density. In addition, this structure of embedding the EVS pixel elements between the RGB pixel elements has a lower limit on the number of EVS pixel elements, that is, more EVS pixel elements can be embedded. The more EVS pixel elements there are, the larger the photosensitive area of ​​the EVS pixel elements. Therefore, when shooting a moving object, more accurate brightness change information can be captured, and richer brightness change information can be captured, thereby providing more accurate and richer information for calibrating the RGB image. Furthermore, the RGB pixel elements and the EVS pixel elements share an optical lens to gather the incident light beam, avoiding the problem of optical crosstalk between multiple optical lenses and improving image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a schematic structural cross-sectional diagram of an image sensor provided by some embodiments of the present application; Figure 2 is a schematic principle diagram of an embedding method of an EVS pixel element with a horizontal straight-line structure provided in some embodiments of the present application; Figure 3 is a schematic principle diagram of an embedding method of an EVS pixel element of a vertical straight-line structure provided in some embodiments of the present application; Figure 4 is a schematic top view of the structure of an image sensor provided by some embodiments of the present application; Figure 5 is a schematic top view of the structure of an image sensor provided by some embodiments of the present application; Fig. 6A is a schematic top view of the structure of an image sensor provided by some embodiments of the present application; Figure 6B is a schematic top view of the structure of an image sensor provided by some embodiments of the present application; Figure 7 is a schematic structural cross-sectional diagram of an image sensor provided by some embodiments of the present application; Figure 8 is a schematic block diagram of a pixel circuit module provided in some embodiments of the present application; Fig. 9 is a schematic block diagram of a pixel circuit module provided in some embodiments of the present application; Fig.10 is a schematic circuit diagram of a pixel circuit module provided in some embodiments of the present application; Fig.11 is a schematic curve diagram of voltage variation over time in a pixel circuit module provided in some embodiments of the present application; Fig.12 is a schematic block diagram of an electronic device provided by some embodiments of the present application; Fig.13 is a schematic flow chart of an image generation method provided by some embodiments of the present application; Fig.14 is a schematic interface diagram of an application scenario of an image generation method provided in some embodiments of the present application; Fig.15 is a schematic output interface diagram of an RGB pixel circuit provided in some embodiments of the present application; Fig.16 is a schematic output interface diagram of an EVS pixel circuit provided in some embodiments of the present application; Fig.17 is a schematic interface diagram of a calibrated image in an image generation method provided in some embodiments of the present application; Fig.18 is a schematic structural diagram of an electronic device provided by some embodiments of the present application; Fig.19 It is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0015] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0016] The image sensor, electronic device and image generation method provided in the embodiments of the present application are described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0017] Figure 1 is a schematic structural diagram of an image sensor according to some embodiments of the present application. Figure 1 As shown, the image sensor includes a photosensitive module 10 and a pixel circuit module 20, which are connected. The photosensitive module 10 includes an optical lens 11, at least two RGB pixel elements 12 and at least one EVS pixel element 13, and the EVS pixel element 13 is embedded between the RGB pixel elements 12.

[0018] The RGB pixel element 12 is used to capture the color light signal of the incident light beam of the optical lens within the exposure time, and transmit the color light signal to the pixel circuit module 20.

[0019] The EVS pixel element 13 is used to capture the brightness signal of the incident light beam within the exposure time and transmit the brightness signal to the pixel circuit module 20.

[0020] The pixel circuit module 20 is used to convert the color light signal and the light brightness signal into a first electrical signal; based on the first electrical signal, an RGB image of the moving object and brightness change information of the moving object during the exposure time are generated, and the brightness change information is used to calibrate the RGB image.

[0021] The RGB pixel element 12 may be a pixel element covered with a color filter, and the color filter may include one or more of a red filter, a green filter, a blue filter, etc. Covering the RGB pixel element 12 on the surface of the pixel circuit module 20 can capture a color image.

[0022] When capturing the light brightness signal, the EVS pixel element 13 does not rely on long exposure, but captures the light brightness signal at a certain frequency. After the light brightness signal is transmitted to the pixel circuit module 20, the pixel circuit module 20 generates event information by detecting the change of pixel brightness in real time. For example, the EVS pixel element 13 captures the light brightness signal once per millisecond and sends the captured light brightness information to the pixel circuit module 20. The pixel circuit module 20 performs real-time detection on the received light brightness signal to determine the brightness change information. The brightness change information captured by the EVS pixel element 13 includes: the brightness change information of each pixel on the moving object during the exposure time. It can be seen that under the synergy of the EVS pixel element 13 and the pixel circuit module 20, it can respond at the moment when the light changes, without extending the exposure time because of the need to capture enough light signals. This event-based capture mechanism greatly reduces the long exposure requirements caused by insufficient photosensitivity, for example, in a low-light environment, due to insufficient photosensitivity, a longer exposure time is required. The use of this event-based capture mechanism of the present embodiment can avoid the problem of image blurring caused by long exposure time.

[0023] In an embodiment of the present application, the image sensor includes a photosensitive module and a pixel circuit module. The photosensitive module is connected to the pixel circuit. The photosensitive module includes an optical lens, at least two RGB pixel elements and at least one EVS pixel element. The EVS pixel element is embedded between the RGB pixel elements. Among them, the RGB pixel element is used to capture the color light signal of the incident light beam of the optical lens during the exposure time, and transmit the color light signal to the pixel circuit module; the EVS pixel element is used to capture the light brightness signal of the incident light beam during the exposure time, and transmit the light brightness signal to the pixel circuit module; the pixel circuit module is used to convert the color light signal and the light brightness signal into a first electrical signal, and generate an RGB image of a moving object and brightness change information of the moving object during the exposure time according to the first electrical signal, and the brightness change information is used to calibrate the RGB image. It can be seen that when generating an image based on an RGB image and brightness change information, there is no need to change the number and position of the RGB pixel elements in the image sensor. It is only necessary to embed the EVS pixel elements between the RGB pixel elements. Therefore, the addition of the EVS pixel elements will not affect the RGB pixel density of the image sensor, thereby reducing the problem of insufficient precision caused by the movement of the pixel elements, and achieving the effect of outputting high-quality images without affecting the RGB pixel density. In addition, this structure of embedding the EVS pixel elements between the RGB pixel elements has a lower limit on the number of EVS pixel elements, that is, more EVS pixel elements can be embedded. The more EVS pixel elements there are, the larger the photosensitive area of ​​the EVS pixel elements. Therefore, when shooting a moving object, more accurate brightness change information can be captured, and richer brightness change information can be captured, thereby providing more accurate and richer information for calibrating the RGB image. Furthermore, the RGB pixel elements and the EVS pixel elements share an optical lens to gather the incident light beam, avoiding the problem of optical crosstalk between multiple optical lenses and improving image quality.

[0024] In some embodiments of the present application, the EVS pixel element 13 is a cross-shaped structure or a straight-line structure. For the straight-line structure of the EVS pixel element 13, it may include a horizontal straight-line structure or a vertical straight-line structure.

[0025] In the case where the EVS pixel elements 13 are in a horizontally inline structure, the EVS pixel elements 13 in the horizontally inline structure are embedded between every two rows of RGB pixel elements 12 . Figure 2 1 is a schematic structural diagram of an embedded method of an EVS pixel element 13 with a horizontal straight-line structure according to some embodiments of the present application. Figure 2As shown, the EVS pixel elements 13 of the horizontal I-shaped structure are embedded between two rows of RGB pixel elements 12, the number of RGB pixel elements 12 in each row of RGB pixel elements 12 is not limited, and a plurality of EVS pixel elements 13 of the horizontal I-shaped structure can be embedded between every two rows of RGB pixel elements 12, and the plurality of EVS pixel elements 13 of the horizontal I-shaped structure are spliced ​​together, and the number of spliced ​​EVS pixel elements 13 depends on the number and length of the RGB pixel elements 12 in each row. Figure 2 The dotted line in the EVS pixel element 13 with a horizontal straight line structure shown indicates that two EVS pixel elements 13 with a horizontal straight line structure are spliced ​​together to form an EVS pixel element 13 with a longer horizontal straight line structure. Figure 2 The structure of some pixel elements is only shown as an example. Figure 2 The structure shown is regarded as a group of pixel elements. In practical applications, multiple groups of pixel elements with such a structure can be provided in the photosensitive module 10, and the arrangement of the multiple groups of pixel elements is not limited.

[0026] When the EVS pixel elements 13 are in a vertical in-line structure, the EVS pixel elements 13 in the vertical in-line structure are embedded between every two columns of RGB pixel elements 12 . Figure 3 1 is a schematic structural diagram of an embedded method of a vertically inline EVS pixel element 13 according to some embodiments of the present application. Figure 3 As shown, the EVS pixel elements 13 of the vertical straight-line structure are embedded between two columns of RGB pixel elements 12, the number of RGB pixel elements 12 in each column of RGB pixel elements 12 is not limited, and a plurality of EVS pixel elements 13 of the vertical straight-line structure can be embedded between every two columns of RGB pixel elements 12, and the plurality of EVS pixel elements 13 of the vertical straight-line structure are spliced ​​together, and the number of spliced ​​EVS pixel elements 13 depends on the number and length of the RGB pixel elements 12 in each column. Figure 3 The dotted line in the vertical in-line EVS pixel element 13 shown indicates that two vertical in-line EVS pixel elements 13 are spliced ​​together to form a vertical in-line EVS pixel element 13 with a longer length. Figure 3 The structure of some pixel elements is only shown as an example. Figure 3 The structure shown is regarded as a group of pixel elements. In practical applications, multiple groups of pixel elements with such a structure can be provided in the photosensitive module 10, and the arrangement of the multiple groups of pixel elements is not limited.

[0027] In the photosensitive module 10, only the EVS pixel elements 13 with a straight line structure or the EVS pixel elements 13 with a cross structure may be embedded, or both the EVS pixel elements 13 with a straight line structure and the EVS pixel elements 13 with a cross structure may be embedded.

[0028] In the case where all the EVS pixel elements 13 in the photosensitive module 10 are in a straight-line structure, an EVS pixel element 13 in a straight-line structure is embedded between at least two RGB pixel elements 12. The embedding method of the EVS pixel element 13 in a straight-line structure is as follows: Figure 2 and Figure 3 The same photosensitive module 10 can be embedded with both the EVS pixel element 13 in a horizontal straight line structure and the EVS pixel element 13 in a vertical straight line structure, or any one of them can be embedded.

[0029] When all the EVS pixel elements 13 in the photosensitive module 10 are of a cross-shaped structure, an EVS pixel element 13 of a cross-shaped structure is embedded between at least four RGB pixel elements 12 . Figure 4 FIG. 1 is a schematic top view showing the embedding method of the cross-shaped EVS pixel element 13 in the RGB pixel element 12. Figure 4 As shown, the dotted circle is an optical lens 11 located above the EVS pixel element 13 and the RGB pixel element 12, which may be a microlens, for example. Below the optical lens 11, the cross-shaped structure embedded in the center is the EVS pixel element 13, and the RGB pixel element 12 is located around the EVS pixel element 13. After the incident light beam is incident on the optical lens 11, the EVS pixel element 13 can capture the brightness signal of the incident light beam, and the RGB pixel element 12 can capture the color light signal of the incident light beam.

[0030] In the case where the photosensitive module 10 includes at least one EVS pixel element 13 with a straight line structure and at least one EVS pixel element 13 with a cross structure, an EVS pixel element 13 with a straight line structure is embedded between at least two RGB pixel elements 12, and an EVS pixel element 13 with a cross structure is embedded between at least four RGB pixel elements 12.

[0031] like Figure 5As shown, it shows the embedding mode of the two structures of EVS pixel elements in the same image sensor. The minimum pixel unit of the image sensor is an RGB pixel element or an EVS pixel element, wherein the area filled with oblique lines is the EVS pixel element, and the unfilled area is the RGB pixel element 12. The left oblique line filling mode indicates the EVS pixel element of the cross structure, and the right oblique line filling mode indicates the EVS pixel element of the straight structure. Considering every four RGB pixel elements as one RGB unit, it can be seen that in Figure 5 In the two RGB units on the left side, a cross-shaped EVS pixel element is embedded in each RGB unit. Taking the RGB unit in the upper left corner as an example, a cross-shaped EVS pixel element 1301 is embedded between four RGB pixel elements 12. Figure 5 In the two right columns of RGB units shown, a straight-line EVS pixel element is embedded in each RGB unit. Taking the RGB unit in the upper right corner as an example, a straight-line EVS pixel element 1302 is embedded between four RGB pixel elements 12, and the length of the straight-line EVS pixel element 1302 is the same as the width of the RGB unit, and the width of the RGB unit is equal to the sum of the widths of the two RGB pixel elements 12. It should be noted that Figure 5 The structure of some pixel elements is only shown as an example, and the filter type of each RGB pixel element 12 in the structure is not limited. For example, the RGB pixel element 12 can be an R pixel element covering a red filter, a GR pixel element covering a green filter, a GB pixel element covering a green filter, or a B pixel element covering a blue filter. In practical applications, the embedding positions of the EVS pixel element 1302 with a straight line structure and the EVS pixel element 1301 with a cross structure in the photosensitive module are not limited. In addition, for the embedding method of the EVS pixel element 1302 with a straight line structure, except for Figure 5 In addition to the above, a plurality of EVS pixel elements 1302 in a straight line structure may be embedded in each RGB unit, and the total length of the plurality of EVS pixel elements 1302 in a straight line structure is the same as the width of the RGB unit.

[0032] Fig. 6A and Figure 6B 1 is a schematic top view of the structure of an image sensor according to some embodiments of the present application. The smallest pixel unit of the image sensor is an RGB pixel element or an EVS pixel element, wherein the area filled with oblique lines is an EVS pixel element 13, and the unfilled area is an RGB pixel element. Fig. 6A and Figure 6BIn the embodiment, the image sensor includes a plurality of RGB pixel elements and a plurality of cross-shaped EVS pixel elements 13. The number of cross-shaped EVS pixel elements 13 is related to the number of RGB pixel elements. Fig. 6A and Figure 6B In the embedding mode, the number of RGB pixel elements is four times the number of EVS pixel elements 13. For example, assuming that the image sensor includes 64 RGB pixel elements, 16 EVS pixel elements 13 with a cross structure can be embedded in the image sensor. If every four RGB pixel elements are regarded as an RGB unit, the image sensor includes a total of 16 RGB units. One EVS pixel element 13 can be embedded in each of the 16 RGB units, or the EVS pixel element 13 can be embedded in only some of the RGB units. In other words, the number of EVS pixel elements 13 can be any number between 1 and 16. Each EVS pixel element 13 with a cross structure is embedded in the center of the four RGB pixel elements. The four RGB pixel elements around each EVS pixel element 13 with a cross structure can be RGB pixel elements of any filtering type.

[0033] Optionally, the filter types of the four RGB pixel elements surrounding the same EVS pixel element 13 are different, such as Fig. 6A As shown, in each RGB unit, the four RGB pixel elements include: an R pixel element covered with a red filter, a GR pixel element covered with a green filter, a GB pixel element covered with a green filter, and a B pixel element covered with a blue filter. Optionally, the four RGB pixel elements surrounding the same EVS pixel element 13 have the same filter type, such as Figure 6B As shown, in each RGB unit, the four RGB pixel elements may include one of an R pixel element covered with a red filter, a GR pixel element covered with a green filter, a GB pixel element covered with a green filter, or a B pixel element covered with a blue filter. Fig. 6A Taking the RGB unit at the upper left corner as an example, the RGB unit at the upper left corner includes four RGB pixel elements, which are respectively an R pixel element 1201 located at the upper left side and covering a red filter, a GR pixel element 1202 located at the upper right side and covering a green filter, a GB pixel element 1203 located at the lower left side and covering a green filter, and a B pixel element 1204 located at the lower right side and covering a blue filter. Figure 6BTaking the RGB unit in the upper left corner as an example, the RGB unit in the upper left corner includes four RGB pixel elements, which are respectively an R pixel element 1201 covering a red filter located on the upper left side, an R pixel element 1202 covering a red filter located on the upper right side, an R pixel element 1203 covering a red filter located on the lower left side, and an R pixel element 1204 covering a red filter located on the lower right side. It should be noted that Fig. 6A and Figure 6B The arrangement of the RGB pixel elements is only an example. In actual applications, there can be many arrangements of the RGB pixel elements. Regardless of the arrangement, the method of embedding the EVS pixel element 13 in multiple RGB pixel elements is similar and will not be listed one by one.

[0034] In some embodiments of the present application, since the EVS pixel element 13 can be a straight-line structure or a cross-shaped structure, the embedding position of the EVS pixel element 13 with a straight-line structure or a cross-shaped structure in the photosensitive module 10 is not limited, and there are multiple embedding methods, thereby achieving the effect of flexibly setting the EVS pixel element 13 in the photosensitive module 10, and this embedding method of the EVS pixel element 13 does not affect the number of RGB pixel elements 12 in the photosensitive module 10, thereby not affecting the RGB pixel density of the image sensor, effectively avoiding the problem of insufficient accuracy caused by changes in pixel density, while capturing the brightness change information of the moving object, it can also ensure the output of high-quality images.

[0035] In some embodiments of the present application, Figure 7 , which schematically shows a cross-sectional view of the structure of the image sensor. The pixel circuit module 20 includes an RGB pixel circuit 21 and an EVS pixel circuit 22, and the photosensitive module 10 is connected to the RGB pixel circuit 21 and the EVS pixel circuit 22 respectively. The RGB pixel circuit 21 and the EVS pixel circuit 22 are connected.

[0036] The RGB pixel element 12 is also used to transmit the color light signal to the RGB pixel circuit 21 .

[0037] The EVS pixel element 13 is also used to transmit the light brightness signal to the RGB pixel circuit 21 .

[0038] The RGB pixel circuit 21 is used to convert the color light signal and the light brightness signal into a first electrical signal, generate an RGB image of the moving object according to the first electrical signal, and transmit the first electrical signal to the EVS pixel circuit.

[0039] The EVS pixel circuit 22 is used to generate brightness change information according to the first electrical signal.

[0040] In some embodiments of the present application, an RGB image of a moving object is generated by an RGB pixel circuit 21, and brightness change information of the moving object is generated by an EVS pixel circuit 22, so that the RGB pixel circuit 21 and the EVS pixel circuit 22 work together to reduce the unclear problem caused by the long exposure time in the RGB image, especially in a low-light environment where the exposure time is longer and the image clarity is lower. In addition, since the RGB pixel circuit 21 can transmit the first electrical signal obtained after the optical signal conversion to the EVS pixel circuit 22 for use, the EVS pixel circuit 22 can share the photoelectric conversion module of the RGB pixel circuit 21, such as a photodiode, a photodetector, etc., to reduce the circuit area of ​​the additional photoelectric conversion module, so that more circuit area can be used to configure pixel elements and improve the pixel density of the image sensor. Furthermore, since the RGB pixel circuit 21 and the EVS pixel circuit 22 are designed in layers, that is, they both adopt independent circuit designs, it is possible to ensure the independent operation of each circuit layer and optimize the overall performance and image generation quality of the image sensor.

[0041] In some embodiments of the present application, Figure 8 As shown, the RGB pixel circuit 21 includes a first photodetection element 211 and a color image generator 212, and the first photodetection element 211 is connected to the color image generator 212. The EVS pixel circuit 22 includes a brightness change detector 221, and the first photodetection element 211 is connected to the brightness change detector 221. The RGB pixel element 12 and the EVS pixel element 13 in the photosensitive module 10 are both connected to the first photodetection element 211, so that the captured color light signal and light brightness signal can be transmitted to the first photodetection element 211.

[0042] The first photodetection element 211 is used to convert the color light signal and the light brightness signal into a first electrical signal, and transmit the first electrical signal to the color image generator 212 and the brightness change detector 221 respectively.

[0043] The color image generator 212 is used to generate an RGB image according to the first electrical signal.

[0044] The brightness change detector 221 is used to generate brightness change information according to the first electrical signal.

[0045] The first photoelectric detection element 211 may be a photodiode, or other components with a photoelectric conversion function.

[0046] Depend on Figure 8It can be seen that based on the connection relationship between the first photodetection element 211 and the brightness change detector 221, the first photodetection element 211 can transmit the first electrical signal obtained after conversion to the EVS pixel circuit 22 for use, so that the EVS pixel circuit 22 can share the first photodetection element 211 of the RGB pixel circuit 21, reducing the circuit area of ​​the additional photodetection element, thereby applying more circuit area to configuring pixel elements and improving the pixel density of the image sensor.

[0047] In some embodiments of the present application, Fig. 9 As shown, the EVS pixel circuit 22 further includes a second photodetection element 222; the second photodetection element 222 is connected to the EVS pixel element 13 and the brightness change detector 221 respectively.

[0048] The EVS pixel element 13 is also used to transmit the light brightness signal to the second photodetection element 222 .

[0049] The second photodetection element 222 is used to convert the light brightness signal into a second electrical signal, and transmit the second electrical signal to the brightness change detector 221 .

[0050] The brightness change detector 221 is used to generate brightness change information according to the first electrical signal and the second electrical signal.

[0051] Optionally, the second photodetection element 222 is further used to convert the light brightness signal into a second electrical signal when the RGB pixel circuit 21 is in a non-conducting state, and transmit the second electrical signal to the brightness change detector 221. The brightness change detector 221 is further used to generate brightness change information according to the second electrical signal.

[0052] In some embodiments of the present application, by configuring the second photodetection element 222 in the EVS pixel circuit 22, the EVS pixel circuit 22 can share the first photodetection element 211 in the RGB pixel circuit 21 in the scenario of working in coordination with the RGB pixel circuit 21. Since sharing the same photodetection element can ensure that the position difference is extremely small, the response accuracy of the EVS pixel circuit 22 can be improved. In addition, this structure enables the EVS pixel circuit 22 to be independent of the RGB pixel circuit 21 when working alone, and can use the second photodetection element 222 to convert the light brightness signal into an electrical signal, thereby obtaining brightness change information.

[0053] In some embodiments of the present application, the brightness change detector 221 includes: an electrical signal conversion element and a comparator. The electrical signal conversion element is used to convert the first electrical signal into an electrical signal reference value. The comparator is used to compare the electrical signal reference value with a reference threshold value and generate brightness change information according to the comparison result.

[0054] Optionally, the electrical signal conversion element is a capacitor, and the electrical signal reference value is a voltage value.

[0055] Optionally, the brightness change information includes: a first event value for characterizing an increase in brightness, or a second event value for characterizing a decrease in brightness. The reference threshold includes an on threshold and an off threshold of the brightness change detector 221. The comparator is used to compare the electrical signal reference value with the on threshold, and output the first event value when the electrical signal reference value is greater than or equal to the on threshold. The electrical signal reference value is compared with the off threshold, and output the second event value when the electrical signal reference value is less than or equal to the off threshold.

[0056] For example, the on threshold is 2.7V and the off threshold is 1V. When the electrical signal reference value is greater than or equal to the on threshold 2.7V, the first event value "1" is output. When the electrical signal reference value is less than or equal to the off threshold 1V, the second event value "0" is output. The values ​​of the on threshold and the off threshold are only examples, and this embodiment does not limit this. It only needs to satisfy that the on threshold is greater than the off threshold.

[0057] Optionally, each time the brightness change information is detected, the EVS pixel circuit 22 outputs an event value, which includes the brightness change information detected at the corresponding moment. The brightness change information can be represented as a frame of binary image, which includes two colors for representing the increase or decrease of brightness. For example, in an event value output once, for a pixel whose brightness increases to exceed the turn-on threshold, the position corresponding to the pixel is represented as black in the binary image, and the corresponding event value is 0; for a pixel whose brightness decreases to below the turn-off threshold, the position corresponding to the pixel is represented as white in the binary image, and the corresponding event value is 1. In this way, the brightness change information of the moving object at the corresponding moment can be obtained through the color information in the binary image, and the effect of outputting the brightness change information in a simple and easy-to-understand manner is achieved. In addition, the brightness change detection effect of the brightness change detector is realized by a comparator, which is also relatively simple in hardware implementation, saving hardware costs.

[0058] In some embodiments of the present application, the RGB pixel circuit 21 and the EVS pixel circuit 22 are connected via a switch device, and the switch device is used to control whether the RGB pixel circuit 21 and the EVS pixel circuit 22 are conductive.

[0059] When the switch device is closed, the RGB pixel circuit 21 and the EVS pixel circuit 22 are connected, and the RGB pixel circuit 21 transmits the first electrical signal to the EVS pixel circuit 22. At this time, the RGB pixel circuit 21 and the EVS pixel circuit 22 can share the same photodetection element, that is, the first photodetection element 211 in the above embodiment. When the switch device is turned off, the RGB pixel circuit 21 and the EVS pixel circuit 22 are disconnected, and the RGB pixel circuit 21 and the EVS pixel circuit 22 can work independently. At this time, the RGB pixel circuit 21 generates an RGB image by processing the color light signal captured by the RGB pixel element 12; the EVS pixel circuit 22 generates brightness change information by processing the light brightness signal captured by the EVS pixel element 13.

[0060] In some embodiments of the present application, the RGB pixel circuit 21 and the EVS pixel circuit 22 are connected by a switch device, so that the RGB pixel circuit 21 and the EVS pixel circuit 22 can work alone in some scenes, and can work together in some scenes, such as scenes where moving objects need to be photographed. In addition, when the EVS pixel circuit 22 works alone, accurate brightness change information can be obtained.

[0061] Fig.10 is a schematic circuit diagram of a pixel circuit module according to some embodiments of the present application. Fig.10 As shown, in the RGB pixel circuit 21, it includes a first photodiode PD1, a transmission gate MOS_2, a reset transistor MOS_1, a source follower MOS_3 and a row selector MOS_4 connected in sequence. One end of the row selector MOS_4 is grounded. The source follower MOS_3 is also connected between the reset transistor MOS_1 and the transmission gate MOS_2. Among them, the first photodiode PD1 is the first photodetection element 211, which is used to perform photoelectric conversion on the light signal captured by the photosensitive module 10 to obtain an electrical signal, and the light signal includes a color light signal and a light brightness signal. The electrical signal is stored and used for subsequent signal processing. The reset transistor MOS_1 is used to reset the electrical signal of the floating diffusion region between the reset transistor MOS_1 and the source follower MOS_3 to a fixed voltage, such as Fig.10 The Vdd shown in the figure clears the previously stored electrical signal and prepares for the next accumulation. The transmission gate MOS_2 is used to control the first photodiode PD1 to transmit the electrical signal to the floating diffusion area. When the transmission gate MOS_2 is turned on, the electrical signal is transmitted to the floating diffusion area. When the transmission gate MOS_2 is turned off, the electrical signal cannot be transmitted. The floating diffusion area is used to store the electrical signal transmitted from the first photodiode PD1. After that, the electrical signal is converted into a voltage signal and amplified. The source follower MOS_3 is used to amplify the voltage signal of the floating diffusion area and output it to the next stage circuit. It has high input impedance and low output impedance. The row selector MOS_4 is used to control the final output path of the signal and decide when to output the amplified voltage signal to the processing circuit.

[0062] When the RGB pixel circuit 21 is working, in the reset stage, when the reset transistor MOS_1 switch is turned on, the floating diffusion area is reset to a fixed voltage Vdd and the previously stored electrical signal is cleared. The first photodiode PD1 accumulates electrical signals under the illumination scene, and after the transmission gate MOS_2 is turned on, the electrical signal is transmitted from the first photodiode PD1 to the floating diffusion area. The electrical signal in the floating diffusion area is amplified by the source follower MOS_3 to generate a corresponding voltage signal. Afterwards, the row selector MOS_4 controls whether to output the amplified voltage signal to the subsequent processing circuit.

[0063] In the EVS pixel circuit 22, a second photodiode PD2, a logic operation device D, a capacitor C, an amplifier A and two comparators are connected in sequence. Among them, the second photodiode PD2 is a second photodetection element 222, which is used to perform photoelectric conversion on the light brightness signal captured by the photosensitive module 10 to obtain an electrical signal, such as a photocurrent Ip. The photocurrent Ip is proportional to the light intensity, and the photocurrent Ip will be superimposed with the dark current Idark. The capacitor C is an electrical signal conversion element, and the photocurrent Ip causes the voltage Vp on the capacitor C to change, that is, the electrical signal reference value changes. As the light increases or decreases, the voltage value on the capacitor C will rise or fall accordingly. After the voltage Vp is amplified by the amplifier A, the obtained voltage Vd enters the two comparators. The two comparators are executed in parallel. Fig.10 As shown in is the opening threshold, When the light changes and the voltage value is greater than or equal to the on threshold, the comparator will output a first event value, such as "ON" or "1", to represent an increase in brightness; when the light changes and the voltage value is less than or equal to the off threshold, the comparator will output a second event value, such as "OFF" or "0", to represent a decrease in brightness.

[0064] In the pixel circuit module 20, the output end of the first photodiode PD1 is connected between the second photodiode PD2 and the logic operation device D, so that the first photodiode PD1 can transmit the obtained electrical signal to the EVS pixel circuit 22, so that the EVS pixel circuit 22 can perform signal processing based on the electrical signal transmitted by the first photodiode PD1, thereby obtaining brightness change information.

[0065] Fig.11 The schematic diagram shows the curve of voltage Vp changing with time t. When the voltage Vp increases and exceeds the turn-on threshold When the voltage Vp decreases and is lower than the off threshold, the EVS pixel circuit 22 outputs the first event value "ON", indicating that the brightness increases. When the light intensity is low, the EVS pixel circuit 22 outputs the second event value "OFF", indicating that the brightness decreases. It can be seen that the EVS pixel circuit 22 can respond to brightness changes, and its response frequency is proportional to the light intensity. That is, the stronger the light, the faster the response speed of the EVS pixel circuit 22.

[0066] Fig.12 is a schematic structural diagram of an electronic device according to some embodiments of the present application. Fig.12 As shown, the electronic device 120 includes an image sensor 121 and an image processor 122. The image sensor 121 may be the image sensor described in any of the above embodiments, and its specific structure will not be repeated.

[0067] Among them, the image sensor 121 is used to capture the color light signal and light brightness signal of the moving object within the exposure time, and convert the color light signal and the light brightness signal into a first electrical signal; generate an RGB image of the moving object and the brightness change information of the moving object within the exposure time according to the first electrical signal; and transmit the RGB image and the brightness change information to the image processor.

[0068] The image processor 122 is used to calibrate the RGB image according to the brightness change information to obtain an object image of the moving object.

[0069] Optionally, the electronic device 120 may be any type of electronic device with an image generation function, such as a camera, a mobile phone, a tablet computer, etc. Taking a camera as an example, when a moving object is photographed by the camera, the image sensor 121 in the camera generates an RGB image and brightness change information of the moving object within the exposure time, and transmits the RGB image and brightness change information to the image processor 121. The image processor 122 calibrates the RGB image according to the brightness change information, thereby obtaining an object image of the moving object.

[0070] In some embodiments of the present application, the color light signal and the light brightness signal of the moving object within the exposure time are captured by the image sensor 121, and the color light signal and the light brightness signal are converted into a first electrical signal, and then an RGB image of the moving object and the brightness change information of the moving object within the exposure time are generated according to the first electrical signal, and the RGB image and the brightness change information are transmitted to the image processor 122. The RGB image is calibrated according to the brightness change information by the image sensor 122 to obtain an object image of the moving object. Since the EVS pixel elements 13 are distributed between the RGB pixel elements 12 in an embedded manner in the image sensor 121, more accurate brightness change information can be obtained, and richer brightness change information can be captured, thereby providing more accurate and richer information for calibrating the RGB image, and optimizing the image quality of the object image finally generated.

[0071] The execution subject in some embodiments of the present application may be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal or other devices other than a terminal. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It may also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiments of the present application.

[0072] The electronic device in some embodiments of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

[0073] Fig.13 is a schematic flow chart of an image generation method according to some embodiments of the present application. Fig.13 As shown, the image generation method is as follows Fig.12 The electronic device shown performs the following steps: Step 132: The image sensor captures the color light signal and the light brightness signal of the moving object within the exposure time, and converts the color light signal and the light brightness signal into a first electrical signal.

[0074] Among them, the color light signal is captured by the RGB pixel element, and the light brightness signal is captured by the EVS pixel element.

[0075] Step 134: The image sensor generates an RGB image of the moving object and brightness change information of the moving object within the exposure time according to the first electrical signal.

[0076] The brightness change information includes: a first event value used to represent an increase in brightness, or a second event value used to represent a decrease in brightness.

[0077] Step 134 can be executed as follows: convert the first electrical signal into an electrical signal reference value. When the electrical signal reference value is greater than or equal to the turn-on threshold, the first event value is output; when the electrical signal reference value is less than or equal to the turn-off threshold, the second event value is output. Optionally, the electrical signal reference value is a voltage value. The representation method of the first event value and the second event value is not limited, for example, it can be numbers, letters, identification symbols, etc. Taking numbers as an example, outputting the first event value "1" indicates that the brightness increases and is greater than or equal to the turn-on threshold. Outputting the second event value "0" indicates that the brightness decreases and is less than or equal to the turn-off threshold.

[0078] Step 136: The image processor calibrates the RGB image according to the brightness change information to obtain an object image of the moving object.

[0079] Optionally, step 136 may be performed as the following steps A1-A2: Step A1, determining the motion trajectory data of the moving object within the exposure time according to the brightness change information; the motion trajectory data is used to characterize the motion trajectory of each pixel point in the area where the moving object is located in the RGB image within the exposure time.

[0080] Step A22, calibrating the image of the motion blur area in the RGB image according to the motion trajectory data to obtain the object image; the motion blur area is the image area formed by the motion blur generated by the motion of the moving object within the exposure time.

[0081] When calibrating the image of the motion blurred area in the RGB image according to the motion trajectory data to obtain the object image, the pixel position coordinates of each pixel point in the motion blurred area within the exposure time can be first determined according to the motion trajectory data, and then the motion end point position coordinates of each pixel point in the motion blurred area within the exposure time can be determined according to the pixel position coordinates of each pixel point in the motion blurred area within the exposure time; according to the motion end point position coordinates, a calibration area image corresponding to the image of the motion blurred area is generated; and then the image of the motion blurred area is replaced with the calibration area image to obtain the target image.

[0082] In some embodiments of the present application, the color light signal and the light brightness signal of the target object are captured within the exposure time, and the color light signal and the light brightness signal are converted into a first electrical signal; based on the first electrical signal, an RGB image of the moving object and the brightness change information of the moving object within the exposure time are generated; the RGB image is calibrated based on the brightness change information to obtain an object image of the moving object. When generating an object image of a moving object, not only the RGB image can be obtained, but also accurate and rich brightness change information can be obtained, providing more accurate and rich information for calibrating the RGB image, and improving the quality of the object image finally generated.

[0083] The image generation method provided by the present application is described below through a specific embodiment.

[0084] In some embodiments of the present application, the electronic device is taken as a camera. The camera is pre-configured with the image sensor 121 and the image processor 122 described in the above embodiments. Fig.14 Taking the scenario shown in the figure as an example, Fig.14 In the example, the shooting scene is a sports scene of throwing a ball, and the ball 100 is a moving object. In this embodiment, under the condition of keeping the image brightness normal, an exposure time of 33 milliseconds is required. Assuming that the speed of the ball 100 is 30 pixels per millimeter, when the RGB pixel element 12 completes the 33 millisecond exposure, the ball 100 has moved 990 pixels. Therefore, the RGB pixel element 12 can only capture the result of the ball 100 after moving 990 pixels, which will cause obvious motion blur and smearing of the ball 100, for example Fig.15 As shown, the solid dot drawn on the sphere 100 represents the center of the sphere 100. Taking the center position of the sphere 100 as the reference, the sphere 100 moves from point A to point B, that is, the center position of the sphere 100 moves from point A to point B. Point A is the position of the sphere 100 at the beginning of the exposure time, and point B is the position of the sphere 100 at the end of the exposure time. At this time, the sphere 100 has moved 990 pixels on the image. Among them, the beginning of the exposure time is the 0th millisecond, and the end of the exposure time is the 33rd millisecond.

[0085] Since the working principle of the EVS pixel element 13 is to capture brightness changes in real time, under the same 33 milliseconds exposure condition, the EVS pixel element 13 can capture brightness changes at a frequency of once per millisecond, so that the EVS pixel circuit 22 outputs the brightness change information of the sphere 100 at a frequency of per millisecond, and finally outputs 33 event frames within 33 milliseconds, each event frame containing the event value corresponding to each pixel on the sphere 100. This method can not only fully record the movement process of the sphere 100, but also fuse these event frame data with the RGB image data through subsequent image processing algorithms, so as to calculate and restore the real position of the sphere 100 in the image, eliminate motion blur, and ensure that the final output image is clear and rich in details.

[0086] When the user turns on the camera and starts the shooting mode, the RGB pixel element 12 and the EVS pixel element 13 in the image sensor 121 are activated and sense light at the same time. The camera program dynamically sets the exposure conditions, including the exposure time, according to the current ambient brightness to ensure the correct exposure of the RGB image. The EVS pixel element 12 continuously tracks the brightness changes of the moving object, and outputs the brightness change information of the moving object through the EVS pixel circuit 22, providing key data for the subsequent calibration of the RGB image. It can be understood that the exposure condition setting action can be set in real time according to the current ambient brightness after turning on the camera, or it can be set in advance. For example, the exposure conditions of the camera are set in advance, and the pre-set exposure conditions will be automatically used each time the camera is turned on.

[0087] The RGB pixel element 12 and the EVS pixel element 13 are simultaneously sensitive to light, wherein the RGB pixel element 12 captures the color light signal of the incident light beam of the optical lens 11. Due to the long exposure time, which is 33 milliseconds in this embodiment, the RGB image output by the RGB pixel circuit 21 may contain a streak or blur of the sphere 100, but the RGB image can provide the overall color information and brightness data of the scene.

[0088] The EVS pixel element 13 captures the brightness signal of the incident light beam. Unlike the RGB pixel element 12, the EVS pixel element 13 has no exposure concept and does not rely on a fixed frequency. When the brightness in the scene changes, the EVS pixel element 13 will respond and capture the brightness signal, and then the EVS pixel circuit 22 will process the brightness signal and output brightness change information. During the entire exposure time of 33 milliseconds, the EVS pixel element 13 captures the brightness signal once per millisecond, so that the EVS pixel circuit 22 outputs brightness change information once per millisecond. Whenever the brightness changes significantly, such as when the brightness change causes an electrical signal reference value, such as Fig.10 and Fig.11 When the voltage value Vp in increases to exceed the turn-on threshold, or decreases to below the turn-off threshold, the EVS pixel circuit 22 will output an event value, which includes a first event value for characterizing an increase in brightness or a second event value for characterizing a decrease in brightness. In this embodiment, each pixel on the sphere 100 corresponds to its own event value. Therefore, at each moment within the exposure time, the EVS pixel circuit 22 will output an event frame, and each event frame contains the event value corresponding to each pixel on the sphere 100 and the position coordinates of the pixel point. If the event value is represented by "0" and "1", the output form of the event frame is a binary image, in which the event value "0" is represented by black and the event value "1" is represented by white.

[0089] After the entire exposure time is over, the EVS pixel circuit 22 outputs 33 event frames. Based on the 33 event frames, the motion trajectory data of the sphere 100 during the entire exposure time can be determined. Since the EVS pixel elements 13 are fewer in number than the RGB pixel elements 12, and an independent EVS pixel circuit 22 is applicable, data can be output at a higher frame rate to capture the brightness change information of the moving object during the entire exposure time. Even in a scene where the sphere 100 moves quickly, the EVS pixel circuit 22 can obtain accurate brightness change information, thereby generating accurate motion trajectory data.

[0090] It can be understood that the RGB pixel circuit 21 outputs a complete static image, which may contain a streaking or blurred area, such as Fig.15 The EVS pixel circuit 22 outputs a series of data streams, which are brightness change information. Based on the data streams, the motion trajectory data of the sphere 100 can be determined, such as Fig.16 As shown. Fig.16 It can be seen that the motion trajectory of the sphere 100 during the entire exposure time is still represented by the solid dot drawn on the sphere 100, and the position of the sphere center is taken as the reference. Fig.16 The position of point B in the image is the end point of the movement of the sphere 100 during the exposure time. Here, the movement trajectory of the sphere is highlighted, so the movement of the person during the exposure time is ignored. In practical applications, if the person moves during the entire exposure time, the data stream output by the EVS pixel circuit 22 can also determine the movement trajectory data of the person.

[0091] Afterwards, the image sensor 121 transmits the RGB image and the brightness change information to the image processor 122, and the image processor 122 calibrates the RGB image according to the brightness change information. Since the RGB pixel element 12 and the EVS pixel element 13 share a position and the position difference is very small, the RGB image can rely on the brightness change information output by the EVS pixel circuit 22 to accurately calculate how many pixels the moving object has moved during the exposure time, thereby calibrating the pixel position, eliminating the image blur problem caused by motion events in the RGB image, and outputting a clear image without blur areas.

[0092] When the image processor 122 calibrates the RGB image according to the brightness change information, it uses the high temporal resolution characteristics of the EVS pixel element 13 to compensate for the motion blur in the RGB image caused by the poor exposure time, thereby obtaining a clear and detailed object image, such as Fig.17 As shown, the object image obtained after calibration is clear and has no blurred areas.

[0093] Specifically, the image processor 122 first determines a motion blur region in the RGB image, where the motion blur region is an image region formed by motion blur caused by the motion of the moving object during the exposure time. Fig.15 As shown, the basketball player 300 throws the basketball, and the throwing action causes the ball 100 to move, and moves from point A to point B within the exposure time. The motion blur area 200 is an image area formed by the motion blur generated by the movement of the ball 100 within the exposure time. Then, the image processor 122 determines the motion trajectory data of the ball 100 within the exposure time based on the brightness change information. The image processor 122 does not limit the order when determining the motion blur area and the motion trajectory data. Since the motion trajectory data can characterize the motion trajectory of each pixel point in the area where the ball 100 is located in the RGB image within the exposure time, the image processor 122 can determine the pixel position coordinates of each pixel point on the ball 100 on the RGB image based on the motion trajectory data, thereby generating a calibration area image corresponding to the image of the motion blur area based on the pixel position coordinates, and then replacing the image of the motion blur area with the calibration area image, thereby obtaining an object image of the ball 100. During the image calibration process, the image processor 122 can retain the color and brightness information of the RGB image, and at the same time combine the motion trajectory data of the EVS to ensure the clarity and accuracy of the moving object in the image, and restore the true position and shape of the fast-moving object. Therefore, the image quality of the final object image can be optimized.

[0094] Optionally, the EVS pixel circuit 22 outputs a plurality of event frames, each of which contains an event value and a pixel position coordinate corresponding to each pixel on the sphere 100 at a corresponding moment, and each event frame is used to represent the brightness change information of the sphere 100 at a corresponding moment. When calibrating the RGB image according to the plurality of brightness change information, since the event frame output at the last moment of the exposure duration represents the final moving position information of the sphere 100, that is, the motion end position coordinate of each pixel point in the motion blur region 200 within the exposure duration. Therefore, the calibration region image corresponding to the image of the motion blur region 200 can be generated according to the motion end position coordinate of the sphere 100, for example Fig.16 The image corresponding to the position of point B where the sphere 100 finally moves is the calibration area image. The image of the motion blur area 200 is replaced by the calibration area image to obtain the following: Fig.17 An object image of a sphere 100 is shown.

[0095] The image processor 122 can further optimize the image through conventional image processing methods, for example, through image processing algorithms such as gamma correction, color matrix adjustment, and interpolation, to generate higher quality images to ensure accurate colors and clear details. Since image processing algorithms such as gamma correction, color matrix adjustment, and interpolation are conventional means of image processing, they will not be described in detail.

[0096] The object image processed by the image processor 122 is displayed on the preview interface of the camera, ensuring that the user can view a clear and detailed image in real time.

[0097] Alternatively, if Fig.18 As shown, some embodiments of the present application also provide an electronic device 1800, including a processor 1801 and a memory 1802, and the memory 1802 stores a program or instruction that can be executed on the processor 1801. When the program or instruction is executed by the processor 1801, the various steps of the above-mentioned image generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0098] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0099] Fig.19 A schematic diagram of the hardware structure of an electronic device for implementing some embodiments of the present application. The electronic device 2000 includes but is not limited to: a radio frequency unit 2001, a network module 2002, an audio output unit 2003, an input unit 2004, a sensor 2005, a display unit 2006, a user input unit 2007, an interface unit 2008, a memory 2009, and a processor 2010 and other components.

[0100] Those skilled in the art will appreciate that the electronic device 2000 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 2010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.19 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here. Among them, the processor 2010 is used to capture the color light signal and the light brightness signal of the moving object within the exposure time, and convert the color light signal and the light brightness signal into a first electrical signal; generate an RGB image of the moving object and the brightness change information of the moving object within the exposure time according to the first electrical signal; calibrate the RGB image according to the brightness change information to obtain the object image of the moving object.

[0101] Optionally, the brightness change information includes: a first event value for representing an increase in brightness, or a second event value for representing a decrease in brightness; Processor 2010 is also used to convert the first electrical signal into an electrical signal reference value; when the electrical signal reference value is greater than or equal to the turn-on threshold, output the first event value; when the electrical signal reference value is less than or equal to the turn-off threshold, output the second event value.

[0102] Optionally, the processor 2010 is further used to determine the motion trajectory data of the moving object within the exposure time according to the brightness change information; the motion trajectory data is used to characterize the motion trajectory of each pixel point in the area where the moving object is located in the RGB image within the exposure time; according to the motion trajectory data, the image of the motion blurred area in the RGB image is calibrated to obtain the object image; the motion blurred area is the image area formed by the motion blur generated by the movement of the moving object within the exposure time.

[0103] Optionally, the processor 2010 is further used to determine the pixel position coordinates of each pixel point in the motion blur area within the exposure time according to the motion trajectory data; determine the motion endpoint position coordinates of each pixel point in the motion blur area within the exposure time according to the pixel position coordinates of each pixel point in the motion blur area within the exposure time; generate a calibration area image corresponding to the image of the motion blur area according to the motion endpoint position coordinates; and replace the image of the motion blur area with the calibration area image to obtain an object image.

[0104] In some embodiments of the present application, the color light signal and the light brightness signal of the moving object are captured within the exposure time, and the color light signal and the light brightness signal are converted into a first electrical signal; based on the first electrical signal, an RGB image of the moving object and the brightness change information of the moving object within the exposure time are generated; the RGB image is calibrated based on the brightness change information to obtain an object image of the moving object. When generating an object image of a moving object, not only the RGB image but also accurate and rich brightness change information can be obtained, providing more accurate and rich information for calibrating the RGB image, and optimizing the image quality of the object image finally generated.

[0105] It should be understood that in the embodiment of the present application, the input unit 2004 may include a graphics processor (Graphics Processing Unit, GPU) 20041 and a microphone 20042, and the graphics processor 2041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 2006 may include a display panel 20061, and the display panel 20061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 2007 includes at least one of a touch panel 20071 and other input devices 20072. The touch panel 20071 is also called a touch screen. The touch panel 20071 may include two parts: a touch detection device and a touch controller. Other input devices 20072 may include but are not limited to physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, and joysticks, which will not be repeated here.

[0106] The memory 2009 can be used to store software programs and various data. The memory 2009 can mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area can store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 2009 can include a volatile memory or a non-volatile memory, or the memory 2009 can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 2009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0107] The processor 2010 may include one or more processing units; optionally, the processor 2010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 2010.

[0108] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image generation method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0109] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0110] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0111] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0112] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0113] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0115] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image sensor, characterized in that: include: A photosensitive module and a pixel circuit module; the photosensitive module is connected to the pixel circuit; The photosensitive module includes an optical lens, at least two RGB pixel elements and at least one EVS pixel element; the EVS pixel element is embedded between the RGB pixel elements; The RGB pixel element is used to capture the color light signal of the incident light beam of the optical lens within the exposure time, and transmit the color light signal to the pixel circuit module; The EVS pixel element is used to capture the brightness signal of the incident light beam within the exposure time and transmit the brightness signal to the pixel circuit module; The pixel circuit module is used to convert the color light signal and the light brightness signal into a first electrical signal; and generate an RGB image of the moving object and brightness change information of the moving object within the exposure time according to the first electrical signal; The brightness change information is used to calibrate the RGB image.

2. The image sensor according to claim 1, characterized in that The EVS pixel element is a cross-shaped structure or a straight-shaped structure.

3. The image sensor according to claim 2, characterized in that The EVS pixel element is a horizontal straight line structure or a vertical straight line structure; Wherein, when the EVS pixel element is a horizontal straight-line structure, the EVS pixel element of the horizontal straight-line structure is embedded between every two rows of RGB pixel elements; In the case where the EVS pixel elements are in a vertical in-line structure, the EVS pixel elements in the vertical in-line structure are embedded between every two columns of RGB pixel elements.

4. The image sensor according to claim 2, characterized in that: When all the EVS pixel elements in the photosensitive module are in a straight-line structure, an EVS pixel element in a straight-line structure is embedded between at least two RGB pixel elements; In the case where all EVS pixel elements in the photosensitive module are of a cross-shaped structure, an EVS pixel element of a cross-shaped structure is embedded between at least four RGB pixel elements; When the photosensitive module includes at least one EVS pixel element with a straight-line structure and at least one EVS pixel element with a cross structure, an EVS pixel element with a straight-line structure is embedded between every at least two RGB pixel elements, and an EVS pixel element with a cross structure is embedded between every at least four RGB pixel elements.

5. The image sensor according to claim 1, characterized in that The pixel circuit module includes an RGB pixel circuit and an EVS pixel circuit; the photosensitive module is connected to the RGB pixel circuit and the EVS pixel circuit respectively, and the RGB pixel circuit is connected to the EVS pixel circuit; The RGB pixel element is further used to transmit the color light signal to the RGB pixel circuit; The EVS pixel element is also used to transmit the light brightness signal to the RGB pixel circuit; The RGB pixel circuit is used to convert the color light signal and the light brightness signal into a first electrical signal, generate an RGB image of the moving object according to the first electrical signal, and transmit the first electrical signal to the EVS pixel circuit; The EVS pixel circuit is used to generate brightness change information according to the first electrical signal.

6. The image sensor according to claim 5, characterized in that The RGB pixel circuit includes a first photodetection element and a color image generator, and the first photodetection element is connected to the color image generator; the EVS pixel circuit includes a brightness change detector; the first photodetection element is connected to the brightness change detector; The first photoelectric detection element is used to convert the color light signal and the light brightness signal into the first electrical signal; transmitting the first electrical signal to the color image generator and the brightness change detector respectively; The color image generator is used to generate an RGB image according to the first electrical signal; The brightness change detector is used to generate brightness change information according to the first electrical signal.

7. The image sensor according to claim 6, characterized in that: The EVS pixel circuit also includes a second photodetection element; the second photodetection element is connected to the EVS pixel element and the brightness change detector respectively; The EVS pixel element is also used to transmit the light brightness signal to the second photoelectric detection element; The second photoelectric detection element is used to convert the light brightness signal into a second electrical signal and transmit the second electrical signal to the brightness change detector; The brightness change detector is used to generate brightness change information according to the first electrical signal and the second electrical signal.

8. The image sensor according to claim 7, characterized in that: The second photodetection element is further used to convert the light brightness signal into the second electrical signal when the RGB pixel circuit is in a non-conducting state, and transmit the second electrical signal to the brightness change detector; The brightness change detector is further used to generate brightness change information according to the second electrical signal.

9. The image sensor according to claim 6, characterized in that: The brightness change detector includes: an electrical signal conversion element and a comparator; The electrical signal conversion element is used to convert the first electrical signal into an electrical signal reference value; The comparator is used to compare the electrical signal reference value with a reference threshold value, and generate brightness change information according to the comparison result.

10. The image sensor according to claim 9, characterized in that: The reference threshold value includes an on threshold value and an off threshold value of the brightness change detector; The brightness change information includes: a first event value used to represent an increase in brightness, or a second event value used to represent a decrease in brightness; The comparator is further used to compare the electrical signal reference value with the turn-on threshold, and output the first event value when the electrical signal reference value is greater than or equal to the turn-on threshold; and to compare the electrical signal reference value with the turn-off threshold, and output the second event value when the electrical signal reference value is less than or equal to the turn-off threshold.

11. The image sensor according to claim 5, characterized in that: The RGB pixel circuit and the EVS pixel circuit are connected via a switch device; the switch device is used to control whether the RGB pixel circuit and the EVS pixel circuit are conductive; Wherein, when the switch device is closed, the RGB pixel circuit and the EVS pixel circuit are connected, and the RGB pixel circuit transmits the first electrical signal to the EVS pixel circuit.

12. An electronic device, characterized in that: An image sensor and an image processor comprising any one of claims 1 to 11; The image sensor is used to capture a color light signal and a light brightness signal of a moving object within an exposure time, and convert the color light signal and the light brightness signal into a first electrical signal; generate an RGB image of the moving object and brightness change information of the moving object within the exposure time according to the first electrical signal; and transmit the RGB image and the brightness change information to the image processor; The image processor is used to calibrate the RGB image according to the brightness change information to obtain the object image of the moving object.

13. An image generation method, executed by the electronic device according to claim 12, characterized in that: The method comprises: The image sensor captures the color light signal and the light brightness signal of the moving object within the exposure time, and converts the color light signal and the light brightness signal into a first electrical signal; The image sensor generates an RGB image of the moving object and brightness change information of the moving object within the exposure time according to the first electrical signal; The image processor calibrates the RGB image according to the brightness change information to obtain an object image of the moving object.

14. The method according to claim 13, characterized in that The brightness change information includes: a first event value used to represent an increase in brightness, or a second event value used to represent a decrease in brightness; The step of generating the RGB image of the moving object and the brightness change information of the moving object within the exposure time according to the first electrical signal includes: converting the first electrical signal into an electrical signal reference value; When the electrical signal reference value is greater than or equal to the start threshold, outputting a first event value; When the electrical signal reference value is less than or equal to the closing threshold, a second event value is output.

15. The method according to claim 13, characterized in that The step of calibrating the RGB image according to the brightness change information to obtain the object image of the moving object includes: Determine, according to the brightness change information, motion trajectory data of the moving object within the exposure time; the motion trajectory data is used to characterize the motion trajectory of each pixel point in the area where the moving object is located in the RGB image within the exposure time; According to the motion trajectory data, the image of the motion blur area in the RGB image is calibrated to obtain the object image; the motion blur area is an image area formed by motion blur generated by the movement of the moving object within the exposure time.

16. The method according to claim 15, characterized in that The step of calibrating the image of the motion blur region in the RGB image according to the motion trajectory data to obtain the object image comprises: Determine the pixel position coordinates of each pixel point in the motion blur area within the exposure time according to the motion trajectory data; Determine the motion endpoint position coordinates of each pixel point in the motion blur area within the exposure time according to the pixel position coordinates of each pixel point in the motion blur area within the exposure time; Generating a calibration area image corresponding to the image of the motion blurred area according to the motion endpoint position coordinates; The image of the motion blurred area is replaced by the image of the calibration area to obtain an object image.

Images