A visual system

Abstract

The present invention discloses a visual system, including an image sensor, which is used to collect original images. When the initial voltage signal of the same pixel point in the n-th frame original image and the n-1-th frame original image changes, the analog signal of the pixel point is converted into a digital signal to obtain the digital signal of the pixel point, and a trigger signal is sent to an event counting module. The event counting module is used to determine the number of target pixels in the original image whose initial voltage signals change by counting the number of trigger signals from the image sensor during the exposure time of the original image. The image processing module is used to output the digital signals of each pixel point of the original image in a synchronous transmission mode when the ratio of the number of target pixels to the total number of pixels of the original image is greater than a set threshold; otherwise, the digital signal of the target pixel point in the original image is output. The visual system provided by the present invention is used to reduce the data transmission bandwidth under the premise of ensuring image resolution and image quality.

Description

Technical Field

[0001] The present invention relates to the field of semiconductor image sensing technology, and in particular to a vision system. Background Art

[0002] In recent years, the field of computer vision has maintained rapid development. A large number of studies in this field use cameras and computers to replace human eyes to intelligently detect, identify and track targets, and have been widely used in security monitoring, autonomous driving, medical care and other scenarios. A typical industrial machine vision system includes an optical system (light source, lens, etc.), an image sensor, an image processing unit, a communication unit, etc. Among them, the image sensor is used to obtain visual data. Complementary Metal Oxide Semiconductor Image Sensor (CIS), as a typical solid imaging sensor, is a key component of digital cameras and is widely used in industrial cameras, consumer electronics, digital products, security monitoring and other fields. Traditional CIS outputs each frame of the image at a fixed frame rate (about 25 frames / second). Each pixel on the image reflects the brightness value of the position. There is a lot of background information in such a picture. Considering that the picture area belonging to the background information is often consistent in multiple consecutive frames, this part of the data is redundant data for the subsequent image processing algorithm. Transmitting such redundant data to the data encoding module not only wastes computing power, but also wastes transmission bandwidth and power.

[0003] Although the prior art provides an event-based vision sensor (Event-based Camera Sensor), also known as an event camera, a typical example of which is a dynamic vision sensor (DVS), each pixel independently generates asynchronous events under the stimulation of light changes. Whenever the light change on a pixel is greater than a set threshold, a corresponding event pulse is emitted. Although the event stream generated in this way eliminates the static redundant data of traditional image sensors, extracts the motion information in the scene, and reduces power consumption and bandwidth occupancy, it still has the following shortcomings: (1) Many tasks in computer vision rely on comprehensive information in the image, including absolute brightness, color, contrast, saturation, texture, etc., while the output of the event camera loses this information and only retains the coordinates and time of the pixel point where the brightness changes; (2) Since the number of events in the event stream is not fixed and the total length of the data is not fixed, it cannot be directly input into the convolutional neural network (Convolutional Neural Network). (3) Although the event stream removes the static redundant information on the two-dimensional screen, it itself has a lot of redundancy in the time dimension. Most monitoring scenarios do not need to pay attention to high-speed motion in microseconds, and the event stream does not have a good grasp of the motion characteristics of low-texture areas.

[0004] Based on the above background, how to control and optimize system power consumption without reducing image resolution and image quality has become a technical challenge currently faced. Summary of the invention

[0005] The embodiment of the present invention provides a visual system, which is used to reduce the data transmission bandwidth and save system power consumption under the premise of ensuring image resolution and image quality.

[0006] In a first aspect, the present invention provides a visual system, comprising an image sensor, an event counting module and an image processing module: the image sensor is used to collect raw images, and when there is a change in the initial voltage signal of the same pixel in the n-1 frame of the raw image, the analog-to-digital conversion unit is triggered to perform analog-to-digital conversion on the analog signal of the pixel to obtain the digital signal of the pixel, and send a trigger signal to the event counting module. The event counting module is used to determine the number of target pixels whose initial voltage signals in the n-1 frame of the raw image have changed by counting the number of trigger signals from the image sensor during the exposure time of the n-1 frame of the raw image. The image processing module is used to output the digital signals of each pixel of the n-1 frame of the raw image in a synchronous transmission mode when the ratio of the number of target pixels to the total number of pixels in the n-1 frame of the raw image is greater than a set threshold; and output the digital signals of the target pixels in the n-1 frame of the raw image in an asynchronous transmission mode when the ratio of the number of target pixels to the total number of pixels in the n-1 frame of the raw image is less than or equal to the set threshold.

[0007] The beneficial effect of the visual system provided by the present invention is that the visual system supports two modes of outputting the data of the CMOS image sensor, which are divided into synchronous and asynchronous modes. When there are many target pixels with changes in the initial voltage signal, the digital signal of the entire frame image is output, which helps to retain the comprehensive information in the image to the maximum extent, including absolute brightness, color, contrast, saturation, texture, etc. When the target pixels with changes in the initial voltage signal are small, the pixel values ​​of the changed pixels are transmitted. Since the data of such pixel points has strong spatial sparsity, the data redundancy can be effectively reduced. Then the visual system provided by the present invention is used to ensure that the data transmission bandwidth is reduced while ensuring the image resolution and image quality, saving system power consumption.

[0008] Optionally, the image processing module outputs the digital signals of each pixel point of the nth frame original image in a synchronous transmission mode, including: for the target pixel point whose initial voltage signal has changed, the image processing module outputs the digital signal of the target pixel point of the nth frame original image in a synchronous transmission mode, and the mark of the change of the initial voltage signal of the target pixel point; for the non-target pixel point whose initial voltage signal has not changed, the image processing module outputs the digital signal of the non-target pixel point of the n-1th frame original image in a synchronous transmission mode. It can be seen that this implementation can realize that when there are many pixels points that have obvious changes in the previous and next frame images, the data generated by the image sensor can be transmitted outward in a synchronous manner, and the comprehensive information of the original image such as image texture and brightness can be retained to a large extent.

[0009] Optionally, the image processing module outputs the digital signal of the target pixel in the nth frame of the original image in an asynchronous transmission mode, including: for the target pixel whose initial voltage signal changes, the image processing module outputs the digital signal of the target pixel in the nth frame of the original image in an asynchronous transmission mode, and the row and column addresses corresponding to the target pixel. It can be seen that this implementation can realize that when the pixel that produces obvious changes in the previous and next frame images is small, the data generated by the image sensor can be transmitted outward in an asynchronous manner, which greatly saves the transmission bandwidth.

[0010] Optionally, the set threshold satisfies the following formula:

[0011]

[0012] Among them, th represents the set threshold, len L Indicates the bit width occupied by the pixel value, H represents the number of pixels in each row of the original image, W represents the number of pixels in each column of the original image, and the length of the information carried by each pixel is len L +1, 1 means Out c The bit width, Indicates rounding up. The larger the image resolution, the smaller the th threshold is.

[0013] Optionally, when the ratio of the number of target pixels to the total number of pixels in the n-th frame of the original image is less than or equal to a set threshold, outputting a digital signal of the target pixel in the n-th frame of the original image in an asynchronous transmission manner includes:

[0014] An image processing module is used to extract the event set generated by all target pixels in the time period corresponding to the acquisition time of the nth frame original image to the acquisition time of the n-1th frame, and the region of interest from the nth frame original image according to the region boundary of the moving object corresponding to the event set, and output the digital signals of each pixel of the region of interest in the nth frame original image in an asynchronous transmission mode; the region of interest includes the digital signals of the target pixels in the nth frame original image. This implementation scheme can pre-process the original image based on the event activity at the edge of the object, extract the region of interest (ROI) from the sensor to the receiver, and the area outside the region of interest is not transmitted. This method further reduces the amount of transmitted data, which is particularly evident in wide-angle monitoring scenes. In addition, it also enhances the effectiveness of the information carried by the data, so that the efficiency of subsequent visual algorithms such as classification and regression is improved.

[0015] Optionally, the image sensor includes a pixel array composed of an analog-to-digital conversion unit and a pixel unit circuit;

[0016] The pixel unit circuit includes a photoelectric signal conversion unit, a voltage amplification unit, a first output unit, a second output unit and a comparison unit.

[0017] A photoelectric signal conversion unit, used to convert exposure information of a pixel point in an image frame into an initial voltage signal; a voltage amplifying unit, used to amplify and output the initial voltage signal; a first output unit connected to the voltage amplifying unit, used to receive and store a first voltage signal of a pixel point in the image frame at the n-1th frame; a second output unit connected to the voltage amplifying unit, used to receive and output a second voltage signal of the pixel point in the image frame at the nth frame; a comparing unit connected to the output end of the first output unit and the output end of the second output unit, respectively, the comparing unit is used to receive the first voltage signal from the first output unit and receive the second voltage signal from the second output unit, and compare the first voltage signal with the second voltage signal to determine whether there is a change between the initial voltage signal of the pixel point in the image frame at the n-1th frame and the initial voltage signal of the pixel point in the image frame at the nth frame.

[0018] The pixel unit circuit provided by the present invention has the beneficial effect that: the pixel unit circuit includes two output units, wherein the second output unit stores the pixel signal of the nth frame, and the first output unit stores the pixel signal of the n-1th frame, and the voltage signals of the two output units are simultaneously output to the comparison unit. The pixel structure is suitable for processing sampling information in parallel with rows, including the subsequent voltage signal comparison and analog-to-digital conversion process. Since the pixels in the same row are operated in parallel, the operation speed can be greatly improved, and the system operation bandwidth can be improved; in addition, the comparison unit can determine whether the initial voltage signal of the pixel point has changed, which is helpful to determine whether to trigger the analog-to-digital conversion.

[0019] Optionally, the photoelectric signal conversion unit includes a photodiode; the voltage amplification unit includes a first MOS tube, a second MOS tube and a third MOS tube; the source or drain of the first MOS tube and the gate of the second MOS tube are both connected to the cathode of the photodiode, the source or drain of the second MOS tube is connected to the source or drain of the third MOS tube, and the drain or source of the second MOS tube and the drain or source of the first MOS tube are both connected to the power supply voltage.

[0020] Optionally, the comparison unit includes a comparator; the first output unit includes a first switch transistor, a capacitor, a second switch transistor, and a fourth MOS transistor and a fifth MOS transistor;

[0021] The capacitor is used to store charge, one end of the capacitor is connected to the first switch transistor and the second switch transistor, and the other end of the capacitor is grounded; the gate of the fifth MOS tube is connected to the second switch transistor, the source or drain of the fifth MOS tube is connected to the source or drain of the fourth MOS tube, and the drain or source of the fifth MOS tube is connected to the first input end of the comparator; the drain or source of the fourth MOS tube is grounded;

[0022] The first switching transistor and the second switching transistor are used to control the charging and discharging state of the capacitor; when the first switching transistor is closed and the second switching transistor is disconnected, the initial voltage signal of the pixel point of the image frame is stored in the capacitor; when the first switching transistor is disconnected and the second switching transistor is closed, the charge in the capacitor is transmitted to the fifth MOS tube through the second switching transistor.

[0023] Optionally, the second output unit includes a sixth MOS tube and a seventh MOS tube; the gate of the seventh MOS tube is connected to the source or drain of the second MOS tube, the source or drain of the seventh MOS tube is connected to the source or drain of the sixth MOS tube, the drain or source of the seventh MOS tube is connected to the second input terminal of the comparator, and the drain or source of the sixth MOS tube is grounded.

[0024] Optionally, the image sensor also includes a latch, which is used to store analog-to-digital conversion results and comparison results of each pixel point, the conversion result includes the digital signal of each pixel point of the original image of the n-1th frame, and the digital signal of the target pixel point of the target original image of the nth frame; the comparison result includes a result identification that there is a change in the initial voltage signal of the pixel point, and a result identification that there is no change in the initial voltage signal of the pixel point. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A schematic diagram of a visual system structure provided by an embodiment of the present invention;

[0027] Figure 2A and Figure 2B Schematic diagrams of two image acquisition scenarios provided by embodiments of the present invention;

[0028] Figure 3 A schematic diagram of another image extraction method provided by an embodiment of the present invention;

[0029] Figure 4A schematic diagram of a pixel unit circuit structure provided by an embodiment of the present invention;

[0030] Figure 5 A schematic diagram of another specific pixel unit circuit structure provided by an embodiment of the present invention;

[0031] Figure 6 A schematic flow chart of a timing control method for a pixel unit circuit provided by an embodiment of the present invention;

[0032] Figure 7 A schematic diagram of the structure of an image sensor provided by an embodiment of the present invention;

[0033] Figure 8 A schematic diagram of timing waveforms of circuit operation of a unit pixel provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] To make the content of the present invention more clear and understandable, the content of the present invention is further described below in conjunction with the accompanying drawings of the specification. Of course, the present invention is not limited to this specific embodiment, and general replacements known to those skilled in the art are also covered within the protection scope of the present invention.

[0035] It should be noted that in the following specific embodiments, when describing the embodiments of the present invention in detail, in order to clearly represent the structure of the present invention for the convenience of explanation, the structures in the accompanying drawings are not drawn according to general proportions, and are partially enlarged, deformed and simplified. Therefore, it should be avoided to understand this as a limitation of the present invention.

[0036] The embodiment of the present invention provides a visual system, which includes: an image sensor 01, an event counting module 02 and an image processing module 03, and optionally, a data encoding module 04. Among them:

[0037] Image sensor 01 is used to collect original images. When there is a change in the initial voltage signal of the same pixel point in the nth frame original image and the n-1th frame original image, the analog-to-digital conversion unit is triggered to perform analog-to-digital conversion on the analog signal of the pixel point to obtain the digital signal of the pixel point, and a trigger signal is sent to the event counting module, where n is a positive integer.

[0038] The event counting module 02 is used to determine the number of target pixels whose initial voltage signals in the nth frame of the original image have changed by counting the number of trigger signals from the image sensor during the exposure time of the nth frame of the original image. For example, the event counting module 02 can be an RS trigger or a counter. When the event counting module 02 is an RS trigger, it can count by receiving event pulses from the image sensor.

[0039] The image processing module 03 is used to output the digital signals of each pixel of the n-th frame original image to the data encoding module 04 in a synchronous transmission mode when the ratio of the number of target pixel points to the total number of pixel points of the n-th frame original image is greater than a set threshold; when the ratio of the number of target pixel points to the total number of pixel points of the n-th frame original image is less than or equal to the set threshold, the digital signals of the target pixel points in the n-th frame original image can be output to the data encoding module 04 in an asynchronous transmission mode.

[0040] It is worth noting that the set threshold in the above embodiment can satisfy the following formula:

[0041]

[0042] Among them, th represents the set threshold, len L Indicates the bit width occupied by the pixel value, H represents the number of pixels in each row of the original image, W represents the number of pixels in each column of the original image, and the length of the information carried by each pixel is len L +1, 1 represents the bit width of the comparator comp, Indicates rounding up. The larger the image resolution, the smaller the th threshold is.

[0043] For example, the image resolution is recorded as H×W, H represents the number of pixels in each row of the original image, W represents the number of pixels in each column of the original image, the number of target pixels is recorded as event_count, and the total number of pixels in the original image of the nth frame is H×W. th represents the set threshold. When , the digital signal of each pixel point of the nth frame original image can be output to the data encoding module 04 in a synchronous transmission mode; when When 04 is transmitted, the digital signal of the target pixel in the nth frame original image can be output to the data encoding module in an asynchronous transmission manner.

[0044] In synchronous mode, since a complete pixel matrix is ​​transmitted, there is no need to encode the address. The address of the changed pixel can be parsed in the subsequent algorithm outside the sensor chip, so the length of the information carried by each pixel is len L +1, here 1 is Out c The total amount of data to be transmitted is fixed (len L +1)×H×W.

[0045] In asynchronous mode, the changed pixel address needs to be additionally encoded, so the length of each event is The total amount of data is When event_count increases to a certain amount, the total amount of data transmitted asynchronously will exceed that transmitted synchronously. The value of is the value of th. The larger the output resolution of the image sensor, the smaller the critical value of th for switching from asynchronous transmission to synchronous transmission. When the image sensor is an event camera, the commonly used resolution is 640×480 (such as SONY IMX637). Based on this resolution, the value of th is calculated to be about 33%. As the image resolution increases, the set threshold will be further reduced.

[0046] Optionally, the image processing module 03 may be an image processor or a circuit for performing image processing.

[0047] In a possible implementation, the image processing module 03 outputs the digital signal of each pixel point of the nth frame original image in a synchronous transmission manner, including: for the target pixel point whose initial voltage signal has changed, the image processing module outputs the digital signal of the target pixel point of the nth frame original image in a synchronous transmission manner, and an indication that the initial voltage signal of the target pixel point has changed; for the non-target pixel point whose initial voltage signal has not changed, the image processing module outputs the digital signal of the non-target pixel point of the n-1th frame original image in a synchronous transmission manner. Exemplarily, the indication that the initial voltage signal of the pixel point has changed may be 0 or 1, and when the comparison result is 0, it indicates that the pixel value has not changed significantly compared with the previous frame, and when the result is 1, it indicates that the pixel value has changed. The information output by the image processing module 03 in a synchronous transmission manner can be represented by p=(L,Out c ) indicates that p represents a pixel, L is its value, Out c is the state comparison result. For example, Figure 2A As shown, Figure 2A (a) and Figure 2A Compared with (b), there are more target pixels with changes in the initial voltage signal, so the synchronous transmission mode is used to output Figure 2A The digital signal of each pixel in (b) and the mark of the change of the initial voltage signal of each pixel. In this embodiment, because the analog-to-digital conversion unit does not perform analog-to-digital conversion on the analog signal of the pixel whose initial voltage signal changes in the nth frame, the digital signal of the non-target pixel of the original image of the n-1th frame is used as the digital signal of the non-target pixel of the original image of the nth frame. In this way, when there are many pixels with obvious changes in the previous and next frame images, the data generated by the image sensor can be transmitted to the outside in a synchronous manner, which can retain the comprehensive information of the original image such as image texture and brightness to a large extent.

[0048] In another possible implementation, the image processing module 03 outputs the digital signal of the target pixel in the nth frame of the original image in an asynchronous transmission manner, including: for the target pixel whose initial voltage signal changes, the image processing module 03 outputs the digital signal of the target pixel in the nth frame of the original image in an asynchronous transmission manner, as well as the row and column addresses corresponding to the target pixel. Exemplarily, asynchronous transmission refers to only outputting pixel information of pixels whose brightness has changed significantly, and the image processing module 03 can send it in the form of an event stream. Each event consists of the analog-to-digital conversion value of the pixel and its corresponding row and column address. For example, the output result is represented by e=(L, x, y), where e represents an event, L is the pixel value at the location where the event occurs, x is the column address where the event occurs, and y is the row address. Reference Figure 2B As shown, Figure 2B (a) and Figure 2B Compared with (b), there are few target pixels with changes in the initial voltage signal, so the output is asynchronously transmitted. Figure 2B The digital signal of each target pixel in (b), and the row and column addresses corresponding to the target pixel.

[0049] Optionally, in the asynchronous transmission mode, the image processing module 03 can extract the region of interest from the nth frame original image according to the event set generated by all target pixels in the corresponding time period from the acquisition time of the nth frame original image to the acquisition time of the n-1th frame, and according to the region boundary of the moving object corresponding to the event set, from the nth frame original image, and output the digital signals of each pixel of the region of interest in the nth frame original image in an asynchronous transmission mode; the region of interest includes the digital signals of the target pixel in the nth frame original image. That is to say, in the asynchronous transmission mode, in this embodiment, the image processing module can extract the region of interest from the original frame based on the edge event of the moving object, and the user can choose to only receive the frame cropped by the region of interest (ROI), which can reduce the data transmission volume and increase the effective information content in the picture, which is conducive to the operation of the subsequent visual algorithm. Specifically, the ROI positioning method is as follows: the event sequence is divided into a series of slices in the time dimension at a small interval, and each slice contains an event set formed by the aggregation of asynchronous events generated by the moving object sensed by the sensor during this period, which is recorded as D={e t =(L t ,x t ,y t )|t start ≤t≤t end}, where e t represents an event occurring at time t, t start and t end are the start and end points of the time slice.

[0050] First, perform outlier detection on the event set to remove outlier events and noise, and then search for the boundaries of the region containing moving objects in the event set:

[0051]

[0052]

[0053]

[0054]

[0055] Thus, the bounding rectangle of the motion area is obtained, (x1, y1) and (x2, y2) are the coordinates of the upper left corner and lower right corner of the rectangle, and the area inside the rectangle is the extracted region of interest. Figure 3 As shown, Figure 3 Column (a) shows 5 consecutive frames of original images in a surveillance video. Figure 3 The (b) column in the figure shows the event set generated by all the changed pixels in the time interval between the current frame and the previous frame, as well as the bounding rectangle corresponding to the event set. Figure 3 Column (c) in the figure shows the region of interest demarcated after the rectangular box is transferred to the original image. It can be seen that most of the static background of the original image (i.e., the part outside the box) has been excluded.

[0056] It can be seen that the visual system provided in this embodiment can support two ways to output the data of the CMOS image sensor, which are divided into synchronous and asynchronous modes. When there are many target pixels with changes in the initial voltage signal, the digital signal of the entire frame image is output, which helps to retain the comprehensive information in the image to the maximum extent, including absolute brightness, color, contrast, saturation, texture, etc. When the target pixel points with changes in the initial voltage signal are small, the pixel values ​​of the changed pixels are transmitted. Because the data of such pixel points has strong spatial sparsity, the data redundancy can be effectively reduced. Then the visual system provided by the present invention is used to ensure that the data transmission bandwidth is reduced while ensuring the image resolution and image quality, saving system power consumption.

[0057] In the above-mentioned visual system, the image sensor includes an analog-to-digital conversion unit 03 and a pixel array composed of a pixel unit circuit. Figure 4 As shown, the pixel unit circuit includes a photoelectric signal conversion unit 401, a voltage amplification unit 402, a first output unit 403, a second output unit 404 and a comparison unit 405.

[0058] The photoelectric signal conversion unit 401 is used to convert the exposure information of the pixel points in the image frame into an initial voltage signal.

[0059] The voltage amplifying unit 402 is used to amplify and output the initial voltage signal.

[0060] The first output unit 403 and the second output unit 404 are both connected to the voltage amplifying unit 402, the first output unit 403 is used to receive and store the first voltage signal of the pixel point of the image frame at the n-1th frame; the second output unit 404 is used to receive and output the second voltage signal of the pixel point of the image frame at the nth frame.

[0061] A comparison unit 405, wherein the comparison unit 405 is connected to the output end of the first output unit 403 and the output end of the second output unit 404, respectively, and is used to receive the first voltage signal from the first output unit 403 and receive the second voltage signal from the second output unit 404, and compare the first voltage signal with the second voltage signal to determine whether there is a change between the initial voltage signal of the pixel point in the image frame at the n-1th frame and the initial voltage signal of the pixel point in the image frame at the nth frame.

[0062] The analog-to-digital conversion unit 406 is connected to the output end of the second output unit 404. The comparison unit 405 is further used to: when the absolute value of the difference between the first voltage signal and the second voltage signal is not within a set range, determine that there is a change between the initial voltage signal of the pixel point in the image frame at the n-1th frame and the initial voltage signal of the pixel point in the image frame at the nth frame, and output a trigger signal, wherein the trigger signal is used to activate the analog-to-digital conversion unit 406 to perform analog-to-digital conversion.

[0063] It can be seen that the above-mentioned pixel unit circuit includes two output units, wherein the second output unit 404 stores and outputs the pixel signal of the nth frame, and the first output unit 403 stores the pixel signal of the n-1th frame, and the voltage signals of the two output units are simultaneously output to the comparison unit 405. The pixel structure is suitable for processing sampling information in parallel with rows, including the subsequent voltage signal comparison and analog-to-digital conversion process. Since the pixels in the same row are operated in parallel, the operation speed can be greatly improved, and the system operation bandwidth can be improved; in addition, the comparison unit 405 can determine whether the initial voltage signal of the pixel point has changed, which is helpful to determine whether to trigger the analog-to-digital conversion. When a pixel signal changes significantly, the comparison unit 405 triggers the analog-to-digital conversion unit 406 to perform analog-to-digital conversion, and when the pixel signal does not change significantly, the analog-to-digital conversion unit 406 is not triggered to perform analog-to-digital conversion, so as to save power consumption while ensuring the performance of the CIS.

[0064] In a specific implementation manner, Figure 5 As shown, the above-mentioned photoelectric signal conversion unit 401 includes a photodiode (PD); the voltage amplification unit 402 includes a first MOS tube M1, a second MOS tube M2 and a third MOS tube M3.

[0065] Specifically, the cathode of the photodiode PD is connected to the source or drain of the first MOS tube M1, the anode of the photodiode PD is grounded to VSS, and the photodiode is used to generate an electrical signal in response to the light signal. The gate of the first MOS tube M1 is connected to the drive circuit, the drive circuit inputs a reset signal (reset) to the first MOS tube M1, and the drain or source of the first MOS tube M1 is connected to the power supply voltage VDD, which is used to reset the photodiode PD.

[0066] The gate of the second MOS transistor M2 is also connected to the cathode of the photodiode, the source or drain of the second MOS transistor M2 is connected to the power supply voltage VDD, and the drain or source is connected to one end of the third MOS transistor M3. The first MOS transistor and the second MOS transistor M2 are used to amplify the initial voltage signal of the pixel point of the image frame to the first node A. The second MOS transistor M2 can be a source follower (SF) amplifying transistor, and the third MOS transistor M3 can be a site selection switch.

[0067] The first output unit 403 is used to output the first voltage signal of the first node A in the n-1th frame. Figure 5 For example, the first output unit 403 includes a first switch transistor S1, a capacitor C, a second switch transistor S2, and a fourth MOS tube M4 and a fifth MOS tube M5. One end of the capacitor C is connected to the first switch transistor S1 and the second switch transistor S2, and the other end is grounded to VSS. The capacitor C is used to store charges. The first switch transistor S1 and the second switch transistor S2 are used to control the charge and discharge state of the capacitor. When the first switch transistor S1 is closed and the second switch transistor S2 is disconnected, the charge at the first node is stored in the capacitor C; when the first switch transistor S1 is disconnected and the second switch transistor S2 is closed, the charge in the capacitor C is transmitted to the fifth MOS tube M5 through the second switch transistor S2. Figure 5 As can be seen, the gate of the fifth MOS tube M5 is connected to the second switch transistor, one end of the fifth MOS tube M5 is connected to one end of the fourth MOS tube M4, and the other end of the fifth MOS tube M5 is connected to the first input end Input1 of the comparator comp. The other end of the fourth MOS tube M4 is grounded VSS.

[0068] The second output unit 404 is used to output the second voltage signal of the first node in the nth frame, where n is a positive integer. Figure 5As shown, the second output unit includes a sixth MOS transistor M6 and a seventh MOS transistor M7. The gate of the seventh MOS transistor M7 is connected to the first node A, one end of the seventh MOS transistor M7 is connected to one end of the sixth MOS transistor M6, and the other end of the seventh MOS transistor M7 is connected to the second input terminal Input2 of the comparator comp. The other end of the sixth MOS transistor M6 is grounded VSS.

[0069] In some optional embodiments, the pixel unit circuit is further connected to a driving circuit, and the driving circuit is connected to the gate of the first MOS tube M1, the gate of the third MOS tube M3, the gate of the fourth MOS tube M4, the first switch transistor S1, the second switch transistor S2 and the gate of the sixth MOS tube M6. The driving circuit is used to drive the first MOS tube M1, the third MOS tube M3, the fourth MOS tube M4, the first switch transistor S1, the second switch transistor S2 and the sixth MOS tube M6.

[0070] like Figure 5 As shown, the comparison unit includes a comparator comp, which is shared by the same column of pixel unit circuits and is used to output a trigger signal when the absolute value of the difference between the first voltage signal and the second voltage signal is not within a set range. The trigger signal is used to activate the analog-to-digital conversion unit to perform analog-to-digital conversion.

[0071] It is worth noting that each transistor mentioned above can be an NMOS transistor or a PMOS transistor. In the present application, one end of any transistor is a source and the other end is a drain, or one end of any transistor is a drain and the other end is a source.

[0072] In this embodiment, optionally, the source or drain of the third MOS tube M3 is connected to the source or drain of the eighth MOS tube M8, the drain or source of the eighth MOS tube M8 is grounded, the eighth MOS tube M8 is shared by the pixel unit circuits in the same column, and the eighth MOS tube M8 is used to provide a bias voltage.

[0073] Optionally, the source or drain of the fifth MOS tube M5 is connected to the source or drain of the ninth MOS tube M9, the drain or source of the ninth MOS tube M9 is grounded, the ninth MOS tube M9 is shared by the pixel unit circuits in the same column, and the ninth MOS tube M9 is used to provide a bias voltage;

[0074] The drain or source of the seventh MOS tube M7 is connected to the source or drain of the tenth MOS tube M10, the drain or source of the tenth MOS tube M10 is grounded, the tenth MOS tube M10 is shared by the pixel unit circuits in the same column, and the tenth MOS tube M10 is used to provide a bias voltage. The eighth MOS tube M8 inputs a bias voltage, the gates of the ninth MOS tube M9 and the tenth MOS tube M10 input a bias voltage, and the drain or source of the ninth MOS tube M9 and the tenth MOS tube M10 is connected to the power supply voltage VDD.

[0075] It is worth noting that the fourth MOS tube M4 and the comparator comp can be shared by the pixel units in the same column, and the ninth MOS tube M9 and the tenth MOS tube M10 can be shared by the pixel units in the same column.

[0076] In this embodiment, the pixel unit circuit includes two output units, wherein the second output unit stores the pixel signal of the nth frame, and the first output unit stores the pixel signal of the n-1th frame, and the voltage signals of the two output units are output simultaneously. The pixel structure is suitable for processing sampling information in parallel with rows, including the subsequent voltage signal comparison and analog-to-digital conversion process. Since the pixels in the same row are operated in parallel, the operation speed can be greatly improved, and the system operation bandwidth can be improved; in addition, when a pixel signal changes significantly, the analog-to-digital conversion is triggered, which helps to save power consumption.

[0077] Based on the above pixel unit circuit, the following is further combined Figure 6 Timing control method in Figure 7 The image sensor structure diagram in , and Figure 8 The timing flow chart in shows a schematic flow chart of a timing control method for a pixel unit circuit, which specifically includes the following steps.

[0078] S601 , for the n-1th frame, in the first period t1 starting from time T1 , the driving circuit controls the reset signal rst input to the first MOS tube M1 of the pixel unit circuit 701 to be high level, turns on the first MOS tube M1 , and resets the photodiode PD.

[0079] S602, in the second period t2 starting from time T2, the driving circuit controls the reset signal rst input to the first MOS transistor M1 to switch from a high level to a low level, the first MOS transistor M1 is disconnected, and the pixel unit circuit starts to be exposed.

[0080] S603, controlling the first switch transistor S1 to be turned on, in the third period t3 starting at time T3, the drive circuit controls the row control signal row to switch to a high potential, the pixels of the corresponding row are selected, and the sampling signals of the pixels in the row are stored on the capacitor C.

[0081] In this step, when the row address selection logic and signal driving component 703 of the image sensor controls the reset signal rst to switch from a high level to a low level, the pixel unit circuit 701 starts to integrate and sample the exposure information, and then the driving circuit controls the first switch transistor S1 to turn on. When the row control signal row switches to a high potential at time T3, the pixels of the corresponding row are selected, and the sampling signals of all pixels in the row are stored on the capacitor C. In addition, the column address encoder 702 of the image sensor is used to implement column address encoding and generate an X column address [M-1:0], and the row address encoder 704 is used to implement row address encoding and generate a Y column address [N-1:0], where M and N are positive integers.

[0082] S604, in the nth frame, in the fourth time period t4 starting from time T4, the driving circuit again controls the reset signal rst input to the first MOS transistor M1 to be high level, turns on the first MOS transistor M1, and resets the photodiode PD.

[0083] S605, in the fifth time period t5 starting from time T5, the driving circuit controls the reset signal rst input to the first MOS tube M1 to switch from a high level to a low level, the first MOS tube M1 is disconnected, the pixel unit circuit starts to be exposed, and the second voltage signal of the nth frame is output to the second input terminal Input2 of the comparator comp through the second transmission unit, and the first voltage signal of the n-1th frame stored in the capacitor C is output to the first input terminal Input1 of the comparator comp through the first transmission unit.

[0084] In this step, after the pixel unit circuit is exposed, since the photodiode PD adopts special semiconductor PN junction technology, the photovoltaic effect under light will accumulate electrons on the surface of the photodiode PD. The photovoltaic special effect refers to: after the PN junction intrinsic layer absorbs photons, photogenerated electrons and photogenerated holes will be generated. The two will be separated in the built-in electric field of the PN junction, thereby forming the phenomenon of photovoltaic voltage and photogenerated current.

[0085] In this embodiment, after a period of photoelectric conversion and integration effect of the pixel unit circuit, the voltage at the sampling point is proportional to the light intensity and reaches a specific value. In the sixth period t6 starting at time T6, the n-1 frame signal stored in the first transmission unit is output to the first input terminal input1. When the row control signal switches to a high potential, the pixel of the corresponding row is selected. At this time, the control signal input to the first switch transistor S1 is a low level, and the first switch transistor S1 is still in the off state. Therefore, the charge will not pass through the first transmission unit, but will be directly output to the second input terminal Input2 of the comparator comp through the second transmission unit. At the same time, the second voltage signal of the nth frame is also output to the input port of the analog-to-digital conversion unit.

[0086] S606, when the difference between the second voltage signal and the first voltage signal is not within the set range, the comparator comp triggers the analog-to-digital conversion unit to perform analog-to-digital conversion at time T7.

[0087] Specifically, the comparator comp determines the difference between the second voltage signal and the first voltage signal collected by the same pixel in the two frames, and determines whether the absolute value of the difference is within the set range. If it exceeds the set range, it means that the pixel signal has a significant change, so the trigger signal (change) output by the comparator switches from a low level to a high level, activating the analog-to-digital conversion unit (ADC) to start the analog-to-digital conversion process, where the gradient waveform generator 709 is connected to the ADC. After a certain period of time, the conversion result is written into the output result latch FIFO of the corresponding column. Combined with Figure 4 For example, Figure 7 The event generator 705 in corresponds to Figure 5 The comparator comp in the comparator output is Figure 5 The trigger signal (change).

[0088] In this embodiment, when a row of pixels is selected, all pixels in the row will input the voltage signal sampled in the current frame and the voltage signal stored on the capacitor C in the previous frame into the comparator for analog signal comparison. When a pixel signal is determined to have a significant change (the criterion is that the absolute value of the difference between the two exceeds the set range), the comparator result of the column corresponding to the pixel will output 1, the analog-to-digital conversion unit is enabled, and the pixel signal is converted into a digital signal by the column analog-to-digital conversion component 706 (A / D) shared by the column, and stored in the column analog-to-digital conversion structure latch 707 of the corresponding column, and the event state of the column is set to 1 (comparison result latch FIFO), and stored in the event comparison result latch 708 of the corresponding column. If the pixel signal is determined not to have a significant change (the criterion is that the absolute value of the difference between the two is within the set range), the comparator of the column corresponding to the pixel outputs 0, the analog-to-digital conversion unit will not be activated, and the pixel signal will not be A / D converted, and the state of the column is set to 0.

[0089] Optionally, S607, when the analog-to-digital conversion of the row of pixels is completed, at time T8, the conversion result and the comparison result stored in the latch are cleared.

[0090] That is, the system finally reads out and clears all conversion results stored in the column analog-to-digital conversion result latch 707 in a fast serial manner. Correspondingly, all comparison results stored in the event result latch 708 are also read out and cleared in a synchronous and fast serial manner.

[0091] It is worth noting that when the value stored in each unit of the latch FIFO is a first value (such as 0), the ADC value of the corresponding column is an invalid value. Only when the value stored in each unit of the latch FIFO is a second value (such as 1), the ADC value of the corresponding column is valid.

[0092] In this embodiment, the output of the image sensor can be output in a synchronous manner or in an asynchronous manner. The so-called synchronous manner means that the output includes the value of each pixel and the value of the corresponding comparator result, that is, the correspondence between the pixel value and the corresponding comparison result. When the value of the comparator result is 0, the value of the pixel is the pixel value collected before and has not changed. When the value of the comparator result is 1, the value of the pixel changes, and the value output by the ADC replaces the value of the pixel collected in the previous frame.

[0093] The so-called asynchronous output means that the output includes the analog-to-digital conversion value of the pixel and the address of the row and column corresponding to the pixel <pixel value, (X column address, Y row address)>. Synchronous and asynchronous refer to whether the image sensor as the sender of information and the computer as the receiver of information need the synchronization signal FRARME_START (or LINE_START) at the beginning of each frame (or each line).

[0094] In summary, the advantage of the present invention is that when a row of pixels is selected, all pixels in the row will input the voltage signal sampled in the current image frame and the voltage signal stored on the capacitor in the previous frame into the comparator for analog signal comparison. When a pixel signal is determined to have changed significantly, the analog-to-digital conversion unit is triggered to perform analog-to-digital conversion, and when the pixel signal has not changed significantly, the analog-to-digital conversion is not triggered, so as to save power consumption while ensuring the performance of the CIS.

[0095] The above description is only the preferred embodiment of the present invention, and the embodiment is not intended to limit the scope of patent protection of the present invention. Therefore, any equivalent structural changes made by using the description and drawings of the present invention should be included in the protection scope of the present invention. The above description of the disclosed embodiments enables professionals in the field to implement or use the present invention.

[0096] Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A visual system, characterized in that: It includes an image sensor, an event counting module and an image processing module; The image sensor is used to collect the original image, and when there is a change in the initial voltage signal of the same pixel point in the nth frame original image and the n-1th frame original image, the analog signal of the pixel point is converted into a digital signal to obtain the digital signal of the pixel point, and a trigger signal is sent to the event counting module, where n is a positive integer; The event counting module is used to determine the number of target pixel points whose initial voltage signals in the nth frame of the original image have changed by counting the number of trigger signals from the image sensor during the exposure time of the nth frame of the original image; An image processing module is used to output the digital signals of each pixel of the n-th frame original image in a synchronous transmission mode when the ratio of the number of target pixel points to the total number of pixel points of the n-th frame original image is greater than a set threshold; and to output the digital signals of the target pixel points in the n-th frame original image in an asynchronous transmission mode when the ratio of the number of target pixel points to the total number of pixel points of the n-th frame original image is less than or equal to the set threshold.

2. The visual system according to claim 1, characterized in that The image processing module outputs the digital signal of each pixel of the nth frame of the original image in a synchronous transmission mode, including: For the target pixel point whose initial voltage signal has changed, the image processing module outputs the digital signal of the target pixel point of the original image of the nth frame and the mark indicating that the initial voltage signal of the target pixel point has changed in a synchronous transmission manner; For non-target pixels whose initial voltage signals have not changed, the image processing module outputs digital signals of the non-target pixels of the original image of the n-1th frame in a synchronous transmission manner.

3. The visual system according to claim 2, characterized in that: The image processing module outputs the digital signal of the target pixel in the nth frame of the original image in an asynchronous transmission mode, including: For the target pixel whose initial voltage signal changes, the image processing module outputs the digital signal of the target pixel of the nth frame of the original image and the row and column addresses corresponding to the target pixel in an asynchronous transmission manner.

4. The visual system according to any one of claims 1 to 3, characterized in that: The threshold value is set to satisfy the following formula: Among them, th represents the set threshold, len L Indicates the bit width occupied by the pixel value, H indicates the number of pixels in each row of the original image, and W indicates the number of pixels in each column of the original image. Indicates rounding up. The larger the image resolution, the smaller the th threshold is.

5. The visual system according to any one of claims 1 to 3, characterized in that: When the ratio of the number of target pixels to the total number of pixels in the n-th frame of the original image is less than or equal to a set threshold, outputting a digital signal of the target pixel in the n-th frame of the original image in an asynchronous transmission manner includes: An image processing module is used to extract an event set generated by all target pixels in a time period corresponding to the acquisition time of the nth frame original image to the acquisition time of the n-1th frame, and a region of interest from the nth frame original image according to the region boundary of the moving object corresponding to the event set, and output digital signals of each pixel in the region of interest in the nth frame original image in an asynchronous transmission mode; the region of interest includes the digital signals of the target pixels in the nth frame original image.

6. The visual system according to claim 1, characterized in that: The image sensor includes a pixel array composed of an analog-to-digital conversion unit and a pixel unit circuit; The pixel unit circuit includes a photoelectric signal conversion unit, a voltage amplification unit, a first output unit, a second output unit and a comparison unit; A photoelectric signal conversion unit, used to convert exposure information of a pixel point in an image frame into an initial voltage signal; A voltage amplifying unit, used to amplify and output the initial voltage signal; The first output unit is connected to the voltage amplifying unit, and is used to receive and store the first voltage signal of the pixel point in the image frame at the n-1th frame; The second output unit is connected to the voltage amplifying unit, and is used for receiving and outputting the second voltage signal of the pixel point in the image frame at the nth frame; a comparing unit, the comparing unit being connected to the output end of the first output unit and the output end of the second output unit respectively, and being used for receiving the first voltage signal from the first output unit and receiving the second voltage signal from the second output unit, and comparing the first voltage signal with the second voltage signal to determine whether there is a change between an initial voltage signal of the pixel point in the image frame at the n-1th frame and an initial voltage signal of the pixel point in the image frame at the nth frame; The comparison unit is further configured to: when the absolute value of the difference between the first voltage signal and the second voltage signal is not within a set range, determine that there is a change between the initial voltage signal of the pixel point in the image frame at the n-1th frame and the initial voltage signal of the pixel point in the image frame at the nth frame, and send a trigger signal to the analog-to-digital conversion unit; The analog-to-digital conversion unit is connected to the output end of the second output unit, and is used to receive the second voltage signal of the pixel point in the image frame at the nth frame; when receiving the trigger signal, the analog-to-digital conversion of the second voltage signal begins.

7. The visual system according to claim 6, characterized in that: The photoelectric signal conversion unit includes a photodiode; the voltage amplification unit includes a first MOS tube, a second MOS tube and a third MOS tube; The source or drain of the first MOS tube and the gate of the second MOS tube are both connected to the cathode of the photodiode, the source or drain of the second MOS tube is connected to the source or drain of the third MOS tube, and the drain or source of the second MOS tube and the drain or source of the first MOS tube are both connected to the power supply voltage.

8. The visual system according to claim 7, characterized in that: The comparison unit includes a comparator; The first output unit includes a first switch transistor, a capacitor, a second switch transistor, and a fourth MOS transistor and a fifth MOS transistor; The capacitor is used to store charge, one end of the capacitor is connected to the first switch transistor and the second switch transistor, and the other end of the capacitor is grounded; the gate of the fifth MOS tube is connected to the second switch transistor, the source or drain of the fifth MOS tube is connected to the source or drain of the fourth MOS tube, and the drain or source of the fifth MOS tube is connected to the first input end of the comparator; the drain or source of the fourth MOS tube is grounded; The first switching transistor and the second switching transistor are used to control the charging and discharging state of the capacitor; when the first switching transistor is closed and the second switching transistor is disconnected, the initial voltage signal of the pixel point of the image frame is stored in the capacitor; when the first switching transistor is disconnected and the second switching transistor is closed, the charge in the capacitor is transmitted to the fifth MOS tube through the second switching transistor.

9. The visual system according to claim 8, characterized in that: The second output unit includes a sixth MOS tube and a seventh MOS tube; the gate of the seventh MOS tube is connected to the source or drain of the second MOS tube, the source or drain of the seventh MOS tube is connected to the source or drain of the sixth MOS tube, the drain or source of the seventh MOS tube is connected to the second input terminal of the comparator, and the drain or source of the sixth MOS tube is grounded.

10. The visual system according to claim 6, characterized in that: The image sensor also includes a latch, which is used to store analog-to-digital conversion results and comparison results of each pixel point, the conversion result includes the digital signal of each pixel point of the n-1th frame original image, and the digital signal of the target pixel point of the nth frame target original image; the comparison result includes a result identification that the initial voltage signal of the pixel point has changed, and a result identification that the initial voltage signal of the pixel point has not changed.

Images