Vision sensor chip based on multi-scale spatiotemporal difference technology

Abstract

Provided in the present application is a vision sensor chip based on a multi-scale spatiotemporal difference technology. The vision sensor chip comprises a pixel array, an intensity path and / or a temporal difference path and / or a spatial difference path, wherein the pixel array comprises a plurality of pixel units; the temporal difference path performs, in a charge domain, an analog domain or a digital domain, weighted difference and quantization operations on signals at the current pixel unit position at the current moment and signals at the current pixel unit position at any previous moment, so as to obtain one or more multi-scale temporal difference values; and the spatial difference path performs, in the charge domain, the analog domain or the digital domain, weighted difference and quantization operations on the signals at the current pixel unit position at the current moment and signals at any spatial pixel unit position at the current moment, so as to obtain one or more multi-scale spatial difference values. In the present application, a temporal difference and a spatial difference are expanded to a multi-scale mode, which facilitates the analysis of spatiotemporal correlations of signals, and provides a larger spatiotemporal receptive field.

Description

A visual sensor chip based on multi-scale spatiotemporal difference technology

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202311423882.8, filed on October 30, 2023, entitled “A visual sensor chip based on multi-scale spatiotemporal difference technology”, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present application relates to the field of visual sensing technology, and in particular to a visual sensor chip based on multi-scale spatiotemporal difference technology. Background Art

[0004] A visual sensor is a photoelectric detection device that senses visible light in the environment and converts it into electrical signals. Various types of visual sensors, such as CIS (CMOS Image Sensor) and DVS (Dynamic Vision Sensor), have been developed to provide high-performance image information of objects. However, existing visual sensors cannot determine spatiotemporal correlations with earlier signals or beyond adjacent pixels, resulting in a limited spatiotemporal receptive field.

[0005] Summary of the Invention

[0006] The present application provides a visual sensor chip based on multi-scale spatiotemporal difference technology to solve the problem that visual sensors in the prior art cannot determine the spatiotemporal correlation between signals at earlier times or beyond adjacent pixels, and have a defect of too small spatiotemporal receptive field. The present application helps to analyze the spatiotemporal correlation of signals by expanding temporal and spatial differences to a multi-scale mode, has a larger spatiotemporal receptive field, and forms an efficient and robust visual representation.

[0007] The present application provides a visual sensor chip based on multi-scale spatiotemporal difference technology, comprising a pixel array, a temporal difference path, and / or a spatial difference path; the pixel array comprises a plurality of pixel units; the temporal difference path is used to perform weighted difference and quantization operations in the charge domain, analog domain, or digital domain on the signal at the current pixel unit position at the current moment and the signal at the current pixel unit position at any previous moment, to obtain one or more multi-scale temporal difference values; the spatial difference path is used to perform weighted difference and quantization operations in the charge domain, analog domain, or digital domain on the signal at the current pixel unit position at the current moment and the signal at any pixel unit position in space at the current moment, to obtain one or more multi-scale spatial difference values.

[0008] According to a visual sensor chip based on multi-scale spatiotemporal difference technology provided by the present application, the time difference path includes multiple time difference storage nodes, a time difference module and a time quantization module; the multiple time difference storage nodes are used to store the electrical signals of the current pixel unit position at multiple moments respectively; the time difference module is used to perform a weighted time difference operation on the electrical signal of the current pixel unit position at the current moment and the electrical signal of the current pixel unit position at any previous moment according to the electrical signals of the current pixel unit position at multiple moments, to obtain the multi-scale time difference value; in the multi-scale time difference process, The time quantization module completes the analog-to-digital signal conversion; the spatial differential path includes a spatial differential storage node, a spatial differential module and a spatial quantization module that are multiplexed with one of the time differential storage nodes; the spatial differential storage node is used to store the electrical signal of the current pixel unit position at the current moment; the spatial differential module is used to perform a weighted time differential operation on the electrical signal of the current pixel unit position at the current moment and the electrical signal of any pixel unit position in the space at the current moment to obtain the multi-scale spatial differential value; the analog-to-digital signal conversion is completed by the spatial quantization module during the multi-scale spatial differential process.

[0009] According to a visual sensor chip based on multi-scale spatiotemporal difference technology provided by the present application, the time difference module and / or the time quantization module are arranged in the pixel unit, and a pixel-level signal readout method is adopted; or the time difference module and / or the time quantization module are arranged outside the pixel unit and shared by the pixel units in the same column, and a column-level signal readout method is adopted; the spatial difference module and / or the spatial quantization module are arranged in the pixel unit, and a pixel-level signal readout method is adopted; or the time difference module and / or the time quantization module are arranged outside the pixel unit, and shared by the pixel units in the same column, and a column-level signal readout method is adopted.

[0010] According to a visual sensor chip based on multi-scale spatiotemporal difference technology provided by the present application, a pulse generating module is set in each pixel unit in the pixel array; or, all pixel units in the pixel array are connected to a pulse generating module; or, the pixel array is divided into multiple sub-areas, and all pixel units in each sub-area are connected to a pulse generating module; wherein, the pulse generating module is used to generate a trigger signal at a fixed time interval, or to generate a trigger signal at an adaptive, programmable variable time interval to control the start exposure moment and exposure duration of the photosensitive module, the pixel units connected to the same pulse generating module are exposed synchronously, and the pixel units connected to different pulse generating modules are exposed synchronously or asynchronously; the photosensitive module is set in the pixel unit, and is used to convert the light signal at the current pixel unit position into an analog electrical signal.

[0011] According to a visual sensor chip based on multi-scale spatiotemporal difference technology provided by the present application, the exposure mode of each pixel unit in the pixel array is global exposure or rolling exposure.

[0012] The present application also provides a multi-channel visual sensor chip based on multi-scale spatiotemporal difference technology, including a pixel array, an intensity channel, a time difference channel and / or a space difference channel; the pixel array includes multiple pixel units; the intensity channel is used to determine the quantized value of the electrical signal converted from the intensity of the incident light at the current pixel unit position at the current moment; the time difference channel is used to perform weighted difference and quantization operations on the signal at the current pixel unit position at the current moment and the signal at the current pixel unit position at any previous moment in the charge domain, analog domain or digital domain to obtain one or more multi-scale time difference values; the space difference channel is used to perform weighted difference and quantization operations on the signal at the current pixel unit position at the current moment and the signal at any pixel unit position in the space at the current moment in the charge domain, analog domain or digital domain to obtain one or more multi-scale spatial difference values.

[0013] According to a multi-channel visual sensor chip based on multi-scale spatiotemporal difference technology provided by the present application, the pixel array includes one type of pixel unit, which has a corresponding intensity channel, a time difference channel and / or a space difference channel; or, the pixel array includes two types of pixel units, the first type of pixel unit has a corresponding intensity channel, and the second type of pixel unit has a corresponding time difference channel and / or a space difference channel.

[0014] According to a multi-channel visual sensor chip based on multi-scale spatiotemporal difference technology provided by the present application, the intensity channel includes an intensity storage node and an intensity quantization module; the intensity storage node is used to store the electrical signal converted from the intensity of the incident light at the current pixel unit position at the current moment; the intensity quantization module is used to perform analog-to-digital conversion on the electrical signal converted from the intensity of the incident light at the current pixel unit position at the current moment, and obtain the quantized value of the electrical signal converted from the intensity of the incident light at the current pixel unit position at the current moment; the time difference channel includes multiple time difference storage nodes, time difference modules and time quantization modules; the multiple time difference storage nodes are used to store the electrical signals of the current pixel unit position at multiple moments respectively; the time difference module is used to, based on the multiple electrical signals of the current pixel unit position at the said moment, A weighted time difference operation is performed on the electrical signal of the current pixel unit position at the current moment and the electrical signal of the current pixel unit position at any previous moment to obtain the multi-scale time difference value; in the multi-scale time difference process, the analog-to-digital signal conversion is completed by the time quantization module; the spatial difference path includes a spatial difference storage node, a spatial difference module and a spatial quantization module that are multiplexed with one of the time difference storage nodes; the spatial difference storage node is used to store the electrical signal of the current pixel unit position at the current moment; the spatial difference module is used to perform a weighted time difference operation on the electrical signal of the current pixel unit position at the current moment and the electrical signal of any pixel unit position in the space at the current moment to obtain the multi-scale spatial difference value; in the multi-scale spatial difference process, the analog-to-digital signal conversion is completed by the spatial quantization module.

[0015] According to a multi-channel visual sensor chip based on multi-scale spatiotemporal difference technology provided by the present application, the intensity quantization module is arranged in the pixel unit and adopts a pixel-level signal readout method; or the intensity quantization module is arranged outside the pixel unit and shared by the pixel units in the same column, and adopts a column-level signal readout method; the time difference module and / or the time quantization module is arranged in the pixel unit and adopts a pixel-level signal readout method; or the time difference module and / or the time quantization module is arranged outside the pixel unit and shared by the pixel units in the same column, and adopts a column-level signal readout method; the spatial difference module and / or the spatial quantization module is arranged in the pixel unit and adopts a pixel-level signal readout method; or the spatial difference module and / or the spatial quantization module is arranged outside the pixel unit and shared by the pixel units in the same column, and adopts a column-level signal readout method.

[0016] According to a multi-channel visual sensor chip based on multi-scale spatiotemporal difference technology provided by the present application, a pulse generating module is set in each pixel unit in the pixel array; or, all pixel units in the pixel array are connected to a pulse generating module; or, the pixel array is divided into multiple sub-areas, and all pixel units in each sub-area are connected to a pulse generating module; wherein, the pulse generating module is used to generate a trigger signal at a fixed time interval, or to generate a trigger signal at an adaptive, programmable variable time interval to control the start exposure time and exposure duration of the photosensitive module, the pixel units connected to the same pulse generating module are exposed synchronously, and the pixel units connected to different pulse generating modules are exposed synchronously or asynchronously; the photosensitive module is set in the pixel unit, and is used to convert the light signal at the current pixel unit position into an analog electrical signal.

[0017] According to a multi-channel visual sensor chip based on multi-scale spatiotemporal difference technology provided by the present application, the exposure mode of each pixel unit in the pixel array is global exposure or rolling exposure.

[0018] The present application provides a visual sensor chip based on multi-scale spatiotemporal differential technology, including a pixel array, a time differential path and / or a spatial differential path; the pixel array includes multiple pixel units; the time differential path performs weighted differential and quantization operations on the signal of the current pixel unit position at the current moment and the signal of the current pixel unit position at any previous moment in the charge domain, analog domain or digital domain to obtain one or more multi-scale time differential values; the spatial differential path performs weighted differential and quantization operations on the signal of the current pixel unit position at the current moment and the signal of any pixel unit position in the space at the current moment in the charge domain, analog domain or digital domain to obtain one or more multi-scale spatial differential values. The present application helps to analyze the spatiotemporal correlation of signals by expanding the time differential and spatial differential to a multi-scale mode, and has a larger spatiotemporal receptive field. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] FIG1 is a schematic diagram of a multi-scale triple-multiplexed pixel provided by the present application;

[0021] FIG2 is a schematic diagram of a one-way expanded multi-scale spatial difference provided by this application;

[0022] FIG3 is a schematic diagram of the free expansion multi-scale spatial difference provided by this application;

[0023] FIG4 is a schematic diagram of a spatiotemporal differential pixel provided by the present application that meets the multi-scale TD and SD output requirements;

[0024] FIG5 is a schematic diagram of a hybrid array three-channel visual sensor based on multi-scale spatiotemporal differential pixels provided by the present application;

[0025] FIG6 is another schematic diagram of multi-scale spatiotemporal difference provided by the present application;

[0026] FIG7 is a multi-scale time difference circuit diagram provided by the present application;

[0027] FIG8 is another multi-scale time difference circuit diagram provided by the present application;

[0028] FIG9 is another multi-scale time difference circuit diagram provided by the present application;

[0029] FIG10 is another multi-scale time difference circuit diagram provided by the present application. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0031] Considering that existing visual sensors derive temporal differential values ​​by subtracting the signal strength at the previous moment from the current pixel unit signal strength, but without considering the signal strength at earlier moments, the temporal correlation is weak. Even if algorithmic processing can be performed later by synthesizing the difference between the three signals, there are problems with large latency and large data transmission volume. If multi-scale differential processing is completed at the sensor perception end, it is expected to reduce back-end processing and transmission costs. At the same time, multi-scale differentials are expected to improve the differential data distribution and reveal more signal temporal correlations.

[0032] The same principle applies to spatial difference values. From the perspective of convolutional neural networks, spatial difference is mathematically similar to the convolutional layer operation. However, simply differencing adjacent pixels has a small receptive field, failing to observe spatial correlations between signals at larger distances. Therefore, multi-scale spatial difference is necessary.

[0033] Please refer to FIG1 , which is a schematic diagram of a multi-scale triple-multiplexed pixel provided by this application.

[0034] Please refer to Figure 2, which is a schematic diagram of unidirectional expansion multi-scale spatial difference provided by this application.

[0035] Please refer to FIG3 , which is a schematic diagram of the freely scalable multi-scale spatial difference provided by this application.

[0036] In order to solve the technical problems existing in the prior art, the present application provides a visual sensor chip based on multi-scale spatiotemporal difference technology, including a pixel array, a time difference path and / or a spatial difference path; the pixel array includes multiple pixel units; the time difference path is used to perform weighted difference and quantization operations on the signal of the current pixel unit position at the current moment and the signal of the current pixel unit position at any previous moment in the charge domain, analog domain or digital domain to obtain one or more multi-scale time difference values; the spatial difference path is used to perform weighted difference and quantization operations on the signal of the current pixel unit position at the current moment and the signal of any pixel unit position in the space at the current moment in the charge domain, analog domain or digital domain to obtain one or more multi-scale spatial difference values.

[0037] The time difference (TD) path of this application outputs the time difference value of the current pixel position (x, y) at different times. The expression of the obtained TD path output is:

[0038] where α i is the weight value, α i Can be any value. TD It is a quantization method, which can be multi-valued (>1 bit) or single-valued (positive and negative pulses). n ,t n-1 ,t n-2 …can be acquired in a manner that the entire array is synchronized with the same time interval, the entire array is synchronized with a variable time interval, or the entire array is asynchronous.

[0039] α i The number of N is not fixed, for example, N = 2, a0 = 1, a1 = -1, TD (x, y, t n )=Q TD (I(x,y,t n )-I(x,y,t n-1 ))

[0040] At this time, the time difference process is the current output value minus the previous output value. N=3,a0=1,a1=-0.8,a2=-0.2, TD(x,y,t n )=QTD(I(x,y,t n )-80%I(x,y,t n-1 )-20%I(x,y,t n-2)) N=3, a0=1, a1=0.2, a2=-1.2, TD(x,y,t n )=Q TD (I(x,y,t n )+20%I(x,y,t n-1 )-120%I(x,y,t n-2 ))The data of quantization and weighted difference can be exchanged, that is, the formula can also be expressed as

[0041] There can be multiple multi-scale time difference channels coexisting, for example

[0042] And so on.

[0043] Furthermore, DVS is asynchronous and outputs only a timestamp and 1-bit information, making it susceptible to noise and low in information content. Since DVS can only output 1-bit time-varying information, it cannot adapt to complex environments. The visual sensing architecture of this application requires acquiring the temporal variation of visual signals in a synchronous or asynchronous manner. This temporal variation is preferably quantized and read out in a high-precision multi-value format.

[0044] The spatial difference (SD) path outputs the current time t n The spatial difference between the current pixel position (x, y) and the adjacent pixel (diagonal or xy direction). For the xy direction difference, the expression of the SD path output is:

[0045] For oblique differential, the expression of the channel output is obtained as

[0046] where Q SD It is a quantization method, which can be multi-valued (>1 bit) or single-valued (positive and negative pulses). n ,t n-1 ,t n-2 …can be acquired in a manner that the entire array is synchronized with the same time interval, the entire array is synchronized with a variable time interval, or the entire array is asynchronous.

[0047] The above multi-scale spatial differentiation is intended to expand along the same direction.

[0048] More generally, the spatial difference can be expressed as

[0049] The purpose of the * subscript in SD is to indicate that SD may have multiple dimensions, for example, x , SD y , SD ↙ , SD ↘ This general formula can be used to express more complex situations.

[0050] SD x ′(x,y,t n )=Q SD (I(x,y,t n )-80%I(x-1,y,t n )-10%I(x-1,y+1,t n )-10%I(x-1,y-1,t n ))

[0051] All signals involved in the above pathways are three-dimensional quantities, including two-dimensional spatial quantities x and y and a time dimension t.

[0052] The number of the time differential path and the space differential path can be one or more, and this application does not impose any particular limitation thereto.

[0053] This application helps analyze the spatiotemporal correlation of signals by extending time difference and spatial difference to multi-scale modes, and has a larger spatiotemporal receptive field.

[0054] Based on the above embodiment:

[0055] Please refer to FIG4 , which is a schematic diagram of spatiotemporal differential pixels provided by the present application that meet the multi-scale TD and SD output requirements.

[0056] As a preferred embodiment, the time difference path includes multiple time difference storage nodes, time difference modules and time quantization modules; the multiple time difference storage nodes are used to store the electrical signals of the current pixel unit positions at multiple moments respectively; the time difference module is used to perform weighted time difference operation on the electrical signal of the current pixel unit position at the current moment and the electrical signal of the current pixel unit position at any previous moment according to the electrical signals of the current pixel unit position at multiple moments, to obtain a multi-scale time difference value; the time quantization module is used to complete the analog-to-digital signal conversion during the multi-scale time difference process; the spatial difference path includes a spatial difference storage node, a spatial difference module and a spatial quantization module reused with a time difference storage node; the spatial difference storage node is used to store the electrical signal of the current pixel unit position at the current moment; the spatial difference module is used to perform weighted time difference operation on the electrical signal of the current pixel unit position at the current moment and the electrical signal of any pixel unit position in the space at the current moment, to obtain a multi-scale spatial difference value; the spatial quantization module is used to complete the analog-to-digital signal conversion during the multi-scale spatial difference process.

[0057] The quantization methods used by the temporal and spatial quantization modules can be either multi-valued (>1 bit) or single-valued (positive and negative pulses). Signal acquisition can be performed synchronously across the entire array with the same time interval, synchronously across the entire array with a variable time interval, or asynchronously across the entire array.

[0058] In the field of digital signal processing, quantization mainly refers to the process of converting analog signals into digital signals. Signal sampling and quantization are usually achieved by analog-to-digital converters (ADCs).

[0059] As a preferred embodiment, the time difference module and / or the time quantization module are arranged in the pixel unit, and a pixel-level signal readout method is adopted; or the time difference module and / or the time quantization module are arranged outside the pixel unit and shared by the pixel units in the same column, and a column-level signal readout method is adopted; the spatial difference module and / or the spatial quantization module are arranged in the pixel unit, and a pixel-level signal readout method is adopted; or the time difference module and / or the time quantization module are arranged outside the pixel unit, and shared by the pixel units in the same column, and a column-level signal readout method is adopted.

[0060] In this embodiment, if the time difference module and the space difference module are set outside the time and space difference pixel unit, each column of time and space difference pixel units shares the same time difference module and the same space difference module, then the time difference values ​​and space difference values ​​of multiple time and space difference pixel units need to be read at the same time, following a certain rule, which can be done at t n , t n-1 , t n-2 ...output information at several fixed moments, which can be fixed time intervals or set to adaptive, programmable variable intervals, reducing the total number of quantization modules and hardware resource consumption.

[0061] If the time difference module and the space difference module are set in the time-space difference pixel unit, the time difference value and the space difference value of the time-space difference pixel unit can be read out synchronously with the entire array or asynchronously with the entire array. The time-space difference value and the space difference value are output specifically according to the triggering moment of each time-space difference pixel unit, which improves flexibility and reduces output delay.

[0062] Of course, the time difference module and the space difference module of the visual sensor chip of the present application can be arbitrarily combined in the case of inside and outside the single element, and the present application does not make any special limitation here.

[0063] As a preferred embodiment, a pulse generation module is provided in each pixel unit in the pixel array; or, all pixel units in the pixel array are connected to a pulse generation module; or, the pixel array is divided into multiple sub-regions, and all pixel units in each sub-region are connected to a pulse generation module; wherein the pulse generation module is used to generate a trigger signal at a fixed time interval, or to generate a trigger signal at an adaptive, programmable variable time interval, to control the start time and exposure duration of the photosensitive module, so that pixel units connected to the same pulse generation module are exposed synchronously, and pixel units connected to different pulse generation modules are exposed synchronously or asynchronously; the photosensitive module is provided in the pixel unit and is used to convert the light signal at the current pixel unit position into an analog electrical signal. In this embodiment, a column-level readout method is used.

[0064] Specifically, in this embodiment, two reading modes may be adopted: full array synchronization with the same time interval, or full array synchronization with a variable time interval.

[0065] The synchronization pulse generation method can be set not only to a fixed interval, but also to an adaptive, programmable variable interval. This adaptive interval can adapt to the changing characteristics of the external visual signal. When the amount of change is large and the frequency of change is high, a higher sampling frequency is used, and when the signal is low frequency, a lower sampling frequency is used to reduce data volume and energy consumption.

[0066] The present application has a time differential path and / or a space differential path, that is, it may have only a time differential path, or only a space differential path, or both a time differential path and a space differential path.

[0067] This application also supports multiple pixels forming a macroblock to share an intra-pixel pulse generation unit to reduce the complexity of chip design and the area occupied.

[0068] As a preferred embodiment, the exposure mode of each pixel unit in the pixel array is global exposure or rolling exposure.

[0069] Of course, the temporal differential path and the spatial differential path can be arbitrarily combined in a global exposure or rolling exposure manner, and this application does not impose any particular limitation thereto.

[0070] Please refer to FIG5 , which is a schematic diagram of a hybrid array three-channel visual sensor based on multi-scale spatiotemporal differential pixels provided by this application.

[0071] The present application also provides a multi-channel visual sensor chip based on multi-scale spatiotemporal difference technology, including a pixel array, an intensity channel, a time difference channel and / or a space difference channel; the pixel array includes multiple pixel units; the intensity channel is used to determine the quantized value of the electrical signal converted from the intensity of the incident light at the current pixel unit position at the current moment; the time difference channel is used to perform weighted difference and quantization operations on the signal of the current pixel unit position at the current moment and the signal of the current pixel unit position at any previous moment in the charge domain, analog domain or digital domain to obtain one or more multi-scale time difference values; the space difference channel is used to perform weighted difference and quantization operations on the signal of the current pixel unit position at the current moment and the signal of any pixel unit position in the space at the current moment in the charge domain, analog domain or digital domain to obtain one or more multi-scale spatial difference values.

[0072] Considering the incomplete information acquisition of existing CIS and DVS vision sensors from the perspective of visual primitives, for example, when there is a large flash or a dramatic change in light intensity in the image, all TD pixels output events, resulting in saturation. The DVS pathway is unable to output valid information, and the CIS pathway, due to frame rate limitations, is unable to respond immediately. Such extreme scenarios are common in autonomous driving and are crucial for driving safety, such as entering and exiting tunnels and capturing camera flashes at night. In contrast, the human visual system can rapidly recognize moving targets, whether at noon or dusk, in open scenes or partially obscured. This achieves robustness and versatility far exceeding that of existing DAVIS or hybrid array systems. This is because the human eye not only outputs an intensity path and a temporal difference path, but also a spatial difference path. These three are organically integrated into distinct primitives, forming an efficient and robust visual representation.

[0073] Inspired by human vision, this application incorporates a spatial difference (SD) pathway modeled after the human retina into existing single-pixel multiplexing or hybrid pixel array solutions. This means the visual sensor simultaneously has three outputs: intensity, TD, and SD.

[0074] Specifically, the intensity path of this application outputs the current time t n The incident light intensity I(x,y,t n ) converted electrical signal, namely A(x,y,t n )=Q A (I(x,y,t n ))

[0075] where Q A A quantification method for the intensity pathway.

[0076] The time difference (TD) path of this application outputs the time difference value of the current pixel position (x, y) at different times. The expression of the obtained TD path output is:

[0077] where α i is the weight value, α i Can be any value. TD Quantization method, which can be multi-valued (>1 bit) or single-valued (positive and negative pulses). The time t at which the signal is acquired n ,t n-1 ,t n-2 …can be acquired in a manner that the entire array is synchronized with the same time interval, the entire array is synchronized with a variable time interval, or the entire array is asynchronous.

[0078] α i The number of N is not fixed, for example, N = 2, a0 = 1, a1 = -1, TD (x, y, t n )=Q TD (I(x,y,t n )-I(x,y,t n-1 ))

[0079] At this time, the time difference process is the current output value minus the previous output value. N=3,a0=1,a1=-0.8,a2=-0.2, TD(x,y,t n )=Q TD (I(x,y,t n )-80%I(x,y,t n-1 )-20%I(x,y,t n-2 )) N=3, a0=1, a1=0.2, a2=-1.2, TD(x,y,t n )=Q TD (I(x,y,t n )+20%I(x,y,t n-1 )-120%I(x,y,t n-2 ))

[0080] The data of quantization and weighted difference can be exchanged, that is, the formula can also be expressed as

[0081] There can be multiple multi-scale time difference channels coexisting, for example

[0082] And so on.

[0083] Furthermore, DVS is asynchronous and outputs only a timestamp and 1-bit information, making it susceptible to noise and low in information content. Since DVS can only output 1-bit time-varying information, it cannot adapt to complex environments. The visual sensing architecture of this application requires acquiring the temporal variation of visual signals in a synchronous or asynchronous manner. This temporal variation is preferably quantized and read out in a high-precision multi-value format.

[0084] The spatial difference (SD) path outputs the current time t n The spatial difference between the current pixel position (x, y) and any pixel unit position in space (diagonal or xy direction). For the xy direction difference, the expression of the SD path output is:

[0085] For oblique differential, the expression of the channel output is obtained as

[0086] where Q SD It is a quantization method, which can be multi-valued (>1 bit) or single-valued (positive and negative pulses). n ,t n-1 ,t n-2 …can be acquired in a manner that the entire array is synchronized with the same time interval, the entire array is synchronized with a variable time interval, or the entire array is asynchronous.

[0087] The above multi-scale spatial differentiation is intended to expand along the same direction.

[0088] More generally, the spatial difference can be expressed as

[0089] The purpose of the * subscript in SD is to indicate that SD may have multiple dimensions, for example, x , SD y , SD ↙ , SD ↘ This general formula can be used to express more complex situations. x ′(x,y,t n )= Q SD (I(x,y,t n )-80%I(x-1,y,t n )-10%I(x-1,y+1,t n )-10%I(x-1,y-1,t n ))

[0090] All signals involved in the above pathways are three-dimensional quantities, including two-dimensional spatial quantities x and y and a time dimension t.

[0091] As a preferred embodiment, the pixel array includes one type of pixel unit having a corresponding intensity path, a time differential path and / or a spatial differential path; or, the pixel array includes two types of pixel units, the first type of pixel unit having a corresponding intensity path, and the second type of pixel unit having a corresponding time differential path and / or a spatial differential path.

[0092] The intensity path, time differential path and / or space differential path involved in this application means that the sensor chip may have an intensity path and a time differential path; or, an intensity path and a space differential path; or, an intensity path and a time differential path and a space differential path.

[0093] When the chip has an intensity path and a time differential path, its pixel array can be composed entirely of one type of pixels, which corresponds to both the intensity path and the time differential path; or, its pixel array can be composed of two types of pixels, with the first type of pixels corresponding to the intensity path and the second type of pixels corresponding to the time differential path; or, the pixel array can also contain three types of pixels at the same time, with the first type of pixels corresponding to the intensity path, the second type of pixels corresponding to the time differential path, and the third type of pixels corresponding to both the intensity path and the time differential path.

[0094] When the chip has an intensity path and a spatial differential path, its pixel array can be composed entirely of one type of pixels, which corresponds to both the intensity path and the spatial differential path; or, its pixel array can be composed of two types of pixels, with the first type of pixels corresponding to the intensity path and the second type of pixels corresponding to the spatial differential path; or, the pixel array can also contain three types of pixels at the same time, with the first type of pixels corresponding to the intensity path, the second type of pixels corresponding to the spatial differential path, and the third type of pixels corresponding to both the intensity path and the spatial differential path.

[0095] When the chip has an intensity path, a time differential path, and a spatial differential path, that is, a three-path chip, its pixel array can be composed entirely of one type of pixels, which corresponds to the intensity path, the time differential path, and the spatial differential path at the same time; or, its pixel array can also be composed of two types of pixels, the first type of pixels corresponding to the intensity path, the second type of pixels corresponding to the time differential path and the spatial differential path; or, its pixel array can also be composed of two types of pixels, the first type of pixels corresponding to the time differential path, the second type of pixels corresponding to the intensity path and the spatial differential path; or, its pixel array can also be composed of two types of pixels, the first type of pixels corresponding to the spatial differential path, the second type of pixels corresponding to the time differential path and the intensity path; or, its pixel array can also be composed of three types of pixels, the first type of pixels corresponding to the intensity path, the second type of pixels corresponding to the time differential path, and the third type of pixels corresponding to the spatial differential path. Furthermore, the pixel array can also include any type of pixels in all the aforementioned methods, as long as at least one type of pixel has the requirement of intensity path output, at least one type of pixel has the requirement of time differential path output, and at least one type of pixel has the requirement of spatial differential path output.

[0096] The three-channel vision sensor chip can adopt a single pixel multiplexing method (that is, the pixel array contains only one type of pixel unit), a mixed pixel array method (that is, the pixel array contains multiple types of pixel units), or a fusion of the two methods.

[0097] Three output multiplexed pixels (intensity, TD, SD);

[0098] Two input multiplexed pixels (intensity, TD), SD pixels, forming a binary hybrid array;

[0099] Two input multiplexed pixels (intensity, SD), TD pixels, forming a binary hybrid array;

[0100] Two input spatiotemporal difference pixels (TD, SD), intensity pixels, form a binary mixed array;

[0101] TD pixels, SD pixels, and intensity pixels are separate and constitute a ternary hybrid array.

[0102] As a preferred embodiment, the intensity path includes an intensity storage node and an intensity quantization module; the intensity storage node is used to store the electrical signal converted from the intensity of the incident light at the current pixel unit position at the current moment; the intensity quantization module is used to perform analog-to-digital conversion on the electrical signal converted from the intensity of the incident light at the current pixel unit position at the current moment, and obtain the quantized value of the electrical signal converted from the intensity of the incident light at the current pixel unit position at the current moment; the time difference path includes multiple time difference storage nodes, time difference modules and time quantization modules; the multiple time difference storage nodes are used to store the electrical signals of the current pixel unit position at multiple moments respectively; the time difference module is used to convert the electrical signal of the current pixel unit position at the current moment into a quantized value based on the electrical signals of the current pixel unit position at multiple moments. The electrical signal of the pixel unit position is subjected to a weighted time difference operation with the electrical signal of the current pixel unit position at any previous moment to obtain a multi-scale time difference value; in the multi-scale time difference process, the analog-to-digital signal conversion is completed through the time quantization module; the spatial difference path includes a spatial difference storage node reused with a time difference storage node, a spatial difference module and a spatial quantization module; the spatial difference storage node is used to store the electrical signal of the current pixel unit position at the current moment; the spatial difference module is used to perform a weighted time difference operation on the electrical signal of the current pixel unit position at the current moment with the electrical signal of any pixel unit position in the space at the current moment to obtain a multi-scale spatial difference value; in the multi-scale spatial difference process, the analog-to-digital signal conversion is completed through the spatial quantization module.

[0103] Specifically, this embodiment adopts a full array asynchronous solution. Based on the intensity output and TD output of the multiplexed pixels, this embodiment adds an output of a time difference storage node, which will be used for SD output. That is, the time difference storage nodes of adjacent pixels are connected to the spatial difference module to obtain the final SD output. The square is a "three-multiplexed pixel", and the arrow represents I (x, y, t n ) information. This pixel supports the multi-scale spatiotemporal differential operation in the embodiment and supports the output intensity path. This embodiment does not provide an array arrangement diagram. The array arrangement diagram should be a densely packed rectangle, and the connection relationship between the rectangles is determined by the I(x, y, t n )Decide.

[0104] Of course, the visual sensor chip of this application can also adopt a full-array synchronization solution. The three-channel visual sensor of this application includes a binary hybrid pixel. This binary hybrid pixel not only has an intensity pixel but also integrates the SD and TD into the same pixel, which is called a multi-scale spatiotemporal differential pixel. Because the connection relationship is determined by the specific multi-scale differential method, the specific connection relationship is not given here.

[0105] Please refer to FIG6 , which is another schematic diagram of multi-scale spatiotemporal difference provided by this application.

[0106] In addition, in multi-scale operations, one intensity pixel can be spanned in the xy direction to obtain the spatial difference output in the xy direction. SD (I(x,yx,t n )-I(x-2,y,t n )). In addition, theoretically, it can also support differential results in any direction, such as Q SD (I(x,y,t n )-I(x-1,y+3,t n )), we can get the spatial difference result under the arctan(3) angle.

[0107] It should be noted that the spatiotemporal difference pixels and the intensity pixels can be arranged in any combination, and this application does not impose any special limitation thereto.

[0108] This application proposes a three-channel vision sensor chip architecture with simultaneous intensity, temporal, and spatial differential outputs, and proposes various pixel and array implementations. This architecture improves the perception capabilities of existing multiplexed pixel and hybrid array sensors, expands their application scenarios, and helps improve the robustness of vision sensors in applications such as autonomous driving. Furthermore, the three-channel vision sensor provides a feasible implementation of representation theory based on visual primitives.

[0109] As a preferred embodiment, the intensity quantization module is arranged in the pixel unit, and a pixel-level signal readout method is adopted; or the intensity quantization module is arranged outside the pixel unit and shared by the pixel units in the same column, and a column-level signal readout method is adopted; the time difference module and / or the time quantization module is arranged in the pixel unit, and a pixel-level signal readout method is adopted; or the time difference module and / or the time quantization module is arranged outside the pixel unit, and shared by the pixel units in the same column, and a column-level signal readout method is adopted; the spatial difference module and / or the spatial quantization module is arranged in the pixel unit, and a pixel-level signal readout method is adopted; or the spatial difference module and / or the spatial quantization module is arranged outside the pixel unit, and shared by the pixel units in the same column, and a column-level signal readout method is adopted.

[0110] In this embodiment, if the time difference module and the space difference module are set outside the time and space difference pixel unit, each column of time and space difference pixel units shares the same time difference module and the same space difference module, then the time difference values ​​and space difference values ​​of multiple time and space difference pixel units need to be read at the same time, following a certain rule, which can be done at t n , t n-1 , t n-2...output information at several fixed moments, which can be fixed time intervals or set to adaptive, programmable variable intervals, reducing the total number of quantization modules and hardware resource consumption.

[0111] If the time difference module and the space difference module are set in the time-space difference pixel unit, the time difference value and the space difference value of the time-space difference pixel unit can be read out synchronously with the entire array or asynchronously with the entire array. The time-space difference value and the space difference value are output specifically according to the triggering moment of each time-space difference pixel unit, which improves flexibility and reduces output delay.

[0112] Of course, the intensity module of the visual sensor chip of the present application can be arranged outside the intensity pixel unit and the time difference module and the space difference module can be arranged inside and outside the time-space difference image element in any combination, and the present application does not make any special restrictions here.

[0113] The visual sensor of this application is a temporal multi-scale and spatial multi-scale visual sensor. The circuit structure of the visual sensor is described below using temporal multi-scale difference as an example. The circuit structure of the spatial multi-scale difference visual sensor can be deduced by analogy and is not specifically limited in this application.

[0114] Please refer to FIG7 , which is a multi-scale time difference circuit diagram provided by this application.

[0115] Please refer to FIG8 , which is another multi-scale time difference circuit diagram provided by the present application.

[0116] This embodiment involves a sensor with only a multi-scale temporal differential path. Based on this structure, we can further infer the possible circuit structure of a sensor with only a multi-scale spatial differential path, or the circuit structure of other multi-scale dual-path or triple-path sensors.

[0117] This application implements a high-precision, multi-value, time-varying vision sensor chip architecture in a fully asynchronous, full-array format. This chip implements self-triggering (supporting internal or external triggering, programmable, and adaptive triggering), signal storage, intra-pixel temporal signal differentiation, and quantized readout within a single pixel. Each pixel directly outputs a high-precision, multi-value time differential (≥2-bit accuracy).

[0118] To support global asynchrony, each pixel in this chip has its own control logic. This control logic, called the intra-pixel pulse generation unit, adaptively adjusts the triggering timing for calculating the temporal difference visual signal based on the light intensity level perceived by the pixel. This unique triggering timing for each pixel achieves global asynchrony.

[0119] As a preferred embodiment, a pulse generating module is provided in each pixel unit in the pixel array; or, all pixel units in the pixel array are connected to a pulse generating module in common; or, the pixel array is divided into multiple sub-areas, and all pixel units in each sub-area are connected to a pulse generating module in common; wherein, the pulse generating module is used to generate a trigger signal at a fixed time interval, or to generate a trigger signal at an adaptive, programmable variable time interval, so as to control the start exposure moment and exposure duration of the photosensitive module, pixel units connected to the same pulse generating module are exposed synchronously, and pixel units connected to different pulse generating modules are exposed synchronously or asynchronously; the photosensitive module is provided in the pixel unit, and is used to convert the light signal at the current pixel unit position into an analog electrical signal.

[0120] Specifically, in this embodiment, two reading modes may be adopted: full array synchronization with the same time interval, or full array synchronization with a variable time interval.

[0121] The synchronization pulse generation method can be set not only to a fixed interval, but also to an adaptive, programmable variable interval. This adaptive interval can adapt to the changing characteristics of the external visual signal. When the amount of change is large and the frequency of change is high, a higher sampling frequency is used, and when the signal is low frequency, a lower sampling frequency is used to reduce data volume and energy consumption.

[0122] The present application has an intensity path and / or a time differential path and / or a space differential path. That is, it may have only a time differential path, or only a space differential path, or only an intensity path, or both an intensity path and a time differential path, or both an intensity path and a space differential path, or both a time differential path and a space differential path, or both an intensity path, a time differential path, and a space differential path.

[0123] This application also supports multiple pixels forming a macroblock to share an intra-pixel pulse generation unit to reduce the complexity of chip design and the area occupied.

[0124] Please refer to FIG9 , which is another multi-scale time difference circuit diagram provided by the present application.

[0125] Please refer to FIG10 , which is another multi-scale time difference circuit diagram provided by the present application.

[0126] The visual sensor chip of this application can support pixel spatial fusion to achieve temporal differential perception with a larger receptive field, larger spatial scale, and higher sensitivity. By sharing readout switches and storage nodes, the photoelectric signals of multiple pixels are fused and then temporal differential calculations are performed.

[0127] This application proposes a multi-value, high-precision time-varying visual sensor chip architecture, and proposes a variety of signal recording and conversion methods, which can greatly improve the visual sensor's ability to reconstruct spatiotemporal dynamic information with high precision.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A visual sensor chip based on multi-scale spatiotemporal difference technology, comprising a pixel array, a temporal difference path and / or a spatial difference path; the pixel array comprises a plurality of pixel units; The time difference path is used to perform weighted difference and quantization operations on the signal of the current pixel unit position at the current moment and the signal of the current pixel unit position at any previous moment in the charge domain, analog domain or digital domain to obtain one or more multi-scale time difference values; The spatial differential path is used to perform weighted differential and quantization operations on the signal of the current pixel unit position at the current moment and the signal of any pixel unit position in the space at the current moment in the charge domain, analog domain or digital domain to obtain one or more multi-scale spatial differential values.

2. The visual sensor chip based on multi-scale spatiotemporal difference technology according to claim 1, wherein: The time difference path includes a plurality of time difference storage nodes, a time difference module and a time quantization module; The multiple time difference storage nodes are used to store electrical signals of the current pixel unit positions at multiple moments respectively; The time difference module is used to perform a weighted time difference operation on the electrical signal at the current pixel unit position at the current moment and the electrical signal at the current pixel unit position at any previous moment according to the electrical signals at the current pixel unit position at the multiple moments, so as to obtain the multi-scale time difference value; the time quantization module is used to complete the analog-to-digital signal conversion in the multi-scale time difference process; The spatial difference path includes a spatial difference storage node multiplexed with one of the temporal difference storage nodes, a spatial difference module and a spatial quantization module; The spatial difference storage node is used to store the electrical signal of the current pixel unit position at the current moment; The spatial difference module is used to perform a weighted time difference operation on the electrical signal of the current pixel unit position at the current moment and the electrical signal of any pixel unit position in the space at the current moment to obtain the multi-scale spatial difference value; In the multi-scale spatial difference process, analog-to-digital signal conversion is completed by the spatial quantization module.

3. The visual sensor chip based on multi-scale spatiotemporal difference technology according to claim 2, wherein: The time difference module and / or the time quantization module are arranged in the pixel unit, and adopt a pixel-level signal readout method; Alternatively, the time difference module and / or the time quantization module are arranged outside the pixel unit and are shared by the pixel units in the same column, and a column-level signal readout method is adopted; The spatial differential module and / or the spatial quantization module are arranged in the pixel unit, and a pixel-level signal readout method is adopted; or the temporal differential module and / or the temporal quantization module are arranged outside the pixel unit and shared by the pixel units in the same column, and a column-level signal readout method is adopted.

4. The visual sensor chip based on multi-scale spatiotemporal difference technology according to claim 1, wherein: A pulse generating module is provided in each pixel unit in the pixel array; Or, all pixel units in the pixel array are connected to a pulse generating module; Or, the pixel array is divided into a plurality of sub-areas, and all pixel units in each sub-area are connected to a pulse generating module; The pulse generating module is used to generate a trigger signal at a fixed time interval, or to generate a trigger signal at an adaptive, programmable variable time interval, so as to control the exposure start time and exposure duration of the photosensitive module, and to synchronously expose the pixel units connected to the same pulse generating module, and synchronously expose or asynchronously expose the pixel units connected to different pulse generating modules; The photosensitive module is disposed in the pixel unit and is used to convert the light signal at the current pixel unit position into an analog electrical signal.

5. The visual sensor chip based on multi-scale spatiotemporal difference technology according to any one of claims 1 to 4, wherein: The exposure mode of each pixel unit in the pixel array is global exposure or rolling exposure.

6. A multi-channel visual sensor chip based on multi-scale spatiotemporal difference technology, comprising a pixel array, an intensity channel, a time difference channel and / or a space difference channel; The pixel array includes a plurality of pixel units; The intensity path is used to determine the quantized value of the electrical signal converted from the light intensity of the incident light at the current pixel unit position at the current moment; The time difference path is used to perform weighted difference and quantization operations on the signal of the current pixel unit position at the current moment and the signal of the current pixel unit position at any previous moment in the charge domain, analog domain or digital domain to obtain one or more multi-scale time difference values; The spatial differential path is used to perform weighted differential and quantization operations on the signal of the current pixel unit position at the current moment and the signal of any pixel unit position in the space at the current moment in the charge domain, analog domain or digital domain to obtain one or more multi-scale spatial differential values.

7. The multi-channel visual sensor chip based on multi-scale spatiotemporal difference technology according to claim 6, wherein: The pixel array includes one type of pixel unit, which has a corresponding intensity path, a time differential path and / or a spatial differential path; or, the pixel array includes two types of pixel units, the first type of pixel unit has a corresponding intensity path, and the second type of pixel unit has a corresponding time differential path and / or a spatial differential path.

8. The multi-channel visual sensor chip based on multi-scale spatiotemporal difference technology according to claim 6, wherein: The intensity path includes an intensity storage node and an intensity quantization module; The intensity storage node is used to store the electrical signal converted from the intensity of the incident light at the current pixel unit position at the current moment; The intensity quantization module is used to perform analog-to-digital conversion on the electrical signal converted from the intensity of the incident light at the current pixel unit position at the current moment, to obtain a quantized value of the electrical signal converted from the intensity of the incident light at the current pixel unit position at the current moment; The time difference path includes a plurality of time difference storage nodes, a time difference module and a time quantization module; The multiple time difference storage nodes are used to store electrical signals of the current pixel unit positions at multiple moments respectively; The time difference module is used to perform a weighted time difference operation on the electrical signal at the current pixel unit position at the current moment and the electrical signal at the current pixel unit position at any previous moment according to the electrical signals at the current pixel unit position at the multiple moments, so as to obtain the multi-scale time difference value; the time quantization module is used to complete the analog-to-digital signal conversion in the multi-scale time difference process; The spatial difference path includes a spatial difference storage node multiplexed with one of the temporal difference storage nodes, a spatial difference module and a spatial quantization module; The spatial difference storage node is used to store the electrical signal of the current pixel unit position at the current moment; The spatial difference module is used to compare the electrical signal of the current pixel unit position at the current moment with the current Performing a weighted time difference operation on the electrical signal of any pixel unit position in the previous moment to obtain the multi-scale spatial difference value; In the multi-scale spatial difference process, analog-to-digital signal conversion is completed by the spatial quantization module.

9. The multi-channel visual sensor chip based on multi-scale spatiotemporal difference technology according to claim 8, wherein: The intensity quantization module is arranged in the pixel unit, and adopts a pixel-level signal readout method; or the intensity quantization module is arranged outside the pixel unit and shared by the pixel units in the same column, and adopts a column-level signal readout method; The time difference module and / or the time quantization module are arranged in the pixel unit, and a pixel-level signal is read out; or the time difference module and / or the time quantization module are arranged outside the pixel unit and shared by the pixel units in the same column, and a column-level signal is read out; The spatial difference module and / or the spatial quantization module are arranged in the pixel unit, and a pixel-level signal readout method is adopted; or the spatial difference module and / or the spatial quantization module are arranged outside the pixel unit and shared by the pixel units in the same column, and a column-level signal readout method is adopted.

10. The multi-channel visual sensor chip based on multi-scale spatiotemporal difference technology according to claim 6, wherein: A pulse generating module is provided in each pixel unit in the pixel array; Or, all pixel units in the pixel array are connected to a pulse generating module; Or, the pixel array is divided into a plurality of sub-areas, and all pixel units in each sub-area are connected to a pulse generating module; The pulse generating module is used to generate a trigger signal at a fixed time interval, or to generate a trigger signal at an adaptive, programmable variable time interval, so as to control the exposure start time and exposure duration of the photosensitive module, and to synchronously expose the pixel units connected to the same pulse generating module, and synchronously expose or asynchronously expose the pixel units connected to different pulse generating modules; The photosensitive module is disposed in the pixel unit and is used to convert the light signal at the current pixel unit position into an analog electrical signal.

11. The multi-channel visual sensor chip based on multi-scale spatiotemporal difference technology according to any one of claims 6 to 10, wherein: The exposure mode of each pixel unit in the pixel array is global exposure or rolling exposure.

Images