Vision sensor chip based on pixel fusion technology

Abstract

The present application provides a vision sensor chip based on pixel fusion technology, comprising: a pixel array, an intensity path, a temporal difference path, and a spatial difference path. The fusion technology is used for fusing signals of a plurality of pixel units within the range of a fused pixel into one signal and then outputting same; the intensity path determines a quantized value of an electric signal converted from the incident light intensity of a fused pixel; the temporal difference path performs difference, fusion and quantization operations on a signal of the current fused pixel position at the current moment and a signal of the current fused pixel position at the previous moment in a charge domain, an analog domain or a digital domain; and the spatial difference path performs difference, fusion and quantization operations on the signal of the current fused pixel position at the current moment and a signal of a space-associated fused pixel position at the current moment in the charge domain, the analog domain or the digital domain. The pixel fusion technology is introduced into a three-path vision sensor chip architecture, so that the output signal-to-noise ratio can be improved, and the volume of transmitted data can be reduced, reducing bandwidth pressure.

Description

A visual sensor chip based on pixel fusion technology

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202311420671.9, filed on October 30, 2023, entitled “A visual sensor chip based on pixel fusion technology”, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present application relates to the field of visual sensing technology, and in particular to a visual sensor chip based on pixel fusion technology. Background Art

[0004] A visual sensor is an instrument that uses a photosensitive element to convert external optical information into an electrical signal image. The most commonly used visual sensor in the prior art is the CIS (CMOS Image Sensor). CIS is an image sensor based on the frame sampling principle and is widely used in camera modules of mobile phones or cameras. This type of image sensor has the advantages of high color reproduction and image quality. However, the dynamic range of the image signal it acquires is small, making it difficult to increase the shooting speed under limited bandwidth. Another new type of image sensor, DVS (Dynamic Vision Sensor), is characterized by its ability to perceive changes in dynamic scenes in the form of a sparse event stream. Due to its fast shooting speed and large dynamic range of the image signal it acquires, this type of sensor has the problems of low resolution and excessive loss of effective information.

[0005] In the existing technology, the frame rate of video frames captured by CIS is not high, and the number of video frames collected within a certain period of time is limited. When there is a large-scale flash or a drastic change in light intensity in the picture, DVS may not be able to output images normally, resulting in the inability to capture spatial changes in the scene. In addition, DVS is easily affected by noise interference. Therefore, the present application urgently needs to provide an improved visual sensor.

[0006] Summary of the Invention

[0007] The present application provides a visual sensor chip based on pixel fusion technology to solve the defects of sensors in the existing technology, such as significant noise problems and limited bandwidth. Introducing pixel fusion technology into the three-channel visual sensor chip architecture can effectively improve the output signal-to-noise ratio and further reduce the transmission bandwidth.

[0008] The present application provides a visual sensor chip based on pixel fusion technology, including a pixel array, an intensity path, a time difference path and a spatial difference path; the pixel array includes multiple pixel units; the fusion technology is used to fuse the signals of multiple pixel units within the fusion pixel range into one signal and then output it; the intensity path is used to determine the quantized value of the electrical signal converted from the intensity of the incident light of the fusion pixel; the time difference path is used to perform time difference, fusion and quantization operations on the signal of the current fusion pixel position at the current moment and the signal of the current fusion pixel position at the previous moment in the charge domain, analog domain or digital domain to obtain the time difference value of the fusion pixel; the signal of the fusion pixel is the signal of multiple pixel units within the fusion pixel range; the spatial difference path is used to perform spatial difference, fusion and quantization operations on the signal of the current fusion pixel position at the current moment and the signal of the spatially associated fusion pixel position at the current moment in the charge domain, analog domain or digital domain to obtain the spatial difference value of the fusion pixel; the spatially associated fusion pixel is any one or more fusion pixels in the array except the current fusion pixel.

[0009] According to a visual sensor chip based on pixel fusion technology provided by the present application, the intensity path includes a first intensity fusion module and a first intensity quantization module; the first intensity fusion module is used to fuse the analog signals of multiple pixel units at the current fusion pixel position at the current moment to obtain a first intensity fusion signal; the first intensity quantization module is used to perform analog-to-digital conversion on the first intensity fusion signal to obtain a quantized value of the electrical signal converted from the incident light intensity of the fusion pixel.

[0010] According to a visual sensor chip based on pixel fusion technology provided by the present application, the intensity path includes a second intensity quantization module and a second intensity fusion module; the second intensity quantization module is used to perform analog-to-digital conversion on the analog signals of multiple pixel units at the current fusion pixel position at the current moment, and obtain the quantized value of the electrical signal converted from the incident light intensity of each pixel unit; the second intensity fusion module is used to fuse the quantized values ​​of the incident light intensity of multiple pixel units, and obtain the quantized value of the electrical signal converted from the incident light intensity of the fused pixel.

[0011] According to a visual sensor chip based on pixel fusion technology provided by the present application, the time differential path includes a first time differential fusion module, a first time differential module and a first time quantization module; the first time differential fusion module is used to fuse the electrical signals of multiple pixel units to obtain the first time differential fusion signal of the current fused pixel position at different times; the first time differential module is used to perform a time differential operation on the first time differential fusion signal of the current fused pixel position at the current moment and the first time differential fusion signal of the current fused pixel position at the previous moment to obtain the time differential value of the fused pixel; the first time quantization module is used to complete the analog-to-digital signal conversion during the fusion and time differential process.

[0012] According to a visual sensor chip based on pixel fusion technology provided by the present application, the time differential path includes a second time differential module, a second time quantization module and a second time differential fusion module; the second time differential module is used to determine the time differential value of each pixel unit; the time differential value of each pixel unit is obtained by performing a time differential operation on the electrical signal of the current pixel unit position at the current moment and the electrical signal of the current pixel unit position at the previous moment; the second time differential fusion module is used to fuse the time differential values ​​of multiple pixel units to obtain the time differential value of the fused pixel; the analog-to-digital signal conversion is completed by the second time quantization module during the fusion and time differential process.

[0013] According to a visual sensor chip based on pixel fusion technology provided by the present application, the spatial differential path includes a first spatial differential fusion module, a first spatial differential module and a first spatial quantization module; the first spatial differential fusion module is used to fuse the electrical signals of multiple pixel units to obtain a first spatial differential fusion signal of the current fused pixel position at the current moment; the first spatial differential module is used to perform spatial differential operation on the first spatial differential fusion signal of the current fused pixel position at the current moment and the first spatial differential fusion signal of the spatially associated fused pixel position at the current moment to obtain the spatial differential value of the fused pixel; the analog-to-digital signal conversion is completed by the first spatial quantization module during the fusion and spatial differential process.

[0014] According to a visual sensor chip based on pixel fusion technology provided by the present application, the spatial differential path includes a second spatial differential module, a second spatial quantization module and a second spatial differential fusion module; the second spatial differential module is used to determine the spatial differential value of each of the pixel units; the spatial differential value of each pixel unit is obtained by performing a spatial differential operation on the electrical signal of the current pixel unit position at the current moment and the electrical signal of the spatially associated pixel position at the current moment; the second spatial differential fusion module is used to fuse the spatial differential values ​​of the multiple pixel units to obtain the spatial differential value of the fused pixel; the analog-to-digital signal conversion is completed by the second spatial quantization module during the fusion and spatial differential process.

[0015] According to a visual sensor chip based on pixel fusion technology provided by the present application, a pulse generating module is set in each pixel unit in the pixel array; or, all pixel units in the pixel array are connected to a pulse generating module; or, the pixel array is divided into multiple sub-areas, and all pixel units in each sub-area are connected to a pulse generating module; wherein, the pulse generating module is used to generate a trigger signal at a fixed time interval, or to generate a trigger signal at an adaptive, programmable variable time interval to control the start exposure moment and exposure duration of the photosensitive module, the pixel units connected to the same pulse generating module are exposed synchronously, and the pixel units connected to different pulse generating modules are exposed synchronously or asynchronously; the photosensitive module is set in the pixel unit, and is used to convert the light signal at the current pixel unit position into an analog electrical signal.

[0016] According to a visual sensor chip based on pixel fusion technology provided by the present application, the exposure mode of each unit pixel in the pixel array is global exposure or rolling exposure.

[0017] According to a visual sensor chip based on pixel fusion technology provided by the present application, when the pixel unit is provided with a color filter, the output color type of the corresponding path of the pixel unit is a color value; when the pixel unit is not provided with a color filter, the output color type of the corresponding path of the pixel unit is a grayscale value.

[0018] The present application provides a visual sensor chip based on pixel fusion technology, comprising: a pixel array, an intensity path, a time difference path, and a space difference path; the pixel array comprises a plurality of pixel units; the fusion technology is used to fuse the signals of the plurality of pixel units within the fusion pixel range into one signal and then output it; the intensity path determines the quantized value of the electrical signal converted from the incident light intensity of the fusion pixel; the time difference path performs differential, fusion, and quantization operations on the signal of the current fusion pixel position at the current moment and the signal of the current fusion pixel position at the previous moment in the charge domain, analog domain, or digital domain; the space difference path performs differential, fusion, and quantization operations on the signal of the current fusion pixel position at the current moment and the signal of the adjacent fusion pixel position associated at the current moment in the charge domain, analog domain, or digital domain. Introducing pixel fusion technology into the three-path visual sensor chip architecture can improve the output signal-to-noise ratio and reduce the amount of transmitted data to alleviate bandwidth pressure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] FIG1 is a schematic diagram of a visual sensor chip based on pixel fusion technology provided by the present application;

[0021] FIG2a is one of the schematic diagrams of the principle of pixel fusion provided by this application;

[0022] FIG2 b is a second schematic diagram of the principle of pixel fusion provided by the present application;

[0023] FIG3 is a schematic structural diagram of a three-channel visual sensor with multiplexed pixels provided by the present application;

[0024] FIG4 is a schematic structural diagram of a hybrid pixel visual sensor provided by the present application;

[0025] FIG5 is a second schematic diagram of the principle of a visual sensor chip based on pixel fusion technology provided by the present application;

[0026] FIG6 is a third schematic diagram of the principle of a visual sensor chip based on pixel fusion technology provided by the present application;

[0027] FIG7 is a schematic diagram of the principle of oblique direction difference provided by the present application;

[0028] FIG8 is a schematic diagram showing the principle of xy direction differentiation provided by this application. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0030] Currently, most CMOS image sensors (CIS) capture video using a frame-based sampling principle. This means that each CIS image frame records the output of all pixels in the pixel array, and each frame is equally spaced. Because transistors are integrated within the pixels to achieve high-performance charge-to-voltage conversion, CIS is also known as an active pixel sensor (APS). By covering the pixel array with a color filter array, CIS can sense visible light of different wavelengths to produce a color image. CIS offers the advantages of high pixel resolution, excellent color reproduction, and high image quality. DVS is a novel imaging system. Unlike traditional cameras, which use a shutter to control the frame rate and record light intensity on a per-frame basis, DVS is sensitive to the rate of change of light intensity. Each pixel independently records the change in the logarithm of the light intensity at that pixel, generating a positive or negative pulse when the change exceeds a threshold. The asynchronous nature of DVS enables it to achieve extremely high temporal resolution. Combined with its sensitivity to change, DVS is naturally suited for tasks such as motion detection. Another camera, called DAVIS, combines a traditional active pixel sensor (APS) with DVS, capable of recording both single-frame images and event information, combining the advantages of high spatial resolution of traditional cameras and high temporal resolution of DVS cameras.

[0031] From the perspective of visual primitives, vision sensors with only CIS and DVS channels provide incomplete information collection. For example, when there is a large flash or a dramatic change in light intensity in the image, all time-differential pixels output events, leading to saturation. The DVS channel is unable to output valid information, and noise becomes more pronounced. The CIS channel, due to frame rate limitations, also cannot respond in a timely manner, making it difficult to increase capture speed within limited bandwidth. Such extreme scenarios are very common in autonomous driving and are crucial to driving safety, such as when entering and exiting tunnels and capturing camera flashes at night.

[0032] Please refer to FIG1 , which is one of the principle schematic diagrams of a visual sensor chip based on pixel fusion technology provided by this application.

[0033] Please refer to FIG2a, which is one of the principle schematic diagrams of pixel fusion provided in this application.

[0034] Please refer to FIG2b, which is a second schematic diagram of the principle of pixel fusion provided in this application.

[0035] In order to solve the technical problems existing in the prior art, the present application provides a visual sensor chip based on pixel fusion technology, including a pixel array, an intensity path, a time difference path and a spatial difference path; the pixel array includes multiple pixel units; the fusion technology is used to fuse the signals of multiple pixel units within the fusion pixel range into one signal and then output it; the intensity path is used to determine the quantized value of the electrical signal converted from the incident light intensity of the fusion pixel; the time difference path is used to perform time difference, fusion and quantization operations on the signal of the current fusion pixel position at the current moment and the signal of the current fusion pixel position at the previous moment in the charge domain, analog domain or digital domain to obtain the time difference value of the fusion pixel; the signal of the fusion pixel is the signal of multiple pixel units within the fusion pixel range; the spatial difference path is used to perform spatial difference, fusion and quantization operations on the signal of the current fusion pixel position at the current moment and the signal of the spatially associated fusion pixel position at the current moment in the charge domain, analog domain or digital domain to obtain the spatial difference value of the fusion pixel; the spatially associated fusion pixel is any one or more fusion pixels in the array except the current fusion pixel.

[0036] In contrast, the human visual system can quickly identify moving targets whether it is at noon or dusk, in an open scene or partially blocked. It achieves robustness and versatility far exceeding the existing DAVIS or hybrid array systems. This is because in addition to outputting color channels and time difference channels, the human eye also has a spatial difference channel. The three are organically integrated and combined into different primitives to form an efficient and robust visual representation. Pixel fusion is mainly used for noise reduction and data bandwidth reduction. Inspired by human vision, this application adds a spatial difference channel that simulates the human retina to the existing single pixel multiplexing or hybrid pixel array solution. That is, the visual sensor has three outputs at the same time: intensity output, time difference (TD, Temporal Difference) output, and spatial difference (SD, Spatial Difference) output.

[0037] This application supports pixel space fusion (binning) to perform temporal and spatial differential perception with a larger receptive field, a larger spatial scale and higher sensitivity. By sharing the readout switch and storage node, multiple similar pixel output values ​​are fused together. The typical fusion range includes 2×2 range pixel unit fusion, 3×3 range pixel unit fusion, and other range pixel unit fusion. The fused pixels are combined into a large fused pixel output. The output value can be the sum of all fused pixel output values, or the average value. In addition, the median, maximum value, minimum value or other functional relationships can also be considered. That is

[0038] I 融合 =f(I1,I2,…I n )

[0039] Among them I 融合 is the result after fusion, I i (i=1…n) is the output value of each small pixel in the fused range, f is the fusion function, and the two most typical ones are summation and averaging.

[0040] The fusion process can be performed in the charge domain, analog domain, or digital domain. Pixel fusion reduces the amount of data that needs to be processed / transmitted, and in some cases can increase the frame rate. Furthermore, the signal-to-noise ratio of the fused pixels is improved.

[0041] The electrical signal here can be a charge, current or voltage signal.

[0042] All signals involved in the three sensing pathways are three-dimensional quantities, including the spatial two-dimensional quantities x and y and the time dimension t.

[0043] The 2×2 fused pixel output value in the dotted box is expressed as

[0044] Among them TD i Indicates the TD path output value of pixel ① at the current moment. i is the weight. Typically, such as n 1～2 =1,n 3～8 =0;n 1～2 =6,n 3～6 =1,n 7～8 =0; The method is not unique.

[0045] Please refer to FIG3 , which is a schematic structural diagram of a three-channel vision sensor with multiplexed pixels provided by the present application.

[0046] Please refer to FIG4 , which is a schematic structural diagram of a hybrid pixel visual sensor provided in this application.

[0047] The fused pixel is a plurality of multiplexed pixels, or a plurality of binary mixed pixels and a plurality of single pixels, or a plurality of ternary mixed pixels; the multiplexed pixel is a pixel multiplexed by intensity and spatiotemporal difference; the binary mixed pixel is a pixel multiplexed by two elements of intensity, time difference and space difference; the single pixel is a pixel with different elements from the binary mixed pixel; the ternary mixed pixel is a pixel in which intensity, time difference and space difference correspond to three different pixels respectively and are mixed and arranged.

[0048] In addition, storage nodes can be flexibly set to store signals according to the needs of signal transmission. This application does not impose any special restrictions on the number and setting positions of storage nodes.

[0049] The intensity quantization module can be set outside the pixel unit; the time difference module, time quantization module, spatial difference module, spatial quantization module, and fusion module can be set outside the pixel unit and shared by pixels in the same column, or they can be set inside the pixel unit. This application does not make any special restrictions here.

[0050] The fusion module can be designed for each pixel unit individually, or distributed in columns, or performed in a later image processing module. This application does not make any special restrictions here.

[0051] Fusion modules with the same fusion logic can be reused.

[0052] This application introduces pixel fusion technology into the three-channel visual sensor chip architecture, which can improve the output signal-to-noise ratio and reduce the amount of transmitted data to alleviate bandwidth pressure.

[0053] Based on the above embodiment:

[0054] As a preferred embodiment, the intensity path includes a first intensity fusion module and a first intensity quantization module; the first intensity fusion module is used to fuse the analog signals of multiple pixel units at the current fusion pixel position at the current moment to obtain a first intensity fusion signal; the first intensity quantization module is used to perform analog-to-digital conversion on the first intensity fusion signal to obtain a quantized value of the electrical signal converted from the incident light intensity of the fusion pixel.

[0055] Among them, xy is the coordinate of the fused pixel, x i ,y i is the coordinate of each pixel unit, Q A Quantification of the intensity pathway.

[0056] Of course, pixel fusion can be done in addition to sum In addition, you can also take the average

[0057] As a preferred embodiment, the intensity path includes a second intensity quantization module and a second intensity fusion module; the second intensity quantization module is used to perform analog-to-digital conversion on the analog signals of multiple pixel units at the current fusion pixel position at the current moment to obtain the quantized value of the electrical signal converted from the incident light intensity of each pixel unit; the second intensity fusion module is used to fuse the quantized values ​​of the incident light intensity of multiple pixel units to obtain the quantized value of the electrical signal converted from the incident light intensity of the fused pixel.

[0058] The second intensity is quantized by the module I(x i ,y i ,t j )=Q A (I(x i ,y i ,t j )) obtaining a quantized value of an electrical signal converted from the incident light intensity of each pixel unit;

[0059] The second strength is achieved by using the fusion module The quantized value of the electrical signal converted from the incident light intensity of the fused pixel is obtained.

[0060] As a preferred embodiment, the time difference path includes a first time difference fusion module, a first time difference module and a first time quantization module; the first time difference fusion module is used to fuse the electrical signals of multiple pixel units to obtain the first time difference fusion signal of the current fusion pixel position at different times; the first time difference module is used to perform a time difference operation on the first time difference fusion signal of the current fusion pixel position at the current moment and the first time difference fusion signal of the current fusion pixel position at the previous moment to obtain the time difference value of the fusion pixel; during the fusion and time difference process, the analog-to-digital signal conversion is completed by the first time quantization module.

[0061] In all the following subscripts, analog indicates that the signal is an analog signal, and digital indicates that the signal is a digital signal. i Represents the initial analog signal, TD and SD represent the final digital signal

[0062] The first time difference is obtained by using the fusion module Obtaining a fusion signal for the first time difference;

[0063] The first time difference module is combined with the first time quantization module to obtain the time difference value TD (x, y, t n )=Q TD (I 融合-analog (x,y,t n )-I 融合-analog (x,y,t n-1 )).

[0064] Or, the first time difference fusion module is combined with the first time quantization module to obtain the first time difference fusion signal

[0065] The first time difference module uses TD(x,y,t n )=I 融合-digital (x,y,t n )-I 融合-digital (x,y,t n-1 ) to obtain the time difference value of the fused pixel.

[0066] Or, use the quantization module to pass I i-digital (x i ,y i ,t j )=Q A (I i (x i ,y i ,t j ) obtain the quantized value of the incident light intensity of each pixel unit;

[0067] The first time difference is obtained by using the fusion module Obtaining a fusion signal for the first time difference;

[0068] The first time difference module uses TD(x,y,t n )=I 融合-digital (x,y,t n )-I 融合-digital (x,y,t n-1 ) to obtain the time difference value of the fused pixel.

[0069] As a preferred embodiment, the time difference path includes a second time difference module, a second time quantization module and a second time difference fusion module; the second time difference module is used to determine the time difference value of each pixel unit; the time difference value of each pixel unit is obtained by performing a time difference operation on the electrical signal of the current pixel unit position at the current moment and the electrical signal of the current pixel unit position at the previous moment; the second time difference fusion module is used to fuse the time difference values ​​of multiple pixel units to obtain the time difference value of the fused pixel; during the fusion and time difference process, the analog-to-digital signal conversion is completed by the second time quantization module.

[0070] The second time difference module uses TD i-analog (x i ,y i ,t n )=I i (x i ,y i ,tn )-I i (x i ,y i ,t n-1 ) determining a time difference value for each pixel unit;

[0071] The second time difference is combined with the quantization module to obtain the time difference value of the fused pixel.

[0072] Alternatively, the second time difference module is combined with the second time quantization module to determine the time difference value TD of each pixel unit. i-digital (x i ,y i ,t n )=Q TD (I i (x i ,y i ,t n )-I i (x i ,y i ,t n-1 ));

[0073] The second time difference is obtained by using the fusion module Get the time difference value of the fused pixel.

[0074] Or, the second time, the quantization module is used to pass I i-digital (x i ,y i ,t j )=Q A (I i (x i ,y i ,t j )) obtain the quantized value of the incident light intensity of each pixel unit;

[0075] The second time difference module uses TD i-digital (x i ,y i ,t n )=(I i-digital (x i ,y i ,t n )-I i-digital (x i ,y i ,t n-1 ))Determine the time difference value of each pixel unit;

[0076] The second time difference is obtained by using the fusion module Get the time difference value of the fused pixel.

[0077] The above content Q TD Quantization of the time difference path.

[0078] Please refer to FIG5 , which is a second schematic diagram of the principle of the visual sensor chip based on pixel fusion technology provided by this application.

[0079] Please refer to FIG6 , which is the third principle schematic diagram of the visual sensor chip based on pixel fusion technology provided by this application.

[0080] The hybrid pixel array involves two different types of pixels, taking TD as an example (the same applies to A and SD).

[0081] Where N is the pixel unit within the selected range of the fused pixel, as well as the surrounding adjacent pixel units. Only pixels with differential output are selected as the selected range.

[0082] The 3×3 fused pixel output value in the dotted box in Figure 5 is expressed as

[0083] Among them TD i Indicates the TD path output value of pixel (i) at the current moment. i is the weight. For example, n 1～5 =1,n 6～13 =0.

[0084] The 3×3 fused pixel output value in the dotted box in Figure 6 is expressed as

[0085] Among them TD i Indicates the TD path output value of pixel (i) at the current moment. i is the weight. For example, n 1～4 =1,n 5～12 =0.

[0086] Please refer to FIG. 7 , which is a schematic diagram of the principle of oblique direction difference provided by this application.

[0087] Please refer to FIG8 , which is a schematic diagram of the principle of xy direction difference provided by this application.

[0088] In addition to these two difference modes, there are other difference modes, such as only performing x difference, or selecting more than two adjacent pixels for difference, etc.

[0089] As a preferred embodiment, the spatial difference path includes a first spatial difference fusion module, a first spatial difference module and a first spatial quantization module; the first spatial difference fusion module is used to fuse the electrical signals of multiple pixel units to obtain the first spatial difference fusion signal of the current fusion pixel position at the current moment; the first spatial difference module is used to perform spatial difference operation on the first spatial difference fusion signal of the current fusion pixel position at the current moment and the first spatial difference fusion signal of the spatially associated fusion pixel position at the current moment to obtain the spatial difference value of the fusion pixel; during the fusion and spatial difference process, the analog-to-digital signal conversion is completed through the first spatial quantization module.

[0090] The first spatial difference is obtained by fusion module Obtaining a fusion signal for a first spatial difference;

[0091] The first spatial difference module is combined with the first spatial quantization module to obtain the spatial difference value SD of the fused pixel X (x,y,t n )=Q SD (I 融合-analog (x,y,t n )-I 融合-analog (x-1,y,t n ))

[0092] SD Y (x,y,t n )=Q SD (I 融合-analog (x,y,t n )-I 融合-analog (x,y-1,t n ))

[0093] SD ↙ (x,y,t n )=Q SD (I 融合 (x,y,t n )-I 融合 (x-1,y-1,t n ))

[0094] SD ↘ (x,y,t n )=Q SD (I 融合 (x,y,t n )-I 融合 (x+1,y-1,t n ))

[0095] Alternatively, the first spatial difference fusion module is combined with the first spatial quantization module to obtain the first spatial difference fusion signal

[0096] The first spatial difference module determines the spatial difference value of the fused pixel

[0097] SD_X(x,y,t n )=I 融合 (x,y,t n )-I 融合 (x-1,y,t n )

[0098] SD_Y(x,y,t n )=I 融合 (x,y,t n )-I 融合 (x,y-1,t n ).

[0099] Or, the first space is quantized by the module I i-analog (x i ,y i ,t j )=Q A (I i (x i ,y i ,t j ) obtain the quantized value of the incident light intensity of each pixel unit;

[0100] The first spatial difference is obtained by fusion module Obtaining a fusion signal for a first spatial difference;

[0101] The first spatial difference module determines the spatial difference value of the fused pixel

[0102] SD_X(x,y,t n )=I 融合 (x,y,t n )-I 融合 (x-1,y,t n )

[0103] SD_Y(x,y,t n )=I 融合 (x,y,t n )-I 融合 (x,y-1,t n ).

[0104] As a preferred embodiment, the spatial difference path includes a second spatial difference module, a second spatial quantization module and a second spatial difference fusion module; the second spatial difference module is used to determine the spatial difference value of each pixel unit; the spatial difference value of each pixel unit is obtained by performing a spatial difference operation on the electrical signal of the current pixel unit position at the current moment and the electrical signal of the spatially associated pixel position at the current moment; the second spatial difference fusion module is used to fuse the spatial difference values ​​of multiple pixel units to obtain the spatial difference value of the fused pixel; during the fusion and spatial difference process, the analog-to-digital signal conversion is completed by the second spatial quantization module.

[0105] The second spatial difference module determines the spatial difference value of each pixel unit

[0106]

[0107]

[0108]

[0109]

[0110] The second spatial difference is combined with the quantization module in the second time to obtain the spatial difference value of the fused pixel.

[0111] Alternatively, the second spatial difference module is combined with the second time quantization module to determine the spatial difference value of each pixel unit.

[0112] SD x (x i ,y i ,t n )=Q SD (I(x i ,y i ,t n )-I(x i -1,y i ,t n ))

[0113] SD y (x i ,y i ,t n )=Q SD (I(x i ,y i ,t n )-I(x i ,y i -1,t n ))

[0114] SD↙ (x i ,y i ,t n )=Q SD (I(x i ,y i ,t n )-I(x i -1,y i -1,t n ))

[0115] SD ↘ (x i ,y i ,t n )=Q SD (I(x i ,y i ,t n )-I(x i +1,y i -1,t n ))

[0116] The second spatial difference is determined by the fusion module to determine the spatial difference value of the fused pixel

[0117] Or, the second time, the quantization module is used to pass I i-digital (x i ,y i ,t j )=Q A (I i (x i ,y i ,t j ))Get the quantized value of the incident light intensity of each pixel unit.

[0118] The second spatial difference module determines the spatial difference value of each pixel unit

[0119] SD x-i (x i ,y i ,t n )=(I i-digital (x i ,y i ,t n )-I i-digital (x i -1,y i ,t n ))

[0120] SD y-i (x i ,y i ,t n )=(Ii-digital (x i ,y i ,t n )-I i-digital (x i ,y i -1,t n ));

[0121] The second spatial difference is determined by the fusion module to determine the spatial difference value of the fused pixel

[0122] As a preferred embodiment, a pulse generating module is provided in each pixel unit in the pixel array;

[0123] Alternatively, all pixel units in the pixel array are connected to a pulse generating module;

[0124] Alternatively, the pixel array is divided into a plurality of sub-regions, and all pixel units in each sub-region are connected to a pulse generating module;

[0125] The pulse generation module is used to generate a trigger signal at a fixed time interval, or to generate a trigger signal at an adaptive, programmable variable time interval, to control the start exposure time and exposure duration of the photosensitive module, and the pixel units connected to the same pulse generation module are exposed synchronously, and the pixel units connected to different pulse generation modules are exposed synchronously or asynchronously;

[0126] The photosensitive module is arranged in the pixel unit and is used to convert the light signal at the current pixel unit position into an analog electrical signal.

[0127] In this embodiment, the visual sensor chip further includes a trigger pulse generating module, which can generate a trigger signal to realize the exposure control of the photosensitive module, that is, determine the time t of collecting the signal. n . If a trigger pulse generation module is designed in each pixel unit, full-array asynchronous exposure can be used. At this time, each trigger pulse generation module can independently and adaptively adjust the moment of triggering the calculation of the spatiotemporal differential signal according to the light intensity level felt by the pixel unit itself. The triggering moment of each pixel unit is different. The pixel unit can output information at any time, which improves flexibility and reduces output delay. Of course, in this case, it can also be set to full-array synchronous exposure as needed. If some pixel units share the same trigger pulse generation module, these pixel units are exposed synchronously.

[0128] The trigger pulse generation module can generate trigger signals at the same time interval or generate trigger signals at adaptive, programmable variable intervals.

[0129] If all pixel units in the array share a trigger pulse generating module, then all pixel units need to be exposed at the same time, and the output of the pixel units needs to follow a certain rule.

[0130] The quantization methods of the temporal and spatial quantization modules can be either multi-valued (>1 bit) or single-valued (positive and negative pulses). The acquisition of signals can be performed synchronously across the entire array with the same time interval, synchronously across the entire array with a variable time interval, or asynchronously across the entire array.

[0131] In the field of digital signal processing, quantization mainly refers to the process of converting analog signals into digital signals. Signal sampling and quantization are usually achieved by analog-to-digital converters (ADCs).

[0132] As a preferred embodiment, the exposure mode of each unit pixel in the pixel array is global exposure or rolling exposure.

[0133] Of course, the intensity path, the time difference path and the space difference path can be arbitrarily combined in a global exposure or rolling exposure manner, and this application does not impose any particular limitation thereto.

[0134] As a preferred embodiment, when the pixel unit is provided with a color filter, the output color type of the path corresponding to the pixel unit is a color value; when the pixel unit is not provided with a color filter, the output color type of the path corresponding to the pixel unit is a grayscale value.

[0135] If a pixel unit is covered with a color filter, the information obtained by the pixel is only that of a specific color channel. A typical color filter is a combination of red, green, and blue, known as the RGB type. Other color channels can also be used, such as a CMY array consisting of the three complementary colors (cyan, magenta, and yellow). Pixels of the same type may differ in color channel, such as a spatiotemporal differential pixel for the X color, a spatiotemporal differential pixel for the Y channel, and a spatiotemporal differential pixel for the Z channel. Spatial differentiation can be performed between pixels of the same color or between pixels of different colors (e.g., subtracting an X pixel from a Y pixel).

[0136] In addition, an external programmable demosaicer may be embedded in the pixel unit to obtain the output values ​​of all other color channels at the pixel position of the X color channel through a demosaicing algorithm, that is, by selecting points of surrounding pixels and performing interpolation.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A visual sensor chip based on pixel fusion technology, comprising a pixel array, an intensity path, a time difference path and a space difference path; The pixel array includes a plurality of pixel units; The fusion technology is used to fuse the signals of multiple pixel units within the fusion pixel range into one signal and then output it; the intensity path is used to determine the quantized value of the electrical signal converted from the incident light intensity of the fusion pixel; The time difference path is used to perform time difference, fusion and quantization operations on the signal of the current fused pixel position at the current moment and the signal of the current fused pixel position at the previous moment in the charge domain, analog domain or digital domain to obtain the time difference value of the fused pixel; The signal of the fused pixel is the signal of a plurality of pixel units within the range of the fused pixel; The spatial difference path is used to perform spatial difference, fusion and quantization operations on the signal of the current fused pixel position at the current moment and the signal of the spatially associated fused pixel position at the current moment in the charge domain, analog domain or digital domain to obtain the spatial difference value of the fused pixel; The spatially associated fused pixels are any one or more fused pixels in the array except the current fused pixel.

2. The visual sensor chip based on pixel fusion technology according to claim 1, wherein: The intensity path includes a first intensity fusion module and a first intensity quantization module; The first intensity fusion module is used to fuse the analog signals of the plurality of pixel units at the current fusion pixel position at the current moment to obtain a first intensity fusion signal; The first intensity quantization module is used to perform analog-to-digital conversion on the first intensity fusion signal to obtain a quantized value of an electrical signal converted from the intensity of the incident light of the fusion pixel.

3. The visual sensor chip based on pixel fusion technology according to claim 1, wherein: The intensity path includes a quantization module for the second intensity and a fusion module for the second intensity; The second intensity quantization module is used to perform analog-to-digital conversion on the analog signals of multiple pixel units at the current fusion pixel position at the current moment, to obtain the quantized value of the electrical signal converted from the incident light intensity of each pixel unit; The second intensity fusion module is used to fuse the quantized values ​​of the incident light intensity of the plurality of pixel units to obtain the quantized value of the electrical signal converted from the incident light intensity of the fused pixel.

4. The visual sensor chip based on pixel fusion technology according to claim 1, wherein the time difference path comprises a first time difference fusion module, a first time difference module and a first time quantization module; The first time difference fusion module is used to fuse the electrical signals of the plurality of pixel units to obtain the first time difference fusion signal of the current fusion pixel position at different times; The first time difference module is used to perform a time difference operation on the first time difference fusion signal of the current fusion pixel position at the current moment and the first time difference fusion signal of the current fusion pixel position at the previous moment to obtain the time difference value of the fusion pixel; In the fusion and time difference process, analog-to-digital signal conversion is completed by the first time quantization module.

5. The visual sensor chip based on pixel fusion technology according to claim 1, wherein: The time difference path includes a second time difference module, a second time quantization module and a second time difference fusion module; The second time difference module is used to determine the time difference value of each pixel unit; the time difference value of each pixel unit is obtained by performing a time difference operation between the electrical signal at the current pixel unit position at the current moment and the electrical signal at the current pixel unit position at the previous moment; The second time difference fusion module is used to fuse the time difference values ​​of the plurality of pixel units to obtain the time difference value of the fused pixel; During the fusion and time difference process, analog-to-digital signal conversion is completed by the second time quantization module.

6. The visual sensor chip based on pixel fusion technology according to claim 1, wherein: The spatial difference path includes a first spatial difference fusion module, a first spatial difference module and a first spatial quantization module; The first spatial difference fusion module is used to fuse the electrical signals of the plurality of pixel units to obtain a first spatial difference fusion signal at the current fusion pixel position at the current moment; The first spatial difference module is used to perform spatial difference operation on the first spatial difference fusion signal of the current fusion pixel position at the current moment and the first spatial difference fusion signal of the spatially associated fusion pixel position at the current moment to obtain the spatial difference value of the fusion pixel; During the fusion and spatial difference process, analog-to-digital signal conversion is completed through the first spatial quantization module.

7. The visual sensor chip based on pixel fusion technology according to claim 1, wherein: The spatial difference path includes a second spatial difference module, a second spatial quantization module and a second spatial difference fusion module; The second spatial difference module is used to determine the spatial difference value of each pixel unit; the spatial difference value of each pixel unit is obtained by performing a spatial difference operation on the electrical signal of the current pixel unit position at the current moment and the electrical signal of the spatially associated pixel position at the current moment; The second spatial difference fusion module is used to fuse the spatial difference values ​​of the plurality of pixel units to obtain the spatial difference value of the fused pixel; During the fusion and spatial difference process, analog-to-digital signal conversion is completed through the second space quantization module.

8. The visual sensor chip based on pixel fusion technology according to claim 1, wherein: A pulse generating module is provided in each pixel unit in the pixel array; Or, all pixel units in the pixel array are connected to a pulse generating module; Or, the pixel array is divided into a plurality of sub-areas, and all pixel units in each sub-area are connected to a pulse generating module; The pulse generating module is used to generate a trigger signal at a fixed time interval, or to generate a trigger signal at an adaptive, programmable and variable time interval, so as to control the exposure start time and exposure duration of the photosensitive module, and to synchronously expose the pixel units connected to the same pulse generating module, and synchronously expose or asynchronously expose the pixel units connected to different pulse generating modules; The photosensitive module is disposed in the pixel unit and is used to convert the light signal at the current pixel unit position into an analog electrical signal.

9. The visual sensor chip based on pixel fusion technology according to claim 1, wherein: The exposure mode of each unit pixel in the pixel array is global exposure or rolling exposure.

10. The visual sensor chip based on pixel fusion technology according to any one of claims 1 to 9, wherein: When the pixel unit is provided with a color filter, the output color type of the path corresponding to the pixel unit is a color value; when the pixel unit is not provided with a color filter, the output color type of the path corresponding to the pixel unit is a gray value.

Images