High dynamic range edge imaging method and device

Abstract

The invention discloses a high-dynamic-range edge imaging method and device. The method comprises the steps that 101, a CMOS image sensor is excited by light to generate an original electric signal; step 102, performing exposure real-time evaluation on different pixel areas by an exposure evaluator based on the original electric signals; step 103, a reference voltage generator generates corresponding reference voltages for different pixel areas based on the exposure real-time evaluation value; step 104, under the regulation and control of the reference voltage, a probability circuit converts the pre-amplified pixel unit electric signals into pulse signals line by line / column by line; and step 105, sending the pulse image data into an edge calculation module for identification, and outputting an image. According to the invention, the delay caused by data transmission is solved, and edge imaging in a high dynamic range can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of artificial intelligence, edge imaging, image coding technology, and the like, and in particular to a high dynamic range edge imaging method and device. BACKGROUND

[0002] Traditional digital imaging devices mostly use 8-bit depth per channel data, so the dynamic range of such imaging equipment is limited to 103orders of magnitude, which is far less than the dynamic range of about 107orders of magnitude of the human eye. When imaging a real environment, bright areas may be overexposed, and dim areas may be underexposed, making it difficult to capture all the details of the scene in one exposure. Therefore, it is extremely important to develop a high dynamic range imaging method and device for obtaining real environment images. Existing HDR (High dynamic range imaging) techniques include tone mapping, inverse tone mapping and other processes, and the processing process is complex, which is difficult to meet the needs of edge computing and edge imaging.

[0003] In edge imaging computing, if the image is first obtained and then sent to the backend computing platform for pulsing, the computing power of the computing platform will be consumed, and the recognition speed of the spiking neural network will be reduced. In addition, taking general network transmission as an example, the image is sent to the computing platform through the network, which puts strict requirements on network bandwidth, connection reliability, etc. In order to reduce the delay caused by image processing, a new method of completing high dynamic range image acquisition and pulsing at the imaging end is needed, which is used to directly generate pulsed image information for the spiking neural network, and to complete image recognition at the detector end.

[0004] Edge computing and edge imaging require intelligent computing to be completed at the device end of image acquisition, so there are higher requirements for the computing power, power consumption and volume of the imaging equipment. In edge computing, spiking neural networks (SNN, Spiking Neuron Networks) are proposed and have important applications. As the third generation of neural networks designed to simulate the mechanism of biological neurons, spiking neural networks have more brain-like characteristics in connection mode, information processing mechanism and synaptic weight learning method. The simulated neurons in the spiking neural network are closer to the actual characteristics: neurons will only be activated when the membrane potential reaches the threshold, and the spiking neural network considers the pulse generation time, enhancing the ability to process spatio-temporal data.

[0005] Pulse neural network processes pulse time series, and there are generally two ways to obtain pulse sequences: one is to convert images into pulse sequences according to certain encoding methods (for example, threshold encoding method, Gaussian difference method, frequency encoding method, etc.), but the above methods have problems such as serious information loss and low encoding efficiency; the other is directly obtained from neuromorphic visual sensors. However, due to the immaturity of the principle and process of neuromorphic visual sensors, the high price and other reasons, the application of neuromorphic visual sensors is not universal. SUMMARY

[0006] In view of the problems in the prior art, the purpose of the present application is to invent a high dynamic range edge imaging method and device based on the electrical characteristics of magnetic tunnel junction, to realize direct acquisition of high dynamic range images on image sensors and corresponding pulsing process.

[0007] To achieve the above-mentioned purpose, the present application provides a high dynamic range edge imaging method, characterized in that the method comprises the following steps: Step 101: the CMOS image sensor generates original electrical signals under light excitation; Step 102: the exposure evaluator performs real-time exposure evaluation on different pixel regions based on the original electrical signals, and obtains real-time exposure evaluation values of each pixel region; Step 103: the reference voltage generator generates corresponding reference voltages for different pixel regions based on the real-time exposure evaluation values; Step 104: under the regulation of the reference voltage, the probability circuit converts the pre-amplified pixel cell electrical signals into pulse signals row by row / column by column through the magnetic tunnel junction; Step 105: the pulsed image data is sent into the edge computing module for identification, the pulsed image data is arranged according to the position coordinates, and is transmitted in row / column order to the intelligent computing network deployed on the upper part of the edge computing module for image recognition, and the image is output.

[0008] Further, step 101 comprises: Step 101-1. During image acquisition, visible light first passes through a color filter for color filtering, and then the filtered visible light enters the pixel sensor; Step 101-2. The pixel sensor generates original electrical signals under light excitation; each pixel generates original electrical signals according to the light intensity of the excitation light at its coordinate position after being excited by light, and the intensity of the original electrical signals is proportional to the light intensity of the received excitation light.

[0009] Further, the total number of horizontal pixels of the pixel sensor is N, and the total number of vertical pixels is M; each image includes N*M pixels, and the coordinate position of each pixel in the pixel sensor is (i, j).

[0010] Furthermore, the CMOS image sensor is compatible with color filter array formats including Bayer, RGBE, RGBW, CYYM, and CYGM, and the color-filtered visible light then enters the pixel sensor.

[0011] Furthermore, step 102 includes: Step 102-1. Based on the original electrical signal I (i,j) The reference electrical signal is calculated; Step 102-2. Compare the original electrical signal of each pixel region with the reference electrical signal, and obtain the real-time exposure evaluation value of the pixel region according to the comparison result.

[0012] Further, in step 102-1, the raw electrical signals generated by the CMOS image sensor are statistically analyzed. The raw electrical signals are I... (i,j) Where (i,j) is the pixel position in the image sensor; the reference electrical signal is , where N and M are the total number of horizontal and vertical pixels in the image sensor, respectively.

[0013] Furthermore, real-time exposure assessment is completed via a dedicated analog circuit.

[0014] Furthermore, step 104 includes: Step 104-1. Amplify the amplitude of the original electrical signal generated by the CMOS pixel unit to obtain the amplified pixel unit electrical signal; Step 104-2. Driven by the timing signal of the timing circuit, the row / column circuit sends the amplified pixel unit electrical signal to the probability circuit; under the control of the reference voltage, the amplified pixel unit electrical signal is sampled and converted into a pulse signal; the corresponding pixel coordinates are added before the converted pulse signal, with each row / column coordinate occupying 12 bits, forming a 64-bit data block that is sent to the buffer.

[0015] On the other hand, the present invention provides a high dynamic range edge imaging apparatus, the apparatus being used to implement the high dynamic range edge imaging method described in the present invention, the apparatus comprising: Color filters are used to filter ambient light into monochromatic light; A pixel sensor is used to acquire image signals; the pixel array generates raw electrical signals when excited by light. An exposure estimator is used to evaluate the exposure intensity distribution on a pixel sensor in real time. An amplifier is used to amplify the raw electrical signals directly generated by the pixel sensor. A reference voltage generator generates the reference voltage required by the probability circuit based on the results of the exposure evaluator. A gate is used to allow the electrical signal generated by a certain row / column pixel to access the probability circuit; A timing circuit is used to generate the timing signal required by the imaging device; A position encoder is used to encode the row / column information of the pixel array into a trigger signal; A probability circuit is used to convert the electrical signal generated by the pixel unit into a pulse signal; A buffer is used to convert the pulse signal into a digital signal and add the pixel coordinates; A calculation module uses the deployed pulse neural network to identify the pulsed image data.

[0016] Further, the probability circuit comprises a matching resistor, an access transistor, a magnetic tunnel junction, a positive voltage supply end, a negative voltage supply end, a comparator and a reference voltage end; the magnetic tunnel junction is a non-stable device composed of two nanometer magnets, which is used to compare with the input voltage and generate a probabilistic pulse signal; the matching resistor works together with the magnetic tunnel junction to perform voltage division, and the non-uniformity introduced by the magnetic tunnel junction in the manufacturing process is eliminated by adjusting the size of the matching resistor value.

[0017] The high dynamic range edge imaging method provided by the application solves the difficulty of obtaining pulsed image data by the pulse neural network in an embedded probabilistic coding manner; at the same time, the reference voltage is obtained by dynamically using the probabilistic sampling and statistical method, which improves the dynamic range of the image; the three edge combination schemes of sensing, coding and calculation are adopted, which solves the delay caused by data transmission and is suitable for scenes such as automatic driving which need real-time identification.

[0018] The high dynamic range edge imaging method and device provided by the application use real-time exposure evaluation to obtain the reference voltage, which improves the dynamic range of the obtained image; the probability circuit is used to realize the embedded probabilistic coding function; the four of sensing, exposure evaluation, coding and calculation are combined at the image acquisition end to perform synchronous processing of the whole row / column pixel information in a parallel manner, which can realize the rapid identification of the image in real-time environment; the probability circuit based on the magnetic tunnel junction can still realize self-oscillation without external triggering, thereby completely solving the problem of data misplacement caused by mismatched trigger signals. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments of the application. Obviously, the drawings described below are only some embodiments of the application, and do not constitute an improper limitation on the application.

[0020] Figure 1 is the flowchart of the high dynamic range edge imaging method provided by the application; Figure 2 is a schematic diagram of a probability circuit structure provided by the present application; Figure 3 is a schematic diagram of a probability circuit output characteristic provided by the present application; Figure 4 is a schematic diagram of a high dynamic range edge imaging device structure and working process provided by the present application; Figure 5 is a schematic diagram of an imaging sensor structure provided by the present application; Figure 6 is a schematic diagram of an integrated high dynamic range edge imaging device structure provided by the present application; Figure 7 is an imaging process and result example of an embodiment of the present application using a high dynamic range edge imaging method; Figure 8 shows a schematic diagram of a curve of a change in electrical signal intensity generated by a CMOS pixel unit in a process in which light intensity gradually increases. DETAILED DESCRIPTION

[0021] The technical solutions of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0022] In the description of the present application, it should be noted that the orientations or positional relationships indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0023] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integrally connected; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0024] The following will be described in detail with reference to the drawings. Figures 1-8The specific embodiments of the present application are described in detail. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0025] As shown in Figure 4 The high dynamic range edge intelligent imaging device according to the present application can effectively image in a larger exposure dynamic range, including: In the field of imaging technology, the range between the maximum brightness value and the minimum brightness value of an image is called the dynamic range of imaging. The high dynamic range of the present application refers to a technology of a larger exposure dynamic range (i.e. imaging can accommodate a larger difference between light and dark). Generally, images with a dynamic range greater than 4 orders of magnitude (80 dB) are called high dynamic range (HDR) images internationally.

[0026] The color filter 401 is used to filter the ambient light into monochromatic light. The pixel sensor 402 is used to acquire an image signal, and the pixel array generates an electrical signal under light excitation. The exposure evaluator 403 is used to evaluate the exposure of different pixel regions in real time. The amplifier 404 is used to amplify the electrical signal directly generated by the pixel sensor. The reference voltage generator 405 is used to generate a corresponding reference voltage for different pixel regions in combination with the exposure evaluator 403. The gate 406 is used to allow the electrical signal generated by a certain row / column pixel to access the probability circuit. The timing circuit 407 is used to generate timing signals required by the imaging device. The position encoder 408, in combination with the gate 406, marks the pixel coordinates on the signals generated by the probability circuits 409-412. The probability circuits 409-412 use magnetic tunnel junction technology to convert the electrical signal generated by the pixel unit into a pulse signal. The buffer 413 is used to convert the pulse signal into a digital signal and add the pixel coordinates. The computing module 414 uses the deployed pulse neural network to identify the pulsed image data.

[0027] Based on the above high dynamic range edge intelligent imaging device, the present application provides a high dynamic range edge imaging method, as shown in Figure 1 The method mainly includes the following steps 101 to 105, as follows: Step 101: The CMOS image sensor generates an electrical signal under light excitation. The CMOS image sensor is the pixel sensor 402.

[0028] Here, the light refers to the light filtered by the color filter 401. Figure 4The light filtered by the middle color filter 401 (and the monochromatic light produced after the visible light passes through the color filter 401, such as the monochromatic light of red, green, and blue).

[0029] Step 101 includes: Step 101-1. When the image is collected, the visible light is first filtered by the color filter 401, and the filtered monochromatic light then enters the pixel sensor 402. Step 101-2. The pixel sensor 402 generates an original electrical signal under light excitation. The total number of horizontal pixels of the pixel sensor 402 is N, and the total number of vertical pixels is M. Each image includes N*M pixels, and each pixel has a coordinate position (i, j) in the pixel sensor 402. Each pixel generates an original electrical signal according to the light intensity of the excitation light at the coordinate position, and the intensity of the original electrical signal is proportional to the light intensity of the received excitation light.

[0030] It should be understood that, in the process of collecting the image, when the excitation light intensity is weak, the intensity of the electrical signal generated by the CMOS image sensor (i.e., the pixel sensor 402) is proportional to the excitation light intensity; but when the excitation light intensity is too strong, the electrical signal saturation is difficult to distinguish the difference in excitation light intensity, and thus there is a problem of insufficient dynamic range.

[0031] It should be understood that the CMOS image sensor is composed of N*M arrayed CMOS pixel units. Visible light can be considered to be composed of monochromatic light of red, green, and blue. Since the CMOS pixel unit itself does not have color resolution capability, it is necessary to first filter out monochromatic light of red, green, and blue from the visible light through a color filter, and then project the monochromatic light onto the CMOS pixel unit, so that the CMOS pixel unit generates an electrical signal related to the intensity of the incident monochromatic light.

[0032] Preferably, the CMOS image sensor is compatible with common color filtering arrangement devices of Bayer / RGBE / RGBW / CYYM / CYGM formats. The visible light filtered by the color filter then enters the pixel sensor. The CMOS image sensor generates an electrical signal under light excitation, including: setting a color filter in front of the CMOS pixel array, filtering out monochromatic light, and exciting the CMOS image sensor with the monochromatic light to obtain an image signal containing color characteristics.

[0033] Step 102: The exposure evaluator 403 performs real-time exposure evaluation on different pixel regions to obtain real-time exposure evaluation values of the pixel regions.

[0034] Specifically, step 102 includes: Step 102-1. Based on the original electrical signal I(i,j), where (i,j) is the pixel position in the image sensor, calculate the reference electrical signal. , The calculation method is as follows: ; Step 102-2. Compare the original electrical signal of each pixel region with the reference electrical signal, and obtain the real-time exposure evaluation value of that pixel region according to the comparison result using the corresponding method; specifically, if the original electrical signal... Greater than the reference electrical signal This indicates strong exposure, so it is done through Methods to obtain real-time exposure evaluation values If the original electrical signal Less than the reference electrical signal This indicates weak exposure, so... Methods to obtain real-time exposure evaluation values If the original electrical signal Equal to reference electrical signal Then the real-time exposure assessment value of that pixel area is equal to the reference electrical signal. .

[0035] In summary, the real-time exposure evaluation process involves performing the following normalization processing on the original electrical signal I(i,j) to obtain the real-time exposure evaluation value. : ; in N and M are the total number of horizontal and vertical pixels in the image sensor, respectively.

[0036] Figure 8 The graph shows the change in electrical signal intensity generated by a CMOS pixel unit as light intensity gradually increases. The dashed line represents a typical imaging scheme, and the solid line represents the technical solution of this invention. Typical imaging schemes have a limited effective light intensity response range. When the light intensity is low, the CMOS pixel unit cannot generate sufficient electrical signals, resulting in underexposure; when the light intensity is high, the electrical signals generated by the CMOS pixel unit saturate, resulting in overexposure. In this invention, the effective light intensity response range is significantly widened, allowing the CMOS pixel unit to generate effective electrical signals across a wider range of light intensities. Preferably, real-time exposure assessment can be performed using a dedicated exposure estimator.

[0037] It should be understood that real-time exposure assessment is a recalculation of the electrical signal intensity generated by the CMOS image sensor in order to reduce the large differences between electrical signal intensities caused by excessive or insufficient exposure.

[0038] Step 103: The reference voltage generator 405 generates corresponding reference voltages for different pixel regions based on the exposure real-time evaluation value.

[0039] Wherein, the CMOS pixel unit generates current signal under the excitation of light intensity. Since the current signal has significant saturation effect and is not easy to alleviate, it leads to technical difficulties. Here, the reference voltage is calculated to input bias voltage for the probability circuit.

[0040] Preferably, the exposure real-time evaluation value is used to generate corresponding reference voltages for different pixel regions according to the following formula: The formula for calculating the reference voltage is as follows: ; Wherein, sigmoid is S-shaped function, a is proportional parameter, and a value is generally set to Figure 2 The off voltage of the access transistor 202.

[0041] Preferably, the range of the reference voltage is determined by the proportional parameter a, and meets the process standard requirements selected during device manufacturing.

[0042] Step 104: Under the regulation of the reference voltage, the probability circuit (409-412) converts the pre-amplified pixel unit electrical signal into pulse signal row by row / column by column.

[0043] Specifically, it includes: Step 104-1. The amplitude of the original electrical signal generated by the CMOS pixel unit is amplified to obtain the amplified pixel unit electrical signal. It should be understood that the original electrical signal generated by the CMOS pixel unit is generally small, which does not meet the requirements of the probability circuit for voltage, and the amplitude of the signal needs to be amplified.

[0044] It should be understood that the regulation of the reference voltage is reflected in that the greater the negative deviation of the input signal from the reference voltage, the smaller the probability of the probability circuit outputting high-level pulse; the greater the positive deviation of the input signal from the reference voltage, the greater the probability of the probability circuit outputting high-level pulse. The influence of the reference voltage on the output of the probability circuit is shown in Figure 3 .

[0045] Preferably, the electrical signal generated by the pixel unit is amplified by the result amplifier 404; Step 104-2. Under the driving of the timing signal of the timing circuit 407, the gate 406 installs the pixel position one by one to send the amplified pixel unit electric signal into the probability circuit 409-412 (the probability circuit 409-412 shown in the figure only represents the probability circuit array, not that the probability circuit has only 4; actually, the number of probability circuits is consistent with the CMOS pixel array); under the regulation of the reference voltage, the amplified pixel unit electric signal is sampled, and the amplified pixel unit electric signal is converted into a pulse signal; the corresponding pixel coordinates are added in front of the converted pulse signal to form a data block and send it into the buffer. Specifically, under the regulation of the reference voltage, the amplified pixel unit electric signal is sampled, and within 40 clock cycles, digital 0 represents low level and digital 1 represents high level, the amplified pixel unit electric signal is converted into a pulse signal; the corresponding pixel coordinates are added in front of the converted pulse signal, and the row / column coordinates each occupy 12 bits to form a 64-bit data block and send it into the buffer 413.

[0046] It should be understood that the row / column coordinates of the pixel unit each occupy 12 bits, which can be compatible with most current CMOS sensors.

[0047] Step 105: sending the pulsed image data into the edge computing module for recognition, including: arranging the pulsed image data according to the position coordinates, and transmitting them in row / column order to the intelligent computing network deployed on the upper part of the edge computing module for image recognition.

[0048] The imaging method provided above uses real-time exposure evaluation to obtain the reference voltage, improves the dynamic range of the obtained image, and adopts the edge combination scheme of sensing, exposure evaluation, pulsing, and computing to solve the delay caused by data transmission, which is suitable for scenes such as automatic driving that require real-time recognition.

[0049] In addition, as shown in Figure 2 , the probability circuit in step 104 is implemented as follows: the probability circuit includes a matching resistor 201, an access transistor 202, a magnetic tunnel junction 203, a positive voltage supply end 204, a negative voltage supply end 205, a comparator 206, and a reference voltage end 207. Among them, the probability circuit outputs a pulse signal, the probability circuit is an array, and the number of probability circuits is consistent with the CMOS pixel array, so the inputs of each probability circuit are arranged according to the position coordinates to form pulsed image data.

[0050] The matching resistor 201 works together with the magnetic tunnel junction 203 to divide the voltage, and the matching resistor value is adjusted to eliminate the non-uniformity introduced by the magnetic tunnel junction 203 in the manufacturing process.

[0051] The access transistor 202 acts as a switch in the circuit.

[0052] Exemplarily, when the input voltage is less than the switching threshold, the access transistor 202 is not turned on; when the input voltage is greater than the switching threshold, the access transistor 202 is turned on.

[0053] Preferably, the access transistor can be selected as an NMOS transistor.

[0054] The magnetic tunnel junction 203 is a non-stable device composed of two nanometer magnets, which can be compared with the input voltage and generate a probabilistic pulse signal. Figure 3 .

[0055] The positive voltage supply end 204 provides positive voltage supply for the probability circuit.

[0056] The negative voltage supply end 205 provides negative voltage supply for the probability circuit.

[0057] The comparator 206 is used for comparing the input voltage and the reference voltage, and outputs the comparison result.

[0058] Exemplarily, when the input voltage is higher than the reference voltage, the comparator 206 outputs high level; when the input voltage is lower than the reference voltage, the comparator 206 outputs low level.

[0059] The reference voltage end 207 is used for accessing the reference voltage provided by an external circuit.

[0060] Figure 2 The overall function of the probability circuit is shown as follows: when there is greater negative deviation of the input voltage relative to the reference voltage, the probability circuit outputs low level signal with greater probability; when there is greater positive deviation of the input voltage relative to the reference voltage, the probability circuit outputs high level signal with greater probability. The present application can complete the pulsing of the image in real time, and the pulsed image completes the conversion of the analog signal to the binary digital signal, and is particularly suitable for recognition by using the edge recognition scheme such as the pulse neural network.

[0061] The present application provides a probability circuit design based on a magnetic tunnel junction. Without external triggering, the magnetic tunnel junction 203 can still realize self-oscillation, thereby completely solving the problem of data misplacement caused by mismatching of the trigger signal.

[0062] Magnetic Tunnel Junction (MTJ) is a sandwich structure device, from top to bottom including reference layer, tunnel barrier layer and free layer. Among them, the free layer and the fixed layer are two layers of magnetic thin film, forming a nanometer magnet; the insulating layer is an oxide film, which separates the electron migration in the free layer and the fixed layer as a tunnel barrier. By applying an external electromagnetic field control, the magnetic moment direction of the free layer can be switched to parallel or anti-parallel direction with the fixed layer. If it is parallel state, the resistance of the magnetic tunnel junction is low, and the resistance is high in anti-parallel state, so as to use the resistance difference as the logic state. When the tunnel barrier in the magnetic tunnel junction is low, the parallel state and the anti-parallel state appear probability jump phenomenon due to thermal noise, generating probability output characteristics.

[0063] Further, in combination with the probability circuit structure shown in Figure 2 , the present application provides the output characteristics of the probability circuit under different bias voltages, as shown in Figure 3 .

[0064] It should be understood that when the input voltage of the probability circuit is small, the average value of the output probability pulse is 0.22; as the input voltage increases, the average value of the output probability pulse rises from 0.22 to 0.49; finally, under a larger voltage, the average value of the output probability pulse signal rises to 0.75.

[0065] It should be understood that Figure 3 The values shown are the average values of the probability pulse in 100 μs sampling time, Figure 3 The ordinate is the normalized value.

[0066] Further, according to the method shown in Figure 1 , in combination with the probability circuit (see Figure 2 ) provided by the present application, a high dynamic range edge imaging device of the present application, Figure 4 shows the specific structure of the edge imaging device, including color filter 401, pixel sensor 402, exposure evaluator 403, amplifier 404, reference voltage generator 405, gate 406, timing circuit 407, position encoder 408, probability circuit 409-412, buffer 413, and calculation module 414.

[0067] Optionally, the color filter 401 is specifically used to filter the ambient light into monochromatic light.

[0068] It should be understood that the light-sensitive element of the CMOS type can only respond to light intensity and cannot respond to color, so the color filter 401 component is set to make the electrical signal output by the pixel unit positively correlated with the intensity of monochromatic light.

[0069] Preferably, the color filter 401 can be selected from common Bayer / RGBE / RGBW / CYYM / CYGM color filter arrangement.

[0070] Optionally, the pixel sensor 402 is specifically configured to acquire image signals, and the pixel array generates electrical signals under light excitation. After passing through the color filter 401, the ambient light is decomposed into monochromatic light and enters the pixel unit, and the corresponding electrical signals are generated after the photoelectric effect occurs.

[0071] Preferably, the pixel sensor 402 adopts a CMOS type photosensitive element, has the ability to distinguish each pixel unit, and realizes one-to-one correspondence between the electrical signals generated by each pixel unit and the physical position thereof.

[0072] Preferably, the exposure evaluator 403 is specifically configured to perform real-time exposure evaluation on different pixel regions, and can perform the following exposure evaluation scheme: the electrical signals generated by the pixel sensor 402 are counted, and the electrical signal is denoted as I (i,j) , wherein (i,j) is the pixel position in the image sensor, and the real-time exposure evaluation process is normalized as follows: ; , wherein N and M are the total number of horizontal and vertical pixels in the image sensor, respectively.

[0073] Optionally, the amplifier 404 is specifically configured to amplify the amplitude of the electrical signals generated by the entire row / column of pixels to meet the requirements of the probability circuits 409-412 on the input voltage.

[0074] Preferably, the imaging device provided by the present application adopts a parallel design, and the number of channels contained in the amplifier 404 is consistent with the row / column value of the pixel array in the pixel sensor 402.

[0075] Optionally, the reference voltage generator 405 is specifically configured to generate corresponding reference voltages for different pixel regions in combination with the exposure evaluator 403. Preferably, the real-time exposure evaluation value generates corresponding reference voltages for different pixel regions according to the following formula: ; , wherein sigmoid is a sigmoid function, and a is a proportional parameter.

[0076] Preferably, the range of the reference voltage is determined by the proportional parameter a, and meets the process standard requirements selected during device manufacturing.

[0077] Optionally, the gate 406 is specifically configured to allow the electrical signals generated by a certain row / column of pixels to access the probability circuits 409-412.

[0078] Optionally, the timing circuit 407 is specifically configured to provide the trigger timing required by the overall imaging device. The components in the overall imaging device that require trigger timing include the pixel sensor 402, the strober 406, the position encoder 408 and the buffer 413.

[0079] Exemplarily, assuming that the pixel sensor contains an N x M pixel array (N and M are both positive integers), and assuming that the trigger frequency required by the pixel sensor 402 is f, then the trigger frequency required by the position encoder 408 is not less than M (times of strobing the entire column of pixels) or (times of strobing the entire row of pixels), and the trigger frequency required by the buffer 413 is not less than .

[0080] Optionally, the position encoder 408 is configured to encode the row / column information of the pixel array into trigger signals; specifically, the position encoder 408 receives the trigger signals provided by the timing circuit, provides the row / column control signals for the strober 406 to the input probability circuit, and provides the corresponding pixel position information to the buffer 413.

[0081] Optionally, the probability circuits 409-412 constitute an array of a certain scale, the array scale is consistent with the pixel sensor 402, and the probability circuits 409-412 are specifically configured to convert the amplified pixel unit signals into pulse signals row by row / column under the regulation of the reference voltage. For each row / column of pixels, the conversion process of the probability circuit lasts for 40 clock cycles.

[0082] It should be understood that the regulation of the reference voltage is reflected in that the greater the negative deviation of the input signal from the reference voltage, the smaller the probability of the probability circuit outputting a high-level pulse; and the greater the positive deviation of the input signal from the reference voltage, the greater the probability of the probability circuit outputting a high-level pulse.

[0083] Preferably, the probability circuits 409-412 should be adjusted to achieve that when the input signal is equal to the reference voltage, the probability of the probability circuit outputting a high-level pulse is 50%.

[0084] Optionally, the buffer 413 is specifically configured to convert the pulse signals into digital signals and add the pixel coordinates.

[0085] Exemplarily, the pulse signals are sampled, and in 40 clock cycles, a digital 0 represents a low level and a digital 1 represents a high level, so as to convert the pulse signals into digital signals; the pixel coordinates generated by the signals are added in front of the converted digital signals, and the row / column coordinates each occupy 12 bits, forming a 64-bit data block.

[0086] Preferably, the row / column coordinates of the pixel unit each occupy 12 bits, which can be compatible with most CMOS sensors at present.

[0087] Optionally, the computing module 414 is specifically configured to input the pulsed image data into the spiking neural network. The pulsed image data is arranged according to the position coordinates, and is transmitted in parallel to the spiking neural network deployed on the computing module in a row / column order for image recognition.

[0088] Preferably, the computing module 414 can be an FPGA, and the trained spiking neural network is deployed on the FPGA.

[0089] Figure 5 An exemplary pixel sensor structure provided by the present application is shown, in which each pixel sensor has a respective row / column coordinate, for embodying a signal generation area in an imaging device.

[0090] Figure 6 An integrated high dynamic range edge imaging device structure provided by the present application is shown, and the main structural components are as follows: Figure 4 The imaging devices shown correspond to each other. Figure 6 The imaging device structure shown provides a structural component layout scheme and data / signal transmission path of the imaging device of the present application. The row selection circuit and the column selection circuit are functionally multiplexed, and can simultaneously perform the functions of Figure 4 The functions of the gate 406 and the position encoder 408 shown in the middle.

[0091] Figure 7 An imaging process and result example using the high dynamic range edge imaging method provided by the present application are shown. Figure 7 The subgraphs (1), (2), (3), and (4) in the middle show images under different exposure intensities, arranged from high to low exposure. It should be noted that under high exposure, the lane lines are indistinguishable; under low exposure, the vehicle details are indistinguishable. Figure 7 The subgraph (5*) in the middle shows the imaging result obtained using the high dynamic range edge imaging method of the present application, and the lane lines, vehicle features, and other details can be seen.

[0092] The high dynamic range edge imaging device provided by the present application breaks through the function of using a random number generator for probability encoding in the prior art, and uses a probability circuit to realize an embedded and efficient probability encoding function; the use of real-time exposure evaluation improves the dynamic range of the acquired image; the combination of sensing, exposure evaluation, pulsing, and computing at the image acquisition end in a parallel manner for synchronous processing of the entire row / column pixel information can realize real-time and rapid identification of the image.

[0093] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the displayed components may or may not be physical units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0094] Any process or method described in the flowcharts of this invention or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, which can be implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device. The computer-readable medium can be any medium containing a program for storage, communication, propagation, or transmission for use by the execution system, apparatus, or device, including read-only memory, magnetic disks, or optical disks.

[0095] In the description of this specification, references to terms such as "embodiment," "example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, those skilled in the art can combine or combine the different embodiments or examples described in this specification and the features therein without causing contradiction.

[0096] While embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and alterations to the above embodiments within the scope of the present invention.

Claims

1. A high dynamic range edge imaging method, characterized by, The method comprises the following steps: Step 101: the CMOS image sensor generates a raw electrical signal under light excitation; Step 102: the exposure evaluator performs real-time exposure evaluation on different pixel regions based on the raw electrical signal, and obtains real-time exposure evaluation values of the pixel regions; Step 103: the reference voltage generator generates corresponding reference voltages for different pixel regions based on the real-time exposure evaluation values; Step 104: under the regulation of the reference voltages, the probability circuit converts the pre-amplified pixel cell electrical signal into a pulse signal row by row or column by column; Step 105: the pulsed image data is sent to the edge computing module for recognition, the pulsed image data is arranged according to the position coordinates, and the pulsed image data is transmitted to the intelligent computing network deployed on the upper part of the edge computing module in a row / column sequence for image recognition, and the image is outputted.

2. The method of claim 1, wherein, Step 101 comprises: Step 101-1. During image acquisition, visible light first passes through a color filter for color filtering, and then the filtered visible light enters the pixel sensor; Step 101-2. The pixel sensor generates a raw electrical signal under light excitation; each pixel generates a raw electrical signal according to the light intensity of the excitation light at its coordinate position after being excited by light, and the intensity of the raw electrical signal is proportional to the light intensity of the received excitation light.

3. The method of claim 2, wherein, The CMOS image sensor generates an electrical signal under light excitation, comprising: a color filter is arranged in front of the CMOS pixel array, the color filter filters out monochromatic light, and the monochromatic light excites the CMOS image sensor to obtain an image signal containing color characteristics.

4. The method of claim 2, wherein the high dynamic range edge imaging method is characterized by, The total number of horizontal pixels of the pixel sensor is N, and the total number of vertical pixels is M; each image includes N*M pixels, and the coordinate position of each pixel in the pixel sensor is (i, j).

5. The method of claim 1, wherein, Step 102 comprises: Step 102-1. Calculate the reference electrical signal Iref based on the original electrical signal I (i,j) The reference electrical signal Iref is calculated. Step 102-2. Compare the raw electrical signal of each pixel region with the reference electrical signal, and obtain the real-time exposure evaluation value of the pixel region according to the comparison result.

6. The method of claim 5, wherein, In step 102-1, statistics are performed on the raw electrical signals generated by the CMOS image sensor, which are I (i,j) where (i,j) is the pixel position in the image sensor; and the reference electrical signal is where N and M are the total number of horizontal and vertical pixels in the image sensor, respectively.

7. The method of claim 5, wherein the high dynamic range edge imaging method further comprises: The real-time exposure evaluation is completed through a special analog circuit.

8. The method of claim 1, wherein, Step 104 comprises: Step 104-1. The amplitude of the raw electrical signal generated by the CMOS pixel cell is amplified to obtain an amplified pixel cell electrical signal; Step 104-2. Under the driving of the timing signal of the timing circuit, the row / column circuit sends the amplified pixel cell electrical signal to the probability circuit; under the regulation of the reference voltage, the amplified pixel cell electrical signal is sampled, and the amplified pixel cell electrical signal is converted into a pulse signal; the corresponding pixel coordinates are added in front of the converted pulse signal to form a data block and send it to the buffer.

9. A high dynamic range edge imaging device, characterized by The device is used to implement the high dynamic range edge imaging method of any one of claims 1-8, and the device comprises: A color filter for filtering ambient light into monochromatic light; A pixel sensor for acquiring an image signal, a pixel array generates a raw electrical signal under light excitation; An exposure evaluator for real-time evaluation of exposure intensity distribution on the pixel sensor; An amplifier for amplifying the raw electrical signal directly generated by the pixel sensor; A reference voltage generator for generating a reference voltage required by the probability circuit according to the exposure evaluator result; A gate is used to allow the electrical signal generated by a certain row / column pixel to access the probability circuit; A timing circuit is used to generate the timing signal required by the imaging device; A position encoder is used to encode the row / column information of the pixel array into a trigger signal; A probability circuit is used to convert the electrical signal generated by the pixel unit into a pulse signal; A buffer is used to convert the pulse signal into a digital signal and add the pixel coordinates; A computing module uses the deployed pulse neural network to identify the pulsed image data.

10. The high dynamic range edge imaging device of claim 9, wherein, The probability circuit includes a matching resistor, an access transistor, a magnetic tunnel junction, a positive voltage supply end, a negative voltage supply end, a comparator, and a reference voltage end.

Images