Panoramic blink event imaging system and method

Abstract

The invention relates to the technical field of optical imaging, and provides a panoramic blink event imaging system, which comprises a dynamic visual sensor used for independently sensing the change of received light intensity by using a basic detection unit and triggering an event when the sensed light intensity change exceeds a preset threshold value; the controllable blinking module comprises a photomask, a shielding rate control unit used for regulating and controlling the shielding rate of the photomask, a shielding area control unit used for regulating and controlling the shielding area of the photomask and a master control part, and the shielding area control unit or the shielding rate control unit drives the photomask to execute one time of'closing-opening 'operation. Wherein the dynamic visual sensor receives the scene light through the controllable blinking module, and the shielding rate of a photomask in the controllable blinking module to the incident light of the photomask is always smaller than 100%. The invention further provides an imaging method based on the panoramic blink event imaging system and an event-based scene editing method based on the imaging method.

Description

Technical Field

[0001] The present invention relates to the field of optical imaging technology, and more specifically, to a panoramic blink event imaging system and method. Background Art

[0002] An event camera is a bionic vision sensor. Its event-based vision is fundamentally different from traditional frame-based vision in the way it processes scenes. The working mechanism of an event camera involves several key steps: each pixel independently, continuously, and asynchronously processes incident light; photons incident on the photoelectric conversion active region in each pixel are converted into current and then into a voltage signal; the generated voltage is continuously compared with a reference voltage at each pixel to detect changes in light intensity, or the logarithm of light intensity, or other functions of light intensity, so as to increase the dynamic contrast range of the perceived scene. Other methods for enhancing the sensitivity and accuracy of detecting changes in light intensity, as well as methods for converting the light intensity change signal into a signal that is convenient to measure, can also achieve the above purposes. The working principle of an event camera enables it to achieve extremely high temporal resolution (in the microsecond range), a wide dynamic range, and low-latency dynamic visual perception. Therefore, event cameras exhibit significant advantages in fields such as robot vision, autonomous driving, and augmented reality.

[0003] Whenever the change in light intensity or its logarithmic change, etc., exceeds a predefined threshold, which often manifests as a voltage difference exceeding the predefined threshold in a specific circuit, an event (x, y, t, p) will be triggered, recording the pixel coordinates (x, y), the timestamp t corresponding to the occurrence of the change, and the polarity p ∈ {-1, +1} indicating a decrease or increase in light intensity. These events are output when they occur and reflect the change of the scene over time through a continuous data stream (referred to as an event stream) rather than a series of static frames. The event stream can be visualized as a two-channel representation in 3D space. Here, two dimensions constitute the spatial component that captures the event position in image coordinates, and the third dimension represents its time coordinate, precisely indicating the time when the event occurs. This spatio-temporal representation minimizes data redundancy and enables efficient recording of the dynamic aspects of the scene through its sparse structure.

[0004] The working principle of event cameras enables them to achieve extremely high temporal resolution (in the microsecond range), a wide dynamic range, and low-latency dynamic vision perception. Therefore, event cameras exhibit significant advantages in fields such as robotic vision, autonomous driving, and augmented reality. However, event cameras still face two major challenges in practical applications: First, the triggering of events depends on the relative motion between the camera and the scene, which restricts the ability of event cameras to acquire the light information of the entire scene. Specifically, in static backgrounds or textureless areas, event cameras often fail to effectively capture light changes, resulting in the loss of information in these areas. Second, the event stream generated by event cameras is a discrete and sparse four-dimensional spatio-temporal signal, and existing image processing techniques (usually based on continuous and dense pixel data) are difficult to directly apply to this type of data. This makes it difficult for event cameras to be directly applied to advanced vision tasks such as object detection and image segmentation, limiting their application potential in these fields.

[0005] In terms of obtaining panoramic information of complex scenes, the visual systems of advanced organisms exhibit significant intelligent characteristics. Taking humans as an example, when observing a complex scene, the human eye and brain usually prioritize dynamic objects or important visual information. However, even so, the visual system still maintains a clear perception of the static background through the "microsaccade mechanism". The "microsaccade mechanism" refers to an unconscious, small-scale, and rapid movement that occurs when the eyeball fixates on a certain fixed point. These tiny eye movements occur multiple times per second, maintaining a slight offset of the eyeball, and even when the visual focus is fixed on an object, it can prevent the complete stillness of visual information. This mechanism helps to continuously obtain information about the static elements in the scene. In addition, when the eyes stare at a specific point for a long time, the visual system gradually adapts to this constant visual input, resulting in a weakened response of neurons in the visual cortex to continuous stimulation, thereby leading to a decline in the ability to perceive image details, and the scene appears blurred or gradually disappears. To counteract this effect, the "blinking mechanism" of humans plays an important role. During the blinking process, the existing visual information on the retina is temporarily interrupted, and then when opening the eyes again, new visual information replaces the old image, thus resetting the visual system. At this time, neurons in the visual cortex respond to the new visual input, restoring a clear perception of the scene. In summary, obtaining panoramic information in complex scenes depends on the "microsaccade" and "blinking mechanisms" of advanced organisms, which optimize the performance of the visual system in processing dynamic and static information. Summary of the Invention

[0006] To overcome the defects that existing event cameras are difficult to acquire the light information of the entire scene and capture light changes, the present invention provides a panoramic blinking event imaging system and method.

[0007] To solve the above technical problems, the technical solution of the present invention is as follows:

[0008] A panoramic blink event imaging system, comprising:

[0009] A dynamic vision sensor, arranged by basic detection units, for receiving optical information; each basic detection unit in the dynamic vision sensor independently senses the change in received light intensity, and triggers an event when the sensed change in light intensity exceeds its preset threshold;

[0010] A controllable blink module, which includes an optical mask, a blocking rate control unit for regulating the blocking rate of the optical mask, a blocking area control unit for regulating the blocking area of the optical mask, and a general control unit; in the controllable blink module, when the general control unit issues a command, the blocking area control unit or the blocking rate control unit drives the optical mask to perform a "closing - opening" operation, that is, the blocking degree of the optical mask on the incident light of the dynamic vision sensor undergoes a change process of small → large → maintaining → small → maintaining;

[0011] Wherein, the dynamic vision sensor receives scene light through the controllable blink module, and the blocking rate of the optical mask in the controllable blink module on its own incident light is always less than 100%.

[0012] Preferably, the "closing - opening" operation includes a closing operation and an opening operation; wherein, the closing operation includes, under the drive of the blocking area control unit, a closing process in which the blocking area of the optical mask changes from the minimum blocking area in this closing operation to the maximum blocking area in this closing operation, and a closing holding process in which it continuously remains at the maximum blocking area in this closing operation after the closing process is completed; name this type of closing operation as a blocking area - type closing operation, and name the closing process it contains as a blocking area - type closing process;

[0013] Or, the closing operation includes, under the drive of the blocking rate control unit, a closing process in which the blocking rate of the optical mask changes from the minimum blocking rate in this closing operation to the maximum blocking rate in this closing operation, and a closing holding process in which it continuously remains at the maximum blocking rate in this closing operation after the closing process is completed; name this type of closing operation as a blocking rate - type closing operation, and name the closing process it contains as a blocking rate - type closing process;

[0014] And, wherein, the opening operation includes, under the drive of the blocking area control unit, an opening process in which the blocking area of the optical mask changes from the maximum blocking area in this opening operation to the minimum blocking area in this opening operation, and an opening holding process in which it continuously remains at the minimum blocking area in this opening operation after the opening process is completed; name this type of opening operation as a blocking area - type opening operation, and name the opening process it contains as a blocking area - type opening process;

[0015] Alternatively, the opening operation includes an opening process in which, under the drive of the occlusion rate control unit, the occlusion rate of the optical mask changes from the maximum occlusion rate in the current opening operation to the minimum occlusion rate in the current opening operation, and an opening holding process in which the occlusion rate remains at the minimum occlusion rate in the current opening operation after the opening process is completed; this type of opening operation is named the occlusion rate type opening operation, and the opening process it contains is named the occlusion rate type opening process;

[0016] The occlusion area type closing operation and the occlusion area type opening operation form an occlusion area type "closing-opening" operation, and the occlusion rate type closing operation and the occlusion rate type opening operation form an occlusion rate type "closing-opening" operation.

[0017] Preferably, the maximum occlusion area of the optical mask includes the maximum occlusion area value in the case of non-full area occlusion, or the occlusion area value in the case of full area occlusion; the minimum occlusion area of the optical mask includes the minimum occlusion area value in the case of partial occlusion, or the occlusion area value in the case of complete non-occlusion, that is, the occlusion area value is zero; the maximum occlusion rate of the optical mask includes the maximum occlusion rate of 100% occlusion; the minimum occlusion rate of the optical mask includes the non-zero minimum occlusion rate, or the zero occlusion rate.

[0018] Preferably, during the occlusion area type closing process or the occlusion area type opening process, the optical mask closes or opens in a rolling shutter manner in one direction, or closes and opens in a radially enclosing manner, or closes and opens in a point-by-point or line-by-line scanning manner, or closes and opens synchronously for the entire maximum occlusion area, or a combination of the above methods.

[0019] Preferably, the system further includes a relay device for modulating or / and guiding the scene light to enter the dynamic vision sensor; the dynamic vision sensor receives the scene light through the controllable blinking module and the relay device.

[0020] Preferably, the optical mask is composed of more than one mask layer; among them, each mask layer has different occlusion rate characteristics, or different color filtering characteristics, or different area characteristics, or different polarization characteristics, or a combined characteristic including multiple characteristics.

[0021] Preferably, the event includes the coordinate information, or / and time information, or / and event polarity information of the triggered basic detection unit; the preset threshold includes the absolute value of the preset light intensity change, or the absolute value of the preset logarithmic light intensity change, or the absolute value of the change of the light intensity function used to enhance the sensitivity and accuracy of the detection of light intensity changes and improve the dynamic contrast range of the perceived scene.

[0022] Preferably, the photomask is placed on the incident path of the scene light to the dynamic vision sensor, and includes the placement of the basic detection unit attached to the dynamic vision sensor.

[0023] Preferably, the relay device includes a lens, or a lens group, or a diffractive optical device, or a diffractive optical device group, or a prism, or a mirror, or a polarizer, or a color filter, or various combinations of the above devices.

[0024] Preferably, the photomask is imaged on the plane where the basic detection unit of the dynamic vision sensor is located through the relay device, or is imaged at a position away from the dynamic vision sensor along the incident light transmission direction.

[0025] Preferably, the system further includes an attenuation sheet with an adjustable attenuation coefficient, which is placed on the incident light transmission path of the dynamic vision sensor to adjust the incident light flux of the dynamic vision sensor.

[0026] Preferably, an aperture stop is introduced into the relay device, and its size is adjusted according to the scene light intensity, so that the entire system can work in a scene where the dynamic contrast range is greater than the intrinsic dynamic contrast range of the dynamic vision sensor.

[0027] Preferably, the relay device is an array-type relay device formed by arranging array units.

[0028] Preferably, the array-type relay device includes a one-dimensional aperture array composed of slits as array units arranged along a one-dimensional direction, or a two-dimensional aperture array composed of small holes as array units arranged along a two-dimensional direction, or a one-dimensional cylindrical lens array composed of cylindrical lenses as array units arranged along a one-dimensional direction, or a two-dimensional microlens array composed of microlenses as array units arranged along a two-dimensional direction.

[0029] Preferably, along at least one direction, for O>1 adjacent array units of the array-type relay device, they respectively have O orthogonal characteristics one by one; the array unit is used to allow the light with the corresponding orthogonal characteristic to pass through, and block or not respond to the other (O-1) non-corresponding orthogonal characteristic lights; among them, the array units with the same orthogonal characteristic form an array unit sub-array; the basic detection units corresponding to each array unit only receive the light with the orthogonal characteristic corresponding to this array unit, block or do not respond to the other orthogonal characteristic lights, and the basic detection units corresponding to the same array unit sub-array form a basic detection unit sub-array corresponding to this array unit sub-array; O basic detection unit sub-arrays and O array unit sub-arrays form O imaging sub-structures one by one; in the imaging sub-structure, any array unit sub-array thereof is composed of at least one array unit.

[0030] Preferably, the orthogonal characteristics include color characteristics of different wavelengths, polarization characteristics of different polarization states, time characteristics that respectively allow light to exit or enter at different time periods, or mixed characteristics composed of the different characteristics.

[0031] Preferably, the dynamic vision sensor includes more than G basic detection unit blocks; the basic detection unit blocks are arranged on a plane or a curved surface without sharing basic detection units at the same time point, and the basic detection unit resolutions, basic detection unit densities, array arrangement patterns, physical properties, and synchronizations of different basic detection unit blocks are the same, or partially the same, or completely different.

[0032] Preferably, the physical properties include color properties of different wavelengths, or polarization properties of different polarization states.

[0033] Furthermore, the present invention also proposes an imaging system, which is composed of more than one panoramic blink event imaging system as proposed by the present invention, and forms a binocular system or a multiocular system through planar or curved arrangement.

[0034] Furthermore, the present invention also proposes a panoramic blink event imaging method, which applies the panoramic blink event imaging system proposed by the present invention. The method includes the following steps:

[0035] S1. Determine the time window [T s , T e of the initial "close-open" operation; wherein, T s is the starting time point of the closing process in the "close-open" operation, and T e is the ending time point of the opening and holding process in the "close-open" operation;

[0036] S2. Set the parameters related to the occlusion area or / and the occlusion rate, and issue a command by the total control unit to implement a corresponding "close-open" operation;

[0037] S3. Select N≥2 time points {t 1 , t 2 , …, t N} within the current time window, and t N = T e ; for any time point t n , use [(t n - Δt n ), t n as the sub-time window; for the events within the sub-time window, use the per-basic-detection-unit event integration algorithm to obtain the absolute light intensity value of each basic detection unit at the time point t n , and according to all the basic detection units at the time point t nThe absolute light intensity value to obtain the time point t n of the light intensity image; where 1 ≤ n ≤ N, Δt n ≤ (tn - Ts) is the preset sub-time window length;

[0038] S4. Using T e + δt as the starting time point T of the closing process in the next "closing - opening" operation s , update the time window of the next "closing - opening" operation; repeat steps S2 - S4 until the imaging task is completed to obtain the light intensity images corresponding to each time point.

[0039] Preferably, for any "closing - opening" operation, for all events within its sub - time window [t n -Δt n , t n , use formulas (1) and (2) to perform the per - basic - detection - unit linear event integration algorithm, calculate the absolute light intensity value of each basic detection unit at the time point t n , and obtain the light intensity image at the time point t n according to the absolute light intensity values of all basic detection units at the time point t n ; its expression is:

[0040]

[0041] H(t) = exp[α·(t i - t i-1 )] (3)

[0042] where formula (1) represents performing the per - basic - detection - unit event integration algorithm for all events within the sub - time window [t n -Δt n , t n to obtain the absolute light intensity value of each basic detection unit at the time point t n ; I represents the light intensity image at the time point t n , ε{e i} represents the set of events triggered by the change in the occluded area or the change in the occlusion rate of the light mask within the sub - time window [t n -Δt n , t n ; formula (2) is the update rule of the per - basic - detection - unit event integration algorithm; I(x, y, t i ) represents the absolute light intensity value of the coordinate (x, y) of the basic detection unit of the dynamic vision sensor at the time stamp t i , t i represents the time stamp of the i - th event e i within ε{e i}; p iIndicates the i-th event e i is the event polarity; ΔC represents a preset constant threshold for event triggering; Equation (3) is the expression of the response function H(t) of the per-basic detection unit event integration algorithm; α is the attenuation factor, and by adjusting the value of the attenuation factor α, Equation (2) is made a linear event integration function.

[0043] Preferably, within any opening time window [T Os , T Oe , using the per-basic detection unit linear event integration algorithm, the absolute light intensity value of each basic detection unit at time point T Oe is obtained, and based on the absolute light intensity values of all basic detection units at time point T Oe , the static background light intensity image corresponding to time point T Oe is obtained; wherein, the opening time window [T Os , T Oe is the time interval corresponding to the opening process of a "close-open" operation.

[0044] Preferably, in any "close-open" operation, the corresponding opening time window [T Os , T Oe is determined based on the fitting of the event number distribution curve; the steps include: using a numerical fitting algorithm to make the number of events triggered by the dynamic vision sensor change with time and approach linearity during each process of the "close-open" operation, and obtaining the start time point T Os and the end time point T Oe of the opening process in the opening operation.

[0045] Preferably, within any opening time window [T Os , T Oe , for the events triggered by the change in the occluded area or the occlusion rate of the light mask, the per-basic detection unit linear event integration algorithm is performed using Equations (4), (5), and (6) to obtain the absolute light intensity value of each basic detection unit at time point T Oe , and based on the absolute light intensity values of all basic detection units at time point T Oe , the static background light intensity image corresponding to time point T Oe is obtained; its expression is:

[0046]

[0047] H(t) =exp[α·(t i -t i-1 )] (6)

[0048] wherein, Equation (4) represents the integration over the opening time window [T Os , TOe Perform per-basic detection unit event integration algorithm for all events within Oe to obtain the absolute light intensity value of each basic detection unit at time point T Oe ; I represents the light intensity image at time point T i}, and ε{e Os} represents the set of events triggered by changes in the occluded area or occlusion rate of the optical mask within the open time window [T Oe , T i ; Formula (5) is the update rule of the per-basic detection unit linear event integration algorithm; I(x, y, t i ) represents the absolute light intensity value of the coordinate (x, y) of the basic detection unit of the dynamic vision sensor at time stamp t i , and t i represents the time stamp of the i-th event e i within ε{e i}; p i represents the event polarity of the i-th event e

[0049] Preferably, within any moving light field time window [D s , D e , take M≥2 time points {s 1 , s 2 , …, s M}, where for any time point s m , take [(s m - Δs m ), s m as the sub-moving light field time window; for other events within the sub-moving light field time window except for the events triggered by changes in the occluded area or occlusion rate of the optical mask, use the per-basic detection unit decaying event integration algorithm to update the absolute light intensity value of each basic detection unit at time point s m , and obtain the moving light field light intensity image corresponding to time point s m according to the absolute light intensity values of all basic detection units at time point s m ; where Δs m ≤ (s m - D s ), or, Δs m is less than or equal to the time point s m and intersects with the moving light field time window [D s , D eThe difference between the minimum time points covered by all time windows, and 1 ≤ m ≤ M; the moving optical field time window [D s , D e , is the time interval corresponding to other events other than the events triggered by the change of the occlusion area or the occlusion rate of the optical mask within the time interval covered by one time window or more than one adjacent time window.

[0050] Preferably, within any moving optical field time window [D s , D e , take K ≥ 2 time points {k 1 , k 2 , …, k K}, where for any time point k k , [(k k - Δk k ), k k is the sub-decaying moving optical field time window; for other events other than the events triggered by the change of the occlusion area or the occlusion rate of the optical mask within the sub-decaying moving optical field time window, use the event integration algorithm of gradually decaying each basic detection unit to update the absolute light intensity value of each basic detection unit at the time point k k , and obtain the decaying moving optical field light intensity image corresponding to the time point k k according to the absolute light intensity values of all basic detection units at the time point k k ; where Δk k ≤ (k k - D s ), or, Δk k is less than or equal to the difference between the time point k k and the minimum time point covered by all time windows intersecting with the moving optical field time window [D s , D e , and the subscript k is a positive integer less than or equal to K.

[0051] Preferably, within any moving optical field time window [D s , D e , divide other events other than the events triggered by the change of the occlusion area or the occlusion rate of the optical mask into Q ≥ 1 moving subsets; let any moving subset q correspond to the moving subset time window Different moving subsets independently adopt their respective corresponding decay event integration algorithms and corresponding decay factors, where 1 ≤ q ≤ Q.

[0052] Preferably, for all events within any time period [T Os , s m , use the time-sharing event integration algorithm of gradually decaying each basic detection unit in formulas (7) to (11) to calculate each basic detection unit at the time point sm and the absolute light intensity value of all basic detection units at time point s m and the absolute light intensity value, to obtain the panoramic trajectory light intensity image at time point s m ; its expression is:

[0053]

[0054] H Θ (t) = exp[α·(t i - t i-1 )] (10)

[0055] H Ω (t) = exp[β·(t j - t j-1 )] (11)

[0056] wherein, formula (7) represents all events within the time period [T Os , s m , and by using the time-sharing event integration algorithm for each basic detection unit, the panoramic trajectory light intensity image I at time point s m is obtained; ε Θ {e i} represents the set of events triggered by the change in the occluded area or the change in the occlusion rate of the optical mask within the time period [T Os , T Oe ; ε Ω (e j ) represents the set of other events other than the events triggered by the change in the occluded area and the change in the occlusion rate of the optical mask within the time period [D s , s m ; formula (8) is the update rule of the linear event integration algorithm for each basic detection unit, where I Θ (x, y, t i ) represents the absolute light intensity value of the coordinate (x, y) of the basic detection unit of the dynamic vision sensor at time stamp t i , t i represents the time stamp of the i-th event e Θ {e i} in ε i ; p i represents the event polarity of the i-th event e i ; ΔC represents the preset constant threshold triggered by the event; H Θ (t) represents the response function of the update rule of the linear event integration algorithm for the basic detection unit, specifically as shown in formula (10); α is the attenuation factor, and by adjusting the value of the attenuation factor α, formula (8) becomes a linear event integration function; formula (9) is the update rule of the decaying event integration algorithm for each basic detection unit, where I Ω(x, y, t j ) represents the coordinates (x, y) of the basic detection unit of the dynamic vision sensor at timestamp t j of the absolute light intensity value, t j represents ε Ω (e j ) within the i-th event e j of the timestamp, p j represents the j-th event e j of the event polarity; H Ω (t) represents the response function of the update rule of the decay event integration algorithm for the basic detection unit, specifically as shown in formula (11); β is the decay factor, and by adjusting the value of the decay factor β, formula (9) becomes the decay event integration function; and α > β.

[0057] Preferably, for the events in ε Ω (e j ), when updating the absolute light intensity value for each event in the time window corresponding to each motion subset using the per-basic-detection-unit event integration algorithm, different values of the decay factor β are selected.

[0058] Preferably, within any motion light field time window [D s , D e , for the events other than those triggered by the change in the occlusion area or occlusion rate of the light mask within the sub-decay motion light field time window, using formulas (12) to (16) for the per-basic-detection-unit decay event integration algorithm, the absolute light intensity value of each basic detection unit at time point k k is obtained, and based on the absolute light intensity values of all basic detection units at time point k k , the decay motion light field intensity image corresponding to time point k k is obtained; then, the decay motion light field intensity images corresponding to each time point k k are stitched together in the time dimension to form the motion light field video of the motion light field time window [D s , D e ; in the decay motion light field intensity image, for the basic detection units that have not triggered events, their corresponding absolute light intensity values are multiplied by the corresponding decay factor as their new absolute light intensity values; its expression is:

[0059]

[0060] H(t) = exp[γ·(t j - t j-1 )] (14)

[0061] ​

[0062] Among them, formula (12) represents the set of events ε{e k -Δk k ,k k within the time period [k i}, using the per-basic detection unit event integration algorithm to calculate the absolute light intensity value of each basic detection unit at time point k k , and based on the absolute light intensity values of all basic detection units at time point k k , obtaining the moving light field light intensity image I at time point k k ; ε{e j} represents the set of other events except for the events triggered by the change in the light mask occlusion area or the change in the occlusion rate within the sub-decaying moving light field time window [k k -Δk k ,k k ; formula (13) is the update rule of the per-basic detection unit decay event integration algorithm, where I(x,y,t j ) represents the absolute light intensity value of the coordinate (x,y) of the basic detection unit of the dynamic vision sensor at time stamp t j , t j represents the time stamp of the j-th event e j within ε{e j}; p j represents the event polarity of the j-th event e j ; ΔC represents the preset constant threshold triggered by the event; H(t) represents the response function of the update rule of the linear event integration algorithm of the basic detection unit, specifically as shown in formula (14); γ is the decay factor, and by adjusting the value of the decay factor γ, formula (14) becomes the decay event integration function; formula (15) represents the final decay moving light field light intensity image obtained by performing a dot product operation on the decay mask F and the absolute light intensity value I Among them, for the sub-decaying moving light field time window [k k -Δk k ,k k , the decay mask value F(x,y) at the position of the basic detection unit with a trigger event is set to 1, and the decay mask value F(x,y) at the position of the basic detection unit without a trigger event is set to η, and 0 < η < 1.

[0063] Preferably, for the events in ε{e j}, when updating the absolute light intensity value using the per-basic detection unit event integration algorithm for the events corresponding to each moving subset time window , different values of the decay factor γ and η are selected.

[0064] Preferably, in any "closing - opening" operation, the occlusion rate control unit sets the occlusion rate of the optical mask such that the number of events triggered by the dynamic vision sensor during the corresponding closing process and opening process approaches a linear change over time.

[0065] Preferably, in any occlusion - area - type "closing - opening" operation, under the illumination condition with external light intensity < 50 lux, the occlusion rate control unit sets the occlusion rate of the optical mask to be < 100% and ≥ 90%, and names this interval as the high - grade occlusion rate; under the illumination condition with external light intensity ≥ 50 lux and < 2000 lux, the occlusion rate control unit sets the occlusion rate of the optical mask to be < 90% and ≥ 50%, and names this interval as the medium - grade occlusion rate; under the illumination condition with external light intensity ≥ 2000 lux, the occlusion rate control unit sets the occlusion rate of the optical mask to be < 50% and > 0%, and names this interval as the low - grade occlusion rate.

[0066] Furthermore, the present invention also proposes a scene editing method, which applies the panoramic blink event imaging system proposed by the present invention. The method includes the following steps:

[0067] A1. Use the controllable blink module (20) to perform a "closing - opening" operation on the static background scene, and the events triggered during the opening process are recorded as the static background event stream ε{e i}, and the corresponding opening time window is recorded as [T Os , T Oe ;

[0068] A2. Stitch the event stream ε s , D e triggered by the movement of the independent object u within the moving light field time window [D u {e i} and the corresponding time period [s us , s ue with the static background event stream ε{e i} in the time dimension, so that the time period [s us , s ue is converted into the moving light field time window [T Oe , t ue ;

[0069] A3. For the stitched time period [T Os , t ue and the stitched event ε{e i}+ε u {e i}, use the linear event integration algorithm of the basic detection unit to calculate the time window [T Os , T OeThe static background light intensity image at time point T Oe ; Then, based on the static background light intensity image at time point T Oe , within the moving light field time window [T Oe , t ue , take M≥2 time points {d u1 , d u2 , …, d uM}, and d uM = t ue . Correspond one by one to the M time points {s us , s ue} within the time period [s u1 , s u2 , …, s uM , and s uM = s ue ; Where for any time point d um , use [d um - Δd um , d um as the corresponding sub - moving light field time window; Finally, use the basic detection unit decay event integration algorithm to update the absolute light intensity value of each basic detection unit at time point d um , and based on the absolute light intensity values of all basic detection units at time point d um , obtain the scene - edited moving light field light intensity image corresponding to time point d um , where 1≤m≤M;

[0070] A4. Stitch the scene - edited moving light field light intensity images corresponding to each time point d um in the time dimension to form the scene - edited moving light field video within the time window [T Os , t ue ;

[0071] A5. For the event streams ε u {e i} triggered by U≥1 independent object movements, where 1≤u≤U, repeat steps A2 - A4 for scene editing to obtain the enhanced scene light intensity image at any time point within the stitched time period.

[0072] Preferably, the expression of the basic detection unit linear event integration algorithm is:

[0073]

[0074] H 1 (t) = exp[α·(t i - t i-1 )] (20)

[0075] H2 I(t) = exp[β·(t r -t r-1 )] (21)

[0076]

[0077] where t i represents the timestamp of the i-th event e i} within ε{e i , I O (x, y, t i ) represents the absolute light intensity value of the basic detection unit at coordinates (x, y) at timestamp t i , p i represents the event polarity of the i-th event e i ; ΔC represents a preset constant threshold for event triggering; t j represents the timestamp of the j-th event e i} within ε{e j , t r represents the timestamp obtained after converting t j through formula (22) after splicing two event streams, I m (x, y, t r ) represents the absolute light intensity value of the basic detection unit at coordinates at timestamp t r , p r represents the event polarity of the r-th event e r ;

[0078] Moreover, among them, formula (18) is the update rule of the linear event integration algorithm for each basic detection unit, where H 1 (t) represents the response function of the update rule of the linear event integration algorithm for the basic detection unit, specifically as shown in formula (20); α represents the attenuation factor, and adjusting the value of the attenuation factor α makes formula (18) a linear event integration function;

[0079] Moreover, among them, formula (19) is the update rule of the decaying event integration algorithm for each basic detection unit, where H 2 (t) represents the response function of the update rule of the decaying event integration algorithm for the basic detection unit, specifically as shown in formula (21), and β represents the attenuation factor, and adjusting the value of the attenuation factor β makes formula (19) a decaying event integration function.

[0080] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0081] The present invention utilizes a controllable blinking module to perform an occlusion area type "closing - opening" operation or an occlusion rate type "closing - opening" operation to simulate the "micro - eye movement mechanism" and "blinking mechanism" of advanced organisms. Further, it cooperates with a dynamic vision sensor to obtain scene light information, and then based on relevant algorithms for reconstruction, it can effectively acquire panoramic light information and capture light changes, thereby optimizing the performance of the vision system in dynamic and static information processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 It is an architecture diagram of a panoramic blinking event imaging system.

[0083] Figure 2 It is the basic optical structure of a panoramic blinking event imaging system.

[0084] Figure 3 It is an example of an occlusion area type "closing - opening" operation.

[0085] Figure 4 It is another example of an occlusion area type "closing - opening" operation.

[0086] Figure 5 It is yet another example of an occlusion area type "closing - opening" operation.

[0087] Figure 6 It is an example of the change mode of the occlusion area of an occlusion area type light mask.

[0088] Figure 7 It is an example of a system structure using an array - type relay device.

[0089] Figure 8 It is an example of a system structure using an array - type relay device where adjacent sub - images overlap.

[0090] Figure 9 It is a schematic diagram of the principle of suppressing the overlap of adjacent sub - images by an orthogonal characteristic array - type relay device.

[0091] Figure 10 It is a schematic diagram of another application example of an orthogonal characteristic array - type relay device.

[0092] Figure 11 It is a schematic example of a complete "closing - opening" operation performed by an area - occlusion type light mask when the static scene and the panoramic blinking event imaging system are stationary.

[0093] Figure 12 It is a schematic example of a static scene with a moving object when the panoramic blinking event imaging system is stationary.

[0094] Figure 13It is an exemplary schematic diagram of the motion of an object in a static scene starting from the opening process in a "closing - opening" operation and continuing in the opening - holding process.

[0095] Figure 14 It is an exemplary schematic diagram of the motion of a very small object in a static scene.

[0096] Figure 15 It is an exemplary schematic diagram of generating relative motion of the dynamic vision sensor with respect to the scene through the active "micro - eye movement" mode.

[0097] Figure 16 It is an exemplary schematic diagram of placing the dynamic vision sensor on a car traveling on a road and having relative motion with the scene.

[0098] Figure 17 It is an exemplary schematic diagram of placing the dynamic vision sensor on a car traveling on a road and having relative motion with the scene, while performing ≥1 occlusion - area - type "closing - opening" operations.

[0099] Figure 18 It is an exemplary schematic diagram of the application of the event - based scene editing method. Detailed implementation manners

[0100] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present invention. On the contrary, they are only examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0101] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the", and "said" used in the present invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0102] The present invention will be described in detail below with reference to the drawings and specific implementation manners.

[0103] Embodiment 1

[0104] As Figure 1 shown, it is the architecture diagram of the panoramic blinking event imaging system. Among them, through the controllable blinking module 20 and the relay device 30, the dynamic vision sensor 10 obtains the scene light information and then performs reconstruction based on relevant algorithms.

[0105] The basic optical structure of the panoramic blinking event imaging system in this embodiment is as follows Figure 1 , 2 shown, including a dynamic vision sensor 10 composed of basic detection units arranged, a controllable blinking module 20, and a relay device 30.

[0106] Each basic detection unit of the dynamic vision sensor 10 independently senses the change in the received light intensity under the control of the total control unit 240, and triggers an event when the sensed change in light intensity exceeds its preset threshold. The basic detection unit can be a sub-pixel, or a pixel formed by arranging sub-pixel planes, or a pixel formed by stacking sub-pixels. Or rather, each basic detection unit of the dynamic vision sensor 10 independently, continuously, and asynchronously processes the change in the incident light intensity. When the absolute value of the change in the light intensity received by the basic detection unit, or the change in the logarithm of the light intensity, or the change in other functions of the light intensity, exceeds the preset constant threshold ΔC, an event e i , is represented by (x, y, t i , p i ); where (x, y) is the coordinate of the basic detection unit, the time stamp t i is the time corresponding to the change in the logarithm of the light intensity exceeding ΔC, and p i ∈ {-1, +1} is the polarity, and p i = 1 indicates an increase in light intensity, which is called a positive event; p i = -1 indicates a decrease in light intensity, which is called a negative event. The event is output when it occurs, and the change of the scene over time is recorded through a continuous data stream (called an event stream). Usually, based on the absolute value of the change in the logarithm of the light intensity, the preset threshold ΔC for the trigger time is set.

[0107] The controllable blinking module 20 includes a light mask 210, a masking rate control unit 220 for regulating the masking rate of the light mask 210, and / or a masking area control unit 230 for regulating the masking area of the light mask 210, and a total control unit 240. Among them, the light mask 210 is on the propagation path of the incident light of the dynamic vision sensor 10. When the total control unit 240 issues an order, the masking area control unit 230 or the masking rate control unit 220 drives the light mask 210 to perform a "closing - opening" operation. During the implementation of this "closing - opening" operation, the degree of occlusion of the incident light of the dynamic vision sensor 10 by the light mask 210 changes in the process of small → large → maintaining → small → maintaining.

[0108] In this embodiment and subsequent embodiments, the masking rate of the light mask 210 for its own incident light is always less than 100%, and the following part will not be repeated.

[0109] Exemplarily, for the photomask 210 with an occlusion rate of 60%, when a beam of light is incident on the photomask 210, only (1 - 60%) = 40% of the incident light intensity remains when it exits the photomask 210. The occlusion rate of the photomask 210 can be changed under the drive of the occlusion rate control unit 220, or the occluded area of the photomask 210 can be changed under the occlusion area control unit 230. The total control unit 240, the occlusion rate control unit 220, the occlusion area control unit 230, and the dynamic vision sensor 10 can all be signal-connected and controlled by it.

[0110] As Figure 2 shown, the occlusion rate control unit 220, the occlusion area control unit 230, and the total control unit 240 are shown as three separate devices; obviously, they can also be combined into a control module to change the occlusion rate of the photomask 210, change the occluded area of the photomask 210, or drive the dynamic vision sensor 10 to work.

[0111] The relay device 30 is used to modulate and guide external light to be incident on the dynamic vision sensor 10.

[0112] Among them, the relay device 30 can be various optical devices, such as diffractive optical devices, or a group of diffractive optical devices, or prisms, or mirrors, or polarizers, or color filters, or various combinations of the above devices, to modulate and guide the light incident on the dynamic vision sensor 10 from the scene. For example, the diffractive device or a group of diffractive devices can guide the spectral information from the observed object to the dynamic vision sensor 10, or guide light of different frequencies to the respective corresponding areas of the dynamic vision sensor 10; for example, prisms and mirrors can modulate the geometric structure of the imaging system by deflecting the light path; for example, the polarizer can select and allow the light with the corresponding deflection characteristics from the observed object to be incident on the dynamic vision sensor 10; for example, the color filter can select and allow the light with the corresponding color from the observed object to the dynamic vision sensor 10. Commonly, the relay device 30 can be various optical devices, which are lenses, lens groups with imaging functions, or diffractive devices or free-form surface groups with imaging functions, called imaging-type relay devices, to image the observed object onto the basic detection unit surface of the dynamic vision sensor 10. This imaging-type relay device can also image the photomask 210, image the photomask 210 onto the basic detection unit surface of the dynamic vision sensor 10, or image it along the incident light transmission direction to a position far from the basic detection unit surface of the dynamic vision sensor 10, such as in front of or behind the basic detection unit surface. For example, a combination of multiple lenses can simultaneously implement the imaging of the observed object and the photomask 210.

[0113] Figure 1In [the figure], the relay device 30 is placed between the optical mask 210 and the dynamic vision sensor 10. In fact, both the optical mask 210 and the relay device 30 are on the incident path of the scene light to the dynamic vision sensor 10, and their positional relationship can be adjusted as needed. Even when the relay device 30 includes multiple optical components, the optical mask 210 can be placed between the optical components of the relay device 30. The optical mask 210 can also be attached to the placement of the basic detection unit of the dynamic vision sensor 10.

[0114] The relay device 30 can also be removed, for example, directly incident the scene light on the dynamic vision sensor 10.

[0115] The relay device 30 can also include an aperture stop 310, the size of which is adaptively adjusted according to the scene light intensity, so that the entire system can work in a scene with a dynamic contrast ratio range greater than the intrinsic dynamic contrast ratio range of the dynamic vision sensor 10. The aperture stop 310 can also be a separately introduced device, as a component of the relay device 30, and manually or by the general control unit 240, the size of its light transmission aperture is changed.

[0116] This system can also include an attenuation sheet 40 with an adjustable attenuation coefficient. Among them, the attenuation sheet 40 is placed on the incident light transmission path of the dynamic vision sensor 10, and its position relative to the optical mask 210 and the relay device 30 can be adjusted as needed, rather than just being placed between the relay device 30 and the dynamic vision sensor 10 as Figure 2 shown.

[0117] In the controllable blinking module 20, when the general control unit 240 issues a command, the occlusion area control unit 230 or the occlusion rate control unit 220 drives the optical mask 210 to perform a "closing - opening" operation. During the implementation of this "closing - opening" operation, the degree of occlusion of the incident light of the optical mask 210 to the dynamic vision sensor 10 undergoes a change process of small → large → maintaining → small → maintaining, that is, it includes a closing process, a closing - maintaining process, an opening process, and an opening - maintaining process. Among them, the closing process and the closing - maintaining process are named the closing operation, and the opening process and the opening - maintaining process are named the opening operation.

[0118] When the light intensity change received by the dynamic vision sensor 10 exceeds a preset threshold, an event is triggered. The triggered event refers to the coordinate information, or / and time information, or / and event polarity information of the triggered basic detection unit. The preset threshold is the absolute value of the preset light intensity change, or the absolute value of the preset logarithmic light intensity change, or the absolute value of the change of a certain function value of light intensity that enhances the sensitivity and accuracy of the detection light intensity change and increases the dynamic contrast ratio range of the perceived scene.

[0119] In the common closing process of the "closing - opening" operation, there are changes in the area blocked by the photomask 210 from small to large, or changes in the blocking rate from small to large; in the common opening process, there are changes in the area blocked by the photomask 210 from large to small, or changes in the blocking rate from large to small. At this time, the events triggered by the dynamic vision sensor 10 include: events triggered by changes in the area blocked by the photomask 210 during the closing and opening processes of the "closing - opening" operation; events triggered by changes in the blocking rate of the photomask 210 during the closing and opening processes of the "closing - opening" operation; events triggered by the movement of objects in the scene; events triggered by the movement of the dynamic vision sensor 10 itself; events triggered by changes in scene illumination; events triggered by scene changes, etc. Among them, the movement of the dynamic vision sensor 10 itself (Ego - motion) includes the local movement of the dynamic vision sensor 10 to generate relative movement with the scene to trigger events, which is called "micro - eye movement"; the overall movement of the dynamic vision sensor 10, such as placing it on a moving vehicle (such as a car, drone, robot, etc.).

[0120] Exemplarily, the main scenarios for the panoramic blinking event imaging system to obtain panoramic light information include:

[0121] ① The dynamic vision sensor 10 is stationary: The controllable blinking module 20 performs the "closing - opening" operation, and the events triggered by driving the change in the degree of blocking of the incident light of the dynamic vision sensor 10 by the photomask 210 are called events triggered by changes in the area blocked or the blocking rate of the photomask 210, and the static background information of the scene is obtained; based on the event triggering principle of the dynamic vision sensor 10, the events triggered by object movement, scene change, scene illumination change, etc. are called other events other than the events triggered by changes in the area blocked or the blocking rate of the photomask 210, and the moving light field information is obtained; the two constitute the panoramic light information.

[0122] ② The photomask 210 is in the open - holding process, and the dynamic vision sensor 10 has local movement (micro - eye movement): Based on the event triggering principle of the dynamic vision sensor 10, the events triggered by the static background, object movement, scene change, scene illumination change, etc. are called other events other than the events triggered by changes in the area blocked or the blocking rate of the photomask 210, and the panoramic light information is obtained.

[0123] ③ The photomask 210 is in the open - holding process, and the dynamic vision sensor 10 moves as a whole in the scene, such as placing it on a moving vehicle; based on the event triggering principle of the dynamic vision sensor 10, the events triggered by the static background, object movement, scene change, scene illumination change, etc. are called other events other than the events triggered by changes in the area blocked or the blocking rate of the photomask 210, and the panoramic light information is obtained.

[0124] ④ When the dynamic vision sensor 10 moves itself, the controllable blinking module 20 performs a "closing - opening" operation, driving the optical mask 210 to implement a cyclic change of small → large → maintaining → small → maintaining in the degree of blocking the incident light of the dynamic vision sensor 10. In this case, during the closing process and the opening process, the events triggered by the change in the degree of blocking of the incident light of the dynamic vision sensor 10 by the optical mask 210 are called the events triggered by the change in the blocked area or blocking rate of the optical mask 210; the events triggered by static background, object movement, scene change, scene illumination change, etc. are called other events other than the events triggered by the change in the blocked area or blocking rate of the optical mask 210; the two constitute the panoramic light information.

[0125] Exemplarily, taking the installation of the panoramic blinking event imaging system on a moving vehicle (such as a car, a drone, a robot, etc.) as an example, the description is as follows:

[0126] Fix the dynamic vision sensor 10 on the vehicle, and through the total control unit 240, command the controllable blinking module 20 to perform a "closing - opening" operation, driving the optical mask 210 to implement a cyclic change of small → large → maintaining → small → maintaining in the degree of blocking the incident light of the dynamic vision sensor 10.

[0127] At the initial moment, before the vehicle starts to move, perform ≥1 "closing - opening" operations to obtain the panoramic light information of the scene, including static background information and the moving light field information in the scene; after the vehicle starts to move, the optical mask 210 is preferably in the open - maintaining process. Based on the event - triggering principle of the dynamic vision sensor 10, due to the relative movement between the scene and the dynamic vision sensor 10, the panoramic light information is obtained. However, considering that there may be moving objects in the scene that are relatively stationary or nearly relatively stationary with respect to the dynamic vision sensor 10, which will not trigger events or only trigger a very small number of events, ≥1 "closing - opening" operations should be appropriately performed to obtain the panoramic light information of the scene; when the vehicle temporarily stops moving (such as: the car stops at a red light, the drone hovers, the robot is stationary, etc.), perform ≥1 "closing - opening" operations to obtain the panoramic light information of the scene, including static background information and the moving light field information in the scene; after the vehicle starts to move again, the optical mask 210 is preferably in the open - maintaining process. Based on the event - triggering principle of the dynamic vision sensor 10, due to the relative movement between the scene and the dynamic vision sensor 10, the panoramic light information is obtained. Similarly, considering that there may be moving objects in the scene that are relatively stationary or nearly relatively stationary with respect to the dynamic vision sensor 10, which will not trigger events or only trigger a very small number of events, ≥1 "closing - opening" operations should be appropriately performed to obtain the panoramic light information of the scene.

[0128] Furthermore, the dynamic vision sensor 10 fixed to the vehicle moves locally as a whole in coordination with the relay device 30, the optical mask 210, and the attenuation sheet 40, which is called "micro eye movement" to expand the field of view. The "micro eye movement" and the "close-open" operation can be carried out in sequence or simultaneously.

[0129] As Figure 3 shown, taking the change of the occlusion area of the optical mask 210 as an example, a specific "close-open" operation is described and named as the occlusion area type "close-open" operation. The closing operation in an occlusion area type "close-open" operation is called the occlusion area type closing operation, its closing process is called the occlusion area type closing process, its opening operation is called the occlusion area type opening operation, and its opening process is called the occlusion area type opening process. Specifically, under the drive of the occlusion area control unit 230, any occlusion area type "close-open" operation includes a closing process in which the occlusion area of the optical mask 210 changes from the minimum occlusion area in this closing operation to the maximum occlusion area in this closing operation, and a closing holding process in which it continuously remains at the maximum occlusion area in this closing operation after the closing process is completed; and an opening process in which the occlusion area of the optical mask 210 changes from the maximum occlusion area in this opening operation to the minimum occlusion area in this closing operation, and an opening holding process in which it continuously remains at the minimum occlusion area in this closing operation after the opening process is completed. Figure 3 The rectangular area marked with S in Figure 3 is the light passing area of the relevant light on the plane where the optical mask 210 is located when the light information of the scene to be detected enters the dynamic vision sensor 10, which is called the full aperture area. When the optical mask 210 is exactly placed on the inherent light passing aperture of the relay device 30 or the introduced aperture stop 310, this full aperture area corresponds to the inherent light passing aperture of the relay device 30 or the aperture of the aperture stop 310.

[0130] At the time point Ts when the closing process starts, that is Figure 3 in state A in 1 the occlusion area of the optical mask 210 is the minimum value 0, that is, there is no occlusion; after a time Δt Figure 3 OS the occlusion area of the optical mask 210 becomes the maximum full aperture area, that is, the full occlusion state, that is in state D in Figure 3 ; then, after a closing holding process of Δt2 time, starting from the full occlusion state at time point T 3 that is, state D' in Figure 3the state A' therein; then, after Δt 4 During the closing hold process of time, at time point T e end this "closing - opening" operation. Before performing the next "closing - opening" operation, there can be an interruption time period of δt≥0, and then start the next similar "closing - opening" operation. Figure 3 For the closing process and the opening process shown, both exemplarily represent the process with 4 time points. Figure 3 In the "closing - opening" operation shown, the durations Δt 1 、Δt 2 、Δt 3 、Δt 4 of the four processes can be different from each other. In fact, the durations of the same type of processes corresponding to different "closing - opening" operations can be the same or different. For example, the duration of one closing process and the duration of the next closing process can be the same or different. At the same time, the total durations corresponding to different "closing - opening" operations can be the same or different. The interruption time periods between different "closing - opening" operations can be the same, for example, all δt = 0, or all other non - zero same values, or can be different. During the closing process of an occlusion - area - type "closing - opening" operation, the change rate of the occluded area of the photomask 210 can be a fixed value, as Figure 3 shown, for states B and C at the same time intervals in sequence, their occluded areas G B and G C increase linearly and uniformly with time; for states B' and C' at the same time intervals in sequence, their occluded areas G B ' and G C ' decrease linearly and uniformly with time.

[0131] Figure 3 In B , "G" represents the occluded area of the photomask 210, and "W" represents the non - occluded area of the photomask 210. For example, G s represents the occluded area of the photomask 210 corresponding to state B at time point T

[0132] +Δ. This representation method will be used in the subsequent parts of this embodiment and will not be elaborated further. Figure 4 In a closing process of an occlusion - area - type or an opening process of an occlusion - area - type, the change rate of the occluded area of the photomask 210 can also be variable, as Figure 4 shown, which is another example of an occlusion - area - type "closing - opening" operation. Among them, Figure 4 for the closing process shown, for states C and B at the same time intervals, the area of the occluded area of the corresponding photomask 210 increases non - linearly; Figure 4In the figure, the area blocked by the photomask 210 is also denoted by "G", and the area not blocked by the photomask 210 is denoted by "W". For the sake of clear illustration, only some blocked areas and some unblocked areas are labeled.

[0133] Among them, the maximum blocked area of the photomask 210 includes the maximum blocked area value in the case of non-full-area blocking, or the blocked area value in the case of full-area blocking; the minimum blocked area of the photomask 210 includes the minimum blocked area value in the case of partial blocking, or the blocked area value in the case of completely unblocked, that is, the blocked area value is zero. As Figure 3 and Figure 4 shown, the maximum blocked area of the photomask 210 is set to the full-blocking state, that is, the coverage rate of the photomask 210 for the full-aperture area is 100%; the minimum is the unblocked state, that is, the coverage rate of the photomask 210 for the full-aperture area is zero.

[0134] In fact, the maximum blocked area of the photomask 210 can be a blockage with a coverage rate of less than 100%, and the minimum blocked area can be a partial blockage with a non-zero coverage rate. As Figure 5 shown, it is another example of the "closing-opening" operation of the blocked area type. At the point T s corresponding to the starting time of the closing process, state A, a partial area G A of the photomask 210 in the full-aperture area is in the blocked state; at state D at the end time of the closing process, the maximum covered area of the photomask 210 in the full-aperture area is G D , and there are still some areas W D1 and W D2 in the unblocked state. At state A' at the end time of the opening process, a partial area G A' of the photomask 210 in the full-aperture area is in the blocked state. That is, the blocked states at the starting time of the closing process and at the end time of the opening process of this "closing-opening" operation can be different.

[0135] Of course, the blocked states at the starting time of the closing process and at the end time of the opening process of a "closing-opening" operation can also be the same. As Figures 4 to 5 shown, the blocked area of the photomask 210 changes in one direction, which is a common rolling-type photomask 210. It can be realized by physical devices such as a mechanically translatable baffle, an electro-controlled liquid crystal light valve, a flexible stretchable film, etc. The rolling-type closing and opening in one direction can make the total amount of events triggered by the change of the blocked area during the closing process and the opening process increase approximately linearly.

[0136] The changing manner of the blocked area of the photomask 210 can also be closing and opening in a radial surrounding manner, or changing in a circular area, or closing and opening in a point-by-point or line-by-line scanning manner, or synchronously closing and opening the entire maximum blocked area, or various combinations of the above manners. For example, Figure 6 (a) shows a schematic diagram of the change of the blocked area of the photomask changing radially. For example, Figure 6 (b) shows a schematic diagram of the change of the blocked area of the photomask changing by point-by-point scanning. Figure 6 In (a), the example of the full-aperture area marked with S is circular. Figure 6 In (b), the example of the full-aperture area marked with S is rectangular. The shaded part is an example of the blocked area in a certain state. In fact, the change of the blocked area of the photomask 210 can be implemented in various possible manners, or even implemented by a combination of different changing manners. This application does not limit the manner in which the blocked area of the photomask 210 changes.

[0137] In this embodiment, the blocking rate of the photomask 210 is always less than 100%. That is to say, for the area blocked by the photomask 210, the light transmittance of the incident light is also greater than zero. Preferably, in the blocking area type closing operation, the blocking rate of the photomask 210 is set to a fixed value P1; in the blocking area type opening operation, the blocking rate of the photomask 210 is set to a fixed value P2. The specific values of P1 and P2 are adjusted according to the scene light intensity. Specifically, for example, under low light conditions where the external light intensity < 50 lux, the blocking rate control unit 220 sets the blocking rates P1 and P2 of the photomask 210 to < 100% and ≥ 90% at the same time, that is, the high gear blocking rate; under the condition where the external light intensity ≥ 50 lux and < 2000 lux, the blocking rate control unit 220 sets the blocking rates P1 and P2 of the photomask 210 to < 90% and ≥ 50% at the same time, that is, the medium gear blocking rate; under the condition where the external light intensity ≥ 2000 lux, the blocking rate control unit 220 sets the blocking rates P1 and P2 of the photomask 210 to < 50% and > 0%, that is, the low gear blocking rate. Commonly, P1 = P2 is set.

[0138] When encountering a sudden change in the scene light intensity in time and / or space, by means of the closing operation, before performing the opening operation, set an appropriate P2. When the scene light intensity is very high, such as far greater than 2000 lux, Figure 2In the system shown, before performing the opening operation, the incident light flux can be adaptively reduced by adjusting the inherent aperture diaphragm of the relay device 30, or the light passing aperture of the introduced aperture diaphragm 310, and / or the attenuation coefficient of the attenuation sheet 40 introduced by the system, by means of the closing operation. In fact, it is preferable that the occlusion rate control unit 220 sets the occlusion rate of the optical mask 210 so that the number of events triggered by the dynamic vision sensor 10 during the corresponding closing process and opening process changes linearly with time.

[0139] The optical mask 210 can include only one mask layer, or can be composed of more than one mask layer. For example, the more than one mask layer has different occlusion rate characteristics, or different color filtering characteristics, or different area characteristics, or different polarization characteristics, or various combined characteristics of the above characteristics. The overlap of the more than one mask layer can endow the optical mask 210 with more flexible occlusion characteristics. For example, for the optical mask 210 composed of two mask layers with different occlusion rates and different areas, the occluded area may be divided into two sub-areas with different occlusion rates. For the optical mask 210 composed of two mask layers with different color filtering characteristics, the occluded area may reconstruct the image information of different color components of the target scene respectively. For two optical masks 210 with different polarization characteristics and joined together, they can reconstruct different polarization images of the target scene respectively.

[0140] Taking the optical mask 210 including two mask layers as an example, they are named mask layer a and mask layer b respectively. Among them, mask layer a is a flexible film with an occlusion rate of 90%, which performs rolling shutter closing and opening along the vertical direction; mask layer b is a rigid film with an occlusion rate of 50% and polarization characteristics, which performs rolling shutter closing and opening along the horizontal direction. Mask layer a and mask layer b can be driven simultaneously during the same "closing - opening" operation, and perform closing and opening independently, or can be driven separately during different "closing - opening" operations.

[0141] For another example, the photomask 210 is composed of 4 mask layers, which are respectively named as mask layer S, mask layer R, mask layer G, and mask layer B. Among them, the mask layer S is a flexible film with a shielding rate of 90%, and it performs rolling shutter closing and opening along the vertical direction; the mask layer R, mask layer G, and mask layer B are rigid color filters with a shielding rate of 10% for red light, green light, and blue light respectively, and a shielding rate of 99% for other light, and they perform rolling shutter closing and opening along the vertical direction, horizontal direction, and horizontal direction respectively. Different mask layers can be driven during the same "closing - opening" operation and perform closing and opening independently; they can also be driven respectively during different "closing - opening" operations. Additionally, different mask layers can perform their corresponding "closing - opening" operations asynchronously. For example, during the opening and holding process of the mask layer S, the "closing - opening" operations of the mask layer R, mask layer G, and mask layer B are executed, and spectral image information of the scene can be obtained.

[0142] The photomask 210 can also have an optical structure. For example, using a diffraction grating with a certain shielding rate as the photomask 210, multi - spectral or hyperspectral image information of the scene can be obtained.

[0143] In the occlusion - area - type "closing - opening" operation, the occlusion area of the photomask 210 changes during the closing process and the opening process. At the same time, the shielding rate of the photomask 310 can also be set to different values during different processes of this "closing - opening" operation. The implementation of this different shielding rate can be carried out by the shielding rate control unit 220 under the drive of the total control unit 240 to change the shielding rate.

[0144] In the above part, for a "closing - opening" operation, the starting time point of its closing process is used as the starting time point of this "closing - opening" operation. In fact, the "closing - opening" operation is carried out cyclically. Although each parameter, including the total time length corresponding to the "closing - opening" operation, the time length corresponding to the closing process, the time length corresponding to the closing - holding process, the time length corresponding to the opening process, the time length corresponding to the opening - holding process, the closing method of the photomask 210, etc., may be different, different "closing - opening" operations can also be considered to occur cyclically. In this case, the starting time point of any "closing - opening" operation can also be taken as other time points during the closing process, closing - holding process, opening process, and opening - holding process, rather than necessarily taking the starting time point of the closing process as the starting time point of the "closing - opening" operation. At the same time, when the system is powered on, an additional system startup process may also be required.

[0145] The "close - open" operation can also be an operation in which the occlusion rate of the photomask 210 changes cyclically, named the occlusion - rate - type "close - open" operation. Its closing operation is called the occlusion - rate - type closing operation, its closing process is called the occlusion - rate - type closing process, its opening operation is called the occlusion - rate - type opening operation, and its opening process is called the occlusion - rate - type opening process. Specifically, under the drive of the occlusion - rate control unit 220, any occlusion - rate - type "close - open" operation includes a closing process in which the occlusion rate of the photomask 210 changes from the minimum occlusion rate in this closing operation to the maximum occlusion rate in this closing operation, and a closing - holding process in which it continuously remains at the maximum occlusion rate in this closing operation after the closing process is completed; and an opening process in which the occlusion rate of the photomask 210 changes from the maximum occlusion rate in this opening operation to the minimum occlusion rate in this closing operation, and an opening - holding process in which it continuously remains at the minimum occlusion rate in this closing operation after the opening process is completed. Similarly, in the same occlusion - rate - type "close - open" operation, the time lengths corresponding to different processes can be different; the minimum occlusion rates corresponding to its closing process and opening process can also be different; the occlusion - rate change in its closing process and the occlusion - rate change in its opening process can be uniform or non - uniform, and the occlusion areas corresponding to different processes can also change. An interruption time period can also be set between adjacent "close - open" operations. Similar processes of different "close - open" operations, such as two different closing processes, can have different corresponding time lengths, occlusion - rate magnitudes, occlusion - rate change rates, and occlusion areas of the photomask 210. Similarly, the photomask 210 can only contain one mask layer or can be composed of more than one mask layer. The photomask 210 with a changing occlusion rate can be implemented by a liquid - crystal device that regulates the light - passing rate by voltage or by any other device with a controllable occlusion rate.

[0146] In the actual process, an occlusion - rate - type "close - open" operation can be inserted into the occlusion - area - type "close - open" operation. For example, during the occlusion - area - type closing process, the occlusion rate of the photomask 210 is P1; during the closing - holding process after the closing process is completed, the total control unit 240 issues a command, and the occlusion - rate control unit 220 drives the photomask 210 to perform ≥1 occlusion - rate - type "close - open" operations. The photomask 210 implements a cyclic change process of small → large → hold → small → hold for the occlusion degree of the incident light on the dynamic vision sensor 10, and the occlusion rate P2 at the end is used as the occlusion rate of the photomask 210 during the occlusion - area - type opening operation process.

[0147] The relay device 30 can also be an array structure formed by arranging array units. For example, a one-dimensional aperture array formed by arranging slits as array units in one dimension, or a two-dimensional aperture array formed by arranging small holes as array units in two dimensions, or a one-dimensional cylindrical lens array formed by arranging cylindrical lenses as array units in one dimension, or a two-dimensional lens array formed by arranging lenses as array units in two dimensions. Such a relay device 30 is named an array-type relay device 30. Then, the reconstructed image of the data output by the dynamic vision sensor 10 can be used to reverse-infer the spatial 3D information through the corresponding array-type relay device 30 in reverse.

[0148] As Figure 7 shown, it is a schematic diagram of the system structure using an array-type relay device. Taking a one-dimensional cylindrical lens array with array units arranged in the x direction as an example, the working principle of the array-type relay device 30 is described. This one-dimensional cylindrical lens array is composed of array units L 1 , L 2 , L 3 , L 4 , …. The optical centers along the x direction are O 1 , O 2 , O 3 , O 4 , …. After passing through the array unit L 1 , the scene light information forms a corresponding sub-image I 1 on the dynamic vision sensor 10; after passing through the array unit L 2 , the scene light information forms a corresponding sub-image I 2 on the dynamic vision sensor 10, and so on. Then, the image reconstructed from the data output by the dynamic vision sensor 10 is formed by arranging multiple sub-images; from the reconstructed image, through the reverse inference of the beam transmission as Figure 7 shown, the reconstruction of the 3D information of the scene can be realized. Obviously, the one-dimensional cylindrical lens array can also be replaced by a two-dimensional lens array. When the array units are arranged in two dimensions, the adjacent array units of an array unit include the array units arranged in two dimensions and the diagonal direction. The one-dimensional aperture array and the two-dimensional aperture array are also the same as the array-type relay device.

[0149] Figure 7 In, the adjacent sub-images are shown not to overlap with each other. In fact, there may be overlap between adjacent sub-images. As Figure 8 shown, it is a system structure example of an array-type relay device where adjacent sub-images overlap. The sub-image I 1 obtained through the array unit L 1 occupies the D 1 D' 1 area on the dynamic vision sensor 10. The sub-image I 2 obtained through the array unit L 2, in the area occupied by the dynamic vision sensor 10 is D 2 D' 2 area, and so on. The sub-image I 1 and the sub-image I 2 there is an overlapping area D 2 D' 1 . At this time, by introducing the orthogonal characteristic design, the overlap between the adjacent sub-images can be avoided. At this time, along at least one direction, the adjacent O>1 array units of the array-type relay device 30 respectively have O orthogonal characteristics one by one, and each array unit only allows the light with the corresponding orthogonal characteristic to pass through, blocks or does not respond to the other (O-1) non-corresponding orthogonal characteristic lights, and the array units with the same orthogonal characteristic form an array unit sub-array. Set the basic detection units corresponding to each array unit, and only receive the light with the orthogonal characteristic corresponding to this array unit, block or do not respond to the other orthogonal characteristic lights, and the basic detection units corresponding to the same array unit sub-array form the basic detection unit sub-array corresponding to this array unit sub-array. The O basic detection unit sub-arrays and the O array unit sub-arrays form O imaging sub-structures one by one. Optionally, a polarizer matching the orthogonal characteristic of the corresponding array unit is arranged on the incident surface of the basic detection unit of the dynamic vision sensor 10, which is used to only allow the corresponding polarized light to pass through.

[0150] As Figure 9 shown, it is a schematic diagram of the principle of suppressing the overlap of adjacent sub-images by the orthogonal characteristic array-type relay device. Among them, O = 2 adjacent array units respectively only allow O = 2 orthogonal characteristic lights to pass through, and the basic detection units in the corresponding areas of each array unit on the dynamic vision sensor 10 also only receive the light with the corresponding orthogonal characteristic, and do not receive the non-corresponding orthogonal characteristic light incident, or do not respond to the non-corresponding orthogonal characteristic light. Here, the orthogonal characteristics corresponding to the adjacent O = 2 array units are respectively represented by "-" and "·", and only allow the "-" light and the "·" light to pass through respectively. Then, the aliasing between the sub-images corresponding to the adjacent array units can be removed. Specifically, taking the array unit L 2 as an example, only allow the "-" light to pass through and block the "·" light, and its corresponding area Z 2 on the dynamic vision sensor 10 only receives "-". The array unit L 2 adjacent to the array unit L 1 and L 3 , however, allow the "·" light to pass through and block the "-" light, and its corresponding area Z 2 on the dynamic vision sensor 10 only receives "·". Then, for the C 1 C 3 area of the scene, the "-" light projected by the array unit L 2 is detected and imaged by the area Z 2 ; but for the C 3 C2 Region via array unit L 2 The projected "-" light will be incident on the array unit L that does not receive the "-" light 1 Corresponding region Z 1 . Each array unit performs a similar operation, and the overlap between the sub-images obtained by adjacent array units can be avoided. At this time, the sub-image corresponding to an array unit may be only an image of a partial scene.

[0151] Figure 9 Taking only two orthogonal characteristics as an example, for more orthogonal characteristics, more adjacent array units can be designed to have different orthogonal characteristics. Figure 9 In, each array unit and its corresponding region on the dynamic vision sensor 10 are shown in a facing relationship; the corresponding region of an array unit on the dynamic vision sensor 10 can also be misaligned in the x direction. In addition, Figure 9 The adjacent array units in are shown as being arranged adjacently; in fact, an opaque region of a certain area can also be set between adjacent array units to restrict the light transmission aperture size of each array unit.

[0152] Figure 9 In, when the sub-array of the same array unit contains more than one array unit, the corresponding regions of adjacent array units on the dynamic vision sensor 10 are separated by the corresponding regions of the array units of other array unit sub-arrays. Another situation is that on the premise that adjacent O>1 array units of the array-type relay device 30 have O orthogonal characteristics one by one along any arrangement direction of the array units, the basic detection units on the dynamic vision sensor 10 are divided into O array unit sub-arrays arranged in an interleaved manner. Among them, along this any arrangement direction, the adjacent two basic detection units of the same basic detection unit sub-array are separated by (O-1) basic detection units. As Figure 10 shown, it is a schematic diagram of another application example of the orthogonal characteristic array-type relay device. Among them, taking O = 2 as an example, the detection units separated by (O-1) = 1 basic detection unit form a basic detection unit sub-array. That is, the basic detection unit p 1 , p 3 , p 4 , … form a basic detection unit sub-array, and the basic detection unit p 2 , p 4 , p 6 , … form another basic detection unit sub-array. At this time, the O imaging sub-structures and the shared controllable blinking module 20, respectively and independently, serve as an equivalent panoramic blinking event imaging system (similar to Figure 7 shown structure), and can perform imaging respectively.

[0153] A special case needs to be noted. Each sub-array of array units can also contain only one array unit. For example, in Figure 9 or Figure 10 , if two array units L1 and L2 are retained as two sub-arrays of array units respectively, the corresponding two imaging sub-structures can also perform imaging separately.

[0154] Figure 9 The "-" light and "·" light shown can be two linearly polarized lights with perpendicular polarization directions, or can represent being activated at different time periods respectively. In fact, the color characteristics of different wavelengths, the polarization characteristics of different polarization states, the time characteristics of allowing light to exit or enter at different time periods respectively, or their combined characteristics, or other various possible and distinguishable characteristics can all be used as the orthogonal characteristics. Among them, the color characteristics of each array unit and each basic detection unit can be achieved by attaching color filters. The polarization characteristics of each array unit and each basic detection unit can be achieved by attaching corresponding polarizers or wave plates. The time characteristics of each array unit can be achieved by attaching a corresponding controllable switch aperture, such as a liquid crystal aperture; the time characteristics of any basic detection unit can be achieved by whether to receive its optical information.

[0155] There is also a case where the array units are given O orthogonal characteristics, and each basic detection unit can receive the O orthogonal characteristic lights but can identify different orthogonal characteristic lights. Taking O = 3 colors red (R), green (G), and blue (B) as the orthogonal characteristics as an example, each basic detection unit is composed of O = 3 component layers stacked, and the O = 3 component layers are respectively based on detecting only the red (R), green (G), and blue (B) light information. At this time, different sub-arrays of basic detection units share basic detection units, but obtain different orthogonal characteristic light information corresponding to the shared basic detection units. That is, the R array unit sub-array corresponds to the R light detected by the shared basic detection unit, the G array unit sub-array corresponds to the G light detected by the shared basic detection unit, and the B array unit sub-array corresponds to the B light detected by the shared basic detection unit.

[0156] Figure 9 and Figure 10 The optical structure with the orthogonal characteristics shown can also be used in other image sensors besides the dynamic vision sensor 10, such as conventional cameras, CCDs, etc., to achieve image reconstruction. In this case, the controllable blinking module 20 can be removed. At this time, Figure 9 In the case shown, a baffle can also be set between different imaging sub-structures, such as Figure 9 the baffle shown, to replace the orthogonal characteristic design and suppress the mutual influence between different imaging sub-structures. Figure 9 and Figure 10 In the case, each sub-array of array units can also contain only one array unit.

[0157] The above array is an array structure formed by arranging array units, and can be used as a relay device 30 together with other optical devices, such as a lens or a lens group.

[0158] The basic detection units of the dynamic vision sensor 10 can also be divided into G>1 basic detection unit blocks arranged in a plane or a curved surface. There are no shared basic detection units between these G basic detection unit blocks at the same time point. The basic detection unit resolution, basic detection unit density, array arrangement pattern, physical properties, and synchronization of different basic detection unit blocks can be the same, not completely the same, or completely different.

[0159] The physical properties refer to color properties of different wavelengths or polarization properties of different polarization states. The basic detection unit blocks with different physical properties respectively only receive light with corresponding physical properties.

[0160] Embodiment 2

[0161] This embodiment proposes a binocular system, which is composed of 2 panoramic blink event imaging systems proposed in Embodiment 1 and arranged in a plane or a curved surface.

[0162] It can be understood that the system in this embodiment corresponds to the panoramic blink event imaging system in Embodiment 1 above, and the optional items in Embodiment 1 above are also applicable to this embodiment, so they will not be repeated here.

[0163] Embodiment 3

[0164] This embodiment proposes a multi-eye system, which is composed of more than 2 panoramic blink event imaging systems proposed in Embodiment 1 and arranged in a plane or a curved surface.

[0165] It can be understood that the system in this embodiment corresponds to the panoramic blink event imaging system in Embodiment 1 above, and the optional items in Embodiment 1 above are also applicable to this embodiment, so they will not be repeated here.

[0166] Embodiment 4

[0167] This embodiment applies the panoramic blink event imaging system proposed in Embodiment 1 and proposes a panoramic blink event imaging method, including the following steps:

[0168] S1. Determine the initial time window [T s , T e ; where T s is the starting time point of the closing process in the "closing-opening" operation, and T e is the ending time point of the opening and holding process in the "closing-opening" operation;

[0169] S2. Set the parameters related to the occlusion area or / and occlusion rate, and issue a command by the total control unit (240) to perform a corresponding "closing - opening" operation once;

[0170] S3. Select N≥2 time points {t 1 , t 2 , …, t N} within the current time window, and t N = T e . For any time point t n , use [(t n - Δt n ), t n as the sub - time window. For the events within the sub - time window, use the per - basic - detector - unit event integration algorithm to obtain the absolute light intensity value of each basic detector unit at the time point t n , and based on the absolute light intensity values of all basic detector units at the time point t n , obtain the light intensity image at the time point t n . Where 1≤n≤N, and Δt n ≤ (tn - Ts) is the preset sub - time window length;

[0171] S4. Use T e + δt as the starting time point T s of the closing process in the next "closing - opening" operation, and update the time window of the next "closing - opening" operation. Repeat steps S2 - S4 until the imaging task is completed to obtain the light intensity images corresponding to each time point. Where δt≥0 is the interruption duration.

[0172] Among them, for any "closing - opening" operation, for all events within its sub - time window [t n - Δt n , t n , use the per - basic - detector - unit linear event integration algorithm with formulas (1) and (2) to calculate the absolute light intensity value of each basic detector unit at the time point t n , and based on the absolute light intensity values of all basic detector units at the time point t n , obtain the light intensity image at the time point t n . Its expression is:

[0173]

[0174] H(t) =exp[α·(t i - t i-1 )] (3)

[0175] Among them, formula (1) represents the sub - time window [t n - Δt n , tn ] are integrated by basic detection unit event integration algorithm to obtain the time point t of each basic detection unit. n The absolute light intensity value; I represents the time point t n The intensity image of ε{e i} means in the sub-time window [t n -Δt n ,t n ] is a set of events triggered by changes in the shielding area or shielding rate of the inner light mask (210); Formula (2) is the update rule of the basic detection unit event integration algorithm; I(x, y, t i ) represents the coordinates (x, y) of the basic detection unit of the dynamic vision sensor (10) at the time stamp t i The absolute light intensity value, t i represents ε{e i}The i-th event e i Timestamp; p i represents the i-th event e i The event polarity; ΔC represents the preset constant threshold value of event triggering; Formula (3) is the expression of the response function H(t) of the event integration algorithm for each basic detection unit; α is the attenuation factor, and by adjusting the value of the attenuation factor α, Formula (2) becomes a linear event integration function.

[0176] Optionally, in each "closing-opening" operation, the occlusion rate of the photomask 210 can be reset or kept unchanged. The occlusion rate of the photomask 210 is preferably set so that the number of events triggered by the dynamic vision sensor 10 during the corresponding closing process and opening process changes with time close to a linear change.

[0177] It can be understood that the system of this embodiment corresponds to the panoramic blink event imaging system of the above-mentioned embodiment 1, and the options in the above-mentioned embodiment 1 are also applicable to this embodiment, so they will not be described repeatedly here.

[0178] Example 5

[0179] This embodiment applies the panoramic blink event imaging system proposed in Example 1 and the panoramic blink event imaging method proposed in Example 4, and is applied to static scenes where the panoramic blink event imaging system is stationary. Taking the occluded area type "close-open" operation as an example, the imaging method of a static background light intensity image based on the panoramic blink event imaging system is exemplarily described.

[0180] In this embodiment, the optical mask 210 is set to be rolled up and down in the vertical direction, with the maximum occlusion area being full occlusion and the minimum occlusion area being zero occlusion; the light shielding rate of the optical mask 210 is set to 95%; the occlusion area control unit 230 drives the optical mask 210 to perform one "closing-opening" operation, increasing uniformly from the current non-occlusion state to full occlusion, maintaining full occlusion, and then decreasing uniformly to the non-occlusion area, and maintaining the non-occlusion state, that is, the opening and maintaining process.

[0181] Based on the events obtained by the panoramic blink event imaging system in ≥1 consecutive "closing-opening" operations, determine the initial time window [T s , T e of one "closing-opening" operation; where T s is the starting time point of the closing process in the "closing-opening" operation, and T e is the ending time point of the opening and maintaining process in the "closing-opening" operation. Determine the opening time window [T s , T e within this time window [T Os , T Oe , that is, the time region corresponding to the opening process of one "closing-opening" operation.

[0182] Among them, the opening time window [T Os , T Oe is determined by curve fitting based on the event number distribution, including the following steps: starting from the opening and maintaining process of the previous "closing-opening" operation and ending at the closing process of the next "closing-opening" operation, within the time window of one "closing-opening" operation, through a numerical fitting algorithm, make the number of trigger events of the dynamic vision sensor 10 approach linearly with time during the closing process, closing and maintaining process, opening process, and opening and maintaining process; use the local inflection point search algorithm to find the inflection point where the curve changes from the closing and maintaining process to the opening process as the time point T Os , and the inflection point where the curve changes from the opening process to the opening and maintaining process as the time point T Oe ; the inflection point where the curve changes from the previous opening and maintaining process to the current closing process is used as the time point T s , and the inflection point where the curve changes from the current opening and maintaining process to the next closing process is used as the time point T e . The starting time point of the closing process of the current "closing-opening" operation and the starting time point of the closing process of the next "closing-opening" operation obtained by this fitting are respectively used as the time window [T s , T e of the current "closing-opening" operation.

[0183] It should be noted that the time point TOs , T Oe and T s , T e respectively correspond to the inflection points of the adjacent approximate linear stage. Different inflection point search methods will obtain similar but not necessarily equal values. The deviation within several milliseconds does not affect the reconstruction effect. The same applies to the time points obtained by searching for inflection points below and will not be elaborated further.

[0184] Within the open time window [T Os , T Oe , using the per-basic detection unit linear event integration algorithm, obtain the absolute light intensity value of each basic detection unit at time point T Oe , and based on the absolute light intensity values of all basic detection units at time point T Oe , obtain the static background light intensity image corresponding to time point T Oe .

[0185] Specifically, for the events triggered by the change in the occluded area of the optical mask 210 within the open time window [T Os , T Oe , use formulas (4), (5), and (6) for the per-basic detection unit linear event integration algorithm to obtain the absolute light intensity value of each basic detection unit at time point T Oe , and based on the absolute light intensity values of all basic detection units at time point T Oe , obtain the static background light intensity image corresponding to time point T Oe . Its expression is:

[0186]

[0187] H(t) =exp[α·(t i -t i-1 )] (6)

[0188] Among them, formula (4) represents performing the per-basic detection unit event integration algorithm on all events within the open time window [T Os , T Oe to obtain the absolute light intensity value of each basic detection unit at time point T Oe ; I represents the light intensity image at time point T Oe ; ε{e i} represents the set of events triggered by the change in the occluded area or the occlusion rate of the optical mask (210) within the open time window [T Os , T Oe ; formula (5) is the update rule of the per-basic detection unit linear event integration algorithm; I(x, y, t i ) represents the coordinates (x, y) of the basic detection unit of the dynamic vision sensor (10) at time stamp ti The absolute light intensity value, t i represents ε{e i} within the i-th event e i 's timestamp; p i represents the i-th event e i 's event polarity; ΔC represents a preset constant threshold for event triggering; Equation (6) is the expression of the response function H(t) of the per-basic detection unit event integration algorithm; α is an attenuation factor, and by adjusting the value of the attenuation factor α, Equation (5) becomes a linear event integration function.

[0189] As an exemplary illustration, in this embodiment, a schematic diagram of an example of a complete "close - open" operation performed by the area occlusion type optical mask 210 is as Figure 11 shown. Among them, Figure 11 (a) is a three-dimensional x - y - t coordinate schematic diagram of the generated event stream, where the x-axis and y-axis represent the resolution of the dynamic vision sensor, the z-axis represents events, and in the illustration, the red part represents positive events and the blue part represents negative events. This representation will be used for subsequent schematic diagrams of the event stream in this embodiment and will not be elaborated further; Figure 11 (b) is a schematic diagram of the opening time window [T Os , T Oe determined by piecewise linear fitting based on the event number distribution curve; Figure 11 (c) is a static background light intensity image obtained by resolving all events within this time window.

[0190] In each "close - open" operation, the occlusion rate of the optical mask 210 can be newly set or remain unchanged. The occlusion rate of the optical mask 210 is set such that the number of events triggered by the dynamic vision sensor 10 during the corresponding closing process and opening process changes linearly with time as much as possible.

[0191] Example 6

[0192] This embodiment applies the panoramic blink event imaging system proposed in Example 1 and the panoramic blink event imaging method proposed in Example 4 to a static scene where the panoramic blink event imaging system is stationary and there are moving objects in the scene. Taking the area occlusion type "close - open" operation as an example, an imaging method of the panoramic trajectory light intensity image based on the panoramic blink event imaging system is exemplarily described.

[0193] Exemplarily, it is set that the time window of the object movement in the static scene is in the opening and holding process of a "close - open" operation.

[0194] As an exemplary illustration, as Figure 12As shown, it is a schematic diagram of an example where there are moving objects in a static scene when the panoramic blinking event imaging system is stationary. Among them Figure 12 (a) is a schematic diagram of the event flow of this process; Figure 12 (b) corresponds to Figure 12 (a) The curve of the number of events changing with time; in this example, the local inflection point search algorithm is used to find the third inflection point of the curve as T Os , the fourth inflection point as T Oe , and the fifth inflection point as D s , then the time point s m > D s ; then for all events within the time period [T Os , s m , using the time-sharing per basic detection unit event integration algorithm of formulas (7) to (11), calculate the absolute light intensity value of each basic detection unit at the time point s m . The absolute light intensity values of all basic detection units are used as the light intensity image at the time point s m , where Figure 12 (c) is the background image reconstructed at the time point T Oe , Figure 12 (d) is the panoramic trajectory light intensity image reconstructed at the time point s m .

[0195]

[0196] H Θ (t) = exp[α·(t i - t i-1 )] (10)

[0197] H Ω (t) = exp[β·(t j - t j-1 )] (11)

[0198] Among them, I represents all events within the time period [T Os , s m . Using the time-sharing per basic detection unit event integration algorithm, calculate the absolute light intensity value of each basic detection unit at the s m time stamp. The absolute light intensity values of all basic detection units are used as the panoramic trajectory light intensity image at the time point s m ; ε Θ {e i} represents the events triggered by the change of the area blocked by the light mask 210 within the open time window [T Os , T Oe ; t i represents ε Θ {ei The timestamp of the i-th event e in { i ; I Θ (x, y, t i ) represents the absolute light intensity value of the basic detection unit with coordinates (x, y) at t i timestamp; p i represents the event polarity of the i-th event e i ; ε Ω (e j ) represents other events other than the events triggered by the change in the occluded area and the change in the occlusion rate of the optical mask 210 within the time period [D s , s m ; t j represents ε Ω (e j ) within the i-th event e j timestamp; I Ω (x, y, t j ) represents the absolute light intensity value of the basic detection unit with coordinates (x, y) at t j timestamp; p j represents the event polarity of the j-th event e j ; ΔC represents a preset constant threshold for event triggering;

[0199] And among them, formula (8) is the update rule of the per-basic detection unit linear event integration algorithm, where H Θ (t) represents the response function of the update rule of the per-basic detection unit linear event integration algorithm, specifically as shown in formula (10), α represents the attenuation factor, and adjusting this value makes formula (8) a linear event integration function;

[0200] And among them, formula (9) is the update rule of the per-basic detection unit decaying event integration algorithm, where H Ω (t) represents the response function of the update rule of the per-basic detection unit decaying event integration algorithm, specifically as shown in formula (11), where β represents the attenuation factor, and adjusting this value makes formula (6) a decaying event integration function; where α > β.

[0201] Exemplarily, it is set that the object movement in the static scene starts from the opening process in a "closing-opening" operation and continues in the opening holding process.

[0202] As an exemplary illustration, as Figure 13 shown, a schematic diagram of an example where the object movement in the static scene starts from the opening process in a "closing-opening" operation and continues in the opening holding process. Among them Figure 13 (a) is the event stream generated by this process; Figure 13 (b) corresponds to Figure 13(a) is a curve showing the change of the number of events over time; in this example, a local inflection point search algorithm is used to find the third inflection point of the curve as T Os , the fourth inflection point is T Oe , since the object movement starts from the opening process, here we take D s =T Oe , take S m >D s As the end time point of the reconstructed image; then for the time period [T Os ,s m ], using the time-sharing basic detection unit event integration algorithm of formula (7) to formula (11), calculate the time of each basic detection unit at time point s m The absolute light intensity value of all basic detection units is taken as the absolute light intensity value at time point s m The intensity image of Figure 13 (c) is the Oe Static background image reconstructed at a time point, Figure 13 (d) is the time at time s m Reconstructed panoramic trajectory intensity image.

[0203] As another example of this example, Figure 14 As shown in Figure 1, it is a schematic diagram of an example of the movement of a very small object in a static scene. Figure 14 (a) the event stream generated for the process; Figure 14 (b) corresponds to Figure 14 (a) is the curve of the number of events changing over time; Figure 14 (c) is the time point T Oe Reconstructed static background intensity image, Figure 14 (d) is the time at time s m Reconstructed panoramic trajectory light intensity image. From this example result, it can be shown that the system proposed by the present invention plays an important role in the motion monitoring of tiny objects.

[0204] Example 7

[0205] This embodiment applies the panoramic blink event imaging system proposed in Embodiment 1 and the panoramic blink event imaging method proposed in Embodiment 4, and is applied to a static scene where the panoramic blink event imaging system is in relative motion with the scene. Taking the occluded area type "close-open" operation as an example, the imaging method of the attenuated motion light field intensity image based on the panoramic blink event imaging system is exemplarily described.

[0206] In this embodiment, the generation of relative motion can be divided into two implementation schemes: active and passive. Among them, the active scheme is mainly used when the dynamic vision sensor 10 is in a static state. By implementing the "micro eye movement" described in Embodiment 1, relative motion with the scene is generated to obtain an event stream triggered by scene light information. The passive scheme is to place the dynamic vision sensor 10 on a moving vehicle as described in Embodiment 1, so as to generate relative motion with the scene and trigger an event stream.

[0207] Exemplarily, it is preferably to execute the active and passive relative motions of the panoramic blink event imaging system and the static scene during the process of keeping the optical mask 210 open to expand the field of view.

[0208] For the time period [D s , D e corresponding to the relative motion, that is, the moving light field time window of this embodiment, take K≥2 time points k 1 , k 2 , …, k k , …, k K from it. Among them, for any time point k k , with [(k k - Δk k ), k k as the corresponding sub-decayed moving light field time window, for other events outside the events triggered by the change of the occluded area of the optical mask 210 within this sub-decayed moving light field time window, here, it is the events triggered by relative motion, object motion, scene change, scene light intensity change, etc. caused by "micro eye movement". Use the formula (12) to (16) to obtain the absolute light intensity value of this basic detection unit at the time point kk through the basic detection unit decay event integration algorithm. From the absolute light intensity values of all basic detection units, it is used as the decayed moving light field light intensity image corresponding to the time point kk, where Δk k ≤ (k k - D s ), or, Δk k is less than or equal to the difference between the time point k k and the minimum time point covered by all time windows intersecting with the moving light field time window [D s , D e ; and the subscript k is a positive integer less than or equal to K. Among them, the characteristic of the decayed moving light field light intensity image is that for the basic detection unit without triggered events, the corresponding absolute light intensity value is multiplied by the corresponding decay factor as its new absolute light intensity value.

[0209] Its expression is:

[0210]

[0211] H(t) = exp[γ·(t j - t j-1 )] (14)

[0212]

[0213] where ε{e j} represents the set of other events outside the events triggered by the change in the occluded area or the change in the occlusion rate of the optical mask (210) within the time window [k k -Δk k , k k of the sub-decaying moving light field; I represents the use of the per-basic detection unit decay event integration algorithm for ε{e j} to calculate the absolute light intensity value of each basic detection unit at time point k k . The absolute light intensity values of all basic detection units are used as the moving light field light intensity image at time point k k ; (x, y) is the coordinate of the basic detection unit of the dynamic vision sensor 10, and t j represents the time stamp of the j-th event e j within ε{e j}, and I(x, y, t j ) represents the absolute light intensity value of the coordinate (x, y) of the basic detection unit of the dynamic vision sensor (10) at time stamp t j ; p j represents the event polarity of the j-th event e j ; ΔC represents the preset constant threshold triggered by the event; Formula (13) is the update rule of the per-basic detection unit decay event integration algorithm, where H(t) represents the response function of the integration algorithm, specifically as shown in Formula (14), γ represents the decay factor, and adjusting this value makes Formula (14) the decay event integration function; F is the decay mask, specifically as shown in Formula (16), and the decay mask value F(x, y) of the position of the basic detection unit with a triggering event within the time window [kk - Δkk, kk] of the sub-decaying moving light field is set to 1, and the decay mask value F(x, y) of the position of the basic detection unit without a triggering event is set to η, where 0 < η < 1; then the dot product operation of the decay mask F and I is performed to obtain the final decay moving light field light intensity image The above reconstruction method can effectively reduce the motion artifact effect caused by the motion of the dynamic vision sensor 10

[0214] Exemplarily, as Figure 15 shown, it is a schematic example diagram of the relative motion of the dynamic vision sensor with respect to the scene through the active "micro eye movement" mode. Among them, Figure 15 (a) is a partial event stream triggered by the scene light information during relative motionFigure 15 (b) is the time point k reconstructed from the event stream k are the intensity images of the decaying motion light field at the 8th ms, 64th ms, 240th ms, and 800th ms, where Δk at each time point k takes 8 ms as an example for all.

[0215] Exemplarily, as Figure 16 shown, a schematic example of placing a dynamic vision sensor on a car moving on a road and having relative motion with the scene. Among them, Figure 16 (a) is a partial event stream triggered by the scene light information during relative motion; Figure 16 (b) is the curve of the number of events corresponding to (a) changing with time; Figure 16 (c) is the time point k reconstructed from the event stream k are the intensity images of the decaying motion light field at the 100th ms, 1440th ms, 3200th ms, and 5600th ms, where Δk at each time point k takes 8 ms as an example for all.

[0216] From the above two examples, it can be seen that in the case of the proposed passive relative motion, in the pure event mode, the light information of the scene can be obtained, and at the same time, high-quality imaging can be achieved, thus maximizing the advantages of high dynamic range, low latency, and low power consumption of the event camera.

[0217] Furthermore, the intensity images of the decaying motion light field corresponding to each time point k k are stitched in the time dimension to form a motion light field video of the motion light field time window [D s , D e .

[0218] Example 8

[0219] This embodiment applies the panoramic blink event imaging system proposed in Example 1 and the panoramic blink event imaging method proposed in Example 4 to the case of a static scene and relative motion between the panoramic blink event imaging system and it. Taking the occlusion area type "close - open" operation as an example, the imaging method based on the intensity image of the motion light field of the panoramic blink event imaging system is exemplarily described.

[0220] In this embodiment, the dynamic vision sensor 10 is placed on a car moving on a road, so as to trigger an event stream due to relative motion with the scene, and at the same time, perform ≥1 occlusion area type "close - open" operation, and obtain a schematic example as Figure 17 shown. Among them, Figure 17 (a) is a partial event stream triggered by the scene light information during relative motion; Figure 17(b) is the curve of the number of events corresponding to (a) varying with time; Figure 17 (c) is at the time point s m The reconstructed intensity image of the moving light field.

[0221] Example 9

[0222] This example applies the panoramic blink event imaging system proposed in Example 1. Taking the occlusion area type "close - open" operation as an example, an event - based scene editing method based on the panoramic blink imaging system is exemplarily described. Its steps include:

[0223] i. Use the controllable blink module 20 to perform a "close - open" operation on a static background scene once. The events triggered during the opening process are denoted as ε{e i}, and the corresponding opening time window is denoted as [T Os , T Oe ;

[0224] ii. Stitch the event stream ε s , D e} triggered by the movement of the independent object u within the time period [D u {e i} and the corresponding time period [s us , s ue with the static background event stream ε{e i} in the time dimension, so that the time period [s us , s ue is converted into the moving light field time window [T Oe , t ue ;

[0225] iii. For the stitched time period [T Os , t ue and the stitched events ε{e i}+ε u {e i}, use the time - sharing and event - by - event basic detection unit integration algorithm: First, use the event - by - event basic detection unit linear event integration algorithm to calculate the static background intensity image at the time point T Os , T Oe in the time window [T Oe , T Oe . Then, based on the static background intensity image at the time point T Oe , t ue within the moving light field time window [T u1 , d u2 , …, d uM = t ue , corresponding to the time period [s us , sue M≥2 time points s within u1 , s u2 , …, s uM = s ue , where for any time point d um , with [d um -Δd um , d um as the corresponding sub - motion light field time window, and finally using the per - basic - detector - decay - event integration algorithm to update the absolute light intensity value of each basic detector at time point d um . The absolute light intensity values of all basic detectors are used as the light intensity image of the scene - editing motion light field corresponding to this time point d um , where 1≤m≤M;

[0226] Among them, the formula of the per - basic - detector linear event integration algorithm is:

[0227]

[0228] H 1 (t) = exp[α·(t i -t i-1 )] (20)

[0229] H 2 (t) = exp[β·(t r -t r-1 )] (21)

[0230]

[0231] Among them, t i represents the timestamp of the i - th event e i} within ε{e i , I O (x, y, t i ) represents the absolute light intensity value of the basic detector with coordinates (x, y) at timestamp t i , p i represents the event polarity of the i - th event e i ; ΔC represents the preset constant threshold for event triggering; t j represents the timestamp of the j - th event e i} within ε{e j , t r represents the timestamp obtained by converting t j through formula (22) after splicing two event streams, I m (x, y, t r ) represents the absolute light intensity value of the basic detector coordinates at timestamp t r , pr represents the r-th event e r of the event polarity;

[0232] Moreover, among them, formula (18) is the update rule of the linear event integration algorithm for each basic detection unit, where H 1 (t) represents the response function of the update rule of the linear event integration algorithm for the basic detection unit, specifically as shown in formula (20); α represents the attenuation factor, and adjusting the value of the attenuation factor α makes formula (18) a linear event integration function;

[0233] Moreover, among them, formula (19) is the update rule of the decaying event integration algorithm for each basic detection unit, where H 2 (t) represents the response function of the update rule of the decaying event integration algorithm for the basic detection unit, specifically as shown in formula (21), and β represents the attenuation factor, and adjusting the value of the attenuation factor β makes formula (19) a decaying event integration function.

[0234] Exemplarily, as Figure 18 shown, it is a schematic diagram of an application example of the event-based scene editing method. Among them, Figure 18 (a) is the event stream ε{e i} triggered by the controllable blinking module 20 performing an opening operation on a static background scene, and the time window of this process is denoted as [T Os , T Oe ; Figure 18 (b) is the event stream ε s , D e within the time period [D u {e i} and the corresponding time period [s us , s ue triggered by the movement of the independent object u; Figure 18 (c) is the event stream within the time period [TOs, tue] after splicing ε{e i} and ε u {e i} in the time dimension; Figure 18 (d) is the result of the final scene editing, that is, the scene editing moving light field intensity image reconstructed from this event stream at time points T Oe , t 1 , t 2 and t r (ms).

[0235] iv. Splicing the scene editing moving light field intensity images corresponding to each time point d um in the time dimension can constitute the time window [T Os , t ueThe motion light field video after scene editing;

[0236] v. The event stream ε triggered by the movement of U≥1 independent objects u {e i}, where 1≤u≤U, and perform scene editing one by one according to the above process to obtain the enhanced scene light intensity image at any time point within the splicing time period.

[0237] It can be seen from this example that by using the described event-based scene editing method, the event streams of different scenes can be edited and fused only in the time dimension, and this method is expected to play an important role in fields such as VR / AR and game rendering.

[0238] The occlusion area type light mask 210 described in Embodiments 5-9 can be replaced with an occlusion rate type light mask 210, that is, the occlusion rate type "close-open" operation is also applicable.

[0239] Each embodiment in the present invention is described in a progressive manner. The same or similar parts among the embodiments can be referred to each other, and the key points of each embodiment are the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, it is described relatively simply, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiments described above are only exemplary. The modules described as separate components may or may not be physically separated. When implementing the solution of the present invention, the functions of each module can be implemented in the same or multiple software and / or hardware. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the solution of this embodiment.

[0240] Obviously, the above embodiments of the present invention are only examples for clearly illustrating the present invention, rather than limiting the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation manners here. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A panoramic blink event imaging system, characterized in that: include: A dynamic vision sensor (10) is composed of an arrangement of basic detection units and is used to receive light information; each basic detection unit in the dynamic vision sensor (10) independently senses a change in received light intensity and triggers an event when the sensed light intensity change exceeds a preset threshold value; A controllable blink module (20) comprises a photomask (210), a blocking rate control unit (220) for adjusting the blocking rate of the photomask, a blocking area control unit (230) for adjusting the blocking area of ​​the photomask, and a general control unit (240); in the controllable blink module (20), the general control unit (240) issues a command once, and the blocking area control unit (230) or the blocking rate control unit (220) drives the photomask (210) to perform a "close-open" operation, that is, the photomask (210) implements a change process of small→large→maintain→small→maintain in the blocking degree of incident light of the dynamic vision sensor (10); The dynamic vision sensor (10) receives scene light via the controllable blink module (20), and the shielding rate of the light mask (210) in the controllable blink module (20) to its own incident light is always less than 100%.

2. The panoramic blink event imaging system according to claim 1, characterized in that: The "close-open" operation includes a closing operation and an opening operation; The closing operation includes a closing process in which the occlusion area of ​​the photomask (210) changes from a minimum occlusion area value in the current closing operation to a maximum occlusion area value in the current closing operation under the drive of the occlusion area control unit (230), and a closing and maintaining process in which the occlusion area value in the current closing operation is maintained at the maximum occlusion area value after the closing process is completed; this type of closing operation is named an occlusion area type closing operation, and the closing process included therein is named an occlusion area type closing process; Or, the closing operation includes a closing process in which the occlusion rate of the photomask (210) changes from a minimum occlusion rate value in this closing operation to a maximum occlusion rate value in this closing operation under the drive of the occlusion rate control unit (220), and a closing and maintaining process in which the occlusion rate value remains at the maximum occlusion rate value in this closing operation after the closing process is completed; this type of closing operation is named an occlusion rate type closing operation, and the closing process included therein is named an occlusion rate type closing process; Furthermore, the opening operation includes an opening process in which the occlusion area of ​​the photomask (210) changes from a maximum occlusion area value in the current opening operation to a minimum occlusion area value in the current opening operation under the drive of the occlusion area control unit (230), and an opening and maintaining process in which the occlusion area remains at the minimum occlusion area value in the current opening operation after the opening process is completed; this type of opening operation is named an occlusion area type opening operation, and the opening process included therein is named an occlusion area type opening process; Or, the opening operation includes an opening process in which the occlusion rate of the photomask (210) changes from a maximum occlusion rate in this opening operation to a minimum occlusion rate in this opening operation under the drive of the occlusion rate control unit (220), and an opening and maintaining process in which the occlusion rate remains at the minimum occlusion rate in this opening operation after the opening process is completed; this type of opening operation is named an occlusion rate type opening operation, and the opening process included therein is named an occlusion rate type opening process; The occlusion area type closing operation and the occlusion area type opening operation are combined into an occlusion area type "close-open" operation, and the occlusion rate type closing operation and the occlusion rate type opening operation are combined into an occlusion rate type "close-open" operation.

3. The panoramic blink event imaging system according to claim 2, characterized in that: The maximum value of the shading area of ​​the photomask (210) includes the maximum shading area value when the entire area is shading, or the shading area value when the entire area is shading; the minimum value of the shading area of ​​the photomask (210) includes the minimum shading area value when the entire area is shading, or the shading area value when there is no shading at all, that is, the shading area value is zero; The maximum shielding rate of the photomask (210) includes a maximum shielding rate of 100% shielding; the minimum shielding rate of the photomask (210) includes a non-zero minimum shielding rate, or a zero shielding rate.

4. The panoramic blink event imaging system according to claim 2, characterized in that: During the blocking area type closing process or the blocking area type opening process, the light mask (210) is closed or opened in a rolling curtain manner in one direction, or is closed and opened in a radially surrounding manner, or is closed and opened in a point-by-point or line-by-line scanning manner, or is closed and opened synchronously in the entire maximum blocking area, or various combinations of the above methods.

5. The panoramic blink event imaging system according to any one of claims 1 to 4, characterized in that: The system further comprises a relay device (30) for modulating and / or guiding scene light to be incident on the dynamic vision sensor (10); the dynamic vision sensor (10) receives the scene light via the controllable blink module (20) and the relay device (30).

6. The panoramic blink event imaging system according to claim 5, characterized in that: The photomask is composed of more than one mask layer; wherein each mask layer has different shielding rate characteristics, or different color filtering characteristics, or different area characteristics, or different polarization characteristics, or a combination of multiple characteristics.

7. The panoramic blink event imaging system according to claim 5, characterized in that: The event includes coordinate information of the triggered basic detection unit, or / and time information, or / and event polarity information; the preset threshold includes a preset absolute value of light intensity change, or a preset absolute value of logarithmic light intensity change, or an absolute value of a light intensity function used to enhance the sensitivity and accuracy of detecting light intensity changes and improve the dynamic contrast range of the perceived scene.

8. The panoramic blink event imaging system according to claim 5, characterized in that: The photomask (210) is placed on the incident path of the scene light to the dynamic vision sensor (10), including a basic detection unit attached to the dynamic vision sensor (10).

9. The panoramic blink event imaging system according to claim 5, characterized in that: The relay device (30) comprises a lens, or a lens group, or a diffractive optical device, or a diffractive optical device group, or a prism, or a reflector, or a polarizer, or a color filter, or various combinations of the above devices.

10. The panoramic blink event imaging system according to claim 9, characterized in that: The photomask (210) is imaged via the relay device on the surface where the basic detection unit of the dynamic vision sensor (10) is located, or is imaged at a position away from the dynamic vision sensor (10) along the transmission direction of the incident light.

11. The panoramic blink event imaging system according to claim 5, characterized in that: The system also includes an attenuation sheet (40) with an adjustable attenuation coefficient, which is placed on the incident light transmission path of the dynamic vision sensor (10) and is used to adjust the incident light flux of the dynamic vision sensor (10).

12. The panoramic blink event imaging system according to claim 5, characterized in that: An aperture stop (310) is introduced into the relay device (30), and its size is adjusted according to the scene light intensity, so that the entire system can work in a scene with a dynamic contrast range greater than the intrinsic dynamic contrast range of the dynamic vision sensor (10).

13. The panoramic blink event imaging system according to claim 5, characterized in that: The relay device (30) is an array-type relay device formed by arranging array units.

14. The panoramic blink event imaging system according to claim 13, characterized in that: The array-type relay device (30) comprises a one-dimensional aperture array composed of slits as array units arranged in a one-dimensional direction, or a two-dimensional aperture array composed of small holes as array units arranged in a two-dimensional direction, or a one-dimensional cylindrical lens array composed of cylindrical lenses as array units arranged in a one-dimensional direction, or a two-dimensional microlens array composed of microlenses as array units arranged in a two-dimensional direction.

15. The panoramic blink event imaging system according to claim 13, characterized in that: Along at least one direction, adjacent O>1 array units of the array-type relay device (30) have O orthogonal characteristics in a one-to-one correspondence; the array units are used to allow corresponding orthogonal characteristic light to pass through, and block or not respond to other non-corresponding (O-1) orthogonal characteristic light; wherein the array units having the same orthogonal characteristics constitute an array unit subarray; The basic detection unit corresponding to each array unit only receives the orthogonal characteristic light corresponding to the array unit, and blocks or does not respond to other orthogonal characteristic light, and the basic detection units corresponding to the same array unit subarray constitute the basic detection unit subarray corresponding to the array unit subarray; O basic detection unit subarrays and O array unit subarrays correspond one to one to form O imaging substructures; in the imaging substructure, any array unit subarray is composed of at least one array unit.

16. The panoramic blink event imaging system according to claim 15, characterized in that: The orthogonal characteristics include color characteristics of different wavelengths, or polarization characteristics of different polarization states, or time characteristics that allow light to be emitted or incident in different time periods, or mixed characteristics composed of a mixture of the above different characteristics.

17. The panoramic blink event imaging system according to claim 5, characterized in that: The dynamic vision sensor (10) comprises G>1 basic detection unit blocks; the basic detection unit blocks are arranged on a plane or a curved surface without sharing basic detection units at the same time point, and the basic detection unit resolution, basic detection unit density, array arrangement mode, physical properties, and synchronization of different basic detection unit blocks are the same, partially the same, or completely different.

18. The panoramic blink event imaging system according to claim 17, characterized in that: The physical properties include color properties of different wavelengths, or polarization properties of different polarization states.

19. An imaging system, characterized in that: More than one panoramic blink event imaging system as described in any one of claims 1 to 18 are arranged in a plane or a curve to form a binocular system or a multi-ocular system.

20. A panoramic blink event imaging method, using the panoramic blink event imaging system according to any one of claims 1 to 18, characterized in that: The following steps are involved: S1. Determine the initial time window of the "close-open" operation [T s ,T e ]; where T s is the starting time point of the closing process in the "close-open" operation, T e The end time point of the open holding process in the "close-open" operation; S2, setting parameters related to the shielding area and / or shielding rate, and the general control unit (240) issues a command to implement a corresponding "close-open" operation; S3, select N ≥ 2 time points {t1, t2, …, t N }, and t N =T e ; For any time point t n , with [(t n -Δt n ),t n ] is a sub-time window; for the events in the sub-time window, the basic detection unit event integration algorithm is used to obtain the event of each basic detection unit at time point t n The absolute light intensity value of all basic detection units at time point t n The absolute light intensity value at time point t n The intensity image of n ≤(tn-Ts) is the preset sub-time window length; S4, T e +δt is the starting time point T of the closing process in the next "close-open" operation s , update the time window for the next "close-open" operation; repeat steps S2 to S4 until the imaging task is completed, and obtain the light intensity image corresponding to each time point; where δt≥0 is the interruption duration.

21. The panoramic blink event imaging method according to claim 20, characterized in that: For any "close-open" operation, the sub-time window [t n -Δt n ,t n ], use formulas (1) and (2) to perform a linear event integration algorithm for each basic detection unit to calculate the time of each basic detection unit at time point t n The absolute light intensity value of all basic detection units at time point t n The absolute light intensity value at time point t n The light intensity image of Its expression is: H(t) =exp[α·(t i -t i-1 )] (3) Wherein, formula (1) represents the sub-time window [t n -Δt n ,t n ] are integrated by basic detection unit event integration algorithm to obtain the time point t of each basic detection unit. n The absolute light intensity value; I represents the time point t n The intensity image of ε{e i } means in the sub-time window [t n -Δt n ,t n ] a set of events triggered by a change in the shading area or shading rate of the inner light mask (210); Formula (2) is the update rule of the basic detection unit event integration algorithm; I(x, y, t i ) represents the coordinates (x, y) of the basic detection unit of the dynamic vision sensor (10) at the time stamp t i The absolute light intensity value, t i represents ε{e i }The i-th event e i Timestamp; p i represents the i-th event e i The event polarity; ΔC represents the preset constant threshold for event triggering; Formula (3) is an expression of the response function H(t) of the basic detection unit event integration algorithm; α is the attenuation factor, and by adjusting the value of the attenuation factor α, formula (2) becomes a linear event integration function.

22. The panoramic blink event imaging method according to claim 20, characterized in that: In any open time window [T Os ,T Oe ], the linear event integration algorithm of each basic detection unit is used to obtain the Oe The absolute light intensity value of all basic detection units at time point T Oe The absolute light intensity value at time point T Oe The corresponding static background light intensity image; Wherein, the opening time window [T Os ,T Oe ] is the time interval corresponding to the opening process of a "close-open" operation.

23. The panoramic blink event imaging method according to claim 22, characterized in that: In any "close-open" operation, the corresponding opening time window [T Os ,T Oe ]; The steps include: By using a numerical fitting algorithm, the number of events triggered by the dynamic vision sensor (10) changes with time, and approaches linearly in each process of the "close-open" operation, and obtains the starting time point T of the opening process in the opening operation. Os and end time point T Oe .

24. The panoramic blink event imaging method according to claim 22 or 23, characterized in that: Any open time window [T Os ,T Oe ], for the event triggered by the change of the shielding area or the change of the shielding rate of the photomask (210), a linear event integration algorithm for each basic detection unit is performed using formulas (4), (5) and (6) to obtain the value of each basic detection unit at the time point T Oe The absolute light intensity value of all basic detection units at time point T Oe The absolute light intensity value at time point T Oe The corresponding static background light intensity image; Its expression is: H(t) =exp[α·(t i -t i-1 )] (6) Wherein, formula (4) represents the opening time window [T Os ,T Oe ] are integrated by basic detection unit event integration algorithm to obtain the time point T of each basic detection unit. Oe The absolute light intensity value; I represents the time point T Oe The intensity image of ε{e i } indicates that when opening the time window [T Os ,T Oe ] a set of events triggered by a change in the shading area or shading rate of the inner light mask (210); Formula (5) is the update rule of the linear event integration algorithm for each basic detection unit; I(x, y, t i ) represents the coordinates (x, y) of the basic detection unit of the dynamic vision sensor (10) at the time stamp t i The absolute light intensity value, t i represents ε{e i }The i-th event e i Timestamp; p i represents the i-th event e i The event polarity; ΔC represents the preset constant threshold for event triggering; Formula (6) is an expression of the response function H(t) of the basic detection unit event integration algorithm; α is the attenuation factor, and by adjusting the value of the attenuation factor α, formula (5) becomes a linear event integration function.

25. The panoramic blink event imaging method according to claim 20, characterized in that: In any moving light field time window [D s ,D e ], take M≥2 time points {s1,s2,…,s M }, where for any time point s m , with [(s m -Δs m ),s m ] is a sub-motion light field time window; for other events in the sub-motion light field time window except for the event triggered by the change of the occlusion area or the change of the occlusion rate of the light mask (210), a basic detection unit attenuation event integration algorithm is used to update each basic detection unit at the time point s m The absolute light intensity value is calculated based on the time point s of all basic detection units. m The absolute light intensity value at time point s m The corresponding motion light field intensity image; where Δs m ≤(s m -D s ), or, Δs m Less than or equal to time point s m and intersect in the motion light field time window [D s ,D e ] is the difference between the minimum time points covered by all time windows of , and 1≤m≤M; The motion light field time window [D s ,D e ] is a time interval corresponding to other events other than events triggered by changes in the occlusion area or occlusion rate of the photomask (210) within a time interval covered by a time window or more than one adjacent time windows.

26. The panoramic blink event imaging method according to claim 25, characterized in that: In any moving light field time window [D s ,D e ], select K ≥ 2 time points {k1, k2, …, k K }, where for any time point k k , with [(k k -Δk k ),k k ] is a sub-attenuation motion light field time window; for other events in the sub-attenuation motion light field time window except for the event triggered by the change of the occlusion area or the change of the occlusion rate of the light mask (210), a basic detection unit attenuation event integration algorithm is used to update each basic detection unit at time point k k The absolute light intensity value of all basic detection units at time point k k The absolute light intensity value at time point k k The corresponding attenuated motion light field intensity image; Where Δk k ≤(k k -D s ), or, Δk k Less than or equal to time point k k and intersect in the motion light field time window [D s ,D e ] is the difference between the minimum time points covered by all time windows of , and the subscript k is a positive integer less than or equal to K.

27. The panoramic blink event imaging method according to claim 25 or 26, characterized in that: In any moving light field time window [D s ,D e ], dividing other events except the events triggered by the changes in the occlusion area or the occlusion rate of the light mask (210) into Q≥1 motion subsets; making any motion subset q correspond to the motion subset time window Different motion subsets independently use their own corresponding decay event integration algorithm and corresponding decay factor, where 1≤q≤Q.

28. The panoramic blink event imaging method according to claim 26, characterized in that: Any time period [T Os ,s m ], using the time-sharing basic detection unit event integration algorithm of formula (7) to formula (11), calculate each basic detection unit at time point s m The absolute light intensity value is calculated based on the time point s of all basic detection units. m The absolute light intensity value at time point s m The panoramic trajectory intensity image of is: H Θ (t)=exp[α·(t i -t i-1 )] (10) H Ω (t) =exp[β·(t j -t j-1 )] (11) Wherein, formula (7) represents the time period [T Os ,s m ], using the time-sharing basic detection unit event integration algorithm, the time point s m Panoramic trajectory intensity image I; ε Θ {e i } indicates that in the time period [T Os ,T Oe ] a set of events triggered by a change in the shading area or shading rate of the inner light mask (210); Ω (e j ) indicates that in the time period [D s ,s m ] A set of other events other than the events triggered by the changes in the shielding area and shielding rate of the inner light mask (210); Formula (8) is the update rule of the linear event integration algorithm for each basic detection unit, where I Θ (x,y,t i ) represents the coordinates (x, y) of the basic detection unit of the dynamic vision sensor (10) at the time stamp t i The absolute light intensity value, t i Represents ε Θ {e i }The i-th event e i Timestamp; p i represents the i-th event e i The event polarity; ΔC represents the preset constant threshold for event triggering; H Θ (t) represents the response function of the update rule of the linear event integration algorithm of the basic detection unit, as shown in formula (10); α is the attenuation factor, and the value of the attenuation factor α is adjusted to make formula (8) a linear event integration function; Formula (9) is the update rule of the basic detection unit decay event integration algorithm, where I Ω (x,y,t j ) represents the coordinates (x, y) of the basic detection unit of the dynamic vision sensor (10) at the time stamp t j The absolute light intensity value, t j Represents ε Ω (e j ) the i-th event e j The timestamp, p j represents the jth event e j The event polarity; H Ω (t) represents the response function of the update rule of the basic detection unit attenuation event integration algorithm, as shown in formula (11); β is the attenuation factor, and by adjusting the value of the attenuation factor β, formula (9) becomes the attenuation event integration function; and α>β.

29. The panoramic blink event imaging method according to claim 28, characterized in that: For ε Ω (e j ), for each motion subset time window When the corresponding event is updated with the absolute light intensity value by the basic detection unit event integration algorithm, different values ​​of the attenuation factor β are selected.

30. The panoramic blink event imaging method according to claim 25 or 26, characterized in that: In any moving light field time window [D s ,D e ], for other events in the sub-attenuated motion light field time window except for the event triggered by the change of the occlusion area or the change of the occlusion rate of the light mask (210), the attenuation event integration algorithm of each basic detection unit is obtained by using formulas (12) to (16) to obtain the attenuation event integration algorithm of each basic detection unit at time point k k The absolute light intensity value of all basic detection units at time point k k The absolute light intensity value at time point k k The corresponding attenuated motion light field intensity image; then, at each time point k k The corresponding attenuated motion light field intensity images are spliced ​​in the time dimension to form the motion light field time window [D s ,D e ]’s motion light field video; In the attenuated motion light field intensity image, for the basic detection unit without a trigger event, the corresponding absolute light intensity value is multiplied by the corresponding attenuation factor as its new absolute light intensity value; Its expression is: H(t)=exp[γ·(t j -t j-1 )] (14) Wherein, formula (12) represents the time period [k k -Δk k ,k k ] in the event set ε{e i }Use the basic detection unit event integration algorithm to calculate the time point k of each basic detection unit k The absolute light intensity value of all basic detection units at time point k k The absolute light intensity value at time point k k The motion light field intensity image I; ε{e j } represents the sub-attenuation motion light field time window [k k -Δk k ,k k ] a set of other events other than the event triggered by a change in the shading area or the shading rate of the photomask (210) within the image processing unit; Formula (13) is the update rule of the basic detection unit decay event integration algorithm, where I(x, y, t j ) represents the coordinates (x, y) of the basic detection unit of the dynamic vision sensor (10) at the time stamp t j The absolute light intensity value, t j represents ε{e j }The jth event e j Timestamp; p j represents the jth event e j The event polarity; ΔC represents the preset constant threshold value of the event trigger; H(t) represents the response function of the update rule of the linear event integration algorithm of the basic detection unit, as shown in formula (14); γ is the attenuation factor, and the value of the attenuation factor γ is adjusted to make formula (14) the attenuation event integration function; Formula (15) represents the final attenuated motion light field intensity image obtained by performing a dot operation on the attenuation mask F and the absolute light intensity value I: Among them, the time window of the attenuated motion light field [k k -Δk k ,k k ] The attenuation mask value F(x,y) of the basic detection unit position with a trigger event is set to 1, and the attenuation mask value F(x,y) of the basic detection unit position without a trigger event is set to η, and 0<η<1.

31. The panoramic blink event imaging method according to claim 30, characterized in that: For ε{e j }, for each motion subset time window When the corresponding event is updated with the absolute light intensity value by the basic detection unit event integration algorithm, different values ​​of the attenuation factors γ and η are selected.

32. The panoramic blink event imaging method according to claim 20, characterized in that: In any "close-open" operation, the occlusion rate control unit (220) sets the occlusion rate of the light mask (210) so that the number of events triggered by the dynamic vision sensor (10) in the corresponding closing process and opening process changes with time close to a linear change.

33. The panoramic blink event imaging method according to claim 20, characterized in that: In any shielding area type "closed-open" operation, under the lighting condition of external light intensity <50lux, the shielding rate control unit (220) sets the shielding rate of the light mask (210) to <100% and ≥90%, and names the interval as the high-level shielding rate; under the lighting condition of external light intensity ≥50lux and <2000lux, the shielding rate control unit (220) sets the shielding rate of the light mask (210) to <90% and ≥50%, and names the interval as the mid-level shielding rate; under the lighting condition of external light intensity ≥2000lux, the shielding rate control unit (220) sets the shielding rate of the light mask (210) to <50% and >0%, and names the interval as the low-level shielding rate.

34. A scene editing method, applied to the panoramic blink event imaging system of claims 1 to 19, characterized in that: The following steps are involved: A1. Using the controllable blinking module (20) to perform a "close-open" operation on the static background scene, the event triggered by the opening process is recorded as the static background event stream ε{e i }, the corresponding open time window is recorded as [T Os ,T Oe ]; A2. Set the motion light field time window [D s ,D e ] is triggered by the movement of an independent object u. u {e i } and the corresponding time period [s us ,s ue ] and the static background event stream ε{e i } is spliced ​​in the time dimension so that the time period [s us ,s ue ] is converted into a motion light field time window [T Oe ,t ue ]; A3. For the splicing time period [T Os ,t ue ] and splicing event ε{e i }+ε u {e i }, the time window [T Os ,T Oe ] Time point T Oe static background intensity image; then at time point T Oe Based on the static background intensity image, in the moving light field time window [T Oe ,t ue ], select M≥2 time points {d u1 ,d u2 ,…,d uM }, and d uM =t ue , one by one corresponding to the time period [s us ,s ue ] within M time points {s u1 ,s u2 ,…,s uM }, and s uM =s ue ; For any time point d um , with [d um -Δd um ,d um ] is the time window of the corresponding sub-motion light field; finally, the basic detection unit attenuation event integration algorithm is used to update each basic detection unit at time point d um The absolute light intensity value of all basic detection units at time point d um The absolute light intensity value at time point d um The corresponding scene edits the motion light field intensity image, where 1≤m≤M; A4、For each time point d um The corresponding scene editing motion light field intensity images are spliced ​​in the time dimension to form a time window [T Os ,t ue ] scene edited motion light field video; A5. Event stream ε triggered by the motion of U ≥ 1 independent object u {e i }, where 1≤u≤U, repeat steps A2 to A4 to perform scene editing to obtain an enhanced scene light intensity image at any time point within the splicing time period.

35. The scene editing method according to claim 34, characterized in that: The expression of the linear event integration algorithm for each basic detection unit is: H1(t)=exp[α·(t i -t i-1 )] (20) H2(t) =exp[β·(t r -t r-1 )] (21) Among them, t i represents ε{e i }The i-th event e i The timestamp of I O (x,y,t i ) indicates that the coordinates of the basic detection unit are (x, y) at timestamp t i The absolute light intensity value, p i represents the i-th event e i The event polarity; ΔC represents the preset constant threshold for event triggering; t j represents ε{e i }The jth event e j The timestamp, t r Indicates that after the two event streams are spliced, t j The timestamp obtained by conversion through formula (22) is m (x,y,t r ) represents the coordinates of the basic detection unit at timestamp t r The absolute light intensity value, p r Represents the rth event e r The polarity of events; And, wherein formula (18) is the update rule of the linear event integration algorithm for each basic detection unit, wherein H1(t) represents the response function of the update rule of the linear event integration algorithm for the basic detection unit, as shown in formula (20); α represents the attenuation factor, and the value of the attenuation factor α is adjusted so that formula (18) is a linear event integration function; And, wherein formula (19) is the update rule of the attenuation event integration algorithm for each basic detection unit, wherein H2(t) represents the response function of the update rule of the attenuation event integration algorithm of the basic detection unit, as shown in formula (21), β represents the attenuation factor, and the value of the attenuation factor β is adjusted so that the formula (19) is the attenuation event integration function.

Images