An event camera based flow field measurement method

Abstract

The application discloses a flow field measurement method based on an event camera, and comprises the following steps: generating a PIV data set, wherein each time sequence sample sequence comprises a plurality of frame-continuous particle images, a corresponding velocity vector field and particle event data at all time points; building a flow field data acquisition device based on the event camera and a high-speed camera, collecting time-synchronized real event data and real image data, adjusting parameters of an event simulator, verifying and updating the particle event data in the PIV data set according to the adjusted event simulator, and obtaining an updated PIV data set as a training data set; building an event camera optical flow method model and training the model by using the training data set; and based on the trained event camera optical flow method model, taking event sequences in two adjacent time periods as inputs of the model to calculate a velocity vector field at a corresponding intermediate time point. The application can obtain a required flow field velocity field at a required time point based on event data in a period of time.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of flow field visualization measurement, and particularly relates to a flow field measurement method based on an event camera. BACKGROUND

[0002] Particle Image Velocimetry (PIV) is a non-contact flow field measurement technology, mainly used for visualization and quantitative measurement of complex fluid motion, and plays a crucial role in fluid mechanics theory and experimental research. PIV technology first adds fluorescent tracer particles to the measured medium to follow the fluid motion, simultaneously uses laser illumination, and uses a digital camera to shoot continuous particle images, and then obtains the global velocity field of the fluid through a cross-frame algorithm.

[0003] Traditional particle image velocimetry technology usually needs to use a high-speed camera for shooting, however, the high-speed camera is costly, and at a higher resolution, the data volume of a single picture is very large, and the requirement for transmission bandwidth is extremely high, so at a high frame rate, real-time transmission cannot be achieved, and thus the memory is also required. The event camera has the advantages of low cost, data reduction, and large dynamic range, while the time resolution is high enough.

[0004] However, due to the difference in principle between the event camera and the traditional camera, the format of the event data and the image are completely different, and a corresponding algorithm needs to be specifically proposed. At present, the flow field measurement method based on the event camera is divided into two categories, one is particle tracking velocimetry, which needs to identify and track each particle in the event data, and has a higher requirement for particle density. The other is a kind of particle image velocimetry, which has a lower sensitivity to particle density, but at present, only one model-based algorithm has been proposed. This method first divides the event data into different rectangular regions according to pixels, and uses a motion compensation method to obtain a velocity vector in each rectangular region, which not only leads to a very high computational load, but also leads to a resolution of the estimated flow field much lower than the original resolution during data acquisition. In addition, due to the strong assumption that the optical flow is constant within the local space-time window, if the spatial non-uniformity and temporal non-stationarity of the optical flow are strong, significant errors will be caused. SUMMARY

[0005] The present application aims to solve the problems of high cost of traditional cameras, excessive data redundancy, and imperfect event camera flow field measurement methods in the prior art, and provides a flow field measurement method based on an event camera. The present application applies the event camera optical flow method to flow field measurement, while taking advantage of the low cost, information reduction, and real-time transmission of the event camera, it makes up for the shortcomings of the current event camera in flow field measurement, so that the event camera flow field measurement method can also output reliable dense flow field estimates like the traditional camera method.

[0006] The object of the present application is achieved by the following technical solution: a flow field measurement method based on an event camera, comprising the following steps:

[0007] (1) generating a particle image velocimetry data set, the particle image velocimetry data set comprising a particle image, particle event data and a corresponding velocity field label; each time sequence sample sequence in the particle image velocimetry data set comprises a plurality of frames of continuous particle images, a velocity vector field at the corresponding time and particle event data at all times;

[0008] (2) building a flow field data acquisition device based on an event camera and a high-speed camera, acquiring time-synchronized real event data and real image data to adjust the parameters of an event simulator, and verifying and updating the particle event data obtained in step (1) according to the adjusted event simulator to obtain an updated particle image velocimetry data set as a training data set;

[0009] (3) building an event camera optical flow method model and training the model using the training data set to obtain a trained event camera optical flow method model;

[0010] (4) based on event sequences in two adjacent time periods, using the trained event camera optical flow method model to calculate a velocity vector field at a corresponding intermediate time.

[0011] Further, step (1) specifically comprises the following sub-steps:

[0012] (1.1) using a particle image simulator to generate a first frame of particle set with random coordinates in a three-dimensional observation area to produce a particle image;

[0013] (1.2) according to the size of the flow field velocity, interpolating the flow field in time by T times and scaling the flow field velocity size to ; obtaining the next frame position of the first frame of particle set according to the integral trajectory of the current time step, generating a particle image in the imaging area, and resetting the particle position outside the imaging area; repeating the particle movement, imaging and resetting of each time step; saving the current real velocity vector field once every T time steps, multiplying the current flow field velocity size by T to scale it back to the original size when saving; obtaining the time-interpolated time sequence particle image and the corresponding real velocity vector field;

[0014] (1.3) based on the time-interpolated time sequence particle image, using an event simulator to simulate the particle event data generated by the particle movement in the entire process;

[0015] (1.4) packing the particle event data, the real velocity vector field and several frames of particle images with the same timestamp of the velocity vector field into a time sequence sample sequence to generate a particle image velocimetry dataset; wherein the velocity vector field is the velocity field label.

[0016] Further, the particle image simulator is used to generate a particle image, wherein the shape of each particle in the particle image in space satisfies a three-dimensional Gaussian distribution, expressed as:

[0017]

[0018] In the formula, represents the three-dimensional space coordinates in the particle image, represents the gray intensity of a single particle, represents the three-dimensional space coordinate position of the particle center, represents the peak intensity value of the particle center, represents the diameter of the particle.

[0019] Further, the event simulator is used to simulate the particle event data of the particle motion based on the time sequence particle image, using the following model of the event simulator:

[0020]

[0021] In the formula, represents the voltage change on the event camera pixel circuit, represents the brightness value considered as a constant in the time period, represents the linear change speed of the local brightness, represents the time interval since the last event trigger, and t represents the timestamp of the current event trigger, represents the timestamp of the last event trigger, is a random term represented by a Wiener process, is a calibration parameter.

[0022] Further, the flow field data acquisition device comprises a sheet-shaped continuous laser source, a control system, an external synchronous triggering device, a high-speed camera, an event camera, a beam splitter and a transparent box body, wherein the sheet-shaped continuous laser source is located on the side of the transparent box body, the control system is in communication connection with the sheet-shaped continuous laser source, the external synchronous triggering device, the high-speed camera and the event camera respectively, the external synchronous triggering device is connected with the high-speed camera and the event camera respectively, and the high-speed camera, the event camera and the beam splitter are all located on the front of the transparent box body, and the optical axes of the high-speed camera and the event camera are aligned with the center of the beam splitter.

[0023] Further, in step (2), the updated particle image velocimetry dataset is obtained, specifically including:

[0024] Particles are implanted into the flow field to be tested in a transparent box. The flow field to be tested is illuminated by a sheet-like continuous laser source. The field of view of the event camera and the high-speed camera is kept consistent by using a beam splitter, controlling the focal length of the lens and the size ratio of the photosensitive element, and calibrating the binocular camera. An external synchronous triggering device is used to synchronously trigger the event camera and the high-speed camera to synchronously acquire real event data and real image data of particles in the flow field to be tested.

[0025] The parameters of the event simulator are adjusted according to the correspondence between real event data and real image data. The particle event data in the particle image velocimetry dataset is then verified using the adjusted event simulator to ensure that the generated particle event data is the same as the corresponding real event data. If there are any discrepancies, the particle image is simulated again using the adjusted event simulator to obtain simulated event data to replace the original particle event data and update the particle image velocimetry dataset.

[0026] Furthermore, the input to the event camera optical flow model is two adjacent three-dimensional voxel grids, each of which is obtained by preprocessing simulated event data within a corresponding time period; the three-dimensional voxel grids are then used via attention-based events... A feature extraction network with a fusion mechanism and shared weights extracts particle features at different time intervals, resulting in feature sequences for different time intervals. Based on these feature sequences, cross-correlation calculations are performed on the features at each time interval of consecutive event frames, yielding cross-correlation volumes for different time intervals. Similarity matrix sequences for different time intervals are obtained from these cross-correlation volumes and input together with the estimated flow sequence from the previous time step into the motion feature encoding layer, resulting in motion feature sequences for different time intervals. These motion feature sequences are then cross-fused using a cross-attention mechanism to obtain fused motion features. A context feature extraction network extracts context features from the current 3D voxel grid and inputs them, along with the fused motion features, into a gated iterative update unit to obtain the updated hidden state. The updated hidden state is then input into the optical flow calculation module to obtain the estimated flow sequence for the current time step. Finally, an upsampling module outputs the velocity vector field estimate for the current time step, and the velocity vector field output in the last iteration is the final velocity vector field estimate.

[0027] Furthermore, in step (3), training is performed using a training dataset, specifically including:

[0028] The simulation event data in the training dataset is preprocessed, that is, according to the frequency of the velocity vector field of the flow field to be measured, the simulation event data is divided into event sequences of equal duration and made into a three-dimensional voxel mesh.

[0029] two three-dimensional voxel grids adjacent to each other are taken as inputs of the event camera optical flow method model, and an estimated value of a velocity vector field at an intermediate time between the two three-dimensional voxel grids is output;

[0030] a training loss function is calculated based on the estimated value of the velocity vector field and a corresponding velocity field label in the training data set;

[0031] With minimizing the training loss function as an optimization goal, parameters of the event camera optical flow method model are optimized through back propagation and using an adaptive moment estimation calculation method until a preset training round is reached, so as to obtain a trained event camera optical flow method model.

[0032] Further, a calculation formula of the training loss function is:

[0033]

[0034] In the formula, denotes the training loss function, denotes the estimated value of the velocity vector field in the i th iteration, denotes the velocity field label in the training data set, denotes the L1 distance, denotes the weight of the error in the i th iteration.

[0035] The present application has the following advantages: the present application breaks through the hardware limitation of the traditional high-speed camera by introducing the event camera optical flow method; compared with the prior art, the device cost is reduced by more than 60% while maintaining sub-pixel level measurement accuracy, and the high dynamic range characteristic is provided, which significantly reduces the sensitivity to light conditions; the event camera optical flow method model is designed in cooperation with a multi-scale feature extraction network and a motion feature coding module, so that large-scale dominant flow (such as vortex structure) and small-scale turbulent pulsation characteristics in the field of view can be captured at the same time, the unique advantages of the microsecond time resolution of the event camera are fully utilized, and the shortcomings of the prior art event camera flow field measurement method in the spatial scale coverage range are made up; the event camera has the advantages of low camera cost, high time resolution, and low information transmission bandwidth requirement. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is an implementation step flowchart of the flow field measurement method based on the event camera of the present application;

[0037] Figure 2 is a generation process schematic diagram of the PIV data of the present application; wherein, Figure 2 (a) in (a) is a previous frame particle set P; Figure 2 (b) in (b) is a known velocity vector field V; Figure 2 (c) in (c) is a next frame particle set Q generated after moving the particle set P using the known velocity vector field V; Figure 2(d) in the (d) is a particle event generated when the particle moves;

[0038] Figure 3 is a structural schematic diagram of the event camera and high-speed camera based flow field data acquisition device of the present application;

[0039] Figure 4 is a structural flow chart of the event camera optical flow method model of the present application.

[0040] In the figure, the sheet-shaped continuous laser source 1, the control system 2, the external synchronization trigger device 3, the high-speed camera 4, the event camera 5, the beam splitter 6, and the transparent box 7. DETAILED DESCRIPTION

[0041] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements throughout the description. The following exemplary embodiments are not intended to represent all implementations consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0042] The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the present application and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0043] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is to be further understood that the term "or" as used herein encompasses both exclusive and inclusive or unless otherwise indicated herein. It is to be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0044] The present application will be described in detail below with reference to the attached drawings. The features of the embodiments and implementation described below can be combined with each other as long as there is no conflict.

[0045] Referring to Figure 1 The event camera based flow field measurement method of the present application specifically comprises the following steps:

[0046] (1) generating a PIV dataset, the PIV dataset comprising particle images, particle event data and corresponding velocity field labels; each time sequence sample sequence in the PIV dataset comprises a plurality of frames of continuous particle images, a velocity vector field corresponding to the time and particle event data with timestamps accurate to microseconds at all time points; wherein the velocity vector field is the velocity field label.

[0047] In this embodiment, the event data of a large number of moving particles in a three-dimensional space and the corresponding velocity field labels (i.e. the velocity vector field corresponding to the time) are artificially synthesized for subsequent training of the event camera optical flow method model; in addition, particle images with the same frame rate as the velocity field labels are generated for reference. The PIV dataset is composed of a plurality of time sequence samples, each time sequence sample corresponds to a set of particle images with a frame rate of 100 fps, a set of N velocity field labels and a set of particle event data generated by particle motion between the timestamp of the first particle image and the timestamp of the Nth particle image. a set of N velocity field labels and a set of particle event data generated by particle motion between the timestamp of the first particle image and the timestamp of the Nth particle image . .

[0048] In this embodiment, a PIV dataset is generated, and the generation process of the PIV dataset is shown in Figure 2 , which specifically includes the following sub-steps:

[0049] (1.1) using a particle image simulator to generate a first frame particle set P with random coordinates in a three-dimensional observation region, as shown in Figure 2 (a), to generate a particle image. The three-dimensional observation region has a size of 300x300x4 pixels, and the first frame particle set P uniformly distributed in the three-dimensional observation region contains more than twenty thousand randomly distributed particles. The imaging region is defined at a specified position of a computational fluid dynamics (CFD) calculation domain, for example, a region of 256x256x1 pixels located at the center of the three-dimensional observation region can be defined as the imaging region, and the size of the imaging region is 256x256x1 pixels.

[0050] It should be understood that the size of the three-dimensional observation region and the imaging region and the size of the flow field velocity can be defined according to requirements. For example, the three-dimensional observation region can be defined as a cuboid region between the coordinate origin [1, 1, 1] and [300, 300, 5], and the imaging region can be defined between [23-278, 23-278, 2.5-3.5], or the imaging layer thickness can be appropriately increased, so that the frequency of disappearance of particles in the particle image sequence in the Z axis is reduced. ​​

[0051] In addition, to ensure the diversity of the data set, different data samples have different parameters, including particle diameter (2-4 voxels), displacement ratio (0.35-6.58), particle peak gray value (200-255), particle imaging layer thickness (1-1.2), and Gaussian blur of particle diameter and intensity (0-0.1), wherein the particle imaging layer thickness and the z-axis velocity of the flow field jointly determine the frequency of the particles entering and leaving the imaging layer on the z-axis, and further determine the particles.

[0052] Further, the particle image simulator is used to generate a particle image, wherein the morphology of each particle in the space in the particle image satisfies a three-dimensional Gaussian distribution, and is expressed as:

[0053]

[0054] In the formula, represents a three-dimensional spatial coordinate in the particle image, represents a gray intensity of a single particle, represents a three-dimensional spatial coordinate position of a particle center, represents a peak intensity value of the particle center, represents a diameter of the particle; wherein the position of each particle per frame is determined by the position of the previous frame and a velocity vector field between every two frames of particle images, and the velocity vector field is the saved velocity field label.

[0055] (1.2) According to the size of the flow field velocity used, the flow field is interpolated in time by T times, and the flow field velocity size is scaled to , so as to ensure that the maximum value of the flow field velocity of each time step is below 0.4 pixels; through experiments, under this condition, the particle event data generated according to the particle image sequence reaches saturation, and therefore, the flow field velocity size is scaled to , so as to ensure that the maximum value of the flow field velocity of each time step is below 0.4 pixels. The next frame position of the first frame particle set P is obtained according to the integral trajectory of the current time step, that is, the second frame particle set Q, as shown in (c) of FIG. 1B, a particle image is generated in the imaging area, and the particle position outside the imaging area is reset, so as to ensure that the imaging area is uniformly covered with particles when the particles are simulated to move at each time step; then the particle movement, imaging and resetting of each time step are repeated; every T time steps, the current true velocity vector field V is saved, and the current flow field velocity size is multiplied by T to scale back to the original size, as shown in (b) of FIG. 1B. Thus, the time-interpolated Figure 2 particle image and the corresponding N true velocity vector fields can be obtained. Figure 2

[0056] ​​Further, the velocity vector field V can be a variety of fluid motion scenarios, including but not limited to: three-dimensional uniform flow field, Beltrami flow field, incompressible isotropic turbulence, magnetohydrodynamic turbulence, turbulent channel flow, transitional boundary layer flow, etc. Figure 2 The velocity vector field shown in (b) of FIG. 1 is a three-dimensional uniform flow field.

[0057] (1.3) Based on the time-interpolated velocity vector field V obtained in step (1.2), the time-sequenced particle image sequence is generated. The first particle image in the time-sequenced particle image sequence is simulated using the event simulator, and all particle event data generated by particle movement between the time stamp of the first particle image and the time stamp of the last particle image in the time-sequenced particle image sequence are simulated. The first particle image in the time-sequenced particle image sequence is simulated using the event simulator, and all particle event data generated by particle movement between the time stamp of the first particle image and the time stamp of the last particle image in the time-sequenced particle image sequence are simulated. Figure 2 As shown in (d) of FIG. 1, a pixel that becomes dark due to the departure of a particle generates a -1 event, and a pixel that becomes bright due to the arrival of a particle generates a +1 event.

[0058] Further, the event simulator is used to generate particle event data based on the time-sequenced particle image sequence, and the particle motion is simulated using the following model of the event simulator:

[0059]

[0060] In the case of only considering the randomness of photon reception, the voltage change on the event camera pixel circuit is modeled as follows: It is assumed that the local brightness changes linearly at a constant speed in a short time; is the brightness value considered constant in a short time; is the time interval since the last event trigger, t represents the time stamp of the current event trigger, represents the time stamp of the last event trigger; is a Wiener process, representing a random term; is a calibration parameter. Specifically, the particle event data is generated by the event simulator based on the time-sequenced particle image sequence. In the case of only considering the randomness of photon reception, it is assumed that the local brightness changes linearly at a constant speed in a short time,

[0061] The current change of the light signal conversion can be represented as follows:

[0062]

[0063] where, is the conversion rate of the light signal to the current, is a random term mainly caused by the randomness of photon reception, which can be represented using a Wiener process as follows: ​​

[0064]

[0065] where, represents the Wiener process, is a constant parameter, is the luminance value considered constant in a short time. Substituting into the ideal model of the voltage variation of the event camera pixel circuit, we get:

[0066]

[0067] and using the first-order Taylor approximation After that, the voltage variation can be modeled as:

[0068]

[0069] where, are circuit element parameters, is the thermal voltage, is the photo current, is the dark current in the absence of light.

[0070] Considering the noise caused by the leakage current, the corresponding voltage variation is:

[0071]

[0072] where, represents the voltage variation caused by the leakage current, and are the event camera constant parameters to be calibrated, , , is the activation energy, is the Boltzmann constant, is the coefficient of the Wiener process assuming that the noise term of the leakage current is white noise, which is a constant parameter to be calibrated. Add and to model, at the same time, is replaced by because it is proportional to , and the coefficient in it is replaced by the calibration parameter , so the model of the event simulator is obtained.

[0073] (1.4) Pack the particle event data obtained in step (1.3), the real velocity vector field obtained in step (1.2), and N+1 frames of particle images with the same timestamp as the velocity vector field into a time sequence sample sequence to generate a PIV data set.

[0074] (2) Build a flow field data acquisition device based on the event camera and the high-speed camera, collect time-synchronized real event data and real image data, adjust the parameters of the event simulator, and verify and update the particle event data obtained in step (1) according to the adjusted event simulator, and obtain the updated PIV data set as the training data set for the subsequent training of the event camera optical flow method model.

[0075] It should be noted that the flow field data acquisition device based on the event camera and the high-speed camera is used to collect real event data and real image data, and the parameters of the event simulator are calibrated using the real event data and the real image data. At the same time, the effectiveness of the data in the PIV data set simulated in step (1) is verified, and when it is invalid, the event simulator with adjusted parameters is used for simulation again to generate simulated event data.

[0076] In this embodiment, as shown in Figure 3 , the flow field data acquisition device includes a sheet-shaped continuous laser source 1, a control system 2, an external synchronization triggering device 3, a high-speed camera 4, an event camera 5, a beam splitter 6, and a transparent box 7. The sheet-shaped continuous laser source 1 is located on the side of the transparent box 7, the control system 2 is in communication connection with the sheet-shaped continuous laser source 1, the external synchronization triggering device 3, the high-speed camera 4, and the event camera 5, the external synchronization triggering device 3 is connected with the high-speed camera 4 and the event camera 5, the optical axes of the high-speed camera 4 and the event camera 5 are aligned with the center of the beam splitter 6, and the high-speed camera 4, the event camera 5, and the beam splitter 6 are located on the front of the transparent box 7.

[0077] Specifically, when the flow field data acquisition device is used for data acquisition, the control system 2 controls the data acquisition of the flow field data acquisition device. First, light-reflecting lightweight particles with uniform size and appropriate diameter are uniformly implanted in the flow field to be measured in the transparent box 7, and the sheet-shaped continuous laser source 1 is used to illuminate the flow field to be measured on the side of the transparent box 7; adjust the experimental parameters to ensure that the particle density, average particle intensity, and particle diameter range of the PIV data set and the collected data are consistent. By using the beam splitter 6, controlling the focal length and the size ratio of the photosensitive element, and binocular camera calibration, the fields of view of the event camera 5 and the high-speed camera 4 are ensured to be consistent. Specifically, as shown in Figure 3 , the high-speed camera 4, the event camera 5, and the beam splitter 6 are placed on the front of the transparent box 7, the optical axes of the high-speed camera 4 and the event camera 5 are aligned with the center of the beam splitter 6 to ensure that the center of the field of view is consistent, and the distance from the illumination plane to the photosensitive element of the high-speed camera 4 and the event camera 5 is controlled. Then, according to the pixel size of the high-speed camera 4 and the event camera 5, the appropriate focal length lens is selected after calculation by the formula , to ensure that the fields of view of the high-speed camera 4 and the event camera 5 are basically the same, where represents the field of view angle, Indicates the size of the photosensitive element. The focal length of the lens is indicated. A calibration board is used to perform dual-target calibration on high-speed camera 4 and event camera 5, achieving pixel-level alignment between them. Then, an external synchronization triggering device 3 is used to synchronously trigger both high-speed camera 4 and event camera 5. Event camera 5 acquires real event data of reflective particles in the flow field under test, while high-speed camera 4 acquires real image data of the same field of view. This ensures that the acquired event data and particle image data are synchronized in time. After obtaining the synchronized real image data and real event data in time steps, the parameters of the event simulator are calibrated according to the correspondence between the real image data and the real event data, and the parameters of the event simulator are adjusted. Then, the adjusted event simulator is used to verify the particle event data in the PIV dataset obtained in step (1) to verify its validity, so as to ensure that the particle event data in the PIV dataset generated in step (1) is the same as the event data in the actual collected data under the same number of particles and the same average motion velocity. If there is unreasonable or different event data, the adjusted event simulator is used to re-simulate the particle image to obtain simulated event data. Finally, the simulated event data obtained from the re-simulation is used to replace the particle event data in the PIV dataset generated in step (1), and the updated PIV dataset is used as the training dataset for the subsequent training of the event camera optical flow model.

[0078] Furthermore, adjusting the parameters of the event simulator At that time, the event simulator model can be used. In summary, it includes drift parameters. and diffusion parameters Brownian motion is represented as: Subsequently, based on the actual collected real image data and real event data, each pair of adjacent image frames was analyzed. and Between events, approximate brightness is extracted at each pixel of the particle image. Brightness change , This represents the brightness of pixel i, and the time interval between adjacent events of the same pixel. Assuming Following a Levy distribution, the fitting parameters are estimated using maximum likelihood estimation. and Using multiple groups Data, through regression calculation Furthermore, manual adjustments are required due to image quality limitations. This is to make the generated data more closely resemble the real data. and Then, using an efficient strategy, the simulation of event generation is completed by alternately executing two parts: determining the event polarity based on the hit probability of Brownian motion and determining the event trigger time based on the first hit time distribution. The probability of a positive polarity event is:

[0079]

[0080] In the formula, It is the probability of producing a positive event. It is the threshold for generating positive polarity events. It is the threshold for generating negative polarity events. and The participants generated during the calculation process each time the polarity of an event is determined. The decision will not be elaborated upon here. Once the complete event simulator with adjusted parameters is obtained, it can be used according to... arrive Temporal particle images between time periods generate simulation event data.

[0081] In this embodiment, the updated PIV dataset contains 60 time-series sample sequences with time steps of 100 or 200, including 8000 velocity vector fields and corresponding particle images, as well as simulation event data of particle motion with a total duration of 80s.

[0082] (3) Build an event camera optical flow model and train it using the training dataset obtained in step (2) to obtain a trained event camera optical flow model.

[0083] It's important to clarify that, from an image perspective, optical flow abstracts object motion into the displacement of each corresponding pixel between two frames; this abstracted displacement field is the optical flow field. Similarly, from an event perspective, events also fall on pixels, and the displacement of the corresponding pixels is also called optical flow, thus allowing the application of optical flow methods. The calculation of the velocity vector field using optical flow is based on the assumption of a constant velocity field over an extremely short time. That is, during the time used to calculate the velocity vector field, the velocity vector field within the entire field of view remains constant. Therefore, the magnitude of the optical flow divided by the time interval gives the velocity magnitude. Thus, optical flow estimation is essentially velocity vector field estimation.

[0084] In this embodiment, the optical flow model for the event camera is constructed, which can be represented as follows: ,in This represents event data of equal duration within two adjacent time periods. This represents the velocity vector field estimate at an intermediate time point from the output of the event camera optical flow model. This represents the mapping function of the event camera optical flow model. The architecture of the event camera optical flow model is as follows: Figure 4As shown, the input of the event camera optical flow model is two adjacent three-dimensional voxel grids, where each three-dimensional voxel grid is obtained by preprocessing the simulation event data in the corresponding time period, i.e., by preprocessing the event sequence into a three-dimensional voxel grid wherein is the time period, is the number of two-dimensional voxel grids in the time period, the three-dimensional voxel grid is an implementation form of the event frame, and the data obtained by processing the events in a certain time period through a certain process and input into the event camera optical flow model for operation is called an event frame. Using the three-dimensional voxel grid, the particle features are extracted on different time intervals through the attention-based event fusion mechanism, and a feature sequence is obtained, in which each pair of features corresponds to a previous event frame and a subsequent event frame, and the order of the feature pairs corresponds to different time intervals , different time intervals refer to that a time period corresponding to a complete three-dimensional voxel grid is equally divided into N time intervals, and the time period with a length of N is the N different time intervals, then the features on the time intervals of the previous and subsequent event frames are pairwise correlated to obtain a cross-correlation body sequence of different time intervals , and then the estimated flow sequence after folding back on different time intervals is obtained according to the low-resolution estimated flow sequence directly output by the optical flow calculation module at the previous moment , in the cross-correlation body, a similarity matrix sequence of different time intervals is obtained by looking up a table for each pixel in a fixed-size window centered on the pixel . The estimated flow sequence and the similarity matrix sequence are input into the motion feature encoding layer to obtain a motion feature sequence of different time intervals , the motion features of different time intervals are compared with the most reliable features through the cross-attention mechanism, and the correct motion patterns are filtered and enhanced through consistency, and the most reliable features are automatically selected through the learning mechanism.

[0085] In addition, a context feature extraction network is used to extract context features in the current three-dimensional voxel grid, and the fused motion features are input into a gated recurrent unit (GRU) iterative update unit together, and the hidden state is updated. In each iteration, the updated hidden state is input into the optical flow calculation module to output the displacement field change amount under low resolution, i.e., the displacement field change amount at the tthiteration , which is added to to obtain the low-resolution estimated flow sequence at the tthiteration, if it is the first iteration, is the optical flow of the next iteration, after upsampling, the velocity vector field estimation at time t is output The velocity vector field output by the last iteration is the final velocity vector field estimation .

[0086] The expression of event data is a three-dimensional voxel grid, which is a specific implementation form of event frame. Two event streams with the same time length and continuous timestamps need to be correlated after the same processing. Specifically, given the event stream from to , , The timestamp of the previous event frame is from to , and the timestamp of the next event frame is from to . They are divided into pieces each, and each piece has a time span of . These pieces are actually generated into two-dimensional voxel grids at their endpoints according to the preprocessing method in step (3.1) below, called event . All events are sequentially arranged and combined, called a three-dimensional voxel grid, which is a specific implementation form of an event frame. The larger is, the finer the segmentation of events in this time period, and the richer the time information that can be retained. In this embodiment, is set to 25, and the event stream with a resolution of is divided into

[0087] three-dimensional voxel grids. Then the attention-based event fusion mechanism further processes the three-dimensional voxel grid. First, define the three-dimensional voxel grid as , where is the spatial resolution, represents the cumulative amount of events at position , represents the event with size , where , In this embodiment, based on the attention-based fusion mechanism, a scoring network with shared weights is first established to score the 25 event , and the score is parameterized to calculate the weight of each weight of each pixel of the previous frame, and the weighted sum of events is denoted by symbol . The previous event frame is added from the last event to the first event, pairwise, triplet, quadruplet, quintuplet, i.e.

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] where to are the fusion events to under the time interval , and are the original events , .

[0094] The next event frame is added from the first event to the last event, pairwise, triplet, quadruplet, quintuplet, to obtain the same fusion events , which are combined into a new three-dimensional voxel grid. The previous and next event frames each obtain five three-dimensional voxel grids containing five different time interval events . Then, they are normalized respectively and input into the shared-weight feature extraction network, which extracts the spatiotemporal features of the particle events while downsampling, and obtains five groups of features of the time interval, i.e. , where represent the height, width and channel number of the features, respectively. Five cross-correlation volumes are calculated using the five groups of features of the two three-dimensional voxel grids, where is defined as:

[0095]

[0096] where D represents the product of the squared lengths of the feature vectors, i.e. . The time-intensive cross-correlation volumes achieve fine recording of relative motion through the similarity comparison of features of multiple time spans.

[0097] Each correlation volume by associating a three-dimensional voxel grid of the corresponding time interval is obtained, meaning that the recorded relative motion is not aligned in time, the search coordinates are reduced in a linear way to match the time stamp of each correlation volume, based on the assumption that the cross-correlation volume is uniformly constructed in time and the optical flow size changes linearly in short time. Specifically, given the current optical flow estimate , it is segmented into segments to sample each cross-correlation volume. The operation of linear search can be represented as:

[0098]

[0099]

[0100] where is the initialization coordinate of the target time , is the current coordinate estimate updated after obtaining a new optical flow estimate each time, and is used to record the optical flow estimate of each iteration, is 5 in this embodiment, and should be a factor of B, i.e., the number of different time intervals, which is also 5 in this embodiment; denotes the search operation, is the search coordinate of , and is the similarity matrix sampled from . After inputting all the correlation maps into the motion feature encoder sharing the weight, the aligned motion features can be obtained. .

[0101] Then the motion features are cross-fused. Specifically, a set of learnable weight coefficients is defined (normalized by Softmax), and the weighted average of all intermediate motion features is calculated as:

[0102]

[0103] wherein denotes the motion feature after weighted average. The weights are dynamically generated through the attention mechanism, allowing the model to adaptively focus on the features of different time scales. Each intermediate feature is paired with the weighted average motion feature , input into the cross-attention module, and the enhanced motion feature is obtained, denoted as:

[0104]

[0105]

[0106] wherein,​ is a weight dynamically generated by the attention mechanism, represents a cross-attention module, , is a dimension, are projection matrices of query, key, value respectively, is a multi-layer perception, is another projection matrix, is the fused motion feature. For simplicity of expression, the normalization function is omitted.

[0107] In this embodiment, when the event camera optical flow method model is trained by using the training data set obtained in step (2), the following sub-steps are specifically included:

[0108] (3.1) Preprocess the simulation event data in the training data set, that is, according to the frequency of the to-be-tested flow field velocity vector field, such as 100 fps, divide the simulation event data into event sequences with equal time length, such as 10 ms, and make a three-dimensional voxel grid.

[0109] Further, in step (3.1), the event sequence is made into a three-dimensional voxel grid according to the following rules: assuming that the number of events in an event sequence is N, then the column of events is ; then the column of events is divided into discrete B containers, the time scale is converted to [0, B-1], and the following operations are performed on the events:

[0110]

[0111]

[0112]

[0113] In the formula, is a new timestamp to which the event is mapped, is the original timestamp of the i-th event in the event sequence, is the event accumulation value of a pixel in the three-dimensional voxel grid at a new timestamp, are the horizontal coordinate, the vertical coordinate, and the timestamp of the three-dimensional voxel grid respectively, is the event polarity, is the horizontal coordinate of the i-th event, is the vertical coordinate of the i-th event, is a bilinear sampling kernel function, is a function argument and has no special meaning. The bilinear sampling kernel function samples the time to merge the timestamp into discrete B values, and samples the coordinates of the event because the coordinate values of the event data after calibration may not be integers.

[0114] (3.2) taking two adjacent three-dimensional voxel grids as inputs of the event camera optical flow method model, and outputting an estimated velocity vector field at an intermediate time between the two three-dimensional voxel grids .

[0115] (3.3) calculating a training loss function based on the estimated velocity vector field output by the event camera optical flow method model and the corresponding velocity field label in the training data set.

[0116] Further, the training loss function can be specified as the error between the estimated velocity vector field output in the iterative training process and the velocity field label, and is calculated according to the following formula:

[0117]

[0118] In the formula, denotes the training loss function, denotes the estimated velocity vector field at the i-th iteration, denotes the velocity field label in the training data set, denotes the L1 distance, denotes the weight of the error at the i-th iteration, obeys an exponential distribution, , is the base of the exponential, which has no practical meaning, is the total number of iterations. In the embodiment, is set to 0.8, is set to 10.

[0119] (3.4) taking minimizing the training loss function as the optimization objective, optimizing the parameters of the event camera optical flow method model through back propagation and using the adaptive moment estimation calculation method until a preset training round is reached, to obtain the trained event camera optical flow method model.

[0120] Specifically, after setting the network structure of the event camera optical flow method model and the training loss function, an optimization strategy is needed to train the network parameters. The event camera optical flow method model is trained using the preprocessed simulation event data and the corresponding velocity field label in the training data set. The preprocessed three-dimensional voxel grid is taken as the input of the event camera optical flow method model, and the real velocity vector field is taken as the corresponding velocity field label. The error between the estimated velocity vector field output by the model and the velocity field label is calculated using the training loss function in step (3.3), and the parameters of the event camera optical flow method model are optimized using the adaptive moment estimation (Adam) algorithm, so as to obtain the trained event camera optical flow method model which can be used for flow field estimation using event data.

[0121] It should be understood that Adam is a widely used optimization algorithm that combines the ideas of momentum and adaptive learning rate, suitable for gradient descent optimization in deep learning. Adam combines the advantages of two methods: ① momentum: accelerate convergence and reduce oscillation by exponentially weighted average gradient (first moment); ② adaptive learning rate (such as RMSProp): adjust the learning rate of each parameter by exponentially weighted average of gradient square (second moment), adapt to different parameter update amplitude.

[0122] (4) Based on the event sequence in two adjacent time periods, the trained event camera optical flow method model is used to calculate the velocity vector field of the corresponding intermediate time.

[0123] In summary, the application can obtain the required two-dimensional flow field motion velocity at a certain time based on event data in a period of time. After training the event camera optical flow method model, the network parameters are obtained. The input is a three-dimensional voxel grid corresponding to the event set in a period of time, and the output is the flow field velocity field at a specified time. Compared with the traditional camera flow field visualization measurement method, the event camera has the advantages of low camera cost, high time resolution, low information simplification requirement for transmission bandwidth, etc.

[0124] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for flow field measurement based on an event camera, characterized in that, The method comprises the following steps: (1) generating a particle image velocimetry data set, wherein the particle image velocimetry data set comprises a particle image, particle event data and a corresponding velocity field label; each time sequence sample sequence in the particle image velocimetry data set comprises a plurality of frames of continuous particle images, a velocity vector field at a corresponding time and particle event data at all times; (2) building a flow field data acquisition device based on an event camera and a high-speed camera, collecting time-synchronized real event data and real image data, adjusting parameters of an event simulator, verifying and updating the particle event data obtained in step (1) according to the adjusted event simulator, and obtaining an updated particle image velocimetry data set as a training data set; (3) building an event camera optical flow method model and training the event camera optical flow method model by using the training data set to obtain a trained event camera optical flow method model; (4) based on event sequences in two adjacent time periods, calculating a velocity vector field at a corresponding intermediate time by using the trained event camera optical flow method model.

2. The event camera based flow field measurement method of claim 1, wherein, The step (1) specifically comprises the following sub-steps: (1.1) generating a first frame of particle set randomly distributed in a three-dimensional observation area by using a particle image simulator to generate a particle image; (1.2) according to the size of the flow field velocity used, the flow field is interpolated in time to T times time steps, and the flow field velocity size is scaled to ; according to the integral trajectory of the current time step, the next frame position of the first frame particle set is obtained, the particle image is generated in the imaging area, and the particle position outside the imaging area is reset; repeat the particle movement, imaging and resetting of each time step; save the current real velocity vector field once every T time steps, multiply the current flow field velocity size by T when saving, and scale back to the original size; obtain the time-interpolated time sequence particle image and the corresponding real velocity vector field; (1.3) based on the time-interpolated time sequence particle image, using an event simulator to simulate particle event data generated by particle movement in the whole process; (1.4) packing the particle event data, the real velocity vector field and a plurality of frames of particle images with the same time stamp as the velocity vector field into a time sequence sample sequence to generate a particle image velocimetry data set; The velocity vector field is the velocity field label.

3. The event camera based flow field measurement method of claim 2, wherein, The particle image simulator is used to generate a particle image, wherein the shape of each particle in space in the particle image satisfies a three-dimensional Gaussian distribution, which is expressed as: ； In the formula, represents a three-dimensional spatial coordinate in a particle image, represents a gray intensity of a single particle, represents a three-dimensional spatial coordinate position of a particle center, represents a peak intensity value of a particle center, represents a diameter of a particle.

4. The event camera based flow field measurement method of claim 2, wherein, The event simulator is used to simulate particle event data of particle movement based on time sequence particle images by using the following model of the event simulator: ； wherein, represents a voltage change on the event camera pixel circuit, represents a luminance value considered constant over a time period, represents a linear change speed of the local luminance, represents a time interval since the last event trigger, t represents a time stamp of the current event trigger, represents a time stamp of the last event trigger, is a random term represented by a Wiener process, is a calibration parameter.

5. The event camera based flow field measurement method of claim 1, wherein, The flow field data acquisition device comprises a sheet-shaped continuous laser source (1), a control system (2), an external synchronization triggering device (3), a high-speed camera (4), an event camera (5), a beam splitter (6) and a transparent box body (7), wherein the sheet-shaped continuous laser source (1) is located on the side of the transparent box body (7), the control system (2) is in communication connection with the sheet-shaped continuous laser source (1), the external synchronization triggering device (3), the high-speed camera (4) and the event camera (5), the external synchronization triggering device (3) is connected with the high-speed camera (4) and the event camera (5), the high-speed camera (4), the event camera (5) and the beam splitter (6) are all located on the front face of the transparent box body (7), and the optical axes of the high-speed camera (4) and the event camera (5) are aligned with the center of the beam splitter (6).

6. The event camera based flow field measurement method of claim 5, wherein, In step (2), the updated particle image velocimetry data set is obtained, specifically comprising: The particle is implanted into the flow field to be measured in the transparent box (7), the flow field to be measured is illuminated by using the sheet-shaped continuous laser source (1), the field of view of the event camera (5) and the high-speed camera (4) is ensured to be consistent by using the beam splitter (6), controlling the ratio of the focal length of the lens and the size of the photosensitive element and binocular camera calibration, the event camera (5) and the high-speed camera (4) are synchronously triggered by using the external synchronous triggering device (3), so as to synchronously collect the real event data and real image data of the particle in the flow field to be measured; The parameters of the event simulator are adjusted according to the correspondence between the real event data and the real image data, and the particle event data in the particle image velocimetry data set is verified according to the adjusted event simulator, so as to ensure that the generated particle event data is the same as the corresponding real event data, if there is a difference, the particle image is simulated again by using the adjusted event simulator, the simulation event data is obtained for replacing the original particle event data, so as to update the particle image velocimetry data set.

7. The event camera based flow field measurement method of claim 1, wherein, The input of the event camera optical flow method model is two adjacent three-dimensional voxel grids, wherein each three-dimensional voxel grid is obtained by preprocessing the simulation event data in the corresponding time period; Using a three-dimensional voxel grid via attention-based events The fusion mechanism and the feature extraction network of shared weights extract particle features on different time intervals to obtain feature sequences of different time intervals; based on the feature sequences of different time intervals, cross-correlation calculation is performed on the features on each time interval of the front and rear event frames to obtain a cross-correlation body sequence of different time intervals; based on the cross-correlation body sequence of different time intervals, a similarity matrix sequence of different time intervals is obtained, and the similarity matrix sequence and an estimated flow sequence of the last moment are input into a motion feature coding layer to obtain a motion feature sequence of different time intervals; the motion feature sequences of different time intervals are cross-fused via a cross-attention mechanism to obtain fused motion features; The context feature extraction network is used to extract the context feature in the current three-dimensional voxel grid, and the context feature and the fused motion feature are input into the gated recurrent iterative update unit, and an updated hidden state is obtained; The updated hidden state is input into the optical flow calculation module to obtain the estimated flow sequence at the current time, and then the velocity vector field estimation at the current time is output through the upsampling module, and the velocity vector field output by the last iteration is the final velocity vector field estimation.

8. The event camera based flow field measurement method of claim 1, wherein, In step (3), the training data set is used for training, specifically including: The simulation event data in the training data set is preprocessed, that is, according to the frequency of the velocity vector field of the flow field to be measured, the simulation event data is divided into event sequences with equal time length, and the three-dimensional voxel grid is made; Two adjacent three-dimensional voxel grids are used as the input of the event camera optical flow method model, and the velocity vector field estimation value at the intermediate time between the two three-dimensional voxel grids is output; The training loss function is calculated based on the velocity vector field estimation value and the corresponding velocity field label in the training data set; The parameters of the event camera optical flow method model are optimized by back propagation and adaptive moment estimation calculation method with the optimization target of minimizing the training loss function until the preset training round is reached, so as to obtain the trained event camera optical flow method model.

9. The event camera based flow field measurement method of claim 8, wherein, The calculation formula of the training loss function is: ； wherein denotes the training loss function, denotes the velocity vector field estimate of the i-th iteration, denotes the velocity field label in the training data set, denotes the L1 distance, denotes the weight of the error of the i-th iteration.

Images