Flow field measurement method based on event camera

Abstract

The invention discloses a flow field measurement method based on an event camera, and the method comprises the steps: generating a PIV data set, each time sequence sample sequence comprising a plurality of frames of continuous particle images, a corresponding velocity vector field, and particle event data at all moments; establishing a flow field data acquisition device based on an event camera and a high-speed camera, acquiring real event data and real image data which are synchronous in time so as to adjust parameters of an event simulator, and verifying and updating particle event data in the PIV data set according to the adjusted event simulator so as to obtain a flow field data acquisition result; obtaining the updated PIV data set as a training data set; building an event camera optical flow method model, and training by adopting the training data set; and on the basis of the trained event camera optical flow method model, event sequences in two adjacent time periods are used as inputs to calculate a velocity vector field corresponding to a middle moment. According to the invention, the flow field velocity field at the required moment can be obtained based on the event data within a period of time.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of flow field visualization measurement, and particularly relates to a flow field measurement method based on an event camera. BACKGROUND

[0002] Particle Image Velocimetry (PIV) is a non-contact flow field measurement technology, mainly used for visualization and quantitative measurement of complex fluid motion, and plays a crucial role in fluid mechanics theory and experimental research. PIV technology first adds fluorescent tracer particles to the measured medium to follow the fluid motion, simultaneously uses laser illumination, and uses a digital camera to shoot continuous particle images, and then obtains the global velocity field of the fluid through a cross-frame algorithm.

[0003] Traditional particle image velocimetry technology usually needs to use a high-speed camera for shooting, however, the high-speed camera is costly, and at a higher resolution, the data volume of a single picture is very large, and the requirement for transmission bandwidth is extremely high, so at a high frame rate, real-time transmission cannot be achieved, and thus the memory is also required. The event camera has the advantages of low cost, data reduction, and large dynamic range, while the time resolution is high enough.

[0004] However, due to the difference in principle between the event camera and the traditional camera, the format of the event data and the image are completely different, and a corresponding algorithm needs to be specifically proposed. At present, the flow field measurement method based on the event camera is divided into two categories, one is particle tracking velocimetry, which needs to identify and track each particle in the event data, and has a higher requirement for particle density. The other is a kind of particle image velocimetry, which has a lower sensitivity to particle density, but at present, only one model-based algorithm has been proposed. This method first divides the event data into different rectangular regions according to pixels, and uses a motion compensation method to obtain a velocity vector in each rectangular region, which not only leads to a very high computational load, but also leads to a resolution of the estimated flow field much lower than the original resolution during data acquisition. In addition, due to the strong assumption that the optical flow is constant within the local space-time window, if the spatial non-uniformity and temporal non-stationarity of the optical flow are strong, significant errors will be caused. SUMMARY

[0005] The present application aims to solve the problems of high cost of traditional cameras, excessive data redundancy, and imperfect event camera flow field measurement methods in the prior art, and provides a flow field measurement method based on an event camera. The present application applies the event camera optical flow method to flow field measurement, while taking advantage of the low cost, information reduction, and real-time transmission of the event camera, it makes up for the shortcomings of the current event camera in flow field measurement, so that the event camera flow field measurement method can also output reliable dense flow field estimates like the traditional camera method.

[0006] The object of the application is achieved by the following technical solution: a flow field measurement method based on an event camera, comprising the following steps: (1) generating a particle image velocimetry data set, the particle image velocimetry data set comprising particle images, particle event data and a corresponding velocity field label; each time sequence sample sequence in the particle image velocimetry data set comprising a plurality of frames of continuous particle images, a velocity vector field at the corresponding time and particle event data at all times; (2) building a flow field data acquisition device based on an event camera and a high-speed camera, acquiring time-synchronized real event data and real image data, adjusting the parameters of an event simulator, and verifying and updating the particle event data obtained in step (1) according to the adjusted event simulator, to obtain an updated particle image velocimetry data set as a training data set; (3) building an event camera optical flow method model and training the model using the training data set to obtain a trained event camera optical flow method model; (4) based on event sequences in two adjacent time periods, using the trained event camera optical flow method model to calculate a velocity vector field at the corresponding intermediate time.

[0007] Further, step (1) specifically comprises the following sub-steps: (1.1) generating a first frame of particle set with random distribution of coordinates in a three-dimensional observation area using a particle image simulator to produce a particle image; (1.2) interpolating the flow field in time to T times of time steps according to the size of the flow field velocity, and scaling the flow field velocity size to ; obtaining the next frame position of the first frame of particle set according to the integral trajectory of the current time step, generating a particle image in the imaging area, and resetting the particle position outside the imaging area; repeating the particle movement, imaging and resetting of each time step; saving the current real velocity vector field once every T time steps, multiplying the current flow field velocity size by T to scale back to the original size when saving; obtaining the time-interpolated time sequence particle image and the corresponding real velocity vector field; (1.3) based on the time-interpolated time sequence particle image, using an event simulator to simulate the particle event data generated by the particle movement in the entire process; (1.4) packing the particle event data, the real velocity vector field and a plurality of frames of particle images with the same time stamp as the velocity vector field into a time sequence sample sequence to generate a particle image velocimetry data set; wherein the velocity vector field is the velocity field label.

[0008] Further, the particle image simulator is used to generate a particle image, wherein the form of each particle in space in the particle image satisfies a three-dimensional Gaussian distribution, represented as:

[0009] In the formula, represents the three-dimensional spatial coordinates in the particle image, represents the gray intensity of a single particle, represents the three-dimensional spatial coordinate position of the particle center, represents the peak intensity value of the particle center, represents the diameter of the particle.

[0010] Further, the event simulator is used for particle image based on timing, and the particle event data simulation of particle motion is completed by using the following model of the event simulator:

[0011] In the formula, represents the voltage change on the event camera pixel circuit, represents the brightness value considered as a constant in the time period, represents the linear change speed of the local brightness, represents the time interval since the last event trigger, and t represents the time stamp of the current event trigger, represents the time stamp of the last event trigger, is a random term represented by a Wiener process, is a calibration parameter.

[0012] Further, the flow field data acquisition device comprises a sheet-shaped continuous laser source, a control system, an external synchronous triggering device, a high-speed camera, an event camera, a beam splitter and a transparent box body, wherein the sheet-shaped continuous laser source is located on the side of the transparent box body, the control system is in communication connection with the sheet-shaped continuous laser source, the external synchronous triggering device, the high-speed camera and the event camera respectively, the external synchronous triggering device is connected with the high-speed camera and the event camera respectively, and the high-speed camera, the event camera and the beam splitter are all located on the front of the transparent box body, and the optical axes of the high-speed camera and the event camera are aligned with the center of the beam splitter.

[0013] Further, in step (2), the updated particle image velocimetry data set is obtained, specifically including: The particles are implanted into the flow field to be measured in the transparent box body, the sheet-shaped continuous laser source is used to illuminate the flow field to be measured, the fields of view of the event camera and the high-speed camera are ensured to be consistent by using the beam splitter, controlling the ratio of the focal length to the size of the light-sensitive element and binocular camera calibration, and the event camera and the high-speed camera are synchronously triggered by using the external synchronous triggering device to synchronously collect the real event data and real image data of the particles in the flow field to be measured. According to the correspondence between the real event data and the real image data, the parameters of the event simulator are adjusted, and the particle event data in the particle image velocimetry data set is verified according to the adjusted event simulator, so as to ensure that the generated particle event data is the same as the corresponding real event data, and if there is a difference, the particle image is simulated again using the adjusted event simulator to obtain simulation event data for replacing the original particle event data, so as to update the particle image velocimetry data set.

[0014] Further, the input of the event camera optical flow method model is two adjacent three-dimensional voxel grids, wherein each three-dimensional voxel grid is obtained by preprocessing the simulation event data in the corresponding time period; the three-dimensional voxel grid is used to obtain the motion feature sequence of different time intervals through the event The fusion mechanism and the shared weight feature extraction network are used to extract particle features in different time intervals to obtain a feature sequence in different time intervals; the features in each time interval of the front and rear event frames are calculated based on the correlation of the feature sequence in different time intervals to obtain a correlation body sequence in different time intervals; the similarity matrix sequence in different time intervals is obtained based on the correlation body sequence in different time intervals, and the estimated flow sequence of the last time is input into the motion feature coding layer together to obtain a motion feature sequence in different time intervals; the motion feature sequence in different time intervals is cross-fused through the cross-attention mechanism to obtain the fused motion feature; the context feature is extracted in the current three-dimensional voxel grid using the context feature extraction network, and the fused motion feature is input into the gated recurrent iterative update unit together to obtain an updated hidden state; the updated hidden state is input into the optical flow calculation module to obtain the estimated flow sequence of the current time, and the velocity vector field estimate of the current time is output through the upsampling module, and the velocity vector field estimate output by the last iteration is the final velocity vector field estimate.

[0015] Further, in step (3), the training data set is used for training, specifically including: The simulation event data in the training data set is preprocessed, that is, according to the frequency of the velocity vector field of the to-be-tested flow field, the simulation event data is divided into event sequences with equal time length, and the three-dimensional voxel grid is made; Two adjacent three-dimensional voxel grids are used as the input of the event camera optical flow method model, and the velocity vector field estimate value of the intermediate time between the two three-dimensional voxel grids is output; The training loss function is calculated based on the velocity vector field estimate value and the corresponding velocity field label in the training data set; The parameters of the event camera optical flow method model are optimized by back propagation and using the adaptive moment estimation calculation method with the optimization goal of minimizing the training loss function until the preset training round is reached, so as to obtain the trained event camera optical flow method model.

[0016] Furthermore, the formula for calculating the training loss function is as follows:

[0017] In the formula, Represents the training loss function. This represents the velocity vector field estimate in the i-th iteration. This represents the velocity field label in the training dataset. Indicates the L1 distance. The weight represents the error of the i-th iteration.

[0018] The beneficial effects of this invention are as follows: By introducing the event camera optical flow method, this invention breaks through the hardware limitations of traditional high-speed cameras; compared with existing methods, while maintaining sub-pixel-level measurement accuracy, the equipment cost is reduced by more than 60%, and it has high dynamic range characteristics, significantly reducing sensitivity to lighting conditions; the event camera optical flow method model, through the collaborative design of a multi-scale feature extraction network and a motion feature encoding module, can simultaneously capture large-scale dominant flows (such as vortex structures) and small-scale turbulent fluctuation features within the field of view, giving full play to the unique advantage of the microsecond-level temporal resolution of the event camera, and making up for the shortcomings of existing event camera flow field measurement methods in terms of spatial scale coverage; it has advantages such as low camera cost, high temporal resolution, and low bandwidth requirements for information conciseness. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the implementation steps of the flow field measurement method based on an event camera according to the present invention. Figure 2 This is a schematic diagram illustrating the PIV data generation process of the present invention; wherein, Figure 2 (a) in the text represents the particle set P from the previous frame; Figure 2 In the diagram, (b) represents the known velocity vector field V; Figure 2 (c) in the figure represents the next frame's particle set Q generated after moving the particle set P using the known velocity vector field V; Figure 2 In this context, (d) represents the particle event that occurs when a particle moves; Figure 3 This is a schematic diagram of the flow field data acquisition device based on an event camera and a high-speed camera according to the present invention. Figure 4 This is a flowchart illustrating the architecture of the event camera optical flow model of the present invention.

[0020] In the figure, there are 1 sheet-like continuous laser source, 2 control system, 3 external synchronous triggering device, 4 high-speed camera, 5 event camera, 6 beam splitter, and 7 transparent box. Detailed Implementation

[0021] The exemplary embodiments will be described in detail herein with reference to the attached drawings. Descriptions of well-known functions and structures are omitted so as not to unnecessarily obscure the application. The examples described herein are intended to cover all aspects of the present application, including those embodiments which are fairly obvious to those skilled in the art.

[0022] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0023] It is to be understood that the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. Pronouns in the masculine form include the feminine form, and vice versa, and the singular form also includes the plural form, unless the context clearly dictates otherwise. It will be further understood that the terms "comprises" and / or "comprising", when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0024] The application will be described in detail herein with reference to the attached drawings. The features described in the following embodiments and / or examples can be combined with each other, if not in conflict.

[0025] Referring to Figure 1 The event camera based flow field measurement method of the present application specifically comprises the following steps: (1) generating a PIV dataset, the PIV dataset comprising particle images, particle event data and corresponding velocity field labels; each time sequence sample sequence in the PIV dataset comprises a plurality of frames of continuous particle images, a velocity vector field at the corresponding time and particle event data at all time points with a timestamp accurate to microseconds; wherein the velocity vector field is the velocity field label.

[0026] In this embodiment, the event data and corresponding velocity field labels (i.e. velocity vector field at the corresponding time) of a large number of moving particles in a three-dimensional space are artificially synthesized for subsequent training of the event camera optical flow method model; in addition, particle images with the same frame rate as the velocity field labels are generated for reference. The PIV dataset is composed of a plurality of time sequence sample sequences, each time sequence sample sequence corresponds to a set of particle images with a frame rate of 100 fps A set of particle images and a set of N velocity field labels and all particle event data (i.e. particle event data at all time instants in the process with timestamps accurate to microseconds) generated by particle motion between the first and the .

[0027] In this embodiment, a PIV dataset is generated, and the process of generating the PIV data is shown in FIG. 1, and specifically includes the following sub-steps: Figure 2 (1.1) Using a particle image simulator, a first frame particle set P is generated in a three-dimensional observation region, as shown in (a) of FIG. 1, to produce a particle image. The three-dimensional observation region has a size of 300x300x4 pixels, and the first frame particle set P uniformly distributed in the three-dimensional observation region contains more than twenty thousand randomly distributed particles. The imaging region is defined at a specified position of a computational fluid dynamics (CFD) calculation domain, for example, a region of 256x256x1 pixels located at the center of the three-dimensional observation region can be defined as the imaging region, and the size of the imaging region is 256x256x1 pixels. Figure 2

[0028] It should be understood that the size of the three-dimensional observation region and the imaging region and the size of the flow field velocity can be defined according to requirements. For example, the three-dimensional observation region can be defined as a cuboid region between the coordinate origin [1, 1, 1] and [300, 300, 5], and the imaging region can be defined between [23-278, 23-278, 2.5-3.5], or the imaging layer thickness can be appropriately increased, so that the frequency of the appearance and disappearance of the particles in the particle image sequence in the z-axis is reduced.

[0029] In addition, in order to ensure the diversity of the dataset, different data samples have different parameters, including particle diameter (2-4 voxels), displacement ratio (0.35-6.58), particle peak gray value (200-255), particle imaging layer thickness (1-1.2), Gaussian blur of particle diameter and intensity (0-0.1), wherein the particle imaging layer thickness and the z-axis velocity of the flow field jointly determine the frequency of the particles entering and leaving the imaging layer in the z-axis, and further determine the particle.

[0030] Further, the particle image simulator is used to generate a particle image, wherein the shape of each particle in the space in the particle image satisfies a three-dimensional Gaussian distribution, and is expressed as:

[0031] In the formula, x, y and z represent the three-dimensional space coordinates in the particle image, ​​​​​gray intensity of a single particle, three-dimensional spatial coordinate position of the center of a particle, peak intensity value of the center of a particle, diameter of a particle; wherein the position of each particle in each frame is determined by the position in the previous frame and the velocity vector field between the particle images of each two frames, which is the saved velocity field tag.

[0032] (1.2) According to the size of the flow field velocity used, the flow field is interpolated in time by T times, and the flow field velocity size is scaled to , so that the maximum flow field velocity of each time step is below 0.4 pixels; through experiments, under this condition, the particle event data generated according to the particle image sequence reaches saturation, therefore, the flow field velocity size is scaled to , so that the maximum flow field velocity of each time step is below 0.4 pixels. The next frame position of the first frame particle set P is obtained according to the integral trajectory of the current time step, that is, the second frame particle set Q, as shown in (c) of Figure 2 , generate particle images in the imaging area, and reset the particle positions outside the imaging area, so that the imaging area is uniformly covered with particles when the particles are simulated to move in each time step; then repeat the particle movement, imaging and resetting of each time step; save the current real velocity vector field V every T time steps, and scale back to the original size by multiplying the current flow field velocity size by T when saving, as shown in (b) of Figure 2 . In this way, the time-interpolated particle images of T frames and the corresponding N real velocity vector fields can be obtained.

[0033] Further, the velocity vector field V can be a variety of fluid motion scenes, including but not limited to: three-dimensional uniform flow field, Beltrami flow field, incompressible isotropic turbulent flow, magnetohydrodynamic turbulent flow, turbulent channel flow, transition boundary layer flow, etc. Figure 2 The velocity vector field shown in (b) of

[0034] (1.3) Based on the time-interpolated particle images of T frames finally obtained in step (1.2), using an event simulator, all particle event data generated by particle movement between the time stamp of the first particle image and the time stamp of the particle image in the whole process are simulated, as shown in (d) of Figure 2 , wherein the pixels that become dark due to the departure of particles generate -1 events, and the pixels that become bright due to the arrival of particles generate +1 events.

[0035] Further, the event simulator is used to simulate the particle event data of the particle motion based on the time-ordered particle images, and the particle event data is simulated by using the following model of the event simulator:

[0036] In the case of only considering the randomness of the photon reception, the voltage change on the event camera pixel circuit is modeled as Assuming that the local brightness changes linearly at a constant speed in a short time The brightness value in a short time is considered as a constant The time interval since the last event trigger is t represents the time stamp of the current event trigger, represents the time stamp of the last event trigger is a Wiener process, representing a random term is a calibration parameter.

[0037] Specifically, the particle event data is generated by the event simulator based on the time-ordered particle image sequence. In the case of only considering the randomness of the photon reception, assuming that the local brightness changes linearly at a constant speed in a short time The current change of the optical signal conversion can be represented as:

[0038] In the formula, is the conversion rate of the optical signal to the current, is a random term mainly caused by the randomness of the photon reception, which can be represented by a Wiener process as shown in the following formula:

[0039] In the formula, represents a Wiener process, is a constant parameter, is the brightness value in a short time which is considered as a constant. Substituting into the ideal model of the voltage change of the event camera pixel circuit, we get:

[0040] and using the first-order Taylor approximation After that, the voltage change can be modeled as:

[0041] In the formula, are circuit element parameters, is the thermal voltage, is the photo current, is the dark current in the absence of illumination.

[0042] Considering the noise caused by the leakage current, the corresponding voltage change is:

[0043] wherein, represents the voltage change caused by the leakage current, and are the event camera constants to be calibrated, , , is the activation energy, is the Boltzmann constant, is the coefficient of the Wiener process assuming that the noise term of the leakage current is white noise, which is a constant to be calibrated. Add and to model, while is replaced by because it is proportional to , and the coefficient in it is replaced by the calibration parameter , so that the model of the event simulator is obtained.

[0044] (1.4) Pack the particle event data obtained in step (1.3), the real velocity vector field obtained in step (1.2), and N+1 frames of particle images with the same time stamp as the velocity vector field into a time sequence sample sequence to generate a PIV data set.

[0045] (2) Build a flow field data acquisition device based on an event camera and a high-speed camera to collect time-synchronized real event data and real image data to adjust the parameters of the event simulator, and verify and update the particle event data obtained in step (1) according to the adjusted event simulator, and obtain the updated PIV data set as a training data set for subsequent training of the event camera optical flow method model.

[0046] It should be noted that the flow field data acquisition device based on the event camera and the high-speed camera is built, the real event data and the real image data are collected by using the flow field data acquisition device, the parameters of the event simulator are calibrated by using the real event data and the real image data, the effectiveness of the data in the PIV data set simulated in step (1) is verified, and when invalid, the event simulator with adjusted parameters is used for simulation again to generate simulated event data.

[0047] In this embodiment, as Figure 3As shown, the flow field data acquisition device includes a sheet-like continuous laser source 1, a control system 2, an external synchronization triggering device 3, a high-speed camera 4, an event camera 5, a beam splitter 6, and a transparent housing 7. The sheet-like continuous laser source 1 is located on the side of the transparent housing 7. The control system 2 is communicatively connected to the sheet-like continuous laser source 1, the external synchronization triggering device 3, the high-speed camera 4, and the event camera 5. The external synchronization triggering device 3 is connected to both the high-speed camera 4 and the event camera 5. The optical axes of the high-speed camera 4 and the event camera 5 are aligned with the center of the beam splitter 6, and the high-speed camera 4, the event camera 5, and the beam splitter 6 are all located on the front of the transparent housing 7.

[0048] Specifically, when using the flow field data acquisition device for data acquisition, the control system 2 controls the data acquisition of the flow field data acquisition device. First, reflective, uniformly sized, and appropriately sized lightweight particles are uniformly implanted into the flow field to be measured in the transparent box 7. The flow field to be measured is illuminated from the side of the transparent box 7 using a sheet-like continuous laser source 1. The experimental parameters are adjusted to ensure that the particle density, average particle intensity, and particle diameter range of the PIV dataset and the acquired data are consistent. By using the beam splitter 6, controlling the focal length of the lens and the size ratio of the photosensitive element, and calibrating the binocular camera, the consistency of the field of view of the event camera 5 and the high-speed camera 4 is ensured. Specifically, on the front of the transparent box 7, the high-speed camera 4, the event camera 5, and the beam splitter 6 are positioned as follows: Figure 3 As shown, align the optical axes of high-speed camera 4 and event camera 5 with the center of beam splitter 6 to ensure consistent field of view. Control the distance from the illuminated plane to the photosensitive elements of high-speed camera 4 and event camera 5, and then, based on the pixel sizes of high-speed camera 4 and event camera 5, use the formula... After calculation, lenses with appropriate focal lengths are selected to ensure that the field of view of high-speed camera 4 and event camera 5 are basically the same. Indicates the field of view. Indicates the size of the photosensitive element. The focal length of the lens is indicated. A calibration board is used to perform dual-target calibration on high-speed camera 4 and event camera 5, achieving pixel-level alignment between them. Then, an external synchronization triggering device 3 is used to synchronously trigger both high-speed camera 4 and event camera 5. Event camera 5 acquires real event data of reflective particles in the flow field under test, while high-speed camera 4 acquires real image data of the same field of view. This ensures that the acquired event data and particle image data are synchronized in time. After obtaining the synchronized real image data and real event data in time steps, the parameters of the event simulator are calibrated according to the correspondence between the real image data and the real event data, and the parameters of the event simulator are adjusted. Then, the adjusted event simulator is used to verify the particle event data in the PIV dataset obtained in step (1) to verify its validity, so as to ensure that the particle event data in the PIV dataset generated in step (1) is the same as the event data in the actual collected data under the same number of particles and the same average motion velocity. If there is unreasonable or different event data, the adjusted event simulator is used to re-simulate the particle image to obtain simulated event data. Finally, the simulated event data obtained from the re-simulation is used to replace the particle event data in the PIV dataset generated in step (1), and the updated PIV dataset is used as the training dataset for the subsequent training of the event camera optical flow model.

[0049] Furthermore, adjusting the parameters of the event simulator At that time, the event simulator model can be used. In summary, it includes drift parameters. and diffusion parameters Brownian motion is represented as: Subsequently, based on the actual collected real image data and real event data, each pair of adjacent image frames was analyzed. and Between events, approximate brightness is extracted at each pixel of the particle image. Brightness change , This represents the brightness of pixel i, and the time interval between adjacent events of the same pixel. Assuming Following a Levy distribution, the fitting parameters are estimated using maximum likelihood estimation. and Using multiple groups Data, through regression calculation Furthermore, manual adjustments are required due to image quality limitations. This is to make the generated data more closely resemble the real data. and Then, using an efficient strategy, the simulation of event generation is completed by alternately executing two parts: determining the event polarity based on the hit probability of Brownian motion and determining the event trigger time based on the first hit time distribution. The probability of a positive polarity event is:

[0050] In the formula, It is the probability of producing a positive event. It is the threshold for generating positive polarity events. It is the threshold for generating negative polarity events. and The participants generated during the calculation process each time the polarity of an event is determined. The determination is not described here. After obtaining the complete event simulator with parameter adjustment, the simulation event data can be generated according to to The time sequence particle image generation simulation event data between the time periods.

[0051] In this embodiment, the updated PIV data set contains 60 time sequence samples of time steps of 100 or 200, including 8000 velocity vector fields and corresponding particle images, and simulation event data of particle motion corresponding to a total time length of 80s.

[0052] (3) Build an event camera optical flow method model and train it using the training data set obtained in step (2) to obtain a trained event camera optical flow method model.

[0053] It should be noted that, in terms of image, the motion of an object is abstracted to the displacement of each corresponding pixel between two frames, and this abstracted displacement field is called an optical flow field; in terms of events, events also fall on pixels, and the displacement of the pixel corresponding to the event is also called optical flow, and then optical flow method can be applied. The calculation of the velocity vector field using the optical flow method is based on the assumption that the velocity field is constant in a very short time, that is, the velocity vector field in the entire field of view is constant in the time period used to calculate the velocity vector field, and then the velocity is the size of the optical flow divided by the time interval. Therefore, the so-called optical flow estimation is actually the estimation of the velocity vector field.

[0054] In this embodiment, the event camera optical flow method model is built, which can be represented as wherein represents event data of equal length in two adjacent time periods, represents the velocity vector field estimation of the intermediate time output by the event camera optical flow method model, represents the mapping function of the event camera optical flow method model. The architecture process of the event camera optical flow method model is shown in Figure 4 The input of the event camera optical flow method model is two adjacent three-dimensional voxel grids, each of which is obtained by preprocessing the simulation event data in the corresponding time period, that is, by preprocessing the event sequence into a three-dimensional voxel grid wherein is the time period, is the number of two-dimensional voxel grids in the time period, and the three-dimensional voxel grid is an implementation form of the event frame, and the data obtained by processing the events in a certain time period through a certain process and input to the event camera optical flow method model for operation is called an event frame. The three-dimensional voxel grid is used to process the event Fusion mechanism, extracting particle features on different time intervals, obtaining a feature sequence with staggered arrangement, each pair of features corresponding to the previous event frame and the next event frame respectively, and the order of the feature pairs corresponding to different time intervals , different time intervals refer to dividing a time period N corresponding to a complete three-dimensional voxel grid into N equal parts, The length of the time period is N different time intervals, and then the features on the time intervals of the previous and next event frames are paired to perform cross-correlation calculation to obtain a cross-correlation body sequence of different time intervals , and then according to the low-resolution estimated flow sequence after folding on different time intervals , in the cross-correlation body, for each pixel, a lookup table is used in a fixed size window centered on the pixel to obtain a similarity matrix sequence of different time intervals . The estimated flow sequence and the similarity matrix sequence are input into the motion feature encoding layer to obtain a motion feature sequence of different time intervals , the motion features of different time intervals are input into the cross-attention mechanism to compare each intermediate feature with the most reliable feature, and the correct motion mode is selected and enhanced through consistency, and the most reliable feature is automatically selected through the learning mechanism.

[0055] In addition, a context feature extraction network is used to extract context features in the current three-dimensional voxel grid, and the motion features after fusion are input into the gated recurrent (GRU) iterative update unit together to update the hidden state. In each iteration, the updated hidden state is input into the optical flow calculation module to output the displacement field change amount under low resolution, i.e. the displacement field change amount , which is added to to obtain the low-resolution estimated flow sequence of the tth iteration, if it is the first iteration, , which is the initial optical flow. , then it is the optical flow of the next iteration, which is output after the upsampling module to output the velocity vector field estimate at time t. The velocity vector field output by the last iteration is the final velocity vector field estimate .

[0056] The expression of event data is a three-dimensional voxel grid, which is an implementation form of event frame. Two event streams with the same time length and continuous timestamps need to be processed in the same way and then cross-correlated. Specifically, given the event stream from to , , , the timestamp of the previous event frame is from to , the timestamp of the latter event frame is from to , divide them into segments each, and each segment has a time span of . This segment is actually generated into a two-dimensional voxel grid at its endpoints by the preprocessing method in step (3.1) as follows, called event , all the event are sequentially arranged and combined, called a three-dimensional voxel grid, which is a specific implementation form of the event frame. The larger the , the finer the segmentation of events in this time period, and the richer the time information that can be retained. In this embodiment, the is set to 25, the event stream with a resolution of is divided into a three-dimensional voxel grid of

[0057] The three-dimensional voxel grid is then further processed by the attention-based event fusion mechanism. First, the three-dimensional voxel grid is defined as , where is the spatial resolution, represents the cumulative amount of the event at position , represents the event with a size of , where , . In this embodiment, based on the attention-based fusion mechanism, a scoring network with shared weights is first established to score the 25 event , and the weight of each pixel of each is calculated according to the score parameterization, and then the weighted sum of the event is calculated, and the symbol represents this series of operations. The former event frame is added two by two from the last , three by three, four by four, and five by five, that is:

[0058]

[0059]

[0060]

[0061] ​​

[0062] wherein, to are to fusion events under time intervals , for addition are original events , .

[0063] The latter event frame starts from the first event , and the two-by-two addition, three-by-three addition, four-by-four addition, and five-by-five addition are performed, to obtain the same fusion event , which is combined into a new three-dimensional voxel grid. The former and latter event frames each obtain five three-dimensional voxel grids containing five events under different time intervals Then, they are each normalized and then input into a feature extraction network sharing weights, to extract the spatiotemporal features of the particle events while down-sampling, and each obtain five groups of features under time intervals, i.e. , wherein respectively represent the height, width, and channel number of the features. Five cross-correlation volumes are calculated using the respective five groups of features of the two three-dimensional voxel grids wherein , which is defined as:

[0064] wherein D represents the product of the squared lengths of the feature vectors, i.e. The time-intensive cross-correlation volumes achieve fine recording of relative motion through feature similarity comparison over multiple time spans.

[0065] Each correlation volume is obtained by correlating of the three-dimensional voxel grids under corresponding time intervals, which means that the recorded relative motion is not aligned in time. Based on the assumption that the cross-correlation volumes are uniformly constructed in time and the optical flow size changes linearly in a short time, the search coordinates are reduced in a linear manner to match the time stamp of each correlation volume. Specifically, given the current optical flow , it is divided into segments to sample each cross-correlation volume. The operation of linear search can be represented as:

[0066]

[0067] wherein is the target time​Initialization coordinates of is the current coordinate estimation updated after obtaining a new optical flow estimation each time, used to record the optical flow estimation of each iteration, in the embodiment, it should be a factor of B, in the embodiment, it is 5, that is, the number of different time intervals, which is also 5 in the embodiment; denotes a search operation, is the search coordinates of is the similarity matrix sampled from After inputting all related graphs into the motion feature encoder sharing weights, the aligned motion features can be obtained.

[0068] Then the motion features need to be cross-fused. Specifically, a set of learnable weight coefficients is defined (normalized by Softmax), and all intermediate motion features are weighted and averaged as follows:

[0069] In the formula, denotes the motion feature after weighted averaging. Among them, the weight is dynamically generated through the attention mechanism, allowing the model to adaptively focus on the features of different time scales. Each intermediate feature is paired with the weighted average motion feature , input into the cross-attention module to obtain the enhanced motion feature, denoted as:

[0070]

[0071] In the formula, is the weight dynamically generated through the attention mechanism, denotes the cross-attention module, , is the dimension of , respectively, are the projection matrices of query, key and value, is a multi-layer perception, is another projection matrix, is the fused motion feature. To simplify the expression, the normalization function is omitted.

[0072] In the embodiment, when the event camera optical flow method model is trained using the training data set obtained in step (2), the following sub-steps are specifically included: (3.1) Preprocess the simulation event data in the training dataset, i.e. divide the simulation event data into event sequences with equal time length, such as 10 ms, according to the frequency of the velocity vector field of the flow field to be tested, such as 100 fps, and make it into a three-dimensional voxel grid.

[0073] Further, in step (3.1), the event sequence is made into a three-dimensional voxel grid according to the following rules: assuming that the number of events in an event sequence is N, the column of events is ; then the column of events is divided into discrete B containers, the time scale is converted to [0, B-1], and the events are operated as follows:

[0074]

[0075]

[0076] In the formula, is the new timestamp of the event mapping, is the original timestamp of the i-th event in the event sequence, is the event cumulative value of a certain pixel in the three-dimensional voxel grid at a certain new timestamp, are the horizontal coordinate, vertical coordinate and timestamp of the three-dimensional voxel grid, respectively, is the event polarity, is the horizontal coordinate of the i-th event, is the vertical coordinate of the i-th event, is a bilinear sampling kernel function, is a function argument, which has no special meaning. The bilinear sampling kernel function samples the time to merge the timestamp into discrete B values, and samples the coordinates of the event because the coordinate values of the event data after calibration may not be integers.

[0077] (3.2) Take two adjacent three-dimensional voxel grids as the input of the event camera optical flow method model, and output the velocity vector field estimation value at the intermediate time between the two three-dimensional voxel grids .

[0078] (3.3) Calculate the training loss function based on the velocity vector field estimation value output by the event camera optical flow method model and the corresponding velocity field label in the training dataset.

[0079] Further, the training loss function can be specified as the error between the velocity vector field estimation value output in the iterative training process and the velocity field label, which is calculated according to the following formula:

[0080] In the formula, denotes a training loss function, denotes the velocity vector field estimate of the i-th iteration, denotes the velocity field label in the training dataset, denotes the L1 distance, denotes the weight of the error of the i-th iteration, obeys an exponential distribution, , is the base of the exponential, which has no practical meaning, is the total number of iterations. In this embodiment, is set to 0.8, is set to 10.

[0081] (3.4) The parameters of the event camera optical flow method model are optimized by back propagation and using the adaptive moment estimation algorithm, with the optimization goal of minimizing the training loss function, until the preset training round is reached, to obtain the trained event camera optical flow method model.

[0082] Specifically, after setting the network structure of the event camera optical flow method model and the training loss function, the network parameters need to be trained using an optimization strategy. The event camera optical flow method model is trained using the preprocessed simulation event data and the corresponding velocity field label in the training dataset. The preprocessed three-dimensional voxel grid is used as the input of the event camera optical flow method model, and the real velocity vector field is used as the corresponding velocity field label. The error between the velocity vector field estimate output by the model and the velocity field label is calculated using the training loss function in step (3.3), and the parameters of the event camera optical flow method model are optimized using the adaptive moment estimation (Adam) algorithm, thereby obtaining a trained event camera optical flow method model that can be used to estimate the flow field using event data.

[0083] It should be understood that Adam is a widely used optimization algorithm that combines the ideas of momentum and adaptive learning rate, and is suitable for gradient descent optimization in deep learning. Adam combines the advantages of two methods: ① momentum: accelerate convergence and reduce oscillation by exponentially weighted average of gradients (first moment); ② adaptive learning rate (such as RMSProp): adjust the learning rate of each parameter by exponentially weighted average of squared gradients (second moment), adapt to different parameter update magnitudes.

[0084] (4) Based on the event sequence in two adjacent time periods, the trained event camera optical flow method model is used to calculate the velocity vector field at the corresponding intermediate time.

[0085] In summary, the application can obtain the two-dimensional flow field motion speed of the required time based on the event data in a period of time, obtain the network parameters after the event camera optical flow method model is trained, input is the three-dimensional voxel grid corresponding to the event set in a period of time, and output is the flow field speed field at the specified time. Compared with the flow field visualization measurement method of the traditional camera, the event camera has the advantages of low camera cost, high time resolution, low information simplification requirement on transmission bandwidth and the like.

[0086] The above examples are only used to illustrate the technical solutions of the present application, but not limit it; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for flow field measurement based on an event camera, characterized in that, The method comprises the following steps: (1) generating a particle image velocimetry data set, wherein the particle image velocimetry data set comprises a particle image, particle event data and a corresponding velocity field label; each time sequence sample sequence in the particle image velocimetry data set comprises a plurality of frames of continuous particle images, a velocity vector field at a corresponding time and particle event data at all times; (2) building a flow field data acquisition device based on an event camera and a high-speed camera, collecting time-synchronized real event data and real image data, adjusting parameters of an event simulator, verifying and updating the particle event data obtained in step (1) according to the adjusted event simulator, and obtaining an updated particle image velocimetry data set as a training data set; (3) building an event camera optical flow method model and training the event camera optical flow method model using the training data set to obtain a trained event camera optical flow method model; (4) based on event sequences in two adjacent time periods, calculating a velocity vector field at a corresponding intermediate time using the trained event camera optical flow method model.

2. The event camera based flow field measurement method of claim 1, wherein, The step (1) specifically comprises the following sub-steps: (1.1) generating a first frame of particle sets randomly distributed in a three-dimensional observation area using a particle image simulator to generate a particle image; (1.2) according to the size of the flow field velocity used, the flow field is interpolated in time to T times time steps, and the flow field velocity size is scaled to ; according to the integral trajectory of the current time step, the next frame position of the first frame particle set is obtained, the particle image is generated in the imaging area, and the particle position outside the imaging area is reset; repeat the particle movement, imaging and resetting of each time step; save the current real velocity vector field once every T time steps, multiply the current flow field velocity size by T when saving, and scale back to the original size; obtain the time-interpolated time sequence particle image and the corresponding real velocity vector field; (1.3) based on the time-interpolated time sequence particle image, using an event simulator, simulating to obtain particle event data generated by particle movement in the entire process; (1.4) packing the particle event data, the real velocity vector field and a plurality of frames of particle images with the same time stamp as the velocity vector field into a time sequence sample sequence to generate a particle image velocimetry data set; The velocity vector field is the velocity field label.

3. The event camera based flow field measurement method of claim 2, wherein, The particle image simulator is used to generate a particle image, wherein the form of each particle in space in the particle image satisfies a three-dimensional Gaussian distribution, which is expressed as: ； In the formula, represents a three-dimensional spatial coordinate in a particle image, represents a gray intensity of a single particle, represents a three-dimensional spatial coordinate position of a particle center, represents a peak intensity value of a particle center, represents a diameter of a particle.

4. The event camera based flow field measurement method of claim 2, wherein, The event simulator is used to simulate particle event data of particle movement based on a time sequence particle image using the following event simulator model: ； wherein, represents a voltage change on the event camera pixel circuit, represents a luminance value considered constant over a time period, represents a linear change speed of the local luminance, represents a time interval since the last event trigger, t represents a time stamp of the current event trigger, represents a time stamp of the last event trigger, is a random term represented by a Wiener process, is a calibration parameter.

5. The event camera based flow field measurement method of claim 1, wherein, The flow field data acquisition device comprises a sheet-shaped continuous laser source (1), a control system (2), an external synchronization triggering device (3), a high-speed camera (4), an event camera (5), a beam splitter (6) and a transparent box body (7), wherein the sheet-shaped continuous laser source (1) is located on the side of the transparent box body (7), the control system (2) is in communication connection with the sheet-shaped continuous laser source (1), the external synchronization triggering device (3), the high-speed camera (4) and the event camera (5), the external synchronization triggering device (3) is connected with the high-speed camera (4) and the event camera (5), the high-speed camera (4), the event camera (5) and the beam splitter (6) are all located on the front face of the transparent box body (7), and the optical axes of the high-speed camera (4) and the event camera (5) are aligned with the center of the beam splitter (6).

6. The event camera based flow field measurement method of claim 5, wherein, In step (2), the updated particle image velocimetry data set is obtained, specifically comprising: The particle is implanted into the flow field to be measured in the transparent box (7), the flow field to be measured is illuminated by using the sheet-shaped continuous laser source (1), the field of view of the event camera (5) and the high-speed camera (4) is ensured to be consistent by using the beam splitter (6), controlling the ratio of the focal length of the lens and the size of the photosensitive element and binocular camera calibration, the event camera (5) and the high-speed camera (4) are synchronously triggered by using the external synchronous triggering device (3), so as to synchronously collect the real event data and real image data of the particle in the flow field to be measured; The parameters of the event simulator are adjusted according to the correspondence between the real event data and the real image data, and the particle event data in the particle image velocimetry data set is verified according to the adjusted event simulator, so as to ensure that the generated particle event data is the same as the corresponding real event data, if there is a difference, the particle image is simulated again by using the adjusted event simulator, the simulation event data is obtained for replacing the original particle event data, so as to update the particle image velocimetry data set.

7. The event camera based flow field measurement method of claim 1, wherein, The input of the event camera optical flow method model is two adjacent three-dimensional voxel grids, wherein each three-dimensional voxel grid is obtained by preprocessing the simulation event data in the corresponding time period; Using a three-dimensional voxel grid via attention-based events The fusion mechanism and the feature extraction network of shared weights extract particle features on different time intervals to obtain feature sequences of different time intervals; based on the feature sequences of different time intervals, cross-correlation calculation is performed on the features on each time interval of the front and rear event frames to obtain a cross-correlation body sequence of different time intervals; based on the cross-correlation body sequence of different time intervals, a similarity matrix sequence of different time intervals is obtained, and the similarity matrix sequence and an estimated flow sequence of the last moment are input into a motion feature coding layer to obtain a motion feature sequence of different time intervals; the motion feature sequences of different time intervals are cross-fused via a cross-attention mechanism to obtain fused motion features; The context feature extraction network is used to extract the context feature in the current three-dimensional voxel grid, and the context feature and the fused motion feature are input into the gated recurrent iterative update unit, and an updated hidden state is obtained; The updated hidden state is input into the optical flow calculation module to obtain the estimated flow sequence at the current time, and then the velocity vector field estimation at the current time is output through the upsampling module, and the velocity vector field output by the last iteration is the final velocity vector field estimation.

8. The event camera based flow field measurement method of claim 1, wherein, In step (3), the training data set is used for training, specifically including: The simulation event data in the training data set is preprocessed, that is, according to the frequency of the velocity vector field of the flow field to be measured, the simulation event data is divided into event sequences with equal time length, and the three-dimensional voxel grid is made; Two adjacent three-dimensional voxel grids are used as the input of the event camera optical flow method model, and the velocity vector field estimation value at the intermediate time between the two three-dimensional voxel grids is output; The training loss function is calculated based on the velocity vector field estimation value and the corresponding velocity field label in the training data set; The parameters of the event camera optical flow method model are optimized by back propagation and adaptive moment estimation calculation method with the optimization target of minimizing the training loss function until the preset training round is reached, so as to obtain the trained event camera optical flow method model.

9. The event camera based flow field measurement method of claim 8, wherein, The calculation formula of the training loss function is: ； wherein denotes the training loss function, denotes the velocity vector field estimate of the i-th iteration, denotes the velocity field label in the training data set, denotes the L1 distance, denotes the weight of the error of the i-th iteration.

Images