Particle flow field reconstruction model training method and device, particle flow field reconstruction method and device

Abstract

The present application relates to a training method and device for a particle flow field reconstruction model, and a particle flow field reconstruction method and device. The initial particle flow field reconstruction model includes a pulse coding module, a cost volume fusion coding module, and a particle flow field reconstruction module. The method includes: acquiring sample pulse image data; performing feature extraction processing on the sample pulse image data collected in different time windows respectively, obtaining multiple pulse feature maps corresponding to different time windows respectively, determining the local cost volume based on the corresponding pulse feature maps in different time windows, and determining the overall cost volume based on the pulse feature maps in all time windows; fusing the local cost volume and the overall cost volume to obtain a multi-time step cost volume memory; reconstructing the multi-time step cost volume memory to obtain a reconstructed particle flow field, and training the initial particle flow field reconstruction model to obtain a particle flow field reconstruction model. The use of this method can improve the accuracy of flow field reconstruction.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method and device for training a particle flow field reconstruction model, and a method and device for reconstructing a particle flow field. Background Art

[0002] Particle flow field reconstruction refers to the process of reconstructing the distribution and motion laws of particles in a specific flow field. This technology has important applications in aerospace, energy and power, environmental science, biomedicine and other fields, and can provide an important theoretical basis for aircraft design, turbomachinery structure design, environmental pollution control, etc.

[0003] In related technologies, particle flow fields are usually reconstructed using PIV (Particle Image Velocimetry) technology. PIV technology captures images of tracer particles in the flow field, determines the displacement of particles between two consecutive frames of particle images, and then reconstructs the flow field based on the displacement of particles.

[0004] However, PIV technology relies on the particle cross-correlation calculation between two frames when reconstructing the flow field. When the displacement of the missing particle between the two frames is less than one pixel, PIV technology cannot predict the precise displacement of the particle well, resulting in a decrease in the accuracy of flow field reconstruction. Therefore, PIV technology has the problem of insufficient flow field reconstruction accuracy. Summary of the invention

[0005] Based on this, it is necessary to provide a training method and device for a particle flow field reconstruction model, and a particle flow field reconstruction method and device to address the above technical problems.

[0006] In a first aspect, the present application provides a method for training a particle flow field reconstruction model. The initial particle flow field reconstruction model includes a pulse encoding module, a cost volume fusion encoding module and a particle flow field reconstruction module, and the method includes:

[0007] Acquire multiple frames of sample pulse image data collected for the sample particle flow field area;

[0008] Through the pulse encoding module, feature extraction processing is performed on the sample pulse image data collected in different time windows to obtain multiple pulse feature maps corresponding to different time windows, and local cost volumes are determined based on the corresponding pulse feature maps in different time windows, and the overall cost volume is determined based on the pulse feature maps in all time windows;

[0009] The local cost volume and the overall cost volume are fused by the cost volume fusion encoding module to obtain a multi-time step cost volume memory;

[0010] The multi-time-step cost volume memory is reconstructed by the particle flow field reconstruction module to obtain a reconstructed particle flow field corresponding to the sample particle flow field area, and based on the difference between the reconstructed particle flow field and the actual particle flow field corresponding to the sample particle flow field area, the initial particle flow field reconstruction model is trained to obtain a particle flow field reconstruction model.

[0011] In one embodiment, the pulse encoding module includes a plurality of pulse encoders, the pulse encoder includes an alternating pulse encoding layer and a dilated convolution layer, and the pulse encoding module performs feature extraction processing on sample pulse image data collected in different time windows to obtain a plurality of pulse feature maps corresponding to different time windows, including:

[0012] The sample pulse image data collected in different time windows are respectively input into different pulse encoders, and sequentially pass through the alternating pulse coding layer and the hole convolution layer in the pulse encoder, and feature extraction processing is performed on the sample pulse image data collected in different time windows to obtain multiple pulse feature maps corresponding to different time windows.

[0013] In one embodiment, performing feature extraction processing on sample pulse image data through the pulse coding layer includes:

[0014] Determine reference pulse image data from each of the sample pulse image data;

[0015] Through the pulse coding layer, the reference pulse image data is subjected to self-attention weighted processing to obtain first reference pulse image data, and the first reference pulse image data is subjected to feature extraction processing to obtain a pulse feature graph corresponding to the reference pulse image data;

[0016] For any sample pulse image data except the reference pulse image data in each of the sample pulse image data, self-attention weighted processing is performed on the sample pulse image data through the pulse coding layer to obtain first sample pulse image data, and cross-attention weighted processing is performed on the sample pulse image data based on the reference pulse image data to obtain second sample pulse image data; the first sample pulse image data and the second sample pulse image data are spliced ​​to obtain third sample pulse image data, and feature extraction processing is performed on the third sample pulse image data to obtain a pulse feature diagram corresponding to the sample pulse image data.

[0017] In one embodiment, the fusing of the local cost volume and the global cost volume to obtain a multi-time-step cost volume memory includes:

[0018] Performing convolution processing on the local cost volume and the overall cost volume to obtain a fused cost volume;

[0019] Segmenting the fused cost volume to obtain a plurality of fused cost volume blocks, and for any of the fused cost volume blocks, performing self-attention downsampling processing on the fused cost volume block to obtain a downsampled feature map corresponding to the fused cost volume block;

[0020] Combining the downsampled feature maps corresponding to the fused cost volume blocks to obtain a downsampled cost volume, and performing self-attention weighted processing on the downsampled cost volume in different dimensions to obtain multiple target feature maps;

[0021] The target feature maps are fused to obtain a multi-time step cost volume memory.

[0022] In one embodiment, the reconstructing the multi-time-step cost volume memory to obtain the reconstructed particle flow field corresponding to the sample particle flow field region includes:

[0023] determining a self-attention weight matrix of the multi-time-step cost volume memory for the overall cost volume;

[0024] The multi-time-step cost volume memory is weighted based on the self-attention weight matrix to obtain a weighted multi-time-step cost volume memory, and the weighted multi-time-step cost volume memory is reconstructed to obtain a reconstructed particle flow field corresponding to the sample particle flow field area.

[0025] In one embodiment, the training of the initial particle flow field reconstruction model according to the difference between the reconstructed particle flow field and the actual particle flow field corresponding to the sample particle flow field region to obtain the particle flow field reconstruction model includes:

[0026] Acquire an actual particle flow field corresponding to the sample particle flow field region, and determine a first loss value based on a difference between a velocity vector at a corresponding position of the reconstructed particle flow field and the actual particle flow field;

[0027] Determine a second loss value based on the difference between velocity vectors at adjacent moments in the same position of the reconstructed particle flow field and the difference between velocity vectors at adjacent positions of the reconstructed particle flow field at the same moment;

[0028] Determining a third loss value based on a difference between velocity vectors corresponding to different illumination conditions at the same position of the reconstructed particle flow field;

[0029] The initial particle flow field reconstruction model is trained according to the first loss value, the second loss value and the third loss value to obtain a particle flow field reconstruction model.

[0030] In a second aspect, the present application provides a particle flow field reconstruction method. The method comprises:

[0031] Collecting multiple frames of pulse image data for the particle flow field area;

[0032] Inputting the multi-frame pulse image data into a particle flow field reconstruction model to obtain a reconstructed particle flow field corresponding to the particle flow field area;

[0033] Wherein, the particle flow field reconstruction model is obtained by training through the method described in any of the aforementioned embodiments.

[0034] In a third aspect, the present application also provides a training device for a particle flow field reconstruction model. The initial particle flow field reconstruction model includes a pulse encoding module, a cost volume fusion encoding module and a particle flow field reconstruction module, and the device includes:

[0035] An acquisition module, used for acquiring multiple frames of sample pulse image data collected from a sample particle flow field area;

[0036] An extraction module, used to perform feature extraction processing on the sample pulse image data collected in different time windows through the pulse encoding module, to obtain a plurality of pulse feature maps corresponding to different time windows, to determine the local cost volume based on the corresponding pulse feature maps in different time windows, and to determine the overall cost volume based on the pulse feature maps in all time windows;

[0037] A fusion module, used for fusing the local cost volume and the overall cost volume through the cost volume fusion encoding module to obtain a multi-time step cost volume memory;

[0038] A reconstruction module is used to reconstruct the multi-time-step cost volume memory through the particle flow field reconstruction module to obtain a reconstructed particle flow field corresponding to the sample particle flow field area, and to train the initial particle flow field reconstruction model based on the difference between the reconstructed particle flow field and the actual particle flow field corresponding to the sample particle flow field area to obtain a particle flow field reconstruction model.

[0039] In one embodiment, the pulse coding module includes a plurality of pulse encoders, the pulse encoder includes alternating pulse coding layers and dilated convolution layers, and the extraction module is further configured to:

[0040] The sample pulse image data collected in different time windows are respectively input into different pulse encoders, and sequentially pass through the alternating pulse coding layer and the hole convolution layer in the pulse encoder, and feature extraction processing is performed on the sample pulse image data collected in different time windows to obtain multiple pulse feature maps corresponding to different time windows.

[0041] In one embodiment, the extraction module is further used to:

[0042] Determine reference pulse image data from each of the sample pulse image data;

[0043] Through the pulse coding layer, the reference pulse image data is subjected to self-attention weighted processing to obtain first reference pulse image data, and the first reference pulse image data is subjected to feature extraction processing to obtain a pulse feature graph corresponding to the reference pulse image data;

[0044] For any sample pulse image data except the reference pulse image data in each of the sample pulse image data, self-attention weighted processing is performed on the sample pulse image data through the pulse coding layer to obtain first sample pulse image data, and cross-attention weighted processing is performed on the sample pulse image data based on the reference pulse image data to obtain second sample pulse image data; the first sample pulse image data and the second sample pulse image data are spliced ​​to obtain third sample pulse image data, and feature extraction processing is performed on the third sample pulse image data to obtain a pulse feature diagram corresponding to the sample pulse image data.

[0045] In one embodiment, the fusion module is further used for:

[0046] Performing convolution processing on the local cost volume and the overall cost volume to obtain a fused cost volume;

[0047] Segmenting the fused cost volume to obtain a plurality of fused cost volume blocks, and for any of the fused cost volume blocks, performing self-attention downsampling processing on the fused cost volume block to obtain a downsampled feature map corresponding to the fused cost volume block;

[0048] Combining the downsampled feature maps corresponding to the fused cost volume blocks to obtain a downsampled cost volume, and performing self-attention weighted processing on the downsampled cost volume in different dimensions to obtain multiple target feature maps;

[0049] The target feature maps are fused to obtain a multi-time step cost volume memory.

[0050] In one embodiment, the reconstruction module is further used to:

[0051] determining a self-attention weight matrix of the multi-time-step cost volume memory for the overall cost volume;

[0052] The multi-time-step cost volume memory is weighted based on the self-attention weight matrix to obtain a weighted multi-time-step cost volume memory, and the weighted multi-time-step cost volume memory is reconstructed to obtain a reconstructed particle flow field corresponding to the sample particle flow field area.

[0053] In one embodiment, the reconstruction module is further used for:

[0054] Acquire an actual particle flow field corresponding to the sample particle flow field region, and determine a first loss value based on a difference between a velocity vector at a corresponding position of the reconstructed particle flow field and the actual particle flow field;

[0055] Determine a second loss value based on the difference between velocity vectors at adjacent moments in the same position of the reconstructed particle flow field and the difference between velocity vectors at adjacent positions of the reconstructed particle flow field at the same moment;

[0056] Determining a third loss value based on a difference between velocity vectors corresponding to different illumination conditions at the same position of the reconstructed particle flow field;

[0057] The initial particle flow field reconstruction model is trained according to the first loss value, the second loss value and the third loss value to obtain a particle flow field reconstruction model.

[0058] In a fourth aspect, the present application also provides a particle flow field reconstruction device. The device comprises:

[0059] An acquisition module, used for acquiring multiple frames of pulse image data for a particle flow field area;

[0060] An input module, used for inputting the multi-frame pulse image data into a particle flow field reconstruction model to obtain a reconstructed particle flow field corresponding to the particle flow field area;

[0061] Wherein, the particle flow field reconstruction model is obtained by training through the method described in any of the aforementioned embodiments.

[0062] In a fifth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, any of the above methods is implemented.

[0063] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements any of the above methods when executed by a processor.

[0064] In a seventh aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0065] The training method and device of the particle flow field reconstruction model, the particle flow field reconstruction method and device, extract the pulse characteristic graph of the sample pulse image data collected in different time windows through the particle flow field reconstruction model, and then determine the local cost volume that can characterize the local changes of the particle flow field in different time windows according to the corresponding pulse characteristic graphs in different time windows, and determine the overall cost volume that can characterize the global changes of the particle flow field in a longer time according to the pulse characteristic graphs in all time windows, and then use the multi-time step cost volume memory that integrates the overall cost volume and each local cost volume to reconstruct the particle flow field, so that the model can simultaneously refer to the change information of the particle flow field on different time scales during the process of reconstructing the particle flow field. Compared with the method of reconstructing the flow field based on the displacement of particles between two adjacent frames of captured images, the time characteristic information used in the embodiment of the present application when reconstructing the flow field is richer, so the reconstruction accuracy of the particle flow field can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A schematic flow chart of a training method for a particle flow field reconstruction model in one embodiment;

[0067] Figure 2 is a schematic diagram of a particle flow field reconstruction model in one embodiment;

[0068] Figure 3 is a schematic diagram of a pulse encoder in one embodiment;

[0069] Figure 4 A schematic diagram of a process of extracting features by a pulse encoder in one embodiment;

[0070] Figure 5 is a flow chart of step 106 in one embodiment;

[0071] Figure 6 is a schematic diagram of a cost volume fusion encoding module in one embodiment;

[0072] Figure 7 is a schematic diagram of a flow chart of reconstructing a flow field in step 108 in one embodiment;

[0073] Figure 8 is a schematic diagram of a particle flow field reconstruction module in one embodiment;

[0074] Fig. 9 1 is a flow chart of training a model in step 108 in one embodiment;

[0075] Fig.10 A schematic diagram of a flow chart of a particle flow field reconstruction method in one embodiment;

[0076] Fig.11 is a structural block diagram of a training device for a particle flow field reconstruction model in one embodiment;

[0077] Fig.12 is a structural block diagram of a particle flow field reconstruction device in one embodiment;

[0078] Fig.13 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0080] In one embodiment, Figure 1 As shown, a training method for a particle flow field reconstruction model is provided. This embodiment takes the method applied to a server as an example. It is understandable that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0081] Step 102: Acquire multiple frames of sample pulse image data collected for the sample particle flow field area.

[0082] In the embodiment of the present application, pulse image data is used as sample data for training, and correspondingly in the application, pulse image data collected by a pulse high-speed camera is also used as input of the model. Since the pulse image data collected by a pulse high-speed camera has the characteristics of high temporal resolution, high sensitivity, and high dynamics, using pulse image data as input of the model can enable the model to obtain more information from the input data and improve the accuracy of reconstructing the flow field.

[0083] The sample particle flow field region can be a real flow field or a simulated flow field. When the sample particle flow field region is a real flow field, the actual particle flow field of the sample particle flow field region can be obtained by numerical simulation, laser velocimetry and other high-precision methods, and multi-frame sample pulse image data of the sample particle flow field region can be obtained by pulse high-speed camera acquisition. When the sample particle flow field region is a simulated flow field, the actual particle flow field of the sample particle flow field region is known, and the light intensity information of each position in the sample particle flow field region can also be simulated to synthesize the sample pulse image data of the sample particle flow field region.

[0084] It should be noted that, since the actual particle flow field obtained for the simulated flow field is generally more accurate than the actual particle flow field obtained for the real flow field, in the process of model training, the model can be first trained using sample data obtained based on the simulated flow field, and then the model can be fine-tuned using sample data obtained based on the real flow field. Alternatively, the model can be trained using all sample data obtained based on the simulated flow field, or using sample data obtained by mixing the simulated flow field and the real flow field, etc., which is not specifically limited in the embodiments of the present application.

[0085] Step 104, through the pulse coding module, feature extraction processing is performed on the sample pulse image data collected in different time windows, and multiple pulse feature maps corresponding to different time windows are obtained. The local cost volume is determined based on the corresponding pulse feature maps in different time windows, and the overall cost volume is determined based on the pulse feature maps in all time windows.

[0086] In the embodiment of the present application, the initial particle flow field reconstruction model includes a pulse coding module, a cost volume fusion coding module and a particle flow field reconstruction module. The pulse coding module is used to extract the features of the sample pulse image data (i.e., the cost volume), the cost volume fusion coding module is used to further process the features extracted by the pulse coding module, and the particle flow field reconstruction module is used to reconstruct the flow field based on the features output by the cost volume fusion coding module. The structure of the initial particle flow field reconstruction model can be found in Figure 2 shown.

[0087] A time window refers to a preset length of time. The time periods represented by different time windows can overlap, for example Figure 2 Time window 1 includes sample pulse image data 1, 2, 3 collected at time points 1, 2, 3, respectively, and time window 2 includes sample pulse image data 2, 3, 4 collected at time points 2, 3, 4, respectively. The specific time window setting method can be determined by those skilled in the art according to actual needs.

[0088] By performing feature extraction processing on all sample pulse image data collected within a time window through the pulse coding module, multiple pulse characteristic graphs corresponding to the time window can be obtained. Different pulse characteristic graphs can correspond to different sample pulse image data, that is, a pulse characteristic graph can be generated for each sample pulse image data in the time window; or different pulse characteristic graphs can also correspond to different feature dimensions, that is, the features of the sample pulse image data can be extracted by different feature extraction methods according to each sample pulse image data in the time window, to obtain pulse characteristic graphs on different feature dimensions. Alternatively, any method that can perform feature extraction processing on an image can be used to generate a pulse characteristic graph, and the embodiments of the present application do not specifically limit this.

[0089] The pulse encoding module performs the same processing on the sample pulse image data corresponding to each time window, so there is a corresponding relationship between the pulse characteristic diagrams of different time windows. For example, when a pulse characteristic diagram is generated for each sample pulse image data, the corresponding pulse characteristic diagrams in different time windows refer to the pulse characteristic diagrams corresponding to the sample pulse image data with the same acquisition order. For example, when time window 1 corresponds to the sample pulse image data A, B, and C acquired in sequence, and time window 2 corresponds to the sample pulse image data D, E, and F acquired in sequence, and the pulse encoding module generates pulse characteristic diagrams a, b, c, d, e, and f for the sample pulse image data A, B, C, D, E, and F, respectively, the corresponding pulse characteristic diagrams in time windows 1 and 2 are pulse characteristic diagrams a and d (the corresponding acquisition order of the sample pulse image data in their respective time windows is 1), b and e (the corresponding acquisition order of the sample pulse image data in their respective time windows is 2), and c and f (the corresponding acquisition order of the sample pulse image data in their respective time windows is 3).

[0090] For another example, when different pulse feature graphs correspond to different feature dimensions, the corresponding pulse feature graphs in different time windows refer to pulse feature graphs with the same corresponding feature dimensions. For example, based on the sample pulse image data in time window 1, a pulse feature graph g1 corresponding to feature dimension G and a pulse feature graph h1 corresponding to feature dimension H are generated respectively, and based on the sample pulse image data in time window 2, a pulse feature graph g2 corresponding to feature dimension G and a pulse feature graph h2 corresponding to feature dimension H are generated respectively, the corresponding pulse feature graphs in time windows 1 and 2 are pulse feature graphs g1 and g2 (both corresponding to feature dimension G), and h1 and h2 (both corresponding to feature dimension H).

[0091] After determining the corresponding pulse feature graphs in different time windows, the local cost volume can be extracted based on the corresponding pulse feature graphs. The embodiment of the present application does not specifically limit how many different time windows are used each time the local cost volume is extracted. The local cost volume can be extracted once for every two adjacent time windows, once for multiple adjacent time windows, and once for time windows with a preset time interval between them. The specific details can be determined according to the actual needs of those skilled in the art.

[0092] The cost volume is used to characterize the correlation between image data, so the local cost volume extracted according to the corresponding pulse feature map can characterize the changes in the pulse feature in different time windows, that is, the local motion information of the particle flow field over a period of time. The cost volume can be obtained by performing a correlation operation on two pulse feature maps, where the correlation operation includes but is not limited to convolution, calculation of similarity, etc., which can extract the correlation between the two pulse feature maps, and the embodiments of the present application do not specifically limit this.

[0093] The overall cost volume can also be determined based on the pulse feature maps in all time windows. For example, the pulse feature maps corresponding to different time windows can be fused (the fusion methods include but are not limited to splicing, superposition, convolution, etc.), and then the fused feature maps corresponding to each time window can be correlated to obtain an overall cost volume. That is, when time windows 1 and 2 correspond to pulse feature maps abc and def respectively, a fused feature map can be generated based on abc. Figure X , a fused feature map Y is generated according to def, and then X and Y are correlated to obtain the overall cost volume. Alternatively, each pulse feature map can be correlated to obtain an overall cost volume, that is, abcdef can be correlated to obtain the overall cost volume. The overall cost volume can characterize the changes in pulse features during the entire sample pulse image data acquisition period, that is, the global motion information of the particle flow field over a longer period of time. This method of encoding long-term pulse image data can extract feature information over a wide range of time dimensions, thereby solving the problem of reconstruction difficulties caused by the displacement of particles less than one pixel at a moment and insufficient reconstruction information for dense flow fields.

[0094] Step 106 , the local cost volume and the overall cost volume are fused by a cost volume fusion encoding module to obtain a multi-time step cost volume memory.

[0095] In the embodiment of the present application, the local cost volume and the overall cost volume are fused to obtain a multi-time step cost volume memory, so that the multi-time step cost volume memory can contain the global motion information of the particle flow field in a long time and the local motion information in a short time. The fusion method includes but is not limited to splicing, superposition, convolution, etc., which is not specifically limited in the embodiment of the present application.

[0096] Step 108, reconstruct the multi-time step cost volume memory through the particle flow field reconstruction module to obtain the reconstructed particle flow field corresponding to the sample particle flow field area, and train the initial particle flow field reconstruction model according to the difference between the reconstructed particle flow field and the actual particle flow field corresponding to the sample particle flow field area to obtain the particle flow field reconstruction model.

[0097] In the embodiment of the present application, a reconstructed particle flow field is obtained based on multi-time step cost volume memory reconstruction. Then, the loss value of the initial particle flow field reconstruction model can be calculated based on the difference between the reconstructed particle flow field and the actual particle flow field corresponding to the sample particle flow field area, and then the pulse coding module, cost volume fusion coding module and particle flow field reconstruction module of the initial particle flow field reconstruction model are trained according to the loss value. The embodiment of the present application does not limit the specific loss value calculation method used in the model training process, and any loss value calculation method is applicable to the embodiment of the present application.

[0098] The training method of the particle flow field reconstruction model provided in the embodiment of the present application extracts the pulse characteristic graph of the sample pulse image data collected in different time windows through the particle flow field reconstruction model, and then determines the local cost volume that can characterize the local changes of the particle flow field in different time windows according to the corresponding pulse characteristic graphs in different time windows, and determines the overall cost volume that can characterize the global changes of the particle flow field in a longer time according to the pulse characteristic graphs in all time windows, and then uses the multi-time step cost volume memory that integrates the overall cost volume and each local cost volume to reconstruct the particle flow field, so that the model can simultaneously refer to the change information of the particle flow field on different time scales in the process of reconstructing the particle flow field. Compared with the method of reconstructing the flow field according to the displacement of particles between two adjacent frames of images, the time characteristic information used in the embodiment of the present application when reconstructing the flow field is richer, so the reconstruction accuracy of the particle flow field can be improved.

[0099] In one embodiment, the pulse coding module includes a plurality of pulse encoders, and the pulse encoder includes an alternating pulse coding layer and a dilated convolution layer. In step 104, the pulse coding module performs feature extraction processing on sample pulse image data collected in different time windows to obtain a plurality of pulse feature maps corresponding to different time windows, including:

[0100] The sample pulse image data collected in different time windows are input into different pulse encoders respectively, and sequentially pass through the alternating pulse coding layer and the hole convolution layer in the pulse encoder to perform feature extraction processing on the sample pulse image data collected in different time windows, so as to obtain multiple pulse feature maps corresponding to different time windows.

[0101] In the embodiment of the present application, multiple pulse encoders are used to synchronously process sample pulse image data collected in different time windows. Figure 3 As shown in Figure 2, each pulse encoder consists of alternating pulse coding layers and dilated convolutional layers ( Figure 3 In the above figure, a convolutional layer is set before the first pulse coding layer to perform a preliminary feature extraction process on the sample pulse image data. The pulse coding layer is used to extract the correlation features and motion features of particles at different times from the sample pulse image data, and the atrous convolutional layer is used to expand the receptive field of the model and extract more global context information.

[0102] After processing the sample pulse image data in a time window through alternating pulse coding layers and void convolution layers multiple times, multiple pulse feature maps corresponding to the time window can be obtained. Then, the local cost volume and the overall cost volume can be calculated based on the pulse feature maps corresponding to different time windows output by different pulse encoders.

[0103] The training method of the particle flow field reconstruction model provided in the embodiment of the present application generates a pulse feature map through alternating pulse coding layers and dilated convolution layers. Since dilated convolution has the function of expanding the receptive field and capturing more contextual information, by repeatedly extracting the features of the sample pulse image data and processing the sample pulse image data by dilated convolution of the features, the pulse feature map finally generated can contain pulse image feature information spanning a longer time period, thereby improving the accuracy of the cost volume generated based on the pulse feature map, and thus improving the accuracy of the particle flow field reconstruction model in reconstructing the flow field.

[0104] In one embodiment, Figure 4 As shown, the sample pulse image data is subjected to feature extraction processing through the pulse coding layer, including:

[0105] Step 402, determining reference pulse image data from each sample pulse image data;

[0106] Step 404, performing self-attention weighted processing on the reference pulse image data through the pulse coding layer to obtain first reference pulse image data, and performing feature extraction processing on the first reference pulse image data to obtain a pulse feature map corresponding to the reference sample data frame;

[0107] Step 406, for any sample pulse image data in each pulse image data except the reference pulse image data, perform self-attention weighted processing on the sample pulse image data through the pulse coding layer to obtain first sample pulse image data, and perform cross-attention weighted processing on the sample pulse image data based on the reference pulse image data to obtain second sample pulse image data; splice the first sample pulse image data and the second sample pulse image data to obtain third sample pulse image data, and perform feature extraction processing on the third sample pulse image data to obtain a pulse feature diagram corresponding to the pulse image data.

[0108] In the embodiment of the present application, the reference pulse image data is any one of the sample pulse image data. The pulse coding layer extracts features for each sample pulse image data through a separate feature extraction network, but different feature extraction networks interact through an attention mechanism, so that the pulse coding layer can capture the correlation information between each sample pulse image data and the reference pulse image data, so that the pulse feature map can reflect the correlation and motion information between particles at different times.

[0109] In order to make the pulse characteristic diagrams output by different pulse encoders correspond to each other, the acquisition order of the reference pulse image data corresponding to each time window can be made the same. For example, for each time window, the sample pulse image data with an acquisition order of 2 can be selected as the reference pulse image data.

[0110] For the reference pulse image data, the self-attention weight matrix of the reference pulse image data can be obtained based on the self-attention mechanism, and then the reference pulse image data is subjected to self-attention weighted processing based on the self-attention weight matrix to obtain the first reference pulse image data. In the self-attention mechanism, the reference pulse image data is first mapped to a query vector (Query), a key vector (Key), and a value vector (Value) respectively through a trainable weight matrix, and then the similarity score between each query vector and each key vector is determined, and the self-attention weight matrix can be obtained after normalizing the similarity score, and finally the self-attention weight matrix and the value vector are multiplied to obtain the first reference pulse image data after self-attention weighted processing.

[0111] Then, a pulse feature map corresponding to the reference pulse image data can be generated according to the first reference pulse image data. The pulse feature map is extracted by the self-attention mechanism, so that the pulse feature map contains the correlation information of different particles at a single moment.

[0112] For other sample pulse image data, the self-attention weight matrix of the sample pulse image data can be obtained based on the self-attention mechanism, and the cross-attention weight matrix of the sample pulse image data with respect to the reference pulse image data can be obtained based on the cross-attention mechanism. In the cross-attention mechanism, the reference pulse image data is first mapped to a query vector (Query) through a trainable weight matrix, and the sample pulse image data is mapped to a key vector (Key) and a value vector (Value), and then the similarity score between each query vector and each key vector is determined. After normalizing the similarity score, the cross-attention weight matrix can be obtained. Finally, the cross-attention weight matrix and the value vector are multiplied to obtain the second reference pulse image data after cross-attention weighting processing.

[0113] The sample pulse image data is processed by self-attention weighting through the self-attention weight matrix to obtain the first sample pulse image data, and the sample pulse image data is processed by cross-attention weighting through the cross-attention weight matrix to obtain the second sample pulse image data, and then the first sample pulse image data and the second sample pulse image data are spliced ​​to obtain the third sample pulse image data, and then the pulse feature map corresponding to the sample pulse image data can be generated according to the third sample pulse image data. The pulse feature map is extracted by the self-attention mechanism and the cross-attention mechanism, so that the pulse feature map contains the association information of different particles at a single moment, as well as the movement information of particles at different moments.

[0114] Taking the input of the pulse coding layer as the sample pulse image data at three times t1, t2 and t3 (respectively represented as S(t1), S(t2), S(t3)), and the reference pulse image data as S(t2), the above process can be expressed by formula (1):

[0115] Formula (I)

[0116] Among them, F(t2) is the pulse feature map generated for the sample pulse image data S(t2), similarly F(t1) is the pulse feature map of S(t1), and F(t3) is the pulse feature map of S(t3). E represents the feature extraction network, SA represents the self-attention weighted operation, and CA represents the cross-attention weighted operation.

[0117] The training method of the particle flow field reconstruction model provided in the embodiment of the present application generates a pulse feature map for each sample pulse image data through a self-attention mechanism and a cross-attention mechanism. Therefore, the pulse feature map can contain not only the correlation information between different particles at a single moment, but also the correlation information between particles at different moments, thereby improving the accuracy of the cost volume generated based on the pulse feature map and improving the reconstruction accuracy of the particle flow field.

[0118] In one embodiment, Figure 5 As shown, in step 106, the local cost volume and the overall cost volume are fused to obtain a multi-time step cost volume memory, including:

[0119] Step 502, performing convolution processing on the local cost volume and the overall cost volume to obtain a fused cost volume;

[0120] Step 504, segmenting the fused cost volume to obtain a plurality of fused cost volume blocks, and for any of the fused cost volume blocks, performing self-attention downsampling processing on the fused cost volume block to obtain a downsampled feature map corresponding to the fused cost volume block;

[0121] Step 506, combining the downsampled feature maps corresponding to the fused cost volume blocks to obtain a downsampled cost volume, and performing self-attention weighted processing on the downsampled cost volume in different dimensions to obtain multiple target feature maps;

[0122] Step 508: performing fusion processing on each of the target feature maps to obtain a multi-time step cost volume memory.

[0123] In the embodiment of the present application, the local cost volume and the overall cost volume are first convolved to obtain a fused cost volume, and then the fused cost volume is segmented to obtain a plurality of fused cost volume blocks, and each fused cost volume block is downsampled by self-attention to obtain a downsampled feature map of each fused cost volume block, and then each downsampled feature map is reassembled. Then, the features of the downsampled cost volume obtained after the splicing are extracted in different dimensions through the self-attention mechanism to obtain a target feature map representing the features of the fused cost volume in different dimensions, and each target feature map is fused to obtain a multi-time step cost volume memory. Through the above method, the multi-time step cost volume memory finally obtained can contain rich cross-time flow field information, so that the subsequent flow field reconstruction can use the information of different time steps to make up for the problem of insufficient reconstruction information caused by the sub-pixel displacement of particles and the uneven distribution of dense flow fields in the current time step, thereby improving the accuracy of flow field reconstruction.

[0124] See also Figure 6 As shown, it is a schematic diagram of the cost volume fusion coding module. After the local cost volume and the overall cost volume are input into the cost volume fusion coding module, the local cost volume and the overall cost volume can be fused sequentially through convolution and dilated convolution operations to obtain a fused cost volume. The fused cost volume can then be divided into multiple blocks, and features can be extracted for each block to speed up the processing speed of the model and make the extracted local features richer. The embodiment of the present application does not specifically limit the specific strategy of block division.

[0125] For each fused cost volume block, self-attention downsampling processing can be performed to obtain a downsampled feature map. Self-attention downsampling processing refers to the fusion of downsampling and self-attention in one step. A key vector that is trainable and has a dimension smaller than the dimension of the fused cost volume block can be pre-set, and then the query vector and value vector are calculated based on the fused cost volume block. The downsampled self-attention feature map is calculated by multiplying the query vector and the key vector, and finally the downsampled feature map of the fused cost volume block is obtained based on the downsampled self-attention feature map and the value vector. Since the dimension of the key vector is smaller than the dimension of the fused cost volume block, the dimension of the downsampled self-attention feature map is also smaller than the dimension of the fused cost volume block. Then, based on the attention mechanism, when the value vector is processed by the downsampled self-attention feature map to obtain the downsampled feature map, the dimension of the final downsampled feature map will also be smaller than the dimension of the fused cost volume block, so that the fusion of downsampling and self-attention is completed in one step.

[0126] After obtaining the downsampled feature maps of different fused cost volume blocks, the inverse operation of the segmentation operation can be performed on each downsampled feature map to combine each downsampled feature map into a downsampled cost volume. Then, by segmenting the downsampled cost volume in different dimensions (for example, the downsampled cost volume can be segmented on the xy axis, on the xz axis, on the yz axis, etc.), and mapping the blocks obtained after segmentation, query vectors located in different dimensions are obtained. Then, the key vector and value vector can be obtained based on the downsampled cost volume mapping. For each key value, the value vector is weighted through the self-attention mechanism to obtain the target feature map.

[0127] After obtaining multiple target feature maps, they can be fused through weighted summation, splicing, convolution, etc. to obtain multi-time step cost volume memory.

[0128] The training method of the particle flow field reconstruction model provided in the embodiment of the present application is to perform preliminary fusion of the local cost volume and the overall cost volume, divide the fused cost volume obtained by fusion into multiple blocks, perform self-attention downsampling on each block, and splice the results obtained by self-attention downsampling to obtain the downsampled cost volume, perform self-attention weighted processing on the downsampled cost volume in different dimensions, obtain multiple target feature maps, and then fuse each target feature map into a multi-time step cost volume memory. Therefore, the information in the local cost volume and the overall cost volume extracted from different dimensions can be used to reconstruct the flow field, thereby improving the flow field prediction accuracy.

[0129] In one embodiment, Figure 7 As shown, in step 108, the multi-time step cost volume memory is reconstructed to obtain a reconstructed particle flow field corresponding to the particle flow field area, including:

[0130] Step 702, determining a self-attention weight matrix of the multi-time step cost volume memory for the overall cost volume;

[0131] Step 704, weighted processing is performed on the multi-time-step cost volume memory based on the self-attention weight matrix to obtain the weighted multi-time-step cost volume memory, and the weighted multi-time-step cost volume memory is reconstructed to obtain the reconstructed particle flow field corresponding to the sample particle flow field area.

[0132] In the present application examples, see Figure 8 The figure shows a schematic diagram of the particle flow field reconstruction module. The particle flow field reconstruction module uses an attention mechanism to reconstruct the flow field. First, the overall cost volume can be mapped to a query vector (Query) through a trainable weight matrix, and the multi-time step cost volume memory can be mapped to a key vector (Key) and a value vector (Value), and then the similarity score between each query vector and each key vector can be determined. After normalizing the similarity score, the self-attention weight matrix of the multi-time step cost volume memory for the overall cost volume can be obtained. Then, the multi-time step cost volume memory and the self-attention weight matrix can be multiplied, and the reconstructed particle flow field corresponding to the particle flow field area can be reconstructed based on the weighted multi-time step cost volume memory obtained after the multiplication.

[0133] Reference Figure 8 As shown, in one embodiment, a complete reconstructed particle flow field can also be reconstructed by multiple iterative cycles. Specifically, in the i-th cycle (i is a positive integer), the i-1th round reconstructed particle flow field reconstructed in the i-1th cycle is obtained (if i is 1, the initial reconstructed particle flow field is used as the i-1th round reconstructed particle flow field, and the initial reconstructed particle flow field can be a flow field with values ​​of 0 at all positions), and based on the i-1th round reconstructed particle flow field, the i-th round query vector is determined from the overall cost volume; based on the i-th round query vector and the key vector, the value vector is self-attention weighted, and the i-th round predicted particle flow field is reconstructed based on the value vector after self-attention weighting (the key vector and the value vector are obtained by memory mapping of the multi-time step cost volume); the i-th round predicted particle flow field is superimposed with the i-1th round reconstructed particle flow field to obtain the i-th round reconstructed particle flow field. The above process can be continuously executed until i reaches the preset number of iterations.

[0134] Among them, the query vector obtained from the overall cost volume in each round of iteration is the part of the overall cost volume that has a strong correlation with the reconstructed particle flow field of the previous round; the embodiment of the present application does not specifically limit how to search for the query vector of the i-th round from the overall cost volume based on the i-1th round of reconstructed particle flow field. For example, the i-1th round of reconstructed particle flow field can be converted into an image vector, and the overall cost volume and the i-1th round of reconstructed particle flow field can be convolved, and the convolution result can be used as the query vector of the i-th round, or a model such as a multi-layer perceptron, a GRU (Gated Recurrent Unit, gated recurrent unit) model can be trained, and the i-1th round of reconstructed particle flow field and the overall cost volume can be input into the model to obtain the query vector. By adopting this cyclic reconstruction method, subtle sub-pixel displacements can be converted into larger-scale, more predictable displacements based on global motion information within a wider time range, thereby effectively improving the accuracy of flow field reconstruction. The iterative prediction architecture based on the self-attention mechanism can also enable the model to learn the physical prior knowledge of the flow field during the training process, further enhancing the flow field reconstruction performance.

[0135] In one embodiment, Fig. 9 As shown, in step 108, the initial particle flow field reconstruction model is trained according to the difference between the reconstructed particle flow field and the actual particle flow field corresponding to the sample particle flow field area to obtain the particle flow field reconstruction model, including:

[0136] Step 902, obtaining an actual particle flow field corresponding to the sample particle flow field region, and determining a first loss value based on a difference between a velocity vector at a corresponding position of the reconstructed particle flow field and the actual particle flow field;

[0137] Step 904, determining a second loss value based on the difference between velocity vectors at adjacent moments in the same position of the reconstructed particle flow field and the difference between velocity vectors at adjacent moments in the reconstructed particle flow field;

[0138] Step 906, determining a third loss value based on the difference between velocity vectors at the same position of the reconstructed particle flow field corresponding to different illumination conditions;

[0139] Step 908: training the initial particle flow field reconstruction model according to the first loss value, the second loss value and the third loss value to obtain the particle flow field reconstruction model.

[0140] In the embodiment of the present application, a loss value of a fusion regularized loss calculation model based on a mixed constraint of temporal fluid smoothness and spatial neighbor fluid smoothness, and a constraint on consistency of flow field reconstruction under different illumination conditions is adopted.

[0141] The first loss value is obtained through the most basic loss function, which is used to measure the difference between the velocity vector at the corresponding position of the reconstructed particle flow field and the actual particle flow field. The difference can be calculated using the Euclidean distance. For details, please refer to formula (II):

[0142] Formula (II)

[0143] in, is the first loss value, H and W are the height and width of the flow field respectively, Represents the velocity vector at the position coordinate (i, j) in the reconstructed particle flow field. Represents the velocity vector of the actual particle flow field at the position coordinate (i, j).

[0144] The second loss value is a mixed constraint of temporal fluid smoothness and spatial neighbor fluid smoothness. Based on the principle of spatiotemporal continuity of the fluid field, the fluid velocity at adjacent locations in space should not change drastically at the same time; the fluid velocity at adjacent times at the same location should not change drastically. Therefore, formulas (3) and (4) can be obtained:

[0145] Formula (III)

[0146] Formula (IV)

[0147] in, represents the loss value calculated based on the fluid velocity at adjacent locations in space, represents the loss value calculated based on the fluid velocity at adjacent locations in time, Represents the velocity vector of the reconstructed particle flow field at time t and position coordinate (i, j). The second loss value can be obtained by adding the above two loss values. By minimizing these two regularization terms, the model can be encouraged to learn a smooth flow field that is more in line with physical laws, especially in areas where particles are sparse, which helps to improve the accuracy and stability of flow field reconstruction.

[0148] The third loss value is the consistency constraint of flow field reconstruction under different illumination conditions, which aims to ensure that the model is robust to illumination changes. For the same flow field and particle distribution, the model should predict the same flow field even if the illumination intensity is different. This constraint is implemented by calculating the difference in the reconstructed flow field under different illumination conditions. In one training process, different training samples can be constructed using sample pulse image data collected under different illumination conditions for the sample particle flow field area, and then the loss is calculated based on the difference between the reconstructed particle flow fields reconstructed from the training samples corresponding to different illumination conditions. By minimizing this loss, the model can be made insensitive to illumination changes, thereby improving its prediction accuracy and generalization ability under different illumination conditions. For details, please refer to formula (V):

[0149] Formula (V)

[0150] in, is the third loss value, Represents the velocity vector of the reconstructed particle flow field at position (i, j) under illumination condition I1. Represents the velocity vector under lighting condition I2.

[0151] The final loss value of the model can be obtained by summing or weighted summing the first, second, and third loss values. The initial particle flow field reconstruction model is trained using the loss value to obtain a particle flow field reconstruction model.

[0152] Compared with traditional image PIV technology and existing deep learning PIV methods based on traditional cameras, the above-mentioned particle flow field reconstruction model training method has the following significant advantages:

[0153] Accurate prediction of particle flow field: Based on the pulse image data that accurately depicts the particle motion trajectory, the complete flow field information is inferred, which makes up for the lack of small-scale features in the flow field reconstruction of traditional PIV technology.

[0154] Detailed analysis capability of high-speed flow fields: Pulse image data has high temporal resolution, which enables the model provided in the embodiments of the present application to analyze high-speed flow fields, break through the limitations of traditional camera frame rates, and accurately capture particle displacements in extremely short periods of time, thereby revealing transient phenomena such as fine vortex structures and shock wave propagation in high-speed flow fields.

[0155] End-to-end solution for pulse flow based on deep learning: The deep learning model is used to directly convert the data collected by the pulse high-speed camera into a flow field map, without the need for complex image pre-processing and post-processing steps, which simplifies the PIV process and improves efficiency. At the same time, the model does not simply accumulate pulse image data into images, but extracts spatiotemporal features from pulse image data of multiple time windows, and generates multiple local cost volumes and an overall cost volume to more finely capture the motion information of particles on different time scales; multiple cost volumes are fused and encoded to extract richer spatiotemporal information, improve the density of flow field reconstruction, and provide more comprehensive information for subsequent flow field reconstruction. By using the attention mechanism and the iterative prediction method, the flow field velocity information is extracted from the fused cost volume, and the prediction accuracy is improved through iterative optimization and learning of physical prior knowledge, which effectively solves the problem of insufficient small-scale feature information of the flow field.

[0156] In one embodiment, a particle flow field reconstruction method is provided, such as Fig.10 As shown, including:

[0157] Step 1002, collecting multiple frames of pulse image data for the particle flow field area;

[0158] Step 1004, inputting the multi-frame pulse image data into the particle flow field reconstruction model to obtain a reconstructed particle flow field corresponding to the particle flow field area;

[0159] The particle flow field reconstruction model is obtained by training using the method of any of the aforementioned embodiments.

[0160] In the embodiment of the present application, when the model is applied, a pulse high-speed camera can be used to collect multiple frames of pulse image data for the area where the flow field needs to be reconstructed, and then the multiple frames of pulse image data are input into the particle flow field reconstruction model trained in the above embodiment, and the reconstructed particle flow field corresponding to the particle flow field area can be obtained. The specific processing process of the particle flow field reconstruction model on the pulse image data can be referred to the description of the above embodiment, and the embodiment of the present application will not be repeated.

[0161] The particle flow field reconstruction method provided in the embodiment of the present application extracts the pulse characteristic graph of the sample pulse image data collected in different time windows through the particle flow field reconstruction model, and then determines the local cost volume that can characterize the local changes of the particle flow field in different time windows according to the corresponding pulse characteristic graphs in different time windows, and determines the overall cost volume that can characterize the global changes of the particle flow field in a longer time according to the pulse characteristic graphs in all time windows, and then uses the multi-time step cost volume memory that integrates the overall cost volume and each local cost volume to reconstruct the particle flow field, so that the model can simultaneously refer to the change information of the particle flow field on different time scales in the process of reconstructing the particle flow field. Compared with the method of reconstructing the flow field based on the displacement of particles between two adjacent frames of images, the time characteristic information used in the embodiment of the present application when reconstructing the flow field is richer, so the reconstruction accuracy of the particle flow field can be improved.

[0162] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0163] Based on the same inventive concept, the embodiment of the present application also provides a training device for a particle flow field reconstruction model for implementing the training method for the particle flow field reconstruction model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more particle flow field reconstruction models provided below can refer to the limitations of the training method for the particle flow field reconstruction model above, and will not be repeated here.

[0164] In one embodiment, Fig.11 As shown, a training device 1100 for a particle flow field reconstruction model is provided, comprising: an acquisition module 1102, an extraction module 1104, a fusion module 1106, and a reconstruction module 1108, wherein:

[0165] An acquisition module 1102 is used to acquire multiple frames of sample pulse image data collected from a sample particle flow field area;

[0166] The extraction module 1104 is used to perform feature extraction processing on the sample pulse image data collected in different time windows through the pulse encoding module, obtain multiple pulse feature maps corresponding to different time windows, determine the local cost volume based on the corresponding pulse feature maps in different time windows, and determine the overall cost volume based on the pulse feature maps in all time windows;

[0167] A fusion module 1106 is used to fuse the local cost volume and the overall cost volume through the cost volume fusion encoding module to obtain a multi-time step cost volume memory;

[0168] The reconstruction module 1108 is used to reconstruct the multi-time-step cost volume memory through the particle flow field reconstruction module to obtain the reconstructed particle flow field corresponding to the sample particle flow field area, and train the initial particle flow field reconstruction model according to the difference between the reconstructed particle flow field and the actual particle flow field corresponding to the sample particle flow field area to obtain the particle flow field reconstruction model.

[0169] The training device of the particle flow field reconstruction model provided in the embodiment of the present application extracts the pulse characteristic graph of the sample pulse image data collected in different time windows through the particle flow field reconstruction model, and then determines the local cost volume that can characterize the local changes of the particle flow field in different time windows according to the corresponding pulse characteristic graphs in different time windows, and determines the overall cost volume that can characterize the global changes of the particle flow field in a longer time according to the pulse characteristic graphs in all time windows, and then uses the multi-time step cost volume memory that integrates the overall cost volume and each local cost volume to reconstruct the particle flow field, so that the model can simultaneously refer to the change information of the particle flow field on different time scales in the process of reconstructing the particle flow field. Compared with the method of reconstructing the flow field according to the displacement of particles between two adjacent frames of images, the time characteristic information used in the embodiment of the present application when reconstructing the flow field is richer, so the reconstruction accuracy of the particle flow field can be improved.

[0170] In one embodiment, the pulse coding module includes a plurality of pulse encoders, the pulse encoder includes alternating pulse coding layers and dilated convolution layers, and the extraction module 1104 is further configured to:

[0171] The sample pulse image data collected in different time windows are respectively input into different pulse encoders, and sequentially pass through the alternating pulse coding layer and the hole convolution layer in the pulse encoder, and feature extraction processing is performed on the sample pulse image data collected in different time windows to obtain multiple pulse feature maps corresponding to different time windows.

[0172] In one embodiment, the extraction module 1104 is further used to:

[0173] Determine reference pulse image data from each of the sample pulse image data;

[0174] Through the pulse coding layer, the reference pulse image data is subjected to self-attention weighted processing to obtain first reference pulse image data, and the first reference pulse image data is subjected to feature extraction processing to obtain a pulse feature graph corresponding to the reference pulse image data;

[0175] For any sample pulse image data except the reference pulse image data in each of the sample pulse image data, self-attention weighted processing is performed on the sample pulse image data through the pulse coding layer to obtain first sample pulse image data, and cross-attention weighted processing is performed on the sample pulse image data based on the reference pulse image data to obtain second sample pulse image data; the first sample pulse image data and the second sample pulse image data are spliced ​​to obtain third sample pulse image data, and feature extraction processing is performed on the third sample pulse image data to obtain a pulse feature diagram corresponding to the sample pulse image data.

[0176] In one embodiment, the fusion module 1106 is further configured to:

[0177] Performing convolution processing on the local cost volume and the overall cost volume to obtain a fused cost volume;

[0178] Segmenting the fused cost volume to obtain a plurality of fused cost volume blocks, and for any of the fused cost volume blocks, performing self-attention downsampling processing on the fused cost volume block to obtain a downsampled feature map corresponding to the fused cost volume block;

[0179] Combining the downsampled feature maps corresponding to the fused cost volume blocks to obtain a downsampled cost volume, and performing self-attention weighted processing on the downsampled cost volume in different dimensions to obtain multiple target feature maps;

[0180] The target feature maps are fused to obtain a multi-time step cost volume memory.

[0181] In one embodiment, the reconstruction module 1108 is further configured to:

[0182] determining a self-attention weight matrix of the multi-time-step cost volume memory for the overall cost volume;

[0183] The multi-time-step cost volume memory is weighted based on the self-attention weight matrix to obtain a weighted multi-time-step cost volume memory, and the weighted multi-time-step cost volume memory is reconstructed to obtain a reconstructed particle flow field corresponding to the sample particle flow field area.

[0184] In one embodiment, the reconstruction module 1108 is further configured to:

[0185] Acquire an actual particle flow field corresponding to the sample particle flow field region, and determine a first loss value based on a difference between a velocity vector at a corresponding position of the reconstructed particle flow field and the actual particle flow field;

[0186] Determine a second loss value based on the difference between velocity vectors at adjacent moments in the same position of the reconstructed particle flow field and the difference between velocity vectors at adjacent positions of the reconstructed particle flow field at the same moment;

[0187] Determining a third loss value based on a difference between velocity vectors corresponding to different illumination conditions at the same position of the reconstructed particle flow field;

[0188] The initial particle flow field reconstruction model is trained according to the first loss value, the second loss value and the third loss value to obtain a particle flow field reconstruction model.

[0189] In one embodiment, Fig.12 As shown, a particle flow field reconstruction device 1200 is provided, including: a collection module 1202, an input module 1204, wherein:

[0190] The acquisition module 1202 is used to acquire multiple frames of pulse image data for the particle flow field area;

[0191] An input module 1204 is used to input the multi-frame pulse image data into a particle flow field reconstruction model to obtain a reconstructed particle flow field corresponding to the particle flow field area;

[0192] The particle flow field reconstruction model is obtained by training using the method described in any of the aforementioned embodiments.

[0193] The particle flow field reconstruction device provided in the embodiment of the present application extracts the pulse characteristic graph of the sample pulse image data collected in different time windows through the particle flow field reconstruction model, and then determines the local cost volume that can characterize the local changes of the particle flow field in different time windows according to the corresponding pulse characteristic graphs in different time windows, and determines the overall cost volume that can characterize the global changes of the particle flow field in a longer time according to the pulse characteristic graphs in all time windows, and then uses the multi-time step cost volume memory that integrates the overall cost volume and each local cost volume to reconstruct the particle flow field, so that the model can simultaneously refer to the change information of the particle flow field on different time scales during the process of reconstructing the particle flow field. Compared with the method of reconstructing the flow field according to the displacement of particles between two adjacent frames of captured images, the time characteristic information used in the embodiment of the present application when reconstructing the flow field is richer, so the reconstruction accuracy of the particle flow field can be improved.

[0194] Each module in the above device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0195] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.13As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training method for a particle flow field reconstruction model is implemented.

[0196] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0197] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0198] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0199] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0200] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0201] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0202] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0203] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for a particle flow field reconstruction model, characterized in that: The initial particle flow field reconstruction model includes a pulse encoding module, a cost volume fusion encoding module and a particle flow field reconstruction module. The method includes: Acquire multiple frames of sample pulse image data collected for the sample particle flow field area; Through the pulse encoding module, feature extraction processing is performed on the sample pulse image data collected in different time windows to obtain multiple pulse feature maps corresponding to different time windows, and local cost volumes are determined based on the corresponding pulse feature maps in different time windows, and the overall cost volume is determined based on the pulse feature maps in all time windows; The local cost volume and the overall cost volume are fused by the cost volume fusion encoding module to obtain a multi-time step cost volume memory; The multi-time-step cost volume memory is reconstructed by the particle flow field reconstruction module to obtain a reconstructed particle flow field corresponding to the sample particle flow field area, and based on the difference between the reconstructed particle flow field and the actual particle flow field corresponding to the sample particle flow field area, the initial particle flow field reconstruction model is trained to obtain a particle flow field reconstruction model.

2. The method according to claim 1, characterized in that: The pulse coding module includes a plurality of pulse encoders, and the pulse encoder includes an alternating pulse coding layer and a hole convolution layer. The pulse coding module performs feature extraction processing on sample pulse image data collected in different time windows to obtain a plurality of pulse feature maps corresponding to different time windows, including: The sample pulse image data collected in different time windows are respectively input into different pulse encoders, and sequentially pass through the alternating pulse coding layer and the hole convolution layer in the pulse encoder, and feature extraction processing is performed on the sample pulse image data collected in different time windows to obtain multiple pulse feature maps corresponding to different time windows.

3. The method according to claim 2, characterized in that The feature extraction process is performed on the sample pulse image data through the pulse coding layer, including: Determine reference pulse image data from each of the sample pulse image data; Through the pulse coding layer, the reference pulse image data is subjected to self-attention weighted processing to obtain first reference pulse image data, and the first reference pulse image data is subjected to feature extraction processing to obtain a pulse feature graph corresponding to the reference pulse image data; For any sample pulse image data except the reference pulse image data in each of the sample pulse image data, self-attention weighted processing is performed on the sample pulse image data through the pulse coding layer to obtain first sample pulse image data, and cross-attention weighted processing is performed on the sample pulse image data based on the reference pulse image data to obtain second sample pulse image data; the first sample pulse image data and the second sample pulse image data are spliced ​​to obtain third sample pulse image data, and feature extraction processing is performed on the third sample pulse image data to obtain a pulse feature diagram corresponding to the sample pulse image data.

4. The method according to claim 1, characterized in that: The fusing process of the local cost volume and the overall cost volume to obtain a multi-time step cost volume memory includes: Performing convolution processing on the local cost volume and the overall cost volume to obtain a fused cost volume; Segmenting the fused cost volume to obtain a plurality of fused cost volume blocks, and for any of the fused cost volume blocks, performing self-attention downsampling processing on the fused cost volume block to obtain a downsampled feature map corresponding to the fused cost volume block; Combining the downsampled feature maps corresponding to the fused cost volume blocks to obtain a downsampled cost volume, and performing self-attention weighted processing on the downsampled cost volume in different dimensions to obtain multiple target feature maps; The target feature maps are fused to obtain a multi-time step cost volume memory.

5. The method according to claim 1, characterized in that The reconstructing the multi-time-step cost volume memory to obtain a reconstructed particle flow field corresponding to the sample particle flow field region includes: determining a self-attention weight matrix of the multi-time-step cost volume memory for the overall cost volume; The multi-time-step cost volume memory is weighted based on the self-attention weight matrix to obtain a weighted multi-time-step cost volume memory, and the weighted multi-time-step cost volume memory is reconstructed to obtain a reconstructed particle flow field corresponding to the sample particle flow field area.

6. The method according to claim 1, characterized in that The step of training the initial particle flow field reconstruction model according to the difference between the reconstructed particle flow field and the actual particle flow field corresponding to the sample particle flow field region to obtain the particle flow field reconstruction model comprises: Acquire an actual particle flow field corresponding to the sample particle flow field region, and determine a first loss value based on a difference between a velocity vector at a corresponding position of the reconstructed particle flow field and the actual particle flow field; Determine a second loss value based on the difference between velocity vectors at adjacent moments in the same position of the reconstructed particle flow field and the difference between velocity vectors at adjacent positions of the reconstructed particle flow field at the same moment; Determining a third loss value based on a difference between velocity vectors corresponding to different illumination conditions at the same position of the reconstructed particle flow field; The initial particle flow field reconstruction model is trained according to the first loss value, the second loss value and the third loss value to obtain a particle flow field reconstruction model.

7. A particle flow field reconstruction method, characterized in that: The method comprises: Collecting multiple frames of pulse image data for the particle flow field area; Inputting the multi-frame pulse image data into a particle flow field reconstruction model to obtain a reconstructed particle flow field corresponding to the particle flow field area; Wherein, the particle flow field reconstruction model is obtained by training through the method described in any one of claims 1-6.

8. A training device for a particle flow field reconstruction model, characterized in that: The initial particle flow field reconstruction model includes a pulse encoding module, a cost volume fusion encoding module and a particle flow field reconstruction module. The device includes: An acquisition module, used for acquiring multiple frames of sample pulse image data collected from a sample particle flow field area; An extraction module, used to perform feature extraction processing on the sample pulse image data collected in different time windows through the pulse encoding module, to obtain a plurality of pulse feature maps corresponding to different time windows, to determine the local cost volume based on the corresponding pulse feature maps in different time windows, and to determine the overall cost volume based on the pulse feature maps in all time windows; A fusion module, configured to fuse the local cost volume and the overall cost volume through the cost volume fusion encoding module to obtain a multi-time step cost volume memory; A reconstruction module is used to reconstruct the multi-time-step cost volume memory through the particle flow field reconstruction module to obtain a reconstructed particle flow field corresponding to the sample particle flow field area, and to train the initial particle flow field reconstruction model based on the difference between the reconstructed particle flow field and the actual particle flow field corresponding to the sample particle flow field area to obtain a particle flow field reconstruction model.

9. A particle flow field reconstruction device, characterized in that: The device comprises: An acquisition module, used for acquiring multiple frames of pulse image data for a particle flow field area; An input module, used for inputting the multi-frame pulse image data into a particle flow field reconstruction model to obtain a reconstructed particle flow field corresponding to the particle flow field area; Wherein, the particle flow field reconstruction model is obtained by training through the method described in any one of claims 1-6.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images