Three-dimensional imaging method fusing frequency domain multiplexing and time frame insertion technology

Abstract

The invention discloses a three-dimensional imaging method fusing frequency domain multiplexing and a time frame insertion technology. According to the method, two information superposition strategies are combined, so that the three-dimensional imaging time resolution which is several times of the limit frame rate of the camera is realized. According to the method, the frequency domain multiplexing technology is used for carrying out time integration on the to-be-measured object in the single-frame exposure time of the camera to superpose the motion states of the object at multiple moments, then the time frame interpolation technology is combined to carry out interpolation on the motion states at adjacent moments, and dynamic three-dimensional imaging exceeding the frame rate limit of the camera by more than one order of magnitude is achieved. According to the invention, high-speed three-dimensional imaging can be realized through a low-cost low-speed camera system, high measurement precision is maintained, and good expandability and system compatibility are achieved. The method is suitable for various dynamic three-dimensional measurement scenes such as aerospace, intelligent manufacturing, damage experiments and the like, and has wide application prospects and engineering value.

Description

Technical Field

[0001] This invention belongs to the field of computational optical imaging, specifically a three-dimensional imaging method that integrates frequency domain multiplexing and time interpolation techniques. Background Technology

[0002] Structured light 3D imaging technology, with its advantages of high precision and non-contact measurement, has been widely applied in fields such as industrial measurement, biological imaging, and autonomous driving. Fringe projection profilometry, as an important method within this technology, typically employs multi-frequency phase-shifting fringes for 3D measurement to obtain detailed topographic information of object surfaces. However, the temporal resolution of this method is limited by the camera frame rate, making it difficult to meet the 3D reconstruction requirements of high-speed moving scenes. In recent years, frequency domain multiplexing technology has been introduced into structured light measurement, which can effectively improve the temporal resolution of imaging while maintaining spatial resolution. However, due to spectral aliasing, the scalability and superposition factor of frequency domain multiplexing are limited. Summary of the Invention

[0003] The purpose of this invention is to provide a three-dimensional imaging method that integrates frequency domain multiplexing and time interpolation techniques.

[0004] The technical solution to achieve the purpose of this invention is: a three-dimensional imaging method that integrates frequency domain multiplexing and time interpolation techniques, comprising:

[0005] Step 1: Construct a stripe pattern sequence, which includes a stripe sequence I composed of stripes with different tilt angles in the same frequency domain, and a stripe sequence II composed of multi-step phase-shifted stripes corresponding to all the stripes with tilt angles in stripe sequence I.

[0006] Step 2: Construct the training dataset. Project fringe sequence I onto the surface of a dynamic object using a projector. Capture multiplexed fringe images and single fringe images modulated by the dynamic object surface using dual cameras under short exposure conditions. The multiplexed fringe images and single fringe images constitute dataset I. Project fringe sequence II onto the surface of the dynamic object using a projector. Simultaneously capture 12-step phase-shifted images corresponding to the single fringe images using dual cameras under short exposure conditions. Solve for the wrapped phase image using the multi-step phase-shifting method. The single fringe images and the wrapped phase corresponding to the single fringe images constitute dataset II.

[0007] Step 3: Construct a multiplexed image decoupling network CNN1, a wrapping phase prediction network CNN2, and a temporal interpolation network CNN3. Use the multiplexed stripe images from dataset I as input to multiplexed image decoupling network CNN1, and use single stripe images as labels to train multiplexed image decoupling network CNN1; use the single stripe images from dataset II as input to wrapping phase prediction network CNN2, and use the wrapping phase as output of wrapping phase prediction network CNN2 to train wrapping phase prediction network CNN2; use a stereo phase unfolding algorithm combined with calibration information to unfold the wrapping phase in dataset II and recover the depth information to obtain a depth image; use the depth images of the first and last two moments in any three adjacent moments as input to temporal interpolation network CNN3, and use the depth image of the middle moment as output of temporal interpolation network CNN3, select multiple sets of data to construct dataset III, and train temporal interpolation network CNN3;

[0008] Step 4: Project the fringe sequence I onto the surface of the object under test using a projector, and re-capture a new set of multiplexed fringe images; input the multiplexed fringe images into the trained multiplexed image decoupling network CNN1 to obtain a set of single fringe images; input the obtained single fringe images into the wrapping phase prediction network CNN2 to obtain the corresponding wrapping phase, and combine the dual-camera wrapping phase data and system calibration data to perform stereo phase unfolding and depth information restoration to obtain a depth image; input the obtained series of depth images into the temporal interpolation network CNN3 to obtain the optical flow field corresponding to the depth images at all adjacent time points, and use the optical flow field to perform temporal interpolation on the depth image sequence to obtain an interpolated image sequence; perform 3D reconstruction on all depth images in the interpolated image sequence, convert them into dynamic results, and achieve high-speed dynamic 3D imaging that exceeds the camera frame rate limit.

[0009] Compared with the prior art, the significant advantages of this invention are: by innovatively combining frequency domain multiplexing technology with time frame interpolation technology, it is possible to effectively improve the temporal resolution of three-dimensional measurement to a level far exceeding the camera's limit frame rate while maintaining spatial resolution, thereby achieving fine three-dimensional dynamic shape capture of fast-moving objects, breaking through the limitation of traditional camera frame rate on the temporal resolution of three-dimensional measurement, and realizing dynamic three-dimensional imaging with high spatiotemporal resolution.

[0010] This method uses deep learning technology to recover high temporal resolution 3D morphology in multiplexed stripe decoupling and depth image temporal interpolation. It avoids dependence on expensive high-speed cameras, high-speed projectors and high-precision complex systems, reduces system hardware costs and simplifies system synchronization design, and has higher feasibility and promotion value.

[0011] The method of this invention has strong system compatibility and high adaptability. It can be easily integrated into various existing structured light 3D measurement platforms without adjusting the original hardware system. It can be quickly applied to existing measurement environments and has good scalability and flexibility. Attached Figure Description

[0012] Figure 1 This is a flowchart of the present invention.

[0013] Figure 2 This is a schematic diagram of image projection acquisition in this invention.

[0014] Figure 3 This is a diagram of the time-interpolation network structure.

[0015] Figure 4 This is a diagram showing the intermediate results of the experiment and the depth changes before and after frame interpolation at selected points. Detailed Implementation

[0016] This invention proposes a three-dimensional imaging method that integrates frequency domain multiplexing and temporal interpolation techniques, such as... Figure 1 As shown, this invention aims to combine frequency domain multiplexing and temporal frame interpolation techniques with traditional structured light 3D imaging methods based on deep learning. It decodes a series of high-precision 3D results at different times from a single-frame multiplexed image captured by a camera, achieving structured light 3D reconstruction that exceeds the camera's frame rate limit by an order of magnitude. This invention mainly includes the following four steps:

[0017] Step 1: Construct the stripe pattern sequence. Construct stripe sequence I and stripe sequence II for projector projection. Stripe sequence I is defined as follows:

[0018]

[0019] Among them, a p b represents the background average intensity of stripe sequence I. p This indicates the modulation density of stripe sequence I. Let x be the frequency of the projected fringes in fringe sequence I in the horizontal and vertical directions. p y p θ represents the pixel coordinates in the horizontal and vertical directions on the projection plane of the projector. m θ represents the angle between the stripe and the vertical direction. Where, when m = 1, 2, 3, θ m = -45°, 0°, 45°.

[0020] The stripes in stripe sequence II are defined as follows:

[0021]

[0022] in, This indicates that when the angle between the stripe and the vertical direction is θ m At that time, the intensity distribution corresponding to the nth image of the 12-step phase-shifting fringes, n=1,2,...,12.

[0023] Stripe sequence I consists of three stripes, namely... The angles between the fringe pattern and the vertical direction are -45°, 0°, and 45°, respectively; fringe sequence II is derived from fringe sequence I. The corresponding 12-step phase-shifting images are as follows: There are a total of 36 stripes, where m = 1, 2, ..., 12.

[0024] Step 2: Construct the training dataset. For example... Figure 2 As shown, the projector first projects a predefined fringe sequence I onto the surface of the object to be measured. The projector cyclically projects all fringe images from fringe sequence I, and the camera exposure time is set to be the same as the projector response time. Assuming that each group of three single-fringe images captured by the camera is I1, I2, and I3, they are combined to form a multiplexed fringe image I. LE . Will I LE I1, I2, and I3 are considered as a subset of dataset I. Several different subsets are combined to form dataset I.

[0025] Next, the fringe sequence II is cyclically projected onto the surface of the object under test using a projector. Simultaneous camera captures images, yielding 12 phase-shifted images corresponding to the single-fringe images in dataset I. After obtaining 12 phase-shifted fringe images in three different directions from fringe sequence II, the least squares method is used to analyze these fringes, yielding the wrapping phase for each direction. Using the single-fringe images and their corresponding wrapping phases as a subset, several different subsets are combined to form dataset II.

[0026] Step 3: Construct and train the neural network. First, construct a multiplexed image decoupling network CNN1 and a wrapper phase prediction network CNN2 based on the MultiResUnet network structure. The multiplexed image decoupling network CNN1 takes the multiplexed stripe image as input, with 1 input channel and 3 output channels, corresponding to three decoupled single-frequency stripe images. CNN1 adopts a multi-scale residual U-Net structure, with the backbone consisting of an encoder and a decoder. The encoder consists of four scale layers, each containing multiple 3×3 convolutions, batch normalization, and ReLU activation. Downsampling is achieved through 2×2 max pooling. The decoder is symmetrically set with four scale layers, each containing upsampling operations, cross-layer feature concatenation, and convolutional residual blocks to gradually recover high-resolution features and pass the encoded feature information to the output. Finally, at the end of the decoder, a 1×1 convolution maps the number of channels to a 3-channel output, resulting in three single-frequency stripe images.

[0027] The wrapping phase prediction network takes a single-frequency stripe image as input, with 1 input channel and 1 output channel, and outputs the wrapping phase map of the corresponding stripe. The network structure is consistent with the decoupled network, including a 4-layer encoder-decoder structure and multi-scale residual blocks. At the end of the network, high-dimensional features are also mapped to a 1-channel output through 1×1 convolution to accurately regress pixel-level phase values ​​and obtain high-precision wrapping phase.

[0028] A temporal frame interpolation network, CNN3, is constructed based on the PWC-Net network architecture. The CNN3 structure is as follows: Figure 3 As shown, the pyramid feature extraction layer generates a four-layer feature pyramid from two consecutive input frames using a weighted convolutional encoder. The feature resolutions of each layer are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image resolution, respectively. The first cost volume layer uses a local window to calculate the correlation of each pixel position on the output feature tensor plane of the top layer of the multi-layer feature pyramid to obtain the cost volume feature. The first optical flow prediction layer uses a multi-layer convolutional network structure to predict the optical flow residual from the cost volume feature output by the first cost volume layer to obtain the first optical flow field. After upsampling the first optical flow field, it is fused with the corresponding size feature tensor output by the multi-layer feature pyramid to obtain the fused feature tensor. The second cost volume layer of the first optical flow prediction and fusion module uses a local window to calculate the correlation of each pixel position on the fused feature tensor plane to obtain the fused cost volume feature. The second optical flow prediction layer employs a multi-layer convolutional network structure to predict optical flow residuals from the fused cost volume features, obtaining the second optical flow field. The context fusion layer of the first optical flow prediction and fusion module performs deep fusion of the second optical flow field with the same-size feature tensor output by the pyramid feature extraction layer to obtain fused context features. The second optical flow field is upsampled and fused with the corresponding-size feature tensor output by the multi-layer feature pyramid to obtain a fused feature tensor. The fused context features output by the context fusion layer of the first optical flow prediction and fusion module are then bilinearly interpolated and superimposed with the fused feature tensor before being input into the second cost volume layer of the second optical flow prediction and fusion module to obtain the third optical flow field. The third optical flow field is then input into the context fusion layer to obtain the fused cost volume features. The third optical flow prediction and fusion module processes the data in the same way as the second optical flow prediction and fusion module, and the final optical flow field is output by the context fusion layer.

[0029] The frame interpolation synthesis layer resamples the two frames of the input CNN3 image based on the predicted final optical flow field and then weights and fuses the resampled results to obtain the synthesized image corresponding to the interpolation time point.

[0030] After constructing the three networks, CNN1 is trained using multiplexed stripe images from dataset I as input and single stripe images as labels; CNN2 is trained using single stripe images from dataset II as input and wrapped phase images as output; a stereo phase unwrapping algorithm combined with system calibration information is used to unwrap the wrapped phase images from dataset II and recover depth information to obtain a depth image. Dataset III is constructed by selecting multiple sets of data, using the depth images of the first and last two time points from any three adjacent time points as input and the depth image of the middle time point as output, and is used to train CNN3.

[0031] The network loss function of the temporal interpolation network CNN3 is a weighted joint loss function consisting of photometric consistency loss, edge-aware smoothing loss, and interpolation reconstruction loss. The photometric consistency loss measures the similarity in pixel brightness between the interpolated result and the real intermediate frame, comparing the interpolated result output by the interpolation synthesis layer with the real intermediate frame pixel-by-pixel to calculate the brightness difference. The edge-aware smoothing loss regularizes the spatial continuity of the predicted optical flow, generating an edge weight map based on the gradient information of the current frame image. The interpolation reconstruction loss measures the absolute difference in pixel space between the interpolated result and the real intermediate frame, backpropagating the pixel-by-pixel difference between the interpolated result and the real frame using the L1 norm, and guiding the interpolation synthesis layer to optimize the accuracy of the interpolation result. The three losses are assigned weights λ1, λ2, and λ3 respectively to form the overall loss function. The total loss function is defined as:

[0032]

[0033] In the formula For loss of photometric uniformity, For edge-aware optical flow smoothing loss, The interpolation reconstruction loss is calculated by subtracting the intermediate frame calculated from the first input frame image combined with the network's predicted forward optical flow field from the intermediate frame calculated from the second frame image combined with the network's predicted backward optical flow field. The resulting photometric consistency loss is defined as:

[0034]

[0035] In the formula, ρ represents the smoothed L1 norm, I1 and I2 represent the two consecutive frames of the input network, respectively, and F 1→2 This represents the optical flow field between two frames of images, where N represents the number of pixels in each input image frame.

[0036] Optical flow smoothing loss applies stronger smoothing constraints to optical flow in non-edge regions by calculating the optical flow gradient and combining it with edge weights generated from image gradients, while allowing variation in edge regions. This preserves image structural details while maintaining the continuity of optical flow. The optical flow smoothing loss is specifically expressed as follows:

[0037]

[0038] In the formula, This represents the gradient value of optical flow at pixel x. Let be the brightness gradient value of the image at pixel x, and α represent the intensity of edge suppression.

[0039] The interpolated frame image output by the network is compared with the real intermediate frame image using pixel-by-pixel difference calculation (L1 loss) to evaluate the interpolation accuracy and obtain the interpolation reconstruction loss, specifically:

[0040]

[0041] In the formula, I represents the predicted intermediate frame image calculated based on the predicted final optical flow field. t This is a real intermediate frame image.

[0042] Step 4: Adjust the camera exposure time to three times the projector response time. During network performance testing, firstly, project the fringe sequence I onto the surface of the object under test using a projector, and then re-capture a new set of multiplexed fringe images. Input the captured multiplexed fringe images into the trained CNN1, which outputs multiple corresponding single-fringe images. Then, input these single-fringe images into CNN2 to predict and generate their respective wrapping phase maps. Subsequently, combining the wrapping phase data acquired by the dual cameras and the system calibration parameters, perform stereo phase unfolding and reconstruct the object surface depth information to obtain a depth image sequence. Input the depth map of this sequence into CNN3 to predict the optical flow field between adjacent depth maps; use the optical flow field to perform temporal interpolation on the depth sequence to generate an interpolated depth image sequence. Finally, perform 3D reconstruction on all depth maps, including the interpolated results, to obtain a dynamic 3D topography, achieving high-speed dynamic 3D imaging that overcomes camera frame rate limitations.

[0043] Specific results are as follows Figure 4As shown, the multiplexed fringe image and its spectrum, single fringe image, wrapping phase, depth image, 3D reconstruction results, and the change of depth value of selected points on the fan surface over time before and after frame interpolation are presented in sequence during the experiment. During the experiment, the projector resolution was 1024×768, the camera image resolution was 640×440, the projector projection frequency was 1000Hz, and the camera exposure time was 3ms. The frame rate of the multiplexed fringe image obtained by the camera was 333Hz, the frame rate of the single fringe image obtained after decoupling was 1000Hz, and the frame rate after time interpolation was 5000Hz. Ultimately, a dynamic 3D result of 5000Hz was achieved by combining 333Hz camera shooting with frequency domain multiplexing and time interpolation techniques, realizing temporal super-resolution structured light 3D imaging that exceeds the camera's frame rate limit by 15 times.

Claims

1. A three-dimensional imaging method integrating frequency domain multiplexing and temporal interpolation techniques, characterized in that, include: Step 1: Construct a stripe pattern sequence, which includes a stripe sequence I composed of stripes with different tilt angles in the same frequency domain, and a stripe sequence II composed of multi-step phase-shifted stripes corresponding to all the stripes with tilt angles in stripe sequence I. Step 2: Construct a training dataset. Project the stripe sequence I onto the surface of a dynamic object using a projector. Capture multiplexed stripe images and single stripe images modulated on the surface of the dynamic object using dual cameras in short exposure. The multiplexed stripe images and single stripe images constitute dataset I. The stripe sequence II is projected onto the surface of a dynamic object using a projector. Two cameras are used to simultaneously capture 12-step phase-shifted images corresponding to single stripe images under short exposure conditions. The wrapping phase image is obtained by solving the multi-step phase-shifting method. The single stripe image and the wrapping phase corresponding to the single stripe image constitute dataset II. Step 3: Construct a multiplexed image decoupling network CNN1, a wrapping phase prediction network CNN2, and a temporal interpolation network CNN3. Use the multiplexed stripe images from dataset I as input to multiplexed image decoupling network CNN1, and use single stripe images as labels to train multiplexed image decoupling network CNN1; use the single stripe images from dataset II as input to wrapping phase prediction network CNN2, and use the wrapping phase as output of wrapping phase prediction network CNN2 to train wrapping phase prediction network CNN2; use a stereo phase unfolding algorithm combined with calibration information to unfold the wrapping phase in dataset II and recover the depth information to obtain a depth image; use the depth images of the first and last two moments in any three adjacent moments as input to temporal interpolation network CNN3, and use the depth image of the middle moment as output of temporal interpolation network CNN3, select multiple sets of data to construct dataset III, and train temporal interpolation network CNN3; Step 4: Project the fringe sequence I onto the surface of the object under test using a projector, and re-capture a new set of multiplexed fringe images; input the multiplexed fringe images into the trained multiplexed image decoupling network CNN1 to obtain a set of single fringe images; input the obtained single fringe images into the wrapping phase prediction network CNN2 to obtain the corresponding wrapping phase, and combine the dual-camera wrapping phase data and system calibration data to perform stereo phase unfolding and depth information restoration to obtain a depth image; input the obtained series of depth images into the temporal interpolation network CNN3 to obtain the optical flow field corresponding to the depth images at all adjacent time points, and use the optical flow field to perform temporal interpolation on the depth image sequence to obtain an interpolated image sequence; perform 3D reconstruction on all depth images in the interpolated image sequence, convert them into dynamic results, and achieve high-speed dynamic 3D imaging that exceeds the camera frame rate limit.

2. The three-dimensional imaging method according to claim 1, which integrates frequency domain multiplexing and time interpolation techniques, is characterized in that... The fringe image to be projected in fringe sequence I is specifically as follows: Among them, a p b represents the background average intensity of stripe sequence I. p Indicates the modulation density of stripe sequence I. These represent the frequencies of the projected fringes in fringe sequence I in the horizontal and vertical directions, respectively, x. p y p θ represents the pixel coordinates in the horizontal and vertical directions on the projection plane of the projector, respectively. m The angle between the stripe and the vertical direction is represented by m, which is the number of stripe images to be projected in stripe sequence I.

3. The three-dimensional imaging method according to claim 2, which integrates frequency domain multiplexing and time interpolation techniques, is characterized in that... The stripes in stripe sequence II are defined as follows: Where, x p y p These represent the pixel coordinates in the horizontal and vertical directions on the projection plane of the projector, respectively. This indicates that when the angle between the stripe and the vertical direction is θ m At that time, the intensity distribution corresponding to the nth image of the 12-step phase-shifting fringes, n=1,2,...,12.

4. The three-dimensional imaging method according to claim 1, which integrates frequency domain multiplexing and time interpolation techniques, is characterized in that... The multiplexed image decoupling network CNN1 takes the multiplexed stripe image as input, with 1 input channel and 3 output channels, corresponding to three decoupled single-frequency stripe images respectively. The multiplexed image decoupling network CNN1 adopts a multi-scale residual U-Net structure, including an encoder and a decoder. The encoder consists of four scale layers, and each scale layer contains multiple 3×3 convolutions, batch normalization and ReLU activation. The decoder is symmetrically set with four scale levels. Each level contains upsampling operations, cross-layer feature concatenation, and convolutional residual blocks to gradually recover high-resolution features and pass the encoded feature information to the output. Finally, at the end of the decoder, the number of channels is mapped to a 3-channel output through a 1×1 convolution to obtain three single-frequency stripe images.

5. The three-dimensional imaging method according to claim 1, which integrates frequency domain multiplexing and time interpolation techniques, is characterized in that... The structure of the wrapping phase prediction network CNN2 is the same as that of the multiplexed image decoupling network CNN1. The wrapping phase prediction network takes a single-image single-frequency stripe image as input, has 1 input channel, 1 output channel, and outputs the wrapping phase map of the corresponding stripe.

6. The three-dimensional imaging method according to claim 1, which integrates frequency domain multiplexing and temporal interpolation techniques, is characterized in that... The temporal interpolation network CNN3 includes a multi-layer feature pyramid, a cost volume and optical flow prediction module, an interpolation synthesis module, and three identical optical flow prediction and fusion modules. The cost volume and optical flow prediction module includes a first cost volume layer and a first optical flow prediction layer. Each optical flow prediction and fusion module includes a second cost volume layer, a second optical flow prediction layer, and a context fusion layer. The temporal interpolation network CNN3 processes depth images as follows: The multi-layer feature pyramid generates four feature tensors from two consecutive input frames using a convolutional encoder with shared weights. The resolutions of each feature tensor are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image resolution, respectively. The first cost volume layer uses a local window to calculate the correlation of each pixel location on the output feature tensor plane at the top of the multi-layer feature pyramid to obtain the cost volume feature. The first optical flow prediction layer adopts a multi-layer convolutional network structure to perform optical flow residual prediction on the cost volume features output by the first cost volume layer, thereby obtaining the first optical flow field. After upsampling the first optical flow field, it is fused with the feature tensor of the corresponding size output by the multi-layer feature pyramid to obtain the fused feature tensor; The second cost volume layer of the first optical flow prediction and fusion module uses a local window to calculate the correlation of each pixel location on the fusion feature tensor plane to obtain the fusion cost volume feature. The second optical flow prediction layer of the first optical flow prediction and fusion module adopts a multi-layer convolutional network structure to predict the optical flow residual of the fusion cost volume feature and obtain the second optical flow field. The context fusion layer of the first optical flow prediction and fusion module performs deep fusion of the second optical flow field with the feature tensor of the same size output by the pyramid feature extraction layer to obtain fused context features; The second optical flow field is upsampled and fused with the corresponding size feature tensor output by the multi-layer feature pyramid to obtain the fused feature tensor. Then, the fused context feature output by the context fusion layer of the first optical flow prediction and fusion module is bilinearly interpolated and superimposed with the fused feature tensor. This is then input into the second cost volume layer of the second optical flow prediction and fusion module to obtain the third optical flow field. The third optical flow field is then input into the context fusion layer to obtain the fused cost volume feature. The third optical flow prediction and fusion module processes the data in the same way as the second optical flow prediction and fusion module, and the final optical flow field is output by the context fusion layer. The frame interpolation synthesis layer resamples the two frames of the input CNN3 image based on the predicted final optical flow field and then weights and fuses the resampled results to obtain the synthesized image corresponding to the interpolation time point.

7. The three-dimensional imaging method according to claim 1, which integrates frequency domain multiplexing and time interpolation techniques, is characterized in that... The total loss function during training of the CNN3 time interpolation network is: In the formula, For loss of photometric uniformity, For edge-aware optical flow smoothing loss, The frame interpolation reconstruction loss is represented by λ1, λ2, and λ3, which are the weights. The photometric uniformity loss is defined as follows: In the formula, ρ represents the smoothed L1 norm, I1 and I2 represent the two consecutive frames of the input network, respectively, and F 1→2 This represents the optical flow field between two frames of images, where N represents the number of pixels in each input image frame. The edge-aware optical flow smoothing loss is expressed as: In the formula, This represents the gradient value of optical flow at pixel x. Let be the brightness gradient value of the image at pixel x, and α represent the intensity of edge suppression. The frame interpolation reconstruction loss is defined as: In the formula, I represents the predicted intermediate frame image calculated based on the predicted final optical flow field. t This is a real intermediate frame image.

Images