A video frame interpolation method and system based on edge computing

Abstract

This invention discloses a video frame interpolation method and system based on edge computing, belonging to the field of intelligent video processing technology. The method includes the following steps: acquiring data and generating video frame sequences, frame timestamps, and edge state tables; extracting change features to generate interpolated frame segments and segment complexity tables; inputting the segment data into an improved PWC-Net network to generate optical flow tables, residual region tables, and candidate frames; calculating the interpolation load to generate a computational cost table; calculating output interval deviation and node switching delay to generate a temporal sequence connection weight table; constructing a DAG graph of the interpolation task and inputting it into the HEFT algorithm to generate a task order and node allocation table; and performing candidate frame correction, splicing, and encoding to generate an interpolated video stream. This invention addresses the problems of computational redundancy in video frame interpolation under fluctuating edge node computing power, unstable interpolation in complex motion regions, and discontinuous temporal sequence connection between adjacent segments by introducing an improved PWC-Net network and the HEFT algorithm.

Description

Technical Field

[0001] This invention relates to the field of intelligent video processing technology, and in particular to a video frame interpolation method and system based on edge computing. Background Technology

[0002] With the rapid development of high-definition video surveillance, live streaming, remote conferencing, intelligent security, and mobile video services, the continuity and smoothness of video footage have become crucial factors affecting user viewing experience and the accuracy of intelligent analysis. Under conditions of low frame rate acquisition, weak network transmission, video compression, or limited edge device processing capabilities, problems such as motion jumps, stuttering, and detail breaks easily occur between adjacent frames in a video sequence. Video frame interpolation technology, by generating intermediate frames between adjacent original frames, increases the video frame rate and improves the continuity of dynamic images, and has become an important technological direction in video enhancement and edge intelligent processing.

[0003] Existing video frame interpolation methods typically generate intermediate frames based on optical flow estimation, convolutional neural networks, or deep learning frame interpolation models. Some methods estimate the motion relationship between preceding and following frames and then combine this with pixel reconstruction or feature fusion to generate the interpolated frame; others use optical flow estimation networks such as PWC-Net to extract multi-scale motion features, improving the inter-frame correspondence capability in complex motion scenes. However, in edge computing scenarios, video frame interpolation tasks often need to be performed on edge nodes with limited computing power, fluctuating link bandwidth, and dynamically changing task queues. If a fixed-scale feature extraction and unified optical flow estimation process are directly adopted, it is easy to cause redundant computation of low-complexity segments and insufficient optical flow estimation of high-complexity segments, thus affecting the frame interpolation efficiency and generation quality.

[0004] Meanwhile, existing edge video processing solutions mostly schedule video segments as ordinary computing tasks, primarily allocating execution nodes based on node computing power, task queue length, or transmission latency, lacking dedicated modeling for the temporal continuity of video frame interpolation. When adjacent interpolated frames are assigned to different edge nodes, node switching delays, output interval deviations, and differences in segment complexity can affect subsequent temporal splicing, easily causing uneven rhythm, abrupt image transitions, or unstable encoded output at segment boundaries in the interpolation results.

[0005] Therefore, how to provide a video frame interpolation method and system based on edge computing is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a video frame interpolation method and system based on edge computing. This invention fully utilizes the improved PWC-Net network, HEFT algorithm, and edge computing technology, and describes in detail the processing steps of completing video frame division, optical flow estimation, interpolation candidate frame generation, interpolation task scheduling, and video encoding output at edge nodes. It has the advantages of good interpolation continuity, high utilization of edge node computing power, low video output latency, and strong image stability in complex motion scenes.

[0007] A video frame interpolation method based on edge computing according to an embodiment of the present invention includes the following steps: Collect raw video stream, node computing power status, task queue records and link bandwidth; decode and segment the raw video stream into frames and extract timestamps to generate video frame sequences, frame timestamps and edge status tables. The video frame sequence is divided into adjacent frame combinations, and the inter-frame brightness difference, texture difference and motion amount are extracted to generate interpolated frame segments and segment complexity tables. An improved PWC-Net network is constructed by adding a scale-gating layer after the pyramid feature extraction layer and a residual partition fusion layer after the cost volume construction layer. The interpolated frame fragments, fragment complexity table and edge state table are input into the improved PWC-Net network to generate a bidirectional optical flow table, residual region table and interpolated frame candidate frames. The computational cost of partitioned frame interpolation is calculated based on the bidirectional optical flow table, residual region table, and fragment complexity table, and a computational cost table is generated. Based on the interpolation candidate frames, frame timestamps, fragment complexity table and edge state table, calculate the output interval deviation between adjacent fragments and node switching delay, and generate a timing connection weight table. The frame insertion task DAG graph is constructed based on the calculation cost table, edge state table, and temporal connection weight table. The graph is then input into the HEFT algorithm. The temporal connection weight table is introduced during the task sorting process, and the node switching delay in the temporal connection weight table is introduced during the node selection process to generate the frame insertion task order and node allocation table. Perform interpolation candidate frame correction, timing splicing, and encoding output according to the interpolation task order and node allocation table to generate an interpolated video stream.

[0008] Optionally, the generation of the video frame sequence, frame timestamps, and edge state table includes: Receive the raw video stream, record the video stream identifier, acquisition start time, frame rate parameters, resolution and encoding format, acquisition node identifier, node computing power status, task queue records and link bandwidth; Decode the original video stream according to the encoding format, write the frame sequence number and timestamp for the video frame according to the acquisition start time and frame rate parameters, and generate a video frame sequence and frame timestamp; Align the node's computing power status, task queue records, and link bandwidth according to the node identifier and collection time to generate an edge status table.

[0009] Optionally, the generation of interpolated frame segments and the segment complexity table includes: Read the video frame sequence and frame timestamp, combine adjacent frames according to frame number, and write the previous frame, next frame, timestamp and insertion frame position into the segment record to generate the inserted frame segment; Extract the luminance channel, texture gradient, and block displacement from the previous and next frames, respectively calculate the average difference of the luminance channel, the average difference of the texture gradient, and the average displacement distance, and generate the inter-frame luminance difference, texture difference, and motion amount; After normalizing the inter-frame brightness difference, texture difference, and motion, sum them according to the index weights to generate a segment complexity value, and write it into the segment complexity table corresponding to the interpolated segment.

[0010] Optionally, generating the bidirectional optical flow table, residual region table, and interpolation candidate frames includes: An improved PWC-Net network is constructed, which includes a pyramid feature extraction layer, a scale gating layer, a feature warping layer, a cost volume construction layer, a residual partition fusion layer, and a frame interpolation synthesis layer. The interpolated frame fragments are input into the pyramid feature extraction layer. The preceding and following frames in the interpolated frame fragments are read, and the scale features of the preceding and following frames are extracted in order from low-resolution scale to high-resolution scale to generate pyramid feature groups. Input the pyramid feature group, fragment complexity table and edge state table into the scale gating layer, read the fragment complexity value in the fragment complexity table, read the node computing power status, task queue record and link bandwidth in the edge state table, perform normalization processing on the fragment complexity value, node computing power margin, task queue length and link bandwidth and perform weighted summation to generate scale retention weights, filter the pyramid feature group according to the scale retention weights, and generate scale-filtered feature groups. The scale-selected feature group is input into the feature warping layer. The zero displacement field of the lowest resolution scale is used as the initial bidirectional estimation flow. The scale-selected features of the next frame and the scale-selected features of the previous frame are backsampled along the initial bidirectional estimation flow in the order from low resolution scale to high resolution scale, respectively, to generate the warping feature group and the scale-by-scale initial optical flow group. The scale-selected feature group, the distortion feature group, and the scale-by-scale initial optical flow group are input into the cost volume construction layer. Local correlation calculation is performed on the scale-selected features of the previous frame and the corresponding distortion features, and local correlation calculation is performed on the scale-selected features of the subsequent frame and the corresponding distortion features to generate a bidirectional cost volume. The bidirectional cost volume, the scale-wise initial optical flow group, and the fragment complexity table are input into the residual partitioning fusion layer. Based on the correlation response of the bidirectional cost volume, the residual amplitude of the scale-wise initial optical flow group, and the fragment complexity value, the fast motion region, the occlusion region, and the static background region are divided to generate the residual region table. Partitioning fusion is then performed on the bidirectional cost volume and the distortion feature group to generate the residual fusion feature group. The residual fusion feature set, residual region table, and scale-wise initial optical flow set are input into the interpolation synthesis layer to generate a bidirectional optical flow table and interpolation candidate frames. The improved PWC-Net network is trained by using interpolation reconstruction error, optical flow consistency error, and residual region partitioning error as joint optimization objectives. The parameters in the pyramid feature extraction layer, scale gating layer, feature distortion layer, cost volume construction layer, residual partition fusion layer, and interpolation synthesis layer are continuously optimized. When the change of the joint optimization objective is less than 0.5% during 5 consecutive training rounds, the trained improved PWC-Net network is obtained.

[0011] Optionally, generating the computational cost table includes: Read the bidirectional optical flow table, residual region table, and fragment complexity table, extract the forward optical flow, backward optical flow, residual region, and fragment complexity values ​​according to the interpolated fragment identifier, and generate partitioned statistical records; The number of regional pixels, average optical flow amplitude, and bidirectional optical flow difference value of each residual region are calculated based on the partition statistics records. These three values ​​are then added to the fragment complexity value after normalization to generate the partition interpolation computation amount. The computational cost of interpolation is summarized by the interpolation segment identifier, and the computational cost record for each interpolation segment is generated by combining the pixel scale of the residual region and the number of interpolated frames. The computational cost table is generated by arranging the records in order of timestamp.

[0012] Optionally, the step of calculating the output interval deviation between adjacent segments and the node switching delay to generate the timing connection weight table includes: Read the candidate frames for interpolation, frame timestamps, and fragment complexity table. Determine the baseline output interval based on the timestamps of adjacent interpolated fragments and the number of interpolated frames. Calculate the absolute difference between the candidate output interval and the baseline output interval to generate the output interval deviation between adjacent fragments. Read the edge state table, determine the fragment migration data amount based on the frame size and encoded data amount of the interpolation candidate frame, divide the fragment migration data amount by the link bandwidth to obtain the transmission delay, add the queuing task duration to the transmission delay to generate the node switching delay; After normalizing the output interval deviation between adjacent segments, node switching delay, and segment complexity value, sum them according to their corresponding weights to generate the timing connection weight value, and write it into the timing connection weight table.

[0013] Optionally, the frame interpolation task order and node allocation table includes: Read the computation cost table, edge state table, and timing connection weight table. Establish frame insertion task nodes according to the frame insertion segment identifier. Establish dependency edges between adjacent frame insertion task nodes according to the frame timestamp order. Write the computation cost record, node computing power status, task queue record, link bandwidth, timing connection weight value, adjacent segment output interval deviation, and node switching delay into the corresponding frame insertion task node to generate a frame insertion task DAG graph. The execution time of each interpolation task node on each edge node is calculated based on the calculation cost record and node computing power status. The communication time of dependent edges is calculated based on the link bandwidth and the data transmission volume between adjacent interpolation task nodes. The execution time and the communication time of dependent edges are written into the interpolation task DAG graph to generate HEFT scheduling input data. The HEFT scheduling input data is input into the HEFT algorithm to calculate the basic upward sorting value of each interpolation task node. The timing connection weight value is multiplied by the output interval deviation of adjacent segments to generate the timing connection correction amount. The basic upward sorting value, the timing connection correction amount and the sorting value of the corresponding dependent subsequent task are added to generate the corrected task sorting value. Arrange the interpolation task nodes in descending order of the corrected task sorting value, calculate the base earliest completion time of each interpolation task node on each edge node in turn, add the node switching delay in the timing connection weight table to the base earliest completion time, and generate the corrected earliest completion time. For each frame insertion task node, select the edge node with the earliest correction completion time and the smallest correction completion time. Write the frame insertion task node number, correction task sorting value, selected edge node, start time and completion time into the scheduling result to generate the frame insertion task order and node allocation table.

[0014] Optionally, generating the interpolated video stream includes: Read the frame insertion task order, node allocation table and frame insertion candidate frames, send the frame insertion candidate frames to the edge nodes corresponding to the node allocation table according to the frame insertion task order, receive the frame insertion candidate frame processing results, and generate a candidate frame result table according to the frame insertion segment identifier. Read the candidate frame result table, video frame sequence, frame timestamp and timing stitching weight table, arrange the original video frames and interpolation candidate frame processing results in the video frame sequence according to the frame timestamp, perform time position correction according to the adjacent segment output interval deviation in the timing stitching weight table, and generate timing correction frame sequence; Read the frame rate parameters, resolution, and encoding format of the timing correction frame sequence and the original video stream, and perform splicing, encoding, and encapsulation according to the frame rate parameters, resolution, and encoding format to generate an interpolated video stream.

[0015] A video frame interpolation system based on edge computing according to an embodiment of the present invention includes the following modules: The video edge access module is used to receive raw video streams and edge node operation data, and generate video frame sequences, frame timestamps, and edge status tables. The segment complexity analysis module is used to create adjacent frame interpolation objects based on the video frame sequence, extract inter-frame change features, and generate interpolated segments and segment complexity tables. The optical flow interpolation generation module is used to build an improved PWC-Net network, perform optical flow estimation and residual partition fusion on the interpolated frame segments, and generate a bidirectional optical flow table, a residual region table, and interpolation candidate frames. The frame interpolation cost evaluation module is used to calculate the computational cost of partitioned frame interpolation based on the bidirectional optical flow table, residual region table, and fragment complexity table, and to generate a computational cost table. The timing alignment evaluation module is used to calculate the output interval deviation between adjacent segments and the node switching delay based on the interpolation candidate frames, frame timestamps, segment complexity table and edge state table, and to generate a timing alignment weight table. The edge task scheduling module is used to construct the DAG graph of the interpolation task based on the calculation cost table, the edge state table and the temporal connection weight table, and to generate the interpolation task order and node allocation table through the HEFT algorithm. The frame interpolation encoding output module is used to perform correction, splicing and encoding on candidate frames for frame interpolation according to the frame interpolation task order and node allocation table, and generate frame interpolation video stream.

[0016] The beneficial effects of this invention are: This invention discloses a video frame interpolation method and system based on edge computing. By acquiring the original video stream, node computing power status, task queue records, and link bandwidth, and performing decoding, frame segmentation, timestamp extraction, and adjacent frame combination on the original video stream, the video frame interpolation process can form schedulable and analyzable interpolated frame segments at the edge node side. By extracting inter-frame brightness differences, texture differences, and motion quantities to generate a segment complexity table, the motion variations and image complexity of different video segments can participate in subsequent network inference and task scheduling. This avoids the problem of traditional frame interpolation methods using a uniform computational intensity for all video segments, resulting in redundant computation of low-complexity segments and insufficient processing of high-complexity segments.

[0017] In the interpolation candidate frame generation stage, this invention constructs an improved PWC-Net network and adds a scale-gating layer after the pyramid feature extraction layer and a residual partitioning fusion layer after the cost volume construction layer. This enables the optical flow estimation process to combine fragment complexity and edge state to filter the scale features participating in the calculation, and to perform differentiated fusion processing on fast-moving regions, occluded regions, and static background regions. By generating bidirectional optical flow tables, residual region tables, and interpolation candidate frames, this invention can improve the ability to express the motion relationship between adjacent frames in complex motion scenes, reduce ghosting, blurring, and edge tearing in the interpolation results, and improve the image stability and continuity of video interpolation on the edge side.

[0018] During the edge task scheduling phase, this invention constructs a DAG graph for frame insertion tasks based on a computation cost table, an edge state table, and a timing concatenation weight table. The timing concatenation weight table is incorporated into the task sorting process of the HEFT algorithm, and node switching latency is introduced into the node selection process. This ensures that frame insertion task scheduling is no longer solely based on node computing power and earliest completion time, but also considers the output interval deviation between adjacent segments, node switching latency, and segment computation cost. By generating a frame insertion task order and node allocation table, and performing correction, timing splicing, and encoding output on candidate frames, this invention reduces the impact of edge node load fluctuations on the frame insertion video stream, improves the rationality of frame insertion task allocation, the real-time performance of video output, and the smoothness of segment concatenation. Compared to traditional cloud-based centralized frame insertion or fixed edge node frame insertion methods, improvements are achieved in transmission pressure, processing latency, frame insertion quality for complex motions, and edge node collaborative efficiency. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a video frame interpolation method based on edge computing proposed in this invention; Figure 2 This is a schematic diagram of the structure of the improved PWC-Net network for a video frame interpolation method based on edge computing proposed in this invention; Figure 3 This is a schematic diagram of the structure of a video frame interpolation system based on edge computing proposed in this invention. Detailed Implementation

[0020] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0021] refer to Figure 1 and Figure 2 A video frame interpolation method based on edge computing includes the following steps: Collect raw video stream, node computing power status, task queue records and link bandwidth; decode and segment the raw video stream into frames and extract timestamps to generate video frame sequences, frame timestamps and edge status tables. The video frame sequence is divided into adjacent frame combinations, and the inter-frame brightness difference, texture difference and motion amount are extracted to generate interpolated frame segments and segment complexity tables. An improved PWC-Net network is constructed by adding a scale-gating layer after the pyramid feature extraction layer and a residual partition fusion layer after the cost volume construction layer. The interpolated frame fragments, fragment complexity table and edge state table are input into the improved PWC-Net network to generate a bidirectional optical flow table, residual region table and interpolated frame candidate frames. The computational cost of partitioned frame interpolation is calculated based on the bidirectional optical flow table, residual region table, and fragment complexity table, and a computational cost table is generated. Based on the interpolation candidate frames, frame timestamps, fragment complexity table and edge state table, calculate the output interval deviation between adjacent fragments and node switching delay, and generate a timing connection weight table. The frame insertion task DAG graph is constructed based on the calculation cost table, edge state table, and temporal connection weight table. The graph is then input into the HEFT algorithm. The temporal connection weight table is introduced during the task sorting process, and the node switching delay in the temporal connection weight table is introduced during the node selection process to generate the frame insertion task order and node allocation table. Perform interpolation candidate frame correction, timing splicing, and encoding output according to the interpolation task order and node allocation table to generate an interpolated video stream.

[0022] In this embodiment, generating the video frame sequence, frame timestamps, and edge state table includes: Receive the raw video stream, record the video stream identifier, acquisition start time, frame rate parameters, resolution and encoding format, acquisition node identifier, node computing power status, task queue records and link bandwidth; Decode the original video stream according to the encoding format, write the frame sequence number and timestamp for the video frame according to the acquisition start time and frame rate parameters, and generate a video frame sequence and frame timestamp; Align the node computing power status, task queue records, and link bandwidth according to the node identifier and collection time to generate an edge status table. The generation of the edge status table is specifically as follows: Using the node identifier as the primary key, the node computing power status, task queue record, and link bandwidth are written to the corresponding node cache area. The acquisition time is converted to a unified time format, and the node computing power status, task queue record, and link bandwidth are matched according to the principle of minimizing the difference between adjacent acquisition times under the same node identifier. For node computing power status, task queue record, and link bandwidth that were not acquired at the same acquisition time, the previous acquisition time record is used to fill in the gaps. The matched node identifier, acquisition time, node computing power status, task queue record, and link bandwidth are merged into a node status record. The node status records are sorted in ascending order of acquisition time to generate an edge status table.

[0023] In this embodiment, the generation of the interpolated frame fragment and the fragment complexity table includes: Read the video frame sequence and frame timestamp, combine adjacent frames according to frame number, and write the previous frame, next frame, timestamp and insertion frame position into the segment record to generate the inserted frame segment; The luminance channel, texture gradient, and block-level displacement are extracted from the previous and subsequent frames. The average difference of the luminance channel, the average difference of the texture gradient, and the average displacement distance are calculated to generate inter-frame luminance difference, texture difference, and motion. Specifically, the calculation of the average difference of the luminance channel, the average difference of the texture gradient, and the average displacement distance to generate inter-frame luminance difference, texture difference, and motion is as follows: According to common RGB-to-luminance channel coefficients, the red channel value of the same pixel position in the previous frame is multiplied by 0.299, the green channel value by 0.587, and the blue channel value by 0.114, and then added together to obtain the luminance channel of the previous frame and the luminance channel of the next frame. The luminance values ​​of the same pixel position in the luminance channel of the previous frame and the luminance channel of the next frame are subtracted, and the absolute value is taken. The absolute values ​​of all pixel positions are accumulated and divided by the total number of pixels to obtain the average luminance channel difference, which is recorded as the inter-frame luminance difference. The horizontal Sobel operator and the vertical Sobel operator are used to perform convolution on the luminance channels of the previous frame and the luminance channel of the next frame, respectively, to obtain the horizontal gradient of the previous frame, the vertical gradient of the previous frame, the horizontal gradient of the next frame, and the vertical gradient of the next frame. The absolute value of the horizontal gradient difference at the same pixel position is then multiplied by... The absolute values ​​of the vertical gradient differences are added together to obtain the texture gradient difference. The texture gradient differences of all pixel positions are accumulated and divided by the total number of pixels to obtain the average texture gradient difference, which is recorded as the texture difference. The brightness channel of the previous frame is divided into 16×16 pixel blocks. In the brightness channel of the subsequent frame, the cumulative value of the absolute brightness difference between each candidate block and the pixel block of the previous frame is calculated within a search radius of 16 pixels with the same block coordinates as the center. The candidate block with the smallest cumulative value of absolute brightness difference is selected as the matching block. The horizontal and vertical differences between the center coordinates of the pixel block of the previous frame and the center coordinates of the matching block are calculated. The square root of the sum of the squares of the horizontal and vertical differences is obtained to obtain the block-level displacement distance. The average displacement distance of all block-level displacement distances is calculated to obtain the average displacement distance, which is recorded as the motion amount. After normalizing the inter-frame brightness difference, texture difference, and motion, the values ​​are summed according to their respective weights to generate a segment complexity value. This value is then written into the segment complexity table along with the corresponding interpolated segments. Specifically, generating the segment complexity value and writing it into the segment complexity table along with the interpolated segments involves: Read the inter-frame brightness difference, texture difference, and motion amount corresponding to the same interpolated frame segment. Read the maximum value, minimum value, maximum value, minimum value, maximum value, and minimum value of inter-frame brightness difference, texture difference, and motion amount of all interpolated frame segments in the current batch, respectively. Subtract the minimum inter-frame luminance difference from the current inter-frame luminance difference, and then divide by the difference between the maximum and minimum inter-frame luminance differences to obtain the normalized inter-frame luminance difference. Calculate the normalized texture difference and normalized motion amount in the same way. The differences between the maximum and minimum inter-frame luminance differences, the maximum and minimum texture differences, and the maximum and minimum motion differences are calculated to obtain luminance discrete values, texture discrete values, and motion discrete values. The total discrete value is obtained by adding the discrete values ​​of brightness, texture, and motion. The total discrete value is obtained by dividing the discrete value of brightness by the total discrete value. The weight of the brightness index is obtained by dividing the discrete value of texture by the total discrete value. The weight of the motion index is obtained by dividing the discrete value of motion by the total discrete value. The normalized inter-frame luminance difference is multiplied by the luminance index weight, the normalized texture difference is multiplied by the texture index weight, and the normalized motion is multiplied by the motion index weight. The three products are then added together to generate the fragment complexity value. Write the interpolated segment identifier, the previous frame number, the next frame number, the previous frame timestamp, the next frame timestamp, the inter-frame brightness difference, the texture difference, the motion amount, and the segment complexity value into the same complexity record. Sort all complexity records in ascending order according to the interpolated segment identifier to generate a segment complexity table.

[0024] In this embodiment, generating the bidirectional optical flow table, the residual region table, and the interpolation candidate frame includes: An improved PWC-Net network is constructed, comprising a pyramid feature extraction layer, a scale-gated layer, a feature warping layer, a cost volume construction layer, a residual partitioning fusion layer, and a frame interpolation synthesis layer. Specifically, the improved PWC-Net network is constructed as follows: Based on the pyramid feature extraction, feature distortion, and cost volume construction of the PWC-Net network, we establish the input terminals for interpolated fragments, fragment complexity tables, and edge state tables. A scale-gated layer is connected to the output of the pyramid feature extraction layer. The pyramid feature group output by the pyramid feature extraction layer, the fragment complexity value in the fragment complexity table, and the node computing power status, task queue record and link bandwidth in the edge state table are all connected to the scale-gated layer. The scale-filtered feature group output by the scale-gated layer is connected to the feature distortion layer. The twisted feature set and the scale-wise initial optical flow set output by the feature twisting layer are connected to the cost volume construction layer, and the cost volume construction layer outputs a bidirectional cost volume. The residual partition fusion layer is connected to the output of the cost volume construction layer. The bidirectional cost volume, the scale-wise initial optical flow group and the fragment complexity value are jointly connected to the residual partition fusion layer. The residual region table and residual fusion feature group output by the residual partition fusion layer are connected to the frame interpolation synthesis layer. The output of the interpolation synthesis layer is set as a bidirectional optical flow table output and an interpolation candidate frame output, forming an improved PWC-Net network that is sequentially connected by a pyramid feature extraction layer, a scale gating layer, a feature warping layer, a cost volume construction layer, a residual partition fusion layer, and an interpolation synthesis layer. The interpolated frame segment is input into the pyramid feature extraction layer. The preceding and following frames are read from the interpolated frame segment, and the scale features of the preceding and following frames are extracted in order from low-resolution to high-resolution scales to generate a pyramid feature set. The pyramid feature extraction layer includes: Frame pair normalization unit: Read the previous frame, the next frame, the interpolated frame segment identifier, the previous frame timestamp, and the next frame timestamp from the interpolated frame segment, perform normalization processing on the pixel values ​​of the previous frame and the next frame, and generate normalized data of the previous frame and normalized data of the next frame. Multi-scale downsampling unit: Reads the normalized data of the previous frame and the normalized data of the next frame, establishes a multi-level scale path in the order from low-resolution scale to high-resolution scale, and generates the previous frame sampling features and the next frame sampling features under the corresponding scale number in each scale path. Shared convolutional coding unit: Read the sampled features of the previous frame and the sampled features of the next frame under the same scale number, and perform convolutional coding respectively using the same set of convolutional parameters to generate the scale features of the previous frame and the scale features of the next frame under the same scale number; Scale recording output unit: Binds the scale features of the previous frame and the scale features of the next frame under the same scale number with the interpolated segment identifier, the previous frame timestamp, the next frame timestamp, and the scale number to generate a pyramid feature group; In the pyramid feature extraction layer, the frame pair normalization unit reads the pixel matrices of the previous and next frames, divides the channel values ​​in the pixel matrices by the maximum channel value to obtain the normalized data of the previous and next frames, and writes the interpolated segment identifier, the timestamp of the previous frame, and the timestamp of the next frame into the frame pair input record. The multi-scale downsampling unit reads the normalized data of the previous and next frames, establishes low-resolution scale paths, intermediate-resolution scale paths, and high-resolution scale paths according to the scale number, and performs stride convolution on the normalized data of the previous and next frames in each scale path to obtain the sampled features of the previous frame and the sampled features of the next frame at the corresponding scale number, and writes the scale number into the sampled feature record. Feature Recording: The shared convolutional coding unit reads the sampled features of the previous frame and the sampled features of the next frame under the same scale number, and processes the sampled features of the previous frame and the next frame respectively using the same convolutional kernel, the same normalization parameters and the same activation function, generating the scale features of the previous frame and the scale features of the next frame with the same number of channels; The scale recording output unit writes the scale features of the previous frame and the scale features of the next frame under the same scale number into the same scale feature record, and writes the interpolated segment identifier, the timestamp of the previous frame, the timestamp of the next frame, the scale number, the feature size and the number of channels, and arranges all scale feature records from low resolution scale to high resolution scale according to the scale number, generating a pyramid feature group for the scale gating layer to call; The pyramid feature groups, fragment complexity table, and edge state table are input into the scale gating layer. Fragment complexity values ​​from the fragment complexity table are read, and node computing power status, task queue records, and link bandwidth from the edge state table are read. Normalization is performed on the fragment complexity values, node computing power margin, task queue length, and link bandwidth, followed by weighted summation to generate scale retention weights. Pyramid feature groups are then filtered according to these scale retention weights to generate scale-filtered feature groups. The scale gating layer includes: Scale feature reading unit: Reads the scale number, previous frame scale feature and next frame scale feature in the pyramid feature group, and establishes a scale feature record according to the scale number; Edge State Encoding Unit: Reads fragment complexity value, node computing power status, task queue record and link bandwidth, subtracts the node's total computing power from the occupied computing power to obtain the node's computing power reserve, and combines the number of queued tasks and the queuing time in the task queue record to form the task queue length. Gated weight calculation unit: Normalizes fragment complexity value, node computing power margin, task queue length and link bandwidth respectively, inputs the normalization result into linear mapping function and performs nonlinear activation to generate scale-preserving weights corresponding to each scale number; Scale feature filtering unit: Performs weighted filtering on the scale features of the previous frame and the scale features of the next frame in the scale feature record according to the scale retention weight, and generates a scale filtering feature group; In the scale gating layer, the scale feature reading unit reads the previous frame scale features and the next frame scale features corresponding to each scale number in the pyramid feature group, and writes the same interpolated segment identifier for the previous frame scale features and the next frame scale features under the same scale number, generating a scale feature record; the edge state encoding unit reads the segment complexity value corresponding to the same interpolated segment, and reads the node computing power status, task queue record, and link bandwidth of the corresponding execution node in the edge state table, subtracts the occupied computing power from the total computing power of the node to obtain the node computing power reserve, combines the number of queued tasks with the queued task duration to form the task queue length, and writes the segment complexity value, node computing power reserve, task queue length, and link bandwidth into the gating input data; the gating weight calculation unit... The unit performs max-min normalization on each value in the gated input data, inputs the normalized values ​​into a linear mapping function to obtain the gated response value corresponding to each scale number, and performs nonlinear activation on the gated response value to generate scale retention weights ranging from 0 to 1. The scale feature filtering unit multiplies the scale retention weights corresponding to the same scale number into the scale features of the previous frame and the scale features of the next frame, respectively, to obtain the gated scale features of the previous frame and the gated scale features of the next frame. It deletes the scale feature records with a scale retention weight of 0, and arranges the retained gated scale features of the previous frame and the gated scale features of the next frame according to the scale number from low resolution scale to high resolution scale to generate a scale filtering feature group for the feature warping layer to use. The scale-selected feature set is input into the feature warping layer. Using the zero-displacement field at the lowest resolution scale as the initial bidirectional estimation flow, the scale-selected features of the subsequent frame and the scale-selected features of the previous frame are backsampled along the initial bidirectional estimation flow in order from low-resolution scale to high-resolution scale, generating a warped feature set and a scale-wise initial optical flow set. The feature warping layer includes: Scale order reading unit: Reads the scale number, previous frame scale filtering feature and next frame scale filtering feature in the scale filtering feature group, and arranges the retained scales in order from low resolution scale to high resolution scale; Initial optical flow generation unit: Reads the feature size corresponding to the lowest resolution preserved scale, sets the lateral and longitudinal displacements of each spatial position to 0, and generates the forward and backward initial optical flow of the lowest resolution scale. Scale-by-scale optical flow transfer unit: reads the forward and backward initial optical flows of adjacent low-resolution preserved scales, performs bilinear interpolation amplification on the forward and backward initial optical flows, and corrects the lateral and longitudinal displacements according to the width and height ratios between adjacent scales to generate the forward and backward initial optical flows of the current scale. Backsampling output unit: Based on the forward initial optical flow and backward initial optical flow at the current scale, backsampling is performed on the subsequent frame scale selection features and the previous frame scale selection features respectively to generate a distortion feature group and a scale-by-scale initial optical flow group; In the feature warping layer, the scale order reading unit reads all the retained scales in the scale-selected feature group, arranges the previous frame scale-selected features and the next frame scale-selected features according to the scale number from low resolution scale to high resolution scale, forming a scale warping input sequence; the initial optical flow generation unit reads the feature width and feature height of the lowest resolution retained scale in the scale warping input sequence, establishes a spatial position grid according to the feature width and feature height, and writes 0 to the lateral and longitudinal displacements of each position in the spatial position grid, forming the forward and backward initial optical flow of the lowest resolution scale; the scale-by-scale optical flow transfer unit reads the forward and backward initial optical flow of the adjacent low resolution retained scales of the current scale, amplifies the lateral displacement according to the width ratio between the current scale and the adjacent low resolution retained scales, and amplifies the longitudinal displacement according to the height ratio between the current scale and the adjacent low resolution retained scales, to obtain the forward and backward initial optical flow of the current scale; the backsampling output unit outputs the spatial coordinates of the previous frame scale-selected features at the current scale. The forward sampling coordinates are obtained by adding the lateral and longitudinal displacements in the forward initial optical flow to the coordinates. The feature values ​​of the four adjacent positions around the forward sampling coordinates are read from the subsequent frame scale filtering features, and the sum is weighted according to the distance between the four adjacent positions and the forward sampling coordinates to generate the forward warp feature. The spatial coordinates of the subsequent frame scale filtering features at the current scale are added to the lateral and longitudinal displacements in the backward initial optical flow to obtain the backward sampling coordinates. The feature values ​​of the four adjacent positions around the backward sampling coordinates are read from the previous frame scale filtering features, and the sum is weighted according to the distance between the four adjacent positions and the backward sampling coordinates to generate the backward warp feature. The forward warp feature and backward warp feature under the same scale number are written into the warp feature record, and the forward initial optical flow and backward initial optical flow under the same scale number are written into the initial optical flow record. All warp feature records and initial optical flow records are arranged from low resolution scale to high resolution scale according to the scale number to generate the warp feature group and scale-wise initial optical flow group for the cost body construction layer to call. The scale-selected feature group, the distortion feature group, and the scale-by-scale initial optical flow group are input into the cost volume construction layer. Local correlation calculations are performed on the scale-selected features of the previous frame and their corresponding distortion features, and on the scale-selected features of the subsequent frame and their corresponding distortion features, to generate a bidirectional cost volume. The cost volume construction layer includes: Feature correspondence establishment unit: Read the scale-filtered feature group, the distortion feature group, and the scale-by-scale initial optical flow group, and establish the correspondence between the previous frame scale-filtered features, the next frame scale-filtered features, the forward distortion features, the backward distortion features, the forward initial optical flow, and the backward initial optical flow according to the scale number; Candidate window generation unit: Reads the spatial position and initial optical flow under the same scale number, adds the spatial position to the lateral displacement and longitudinal displacement in the initial optical flow to generate a matching reference position, and establishes a local candidate window around the matching reference position; Local correlation calculation unit: Reads the distorted feature vector within the local candidate window, multiplies the scale-filtered feature vector and the distorted feature vector channel by channel and accumulates them, then divides by the number of channels to generate the local correlation value; Two-way cost volume output unit: Arrange the forward local correlation value and backward local correlation value according to the scale number, spatial location and candidate displacement to generate a two-way cost volume; In the cost volume construction layer, the feature correspondence establishment unit reads the scale number, previous frame scale selection feature, and subsequent frame scale selection feature from the scale selection feature group, reads the forward and backward distortion features from the distortion feature group, and reads the forward and backward initial optical flow from the scale-by-scale initial optical flow group. Data under the same scale number is written into the same cost volume input record. The candidate window generation unit reads the cost volume input record, uses each spatial position in the previous frame scale selection feature as the forward matching center, adds the lateral and longitudinal displacements in the forward initial optical flow to the forward matching center to obtain the forward matching reference position, and establishes a forward local candidate window around the forward matching reference position. Simultaneously, it uses each spatial position in the subsequent frame scale selection feature as the backward matching center, adds the lateral and longitudinal displacements in the backward initial optical flow to the backward matching center to obtain the backward matching reference position, and establishes a backward local candidate window around the backward matching reference position. Select a window; the local correlation calculation unit reads the backward distortion feature candidate vector in the forward local candidate window, multiplies the previous frame scale-filtered feature vector with the backward distortion feature candidate vector channel by channel and accumulates them, then divides by the number of channels to obtain the forward local correlation value; at the same time, it reads the forward distortion feature candidate vector in the backward local candidate window, multiplies the next frame scale-filtered feature vector with the forward distortion feature candidate vector channel by channel and accumulates them, then divides by the number of channels to obtain the backward local correlation value; the bidirectional cost body output unit writes the forward local correlation value corresponding to the same scale number, the same spatial position and the same candidate displacement into the forward cost body, writes the backward local correlation value corresponding to the same scale number, the same spatial position and the same candidate displacement into the backward cost body, and binds the forward cost body and the backward cost body according to the scale number, arranges them in order from low resolution scale to high resolution scale, and generates a bidirectional cost body for the residual partition fusion layer to call; The bidirectional cost volume, the scale-wise initial optical flow set, and the fragment complexity table are input into the residual partitioning fusion layer. Based on the correlation response of the bidirectional cost volume, the residual amplitude of the scale-wise initial optical flow set, and the fragment complexity value, the layer divides the region into fast-moving regions, occlusion regions, and static background regions, generating a residual region table. Partitioning fusion is then performed on the bidirectional cost volume and the distortion feature set to generate a residual fused feature set. The residual partitioning fusion layer includes: Correlation response analysis unit: Reads the forward cost volume and backward cost volume in the bidirectional cost volume, and extracts the maximum correlation response, response peak position and bidirectional response difference corresponding to the same scale number and the same spatial location; Optical flow residual calculation unit: Reads the forward initial optical flow and backward initial optical flow from the scale-by-scale initial optical flow group, calculates the forward optical flow amplitude, backward optical flow amplitude and bidirectional optical flow amplitude difference at the same spatial location, and generates optical flow residual amplitude; Residual region segmentation unit: Based on the maximum correlation response, bidirectional response difference, optical flow residual amplitude, and fragment complexity value, each spatial location is divided into fast-moving region, occlusion region, and static background region, generating a residual region table; Partition Feature Fusion Unit: Reads the bidirectional cost volume and distortion feature group in the corresponding region according to the residual region table, performs weighted fusion on the fast motion region, occlusion region and static background region respectively, and generates residual fusion feature group; In the residual partitioning fusion layer, the correlation response parsing unit traverses the bidirectional cost volume according to scale number and spatial location, records the highest correlation response and corresponding candidate displacement in the forward and backward cost volumes, and calculates the difference between the highest forward correlation response and the highest backward correlation response. The optical flow residual calculation unit reads the forward and backward initial optical flows at the same scale number and spatial location, adds the square root of the square of the lateral and longitudinal displacements of the forward initial optical flow to obtain the forward optical flow amplitude, adds the square root of the square of the lateral and longitudinal displacements of the backward initial optical flow to obtain the backward optical flow amplitude, subtracts the forward and backward optical flow amplitudes and takes the absolute value to obtain the optical flow residual amplitude. The residual region partitioning unit normalizes the maximum correlation response, the bidirectional response difference, the optical flow residual amplitude, and the fragment complexity value and combines them into a region discrimination record, according to... The region discrimination record writes the spatial locations with large optical flow residual amplitude and high segment complexity value into the fast motion region, the spatial locations with low maximum correlation response and large bidirectional response difference into the occluded region, and the spatial locations with small optical flow residual amplitude and small bidirectional response difference into the static background region. It also generates a residual region table according to the interpolated segment identifier, scale number, and spatial location. The partition feature fusion unit reads the bidirectional cost volume, forward distortion feature, and backward distortion feature corresponding to the fast motion region, occluded region, and static background region according to the residual region table. It normalizes the bidirectional cost volume response value in the same region to obtain the region fusion weight. The region fusion weight is multiplied by the forward distortion feature and backward distortion feature respectively and then added to generate the fusion feature of the corresponding region. The fusion features of each region are written according to the scale number and spatial location to generate a residual fusion feature group for the interpolation synthesis layer to call. The residual fusion feature set, residual region table, and scale-wise initial optical flow set are input into the interpolation synthesis layer to generate a bidirectional optical flow table and interpolation candidate frames. The improved PWC-Net network is trained by using interpolation reconstruction error, optical flow consistency error, and residual region partitioning error as joint optimization objectives. The parameters in the pyramid feature extraction layer, scale gating layer, feature distortion layer, cost volume construction layer, residual partition fusion layer, and interpolation synthesis layer are continuously optimized. When the change of the joint optimization objective is less than 0.5% during 5 consecutive training rounds, the trained improved PWC-Net network is obtained.

[0025] In this embodiment, generating the computational cost table includes: Read the bidirectional optical flow table, residual region table, and fragment complexity table, extract the forward optical flow, backward optical flow, residual region, and fragment complexity values ​​according to the interpolated fragment identifier, and generate partitioned statistical records; Based on the partitioned statistical records, the number of pixels in each residual region, the average optical flow amplitude, and the bidirectional optical flow difference value are calculated. These three values ​​are then normalized and added to the segment complexity value to generate the partitioned frame interpolation computational cost. Specifically, the partitioned frame interpolation computational cost is as follows: Read the pixel coordinate set, forward optical flow, backward optical flow, and segment complexity value under the same interpolated segment identifier and the same residual region identifier in the partition statistics record; count the number of coordinates in the pixel coordinate set to obtain the number of pixels in the region; for each pixel coordinate, read the forward optical flow lateral displacement, forward optical flow longitudinal displacement, backward optical flow lateral displacement, and backward optical flow longitudinal displacement; add the square of the forward optical flow lateral displacement and the square of the forward optical flow longitudinal displacement, and take the square root to obtain the forward optical flow amplitude; add the square of the backward optical flow lateral displacement and the square of the backward optical flow longitudinal displacement, and take the square root to obtain the backward optical flow amplitude; add the forward optical flow amplitude and the backward optical flow amplitude and divide by the number of optical flow directions involved in the summation. The number of optical flow directions involved in the summation is 2, so the single pixel optical flow amplitude is obtained; for the same The average optical flow amplitude of all single pixels within the residual region is calculated. The absolute values ​​of the forward and backward optical flow lateral displacements at the same pixel coordinates are added together to obtain the lateral bidirectional difference. The absolute values ​​of the forward and backward optical flow longitudinal displacements are added together to obtain the longitudinal bidirectional difference. The bidirectional and longitudinal differences are added together to obtain the single-pixel bidirectional optical flow difference. The bidirectional optical flow difference of all single pixels within the same residual region is averaged to obtain the bidirectional optical flow difference value. Max-min normalization is performed on the number of pixels in the region, the average optical flow amplitude, the bidirectional optical flow difference value, and the segment complexity value. The normalized number of pixels in the region, the average optical flow amplitude, the bidirectional optical flow difference value, and the segment complexity value are added together to generate the partitioned frame interpolation computation amount. The computational cost of interpolation is summarized by segment identifier, and the computational cost for each interpolated segment is generated by combining the pixel size of the residual region and the number of interpolated frames. These records are then arranged in timestamp order to form a computational cost table. Specifically, the computational cost table is generated in timestamp order as follows: Read the interpolation computation amount, residual region pixel size, and number of inserted frames for each residual region partition corresponding to the same interpolated segment identifier. Add the interpolation computation amounts for each residual region partition to obtain the basic computation amount for the segment. Multiply the sum of the residual region pixel sizes by the number of inserted frames to obtain the segment size correction amount. Add the basic computation amount and the segment size correction amount to generate a computation cost record. Read the previous frame timestamp and the next frame timestamp from the segment record corresponding to the interpolated segment. Add the previous frame timestamp and the next frame timestamp and divide by 2 to obtain the segment sorting timestamp. Write the interpolated segment identifier, segment sorting timestamp, basic computation amount, segment size correction amount, number of inserted frames, and computation cost record into the same cost record item. Sort all cost record items in ascending order of segment sorting timestamp. For cost record items with the same segment sorting timestamp, sort them in ascending order of interpolated segment identifier to generate a computation cost table.

[0026] In this embodiment, the step of calculating the output interval deviation between adjacent segments and the node switching delay to generate a timing connection weight table includes: Read the candidate interpolation frames, frame timestamps, and fragment complexity table. Determine the baseline output interval based on the timestamps of adjacent interpolation fragments and the number of interpolated frames. Calculate the absolute difference between the candidate output interval and the baseline output interval to generate the adjacent fragment output interval deviation. Specifically, the calculation of the absolute difference between the candidate output interval and the baseline output interval to generate the adjacent fragment output interval deviation is as follows: Read the interpolation segment identifier, previous frame timestamp, next frame timestamp, number of interpolated frames, and generation time of interpolation candidate frames for two adjacent interpolation segments; subtract the previous frame timestamp from the next frame timestamp of the same interpolation segment to obtain the original time span of the segment; divide the original time span of the segment by the number of interpolated frames plus 1 to obtain the baseline output interval; sort the interpolation candidate frames in two adjacent interpolation segments according to the generation time of the interpolation candidate frames, subtract the generation times of the interpolation candidate frames in two adjacent frames to obtain the candidate output interval; subtract the baseline output interval from the candidate output interval and take the absolute value to obtain the single output interval deviation; average all single output interval deviations at the boundary of the same adjacent segment to generate the adjacent segment output interval deviation; write the adjacent segment output interval deviation, the corresponding previous interpolation segment identifier, next interpolation segment identifier, and segment complexity value into the timing deviation record; Read the edge state table, determine the fragment migration data volume based on the frame size and coded data volume of the interpolation candidate frame, divide the fragment migration data volume by the link bandwidth to obtain the transmission delay, and add the queuing task duration to the transmission delay to generate the node switching delay. Specifically, the node switching delay is generated as follows: Read the link bandwidth, task queue record, and node identifier corresponding to the same acquisition time in the edge status table; read the frame size, encoded data volume, and intercalation fragment identifier corresponding to the intercalation candidate frame; sum the encoded data volume of all intercalation candidate frames within the same intercalation fragment to obtain the candidate frame data volume; sum the task description data volume, candidate frame data volume, and scheduling result feedback data volume corresponding to the intercalation fragment to obtain the fragment migration data volume; divide the fragment migration data volume by the link bandwidth to obtain the fragment transmission delay; read the number of queued tasks preceding the current intercalation task and the estimated execution time of each queued task in the task queue record; sum the estimated execution times of each queued task to obtain the queued task duration; add the queued task duration to the fragment transmission delay to generate the node switching delay; write the node switching delay, node identifier, intercalation fragment identifier, and acquisition time into the node switching delay record; After normalizing the output interval deviation between adjacent segments, node switching delay, and segment complexity value, sum them according to their corresponding weights to generate timing connection weight values, which are then written into the timing connection weight table. Specifically, generating timing connection weight values ​​and writing them into the timing connection weight table involves: Read the adjacent segment output interval deviation, node switching delay, and segment complexity values ​​corresponding to the same interpolated segment identifier; read the maximum and minimum adjacent segment output interval deviations, maximum and minimum node switching delays, maximum and minimum segment complexities from all interpolated segments in the current batch; subtract the minimum adjacent segment output interval deviation from the current interpolated segment's adjacent segment output interval deviation, and then divide by the difference between the maximum and minimum adjacent segment output interval deviations to obtain the normalized output interval deviation; calculate the normalized node switching delay and normalized segment complexity in the same way; calculate the difference between the maximum and minimum adjacent segment output interval deviations, the difference between the maximum and minimum node switching delays, and the minimum segment complexity value, respectively. The difference between the maximum value and the minimum fragment complexity is used to obtain the interval discrete value, switching discrete value, and complexity discrete value. These three discrete values ​​are then added together to obtain the total discrete value. The interval discrete value is divided by the total discrete value to obtain the interval weight, the switching discrete value by the total discrete value to obtain the switching weight, and the complexity discrete value by the total discrete value to obtain the complexity weight. The normalized output interval deviation is multiplied by the interval weight, the normalized node switching delay by the switching weight, and the normalized fragment complexity by the complexity weight. These three products are then added together to generate the timing connection weight value. The interpolated fragment identifier, the previous interpolated fragment identifier, the next interpolated fragment identifier, the adjacent fragment output interval deviation, the node switching delay, the fragment complexity value, and the timing connection weight value are written into the same timing connection record. All timing connection records are sorted in ascending order by the interpolated fragment identifier to generate a timing connection weight table.

[0027] In this embodiment, the generation of the frame interpolation task order and node allocation table includes: Read the computation cost table, edge state table, and temporal connection weight table; establish interpolation task nodes according to the interpolated segment identifiers; establish dependency edges between adjacent interpolation task nodes according to frame timestamp order; write the computation cost record, node computing power status, task queue record, link bandwidth, temporal connection weight value, adjacent segment output interval deviation, and node switching delay into the corresponding interpolation task node; and generate the interpolation task DAG graph. Specifically, generating the interpolation task DAG graph involves: Read the interpolated segment identifier, segment sorting timestamp, and computational cost record from the computational cost table; read the node identifier, node computing power status, task queue record, and link bandwidth from the edge status table; read the timing connection weight value, adjacent segment output interval deviation, and node switching delay from the timing connection weight table; using the interpolated segment identifier as the task node number, establish an interpolated task node for each interpolated segment, and write the segment sorting timestamp, computational cost record, timing connection weight value, adjacent segment output interval deviation, and node switching delay into the interpolated task node attributes; establish edge node attribute records according to the node identifier, including the node computing power status, task queue record, and link bandwidth. Bandwidth is written into the edge node attribute record, and the edge node attribute record is used to establish candidate execution relationships with all interpolation task nodes; all interpolation task nodes are sorted in ascending order of segment sorting timestamp, and directed dependency edges are established between adjacent interpolation task nodes, with the direction of the dependency edges pointing from the interpolation task node with the smaller timestamp to the interpolation task node with the larger timestamp; the data transmission volume between adjacent interpolation task nodes, the output interval deviation between adjacent segments, and the node switching delay are written into the dependency edge attributes; an acyclic task graph is generated according to the interpolation task nodes, candidate execution relationships, dependency edges, and dependency edge attributes, and topological sorting is performed according to the segment sorting timestamp to generate the interpolation task DAG graph; The execution time of each interpolation task node on each edge node is calculated based on the computation cost record and node computing power status. The communication time of dependent edges is calculated based on the link bandwidth and the data transmission volume between adjacent interpolation task nodes. The execution time and the communication time of dependent edges are written into the interpolation task DAG graph to generate HEFT scheduling input data. Specifically, the generation of HEFT scheduling input data is as follows: Read the interpolation task nodes, dependent edges, computation cost records, node computing power status, link bandwidth, and data transmission volume between adjacent interpolation task nodes from the DAG graph of the interpolation task; extract the total computing power and occupied computing power of the node from the node computing power status, subtract the occupied computing power from the total computing power to obtain the effective computing power of the node, divide the computation cost record of each interpolation task node by the effective computing power of each edge node to obtain the execution time of each interpolation task node on each edge node; read the preceding interpolation task nodes and the succeeding interpolation task nodes at both ends of the dependent edge, and connect the preceding interpolation task nodes to the succeeding interpolation task nodes. The data transmission volume transmitted by the node is divided by the link bandwidth between the corresponding edge nodes to obtain the communication duration of the dependent edge; the node number of the frame insertion task, the direction of the dependent edge, the execution duration, the communication duration of the dependent edge, the edge node number, and the effective computing power of the node are written into the DAG graph of the frame insertion task; the execution duration is arranged according to the node number of the frame insertion task to generate a task execution duration matrix, and the communication duration of the dependent edge is arranged according to the direction of the dependent edge to generate a dependent edge communication duration table; the DAG graph of the frame insertion task, the task execution duration matrix, the communication duration table of the dependent edge, the edge node number, and the effective computing power of the node are combined to generate HEFT scheduling input data; The HEFT scheduling input data is input into the HEFT algorithm to calculate the basic upward sorting value for each interpolation task node. The timing stitching weight value is multiplied by the output interval deviation of adjacent segments to generate the timing stitching correction. The basic upward sorting value, the timing stitching correction, and the sorting value of the corresponding dependent subsequent task are added to generate the corrected task sorting value. Specifically, the generation of the corrected task sorting value is as follows: Read the DAG graph of the interpolation task, the task execution duration matrix, the dependency edge communication duration table, and the timing connection weight table from the HEFT scheduling input data; for each interpolation task node, read the execution duration on all edge nodes and calculate the average execution duration of the node; traverse from the node without successor tasks to the dependent predecessor task nodes according to the dependency edge direction in the interpolation task DAG graph; for nodes without successor tasks, record the average execution duration of the node as the base upward sorting value; for interpolation task nodes with successor task nodes, read the base upward sorting value and dependency edge communication duration of each direct successor task node, and compare the dependency edge communication duration with the direct successor task node... The basic upward sorting values ​​are added together to obtain the successor sorting candidate values. The maximum value is selected from all the successor sorting candidate values ​​and added to the average execution time of the node to obtain the basic upward sorting value. The timing connection weight value and the output interval deviation of adjacent segments under the same interpolation segment identifier in the timing connection weight table are read. The timing connection weight value and the output interval deviation of adjacent segments are multiplied to generate the timing connection correction amount. The basic upward sorting value and the timing connection correction amount are added together to generate the correction task sorting value. The interpolation task node number, basic upward sorting value, timing connection correction amount and correction task sorting value are written into the same sorting record to generate a task sorting record table for the node selection process. Arrange the frame interpolation task nodes in descending order of their corrected task sorting values. Calculate the base earliest completion time for each frame interpolation task node at each edge node. Add the node switching delay from the timing alignment weight table to the base earliest completion time to generate the corrected earliest completion time. The specific steps for generating the corrected earliest completion time are as follows: Read the task sorting record table and sort the interpolated task nodes from largest to smallest according to the corrected task sorting value; read the execution duration of the current interpolated task node on the target edge node, the completion time of existing tasks on the target edge node, the completion time of dependent predecessor task nodes, and the communication duration of dependent edges; add the completion time of dependent predecessor task nodes to the communication duration of dependent edges to obtain the dependency arrival time; select the maximum value from all dependency arrival times to obtain the dependency constraint time; compare the completion time of existing tasks on the target edge node with the dependency constraint time and take the larger value as the task start time; add the task start time to the execution duration of the current interpolated task node on the target edge node to generate the basic earliest completion time; read the node switching delay corresponding to the current interpolated task node in the timing connection weight table, add the basic earliest completion time to the node switching delay to generate the corrected earliest completion time; write the interpolated task node number, target edge node number, task start time, basic earliest completion time, node switching delay, and corrected earliest completion time into the node completion time record; For each frame insertion task node, select the edge node with the earliest correction completion time. Write the frame insertion task node number, correction task sorting value, selected edge node, start time, and completion time into the scheduling result to generate a frame insertion task order and node allocation table. Specifically, generating the frame insertion task order and node allocation table involves: Read the edge node numbers, task start times, earliest corrected completion times, and node switching delays corresponding to the same interpolation task node from the node completion time record; compare the earliest corrected completion times of the same interpolation task node on each edge node, and select the edge node with the smallest earliest corrected completion time as the execution node; if the earliest corrected completion times are the same, select the edge node with the smaller node switching delay as the execution node; write the interpolation task node number, corrected task sorting value, execution node number, task start time, earliest corrected completion time, and corresponding interpolation fragment identifier into the task scheduling record; update the existing task completion times of the execution nodes according to the task scheduling record, and continue processing the remaining interpolation task nodes according to the corrected task sorting value from largest to smallest; sort all task scheduling records in ascending order of task start time to generate the interpolation task order; write the interpolation task node number, interpolation fragment identifier, execution node number, task start time, earliest corrected completion time, and node switching delay into the same allocation record according to the interpolation task node number to generate a node allocation table.

[0028] In this embodiment, generating the interpolated video stream includes: Read the frame insertion task order, node allocation table, and candidate frames. Send the candidate frames to the corresponding edge nodes in the node allocation table according to the frame insertion task order. Receive the processing results of the candidate frames and generate a candidate frame result table according to the frame insertion segment identifier. Specifically, the generation of the candidate frame result table is as follows: The system reads the interpolation task node numbers and execution order from the interpolation task sequence, reads the interpolation fragment identifier, execution node number, task start time, and earliest corrected completion time from the node allocation table, and reads the interpolation fragment identifier, candidate frame sequence number, and candidate frame data corresponding to the interpolation candidate frames. It then encapsulates the candidate frame data, candidate frame sequence number, interpolation fragment identifier, and task node number into a candidate frame processing package according to the interpolation task sequence. The package is sent to the edge node corresponding to the execution node number. The system receives the candidate frame processing results returned by the edge node and extracts the interpolation fragment identifier, candidate frame sequence number, processing completion time, processing status, and processed candidate frame data from the processing results. Candidate frame processing results with abnormal processing status are marked as records to be reprocessed, while candidate frame processing results with normal processing status are written to the same result record according to the interpolation fragment identifier and candidate frame sequence number. All result records are sorted in ascending order by interpolation fragment identifier and candidate frame sequence number to generate a candidate frame result table. Read the candidate frame result table, video frame sequence, frame timestamp, and timing stitching weight table. Arrange the original video frames and interpolation candidate frame processing results in the video frame sequence according to the frame timestamp. Perform time position correction based on the output interval deviation between adjacent segments in the timing stitching weight table to generate a timing correction frame sequence. Specifically, the generation of the timing correction frame sequence is as follows: Read the original video frames, frame timestamps, processed candidate frame data from the candidate frame result table, and adjacent segment output interval deviations from the temporal connection weight table in the video frame sequence; establish the original frame sorting position for the original video frames according to the frame timestamps, and establish the candidate frame sorting position for the processed candidate frame data according to the interpolated frame segment identifier and candidate frame sequence number; read the previous frame timestamp, next frame timestamp, and number of interpolated frames corresponding to the same interpolated frame segment; subtract the previous frame timestamp from the next frame timestamp to obtain the segment time span; divide the segment time span by the number of interpolated frames plus 1 to obtain the baseline interpolation interval; and sort the adjacent segments according to the candidate frame sequence number. The reference interpolation interval is accumulated to the timestamp of the previous frame to obtain the reference time position of the candidate frame; the output interval deviation between adjacent segments is read, and half of the output interval deviation between adjacent segments is subtracted from the reference time position of the candidate frame to obtain the corrected time position of the candidate frame; the original video frames are written into the sequence buffer according to their original frame sorting positions, and the processed candidate frame data is written into the sequence buffer according to the corrected time position of the candidate frame; for frame data with the same time position in the sequence buffer, they are arranged in the order of the original video frame and the processed candidate frame data; all frame data in the sequence buffer are output in ascending order of time position to generate a time-corrected frame sequence; The system reads the frame rate parameters, resolution, and encoding format of the timing correction frame sequence and the original video stream. It then performs splicing, encoding, and encapsulation according to the frame rate parameters, resolution, and encoding format to generate an interpolated video stream. Specifically, generating the interpolated video stream involves: The process involves: reading frame data and corresponding time positions from the timing correction frame sequence; reading the frame rate parameters, resolution, and encoding format of the original video stream; traversing the timing correction frame sequence in ascending order of time position and writing a consecutive frame number for each frame; performing scale resampling on frame data with inconsistent size and resolution to make the frame data width and height consistent with the resolution of the original video stream; calculating the playback time interval between adjacent output frames according to the frame rate parameters and writing a playback timestamp for each frame based on the consecutive frame number; arranging the frame data with the written consecutive frame number and playback timestamp into a sequence of frames to be encoded; performing intra-frame prediction, inter-frame prediction, transform quantization, and entropy encoding on the sequence of frames to be encoded according to the encoding format to generate encoded video data; and writing the encoded video data, frame rate parameters, resolution, encoding format, and playback timestamp into a video encapsulation container to generate an interpolated video stream.

[0029] refer to Figure 3 A video frame interpolation system based on edge computing includes the following modules: The video edge access module is used to receive raw video streams and edge node operation data, and generate video frame sequences, frame timestamps, and edge status tables. The segment complexity analysis module is used to create adjacent frame interpolation objects based on the video frame sequence, extract inter-frame change features, and generate interpolated segments and segment complexity tables. The optical flow interpolation generation module is used to build an improved PWC-Net network, perform optical flow estimation and residual partition fusion on the interpolated frame segments, and generate a bidirectional optical flow table, a residual region table, and interpolation candidate frames. The frame interpolation cost evaluation module is used to calculate the computational cost of partitioned frame interpolation based on the bidirectional optical flow table, residual region table, and fragment complexity table, and to generate a computational cost table. The timing alignment evaluation module is used to calculate the output interval deviation between adjacent segments and the node switching delay based on the interpolation candidate frames, frame timestamps, segment complexity table and edge state table, and to generate a timing alignment weight table. The edge task scheduling module is used to construct the DAG graph of the interpolation task based on the calculation cost table, the edge state table and the temporal connection weight table, and to generate the interpolation task order and node allocation table through the HEFT algorithm. The frame interpolation encoding output module is used to perform correction, splicing and encoding on candidate frames for frame interpolation according to the frame interpolation task order and node allocation table, and generate frame interpolation video stream.

[0030] Example 1: To verify the feasibility of this invention in practice, it was applied to a multi-channel edge video surveillance frame interpolation scenario in a park. The video acquisition terminal received six low-frame-rate video streams, each 10 minutes long, with an original frame rate of 15fps and a target output frame rate of 30fps. The resolution was 1280×720 pixels, and the encoding format was H.264. The park included vehicle traffic areas, pedestrian areas, and entrance / exit gate areas. The video showed vehicles passing quickly, pedestrian obstruction, nighttime lighting changes, and network transmission fluctuations. Four edge nodes were deployed at the edge, with node computing power levels of 92%, 78%, 65%, and 54%, respectively. The link bandwidth ranged from 18Mbps to 45Mbps, and the number of queued tasks ranged from 3 to 11.

[0031] After receiving the raw video stream, the system records the video stream identifier, acquisition start time, frame rate parameters, resolution, and encoding format, and simultaneously acquires the node identifier, node computing power status, task queue records, and link bandwidth. The system decodes and frames the raw video stream according to the H.264 encoding format, resulting in 9000 raw video frames for each stream. Each video frame is assigned a frame number and timestamp according to the acquisition start time and 15fps frame rate parameter, generating a video frame sequence and frame timestamp. The edge node operation data, aligned with the node identifier and acquisition time, forms an edge status table. Each node status record includes the node computing power status, queue length, queued task duration, and link bandwidth.

[0032] During the frame interpolation segmentation stage, the system divides the video frame sequence according to the adjacent frame combination method. Every two adjacent video frames form an interpolation segment, resulting in 8999 interpolation segments per video stream. For interpolation segment numbered 3260, the timestamp of the preceding frame is 217.333s, the timestamp of the following frame is 217.400s, and the interpolation frame position is located between the two frames. The system extracts the luminance channel, texture gradient, and block-level displacement of the preceding and following frames, calculating an inter-frame luminance difference of 0.18, a texture difference of 0.24, and a motion of 7.6 pixels. After normalization and index weighting, the segment complexity value is 0.62, belonging to the medium-high complexity segment category. For static background segments, the segment complexity values ​​are concentrated between 0.12 and 0.25; for segments with vehicles passing by quickly, the segment complexity values ​​are concentrated between 0.68 and 0.91.

[0033] After the interpolated frame segments enter the improved PWC-Net network, the system extracts scale features from the previous frame and the next frame in the pyramid feature extraction layer according to the scale from low resolution to high resolution, forming a pyramid feature group with five scales: 1 / 16, 1 / 8, 1 / 4, 1 / 2, and the original scale. For segments with a complexity value of 0.62, the scale gating layer reads the segment complexity value, node computing power margin, task queue length, and link bandwidth, normalizes each indicator, and performs a weighted sum to generate scale retention weights. When the edge node computing power is 65%, the queue length is 8, and the link bandwidth is 22Mbps, the system retains the 1 / 16, 1 / 8, 1 / 4, and 1 / 2 scale features, reduces the proportion of original scale features, and generates a scale-selected feature group.

[0034] In the feature distortion layer, the system uses the zero-displacement field at the lowest resolution scale as the initial bidirectional estimation flow. Following the order from low-resolution to high-resolution scales, it performs backsampling on the features selected at the previous frame scale and the features selected at the next frame scale, generating a distortion feature set and a scale-wise initial optical flow set. Taking a segment of a vehicle moving quickly from left to right as an example, the average optical flow amplitude at the low-resolution scale is 3.2 pixels, which increases to 11.7 pixels at the 1 / 2 scale after scale-wise refinement. The cost volume construction layer performs local correlation calculations on the previous frame scale selection features and their corresponding distortion features, and the next frame scale selection features and their corresponding distortion features, to obtain the bidirectional cost volume. The residual partitioning fusion layer divides the image into fast-moving regions, occluded regions, and static background regions based on the correlation response of the bidirectional cost volume, the residual amplitude of the scale-wise initial optical flow set, and the segment complexity value. In segment number 3260, the fast-moving region accounts for 18.4%, the occluded region accounts for 6.7%, and the static background region accounts for 74.9%.

[0035] In the frame interpolation synthesis layer, the system inputs the residual fusion feature set, residual region table, and scale-wise initial optical flow set to generate forward optical flow, backward optical flow, and corresponding interpolation candidate frames. For segment number 3260, the average gradient value of the vehicle contour boundary in the interpolation candidate frame is 0.71, and the average texture difference in the background region is 0.09. Compared with the PWC-Net processing results without the scale gating layer and residual partitioning fusion layer, the width of the ghosting region at the vehicle edge is reduced from an average of 5.8 pixels to 2.1 pixels, and the local tearing region at the occlusion boundary is reduced by approximately 46%.

[0036] The system calculates the computational cost table based on the bidirectional optical flow table, residual region table, and fragment complexity table. For each interpolated frame fragment, the system extracts the forward optical flow, backward optical flow, residual region, and fragment complexity values, and calculates the number of pixels, average optical flow amplitude, and bidirectional optical flow difference value for the fast-moving region, occluded region, and static background region, respectively. For fragment number 3260, the fast-moving region has 152,640 pixels, an average optical flow amplitude of 11.7 pixels, and a bidirectional optical flow difference value of 2.4; the occluded region has 55,580 pixels, an average optical flow amplitude of 8.9 pixels, and a bidirectional optical flow difference value of 3.1. The system normalizes the number of pixels, average optical flow amplitude, bidirectional optical flow difference value, and fragment complexity value, then sums them to generate the computational cost for each partitioned frame interpolation, and summarizes these to form a computational cost record.

[0037] During the timing stitching weight table generation stage, the system calculates the output interval deviation between adjacent segments and the node switching delay based on the interpolation candidate frames, frame timestamps, segment complexity table, and edge state table. For two adjacent interpolation segments, the baseline output interval is 0.033s, and the actual candidate output interval for the edge node returning candidate frames is 0.041s, with an absolute difference of 0.008s. Based on the frame size and coded data volume of the interpolation candidate frames, the system determines the segment migration data volume to be 1.8MB. With a link bandwidth of 22Mbps and a transmission delay of 0.65s, combined with a queuing task duration of 0.42s, a node switching delay of 1.07s is generated. The system normalizes the output interval deviation, node switching delay, and segment complexity value and sums them according to their weights to generate a timing stitching weight value of 0.58.

[0038] During the edge task scheduling phase, the system establishes interpolation task nodes according to the interpolated segment identifier and establishes dependency edges between adjacent interpolation task nodes according to the frame timestamp order, generating an interpolation task DAG graph. For a 60-second video segment, the system establishes 899 interpolation task nodes and 898 dependency edges. The HEFT algorithm reads the calculation cost table, edge state table, and temporal connection weight table, first calculates the basic upward sorting value, and then multiplies the temporal connection weight value by the output interval deviation of adjacent segments to generate the temporal connection correction amount. The basic upward sorting value of task node number 3260 is 13.6, the temporal connection correction amount is 0.46, and after combining the dependent successor task sorting value, the corrected task sorting value is 18.9. The system arranges the interpolation task nodes in descending order of the corrected task sorting value and adds the node switching delay to the basic earliest completion time during the node selection process. The earliest completion time of edge node 2 is 2.84s, which becomes 3.91s after adding the node switching delay; the earliest completion time of edge node 3 is 3.10s, which becomes 3.36s after adding the node switching delay. Therefore, the system selects edge node 3 to perform the corresponding frame interpolation task.

[0039] During the frame interpolation output stage, the system sends candidate frames for interpolation to the corresponding edge nodes for processing according to the interpolation task order and node allocation table, and receives the processing results of the candidate frames. It then generates a candidate frame result table based on the interpolation segment identifier. The system arranges the original video frames and candidate frame processing results in the video frame sequence according to the frame timestamp, performs time position correction based on the output interval deviation between adjacent segments in the timing concatenation weight table, and generates a timing-corrected frame sequence. After correction, the frame rate of the interpolated video stream stabilizes at 30fps, with an average end-to-end latency of 186ms and a maximum latency of 241ms. Finally, the system performs splicing, encoding, and encapsulation according to the frame rate parameters, resolution, and encoding format to generate the interpolated video stream.

[0040] In the comparative experiment, the system compared the method of this invention with the fixed edge node PWC-Net frame interpolation method. The test video consisted of 36 segments, each 60 seconds long. Experimental results showed that the average end-to-end latency of the fixed edge node method was 312ms, while that of this invention was 186ms; the average ghost width of the fixed edge node method in fast-moving regions was 5.6 pixels, while that of this invention was 2.3 pixels; the average segment boundary output interval deviation of the fixed edge node method was 0.018s, while that of this invention was 0.006s; the average load fluctuation of edge nodes in the fixed edge node method was 34.8%, while that of this invention was 17.5%; and the subjective continuity score of the interpolated video was 82.1 points, while that of this invention was 91.4 points. As can be seen from Example 1, this invention can complete optical flow estimation, interpolation candidate frame generation, interpolation task scheduling, and encoding output even in scenarios with fluctuating edge node computing power, differences in video segment complexity, and temporal continuity between adjacent segments, verifying the engineering feasibility of this method in edge video interpolation scenarios.

[0041] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An edge computing based video interpolation method, characterized in that, Includes the following steps: Collect raw video stream, node computing power status, task queue records and link bandwidth; decode and segment the raw video stream into frames and extract timestamps to generate video frame sequences, frame timestamps and edge status tables. The video frame sequence is divided into adjacent frame combinations, and the inter-frame brightness difference, texture difference and motion amount are extracted to generate interpolated frame segments and segment complexity tables. An improved PWC-Net network is constructed by adding a scale-gating layer after the pyramid feature extraction layer and a residual partition fusion layer after the cost volume construction layer. The interpolated frame fragments, fragment complexity table and edge state table are input into the improved PWC-Net network to generate a bidirectional optical flow table, residual region table and interpolated frame candidate frames. The computational cost of partitioned frame interpolation is calculated based on the bidirectional optical flow table, residual region table, and fragment complexity table, and a computational cost table is generated. Based on the interpolation candidate frames, frame timestamps, fragment complexity table and edge state table, calculate the output interval deviation between adjacent fragments and node switching delay, and generate a timing connection weight table. The frame insertion task DAG graph is constructed based on the calculation cost table, edge state table, and temporal connection weight table. The graph is then input into the HEFT algorithm. The temporal connection weight table is introduced during the task sorting process, and the node switching delay in the temporal connection weight table is introduced during the node selection process to generate the frame insertion task order and node allocation table. Perform interpolation candidate frame correction, timing splicing, and encoding output according to the interpolation task order and node allocation table to generate an interpolated video stream.

2. The video frame interpolation method based on edge computing according to claim 1, characterized in that, The generated video frame sequence, frame timestamps, and edge state table include: Receive the raw video stream, record the video stream identifier, acquisition start time, frame rate parameters, resolution and encoding format, acquisition node identifier, node computing power status, task queue records and link bandwidth; Decode the original video stream according to the encoding format, write the frame sequence number and timestamp for the video frame according to the acquisition start time and frame rate parameters, and generate a video frame sequence and frame timestamp; Align the node's computing power status, task queue records, and link bandwidth according to the node identifier and collection time to generate an edge status table.

3. The video frame interpolation method based on edge computing according to claim 1, characterized in that, The generated interpolated fragments and fragment complexity table include: Read the video frame sequence and frame timestamp, combine adjacent frames according to frame number, and write the previous frame, next frame, timestamp and insertion frame position into the segment record to generate the inserted frame segment; Extract the luminance channel, texture gradient, and block displacement from the previous and next frames, respectively calculate the average difference of the luminance channel, the average difference of the texture gradient, and the average displacement distance, and generate the inter-frame luminance difference, texture difference, and motion amount; After normalizing the inter-frame brightness difference, texture difference, and motion, sum them according to the index weights to generate a segment complexity value, and write it into the segment complexity table corresponding to the interpolated segment.

4. The video frame interpolation method based on edge computing according to claim 1, characterized in that, The generation of the bidirectional optical flow table, residual region table, and interpolation candidate frames includes: An improved PWC-Net network is constructed, which includes a pyramid feature extraction layer, a scale gating layer, a feature warping layer, a cost volume construction layer, a residual partition fusion layer, and a frame interpolation synthesis layer. The interpolated frame fragments are input into the pyramid feature extraction layer. The preceding and following frames in the interpolated frame fragments are read, and the scale features of the preceding and following frames are extracted in order from low-resolution scale to high-resolution scale to generate pyramid feature groups. Input the pyramid feature group, fragment complexity table and edge state table into the scale gating layer, read the fragment complexity value in the fragment complexity table, read the node computing power status, task queue record and link bandwidth in the edge state table, perform normalization processing on the fragment complexity value, node computing power margin, task queue length and link bandwidth and perform weighted summation to generate scale retention weights, filter the pyramid feature group according to the scale retention weights, and generate scale-filtered feature groups. The scale-selected feature group is input into the feature warping layer. The zero displacement field of the lowest resolution scale is used as the initial bidirectional estimation flow. The scale-selected features of the next frame and the scale-selected features of the previous frame are backsampled along the initial bidirectional estimation flow in the order from low resolution scale to high resolution scale, respectively, to generate the warping feature group and the scale-by-scale initial optical flow group. The scale-selected feature group, the distortion feature group, and the scale-by-scale initial optical flow group are input into the cost volume construction layer. Local correlation calculation is performed on the scale-selected features of the previous frame and the corresponding distortion features, and local correlation calculation is performed on the scale-selected features of the subsequent frame and the corresponding distortion features to generate a bidirectional cost volume. The bidirectional cost volume, the scale-wise initial optical flow group, and the fragment complexity table are input into the residual partitioning fusion layer. Based on the correlation response of the bidirectional cost volume, the residual amplitude of the scale-wise initial optical flow group, and the fragment complexity value, the fast motion region, the occlusion region, and the static background region are divided to generate the residual region table. Partitioning fusion is then performed on the bidirectional cost volume and the distortion feature group to generate the residual fusion feature group. The residual fusion feature set, residual region table, and scale-wise initial optical flow set are input into the interpolation synthesis layer to generate a bidirectional optical flow table and interpolation candidate frames. The improved PWC-Net network is trained by using interpolation reconstruction error, optical flow consistency error, and residual region partitioning error as joint optimization objectives. The parameters in the pyramid feature extraction layer, scale gating layer, feature distortion layer, cost volume construction layer, residual partition fusion layer, and interpolation synthesis layer are continuously optimized. When the change of the joint optimization objective is less than 0.5% during 5 consecutive training rounds, the trained improved PWC-Net network is obtained.

5. The video frame interpolation method based on edge computing according to claim 1, characterized in that, The generated computational cost table includes: Read the bidirectional optical flow table, residual region table, and fragment complexity table, extract the forward optical flow, backward optical flow, residual region, and fragment complexity values ​​according to the interpolated fragment identifier, and generate partitioned statistical records; The number of regional pixels, average optical flow amplitude, and bidirectional optical flow difference value of each residual region are calculated based on the partition statistics records. These three values ​​are then added to the fragment complexity value after normalization to generate the partition interpolation computation amount. The computational cost of interpolation is summarized by the interpolation segment identifier, and the computational cost record for each interpolation segment is generated by combining the pixel scale of the residual region and the number of interpolated frames. The computational cost table is generated by arranging the records in order of timestamp.

6. The video frame interpolation method based on edge computing according to claim 1, characterized in that, The calculation of the output interval deviation between adjacent segments and the node switching delay, and the generation of the timing connection weight table, includes: Read the candidate frames for interpolation, frame timestamps, and fragment complexity table. Determine the baseline output interval based on the timestamps of adjacent interpolated fragments and the number of interpolated frames. Calculate the absolute difference between the candidate output interval and the baseline output interval to generate the output interval deviation between adjacent fragments. Read the edge state table, determine the fragment migration data amount based on the frame size and encoded data amount of the interpolation candidate frame, divide the fragment migration data amount by the link bandwidth to obtain the transmission delay, add the queuing task duration to the transmission delay to generate the node switching delay; After normalizing the output interval deviation between adjacent segments, node switching delay, and segment complexity value, sum them according to their corresponding weights to generate the timing connection weight value, and write it into the timing connection weight table.

7. The video frame interpolation method based on edge computing according to claim 1, characterized in that, The frame interpolation task order and node allocation table includes: Read the computation cost table, edge state table, and timing connection weight table. Establish frame insertion task nodes according to the frame insertion segment identifier. Establish dependency edges between adjacent frame insertion task nodes according to the frame timestamp order. Write the computation cost record, node computing power status, task queue record, link bandwidth, timing connection weight value, adjacent segment output interval deviation, and node switching delay into the corresponding frame insertion task node to generate a frame insertion task DAG graph. The execution time of each interpolation task node on each edge node is calculated based on the calculation cost record and node computing power status. The communication time of dependent edges is calculated based on the link bandwidth and the data transmission volume between adjacent interpolation task nodes. The execution time and the communication time of dependent edges are written into the interpolation task DAG graph to generate HEFT scheduling input data. The HEFT scheduling input data is input into the HEFT algorithm to calculate the basic upward sorting value of each interpolation task node. The timing connection weight value is multiplied by the output interval deviation of adjacent segments to generate the timing connection correction amount. The basic upward sorting value, the timing connection correction amount and the sorting value of the corresponding dependent subsequent task are added to generate the corrected task sorting value. Arrange the interpolation task nodes in descending order of the corrected task sorting value, calculate the base earliest completion time of each interpolation task node on each edge node in turn, add the node switching delay in the timing connection weight table to the base earliest completion time, and generate the corrected earliest completion time. For each frame insertion task node, select the edge node with the earliest correction completion time and the smallest correction completion time. Write the frame insertion task node number, correction task sorting value, selected edge node, start time and completion time into the scheduling result to generate the frame insertion task order and node allocation table.

8. The video frame interpolation method based on edge computing according to claim 1, characterized in that, The generated interpolated video stream includes: Read the frame insertion task order, node allocation table and frame insertion candidate frames, send the frame insertion candidate frames to the edge nodes corresponding to the node allocation table according to the frame insertion task order, receive the frame insertion candidate frame processing results, and generate a candidate frame result table according to the frame insertion segment identifier. Read the candidate frame result table, video frame sequence, frame timestamp and timing stitching weight table, arrange the original video frames and interpolation candidate frame processing results in the video frame sequence according to the frame timestamp, perform time position correction according to the adjacent segment output interval deviation in the timing stitching weight table, and generate timing correction frame sequence; Read the frame rate parameters, resolution, and encoding format of the timing correction frame sequence and the original video stream, and perform splicing, encoding, and encapsulation according to the frame rate parameters, resolution, and encoding format to generate an interpolated video stream.

9. A video frame interpolation system based on edge computing, comprising executing the video frame interpolation method based on edge computing as described in any one of claims 1 to 8, characterized in that, Includes the following modules: The video edge access module is used to receive raw video streams and edge node operation data, and generate video frame sequences, frame timestamps, and edge status tables. The segment complexity analysis module is used to create adjacent frame interpolation objects based on the video frame sequence, extract inter-frame change features, and generate interpolated segments and segment complexity tables. The optical flow interpolation generation module is used to build an improved PWC-Net network, perform optical flow estimation and residual partition fusion on the interpolated frame segments, and generate a bidirectional optical flow table, a residual region table, and interpolation candidate frames. The frame interpolation cost evaluation module is used to calculate the computational cost of partitioned frame interpolation based on the bidirectional optical flow table, residual region table, and fragment complexity table, and to generate a computational cost table. The timing alignment evaluation module is used to calculate the output interval deviation between adjacent segments and the node switching delay based on the interpolation candidate frames, frame timestamps, segment complexity table and edge state table, and to generate a timing alignment weight table. The edge task scheduling module is used to construct the DAG graph of the interpolation task based on the calculation cost table, the edge state table and the temporal connection weight table, and to generate the interpolation task order and node allocation table through the HEFT algorithm. The frame interpolation encoding output module is used to perform correction, splicing and encoding on candidate frames for frame interpolation according to the frame interpolation task order and node allocation table, and generate frame interpolation video stream.

Images