Video slow motion frame interpolation and playback system based on generative adversarial network

Abstract

The application relates to the technical field of image communication, in particular to a video slow-motion frame insertion playback system based on a generative adversarial network, which comprises a video signal acquisition module, a signal processing module, a frame rate conversion module and a playback encoding output module. The acquisition module outputs an original video frame sequence through a ring buffer; the processing module parallelly separates a detail component after denoising and an edge gradient feature; the frame rate conversion module uses a geometric structure as a boundary locking condition, guides the detail component to execute nonlinear motion compensation and intermediate frame reconstruction; and the playback module executes time base remapping and distributed encoding transmission. Through a structure constraint texture depth generation architecture, in combination with a space-time consistency check and a nearest neighbor pixel backfilling mechanism and a virtual time base rate decoupling technology, the application realizes real-time slow-motion video review for high-speed moving targets with high signal-to-noise ratio and no artifacts.

Description

Technical Field

[0001] This invention relates to the field of image communication technology, specifically to a video slow-motion frame interpolation playback system based on generative adversarial networks. Background Technology

[0002] In video surveillance and industrial site visual analysis systems, signal acquisition and real-time slow-motion playback of high-speed moving targets are core functions for manual monitoring and assisted interpretation of critical events (such as equipment transient failures and trajectory attitude anomalies). Existing video playback systems mainly adopt two approaches: one is to rely on high-speed front-end cameras, but this generates huge data throughput, which can easily cause transmission bandwidth congestion and storage resource exhaustion; the other is to use back-end workstations to perform frame interpolation and up-conversion on standard frame rate signals.

[0003] However, existing solutions have significant systemic flaws in practical video communication links. Because video signals are easily affected by the environment during acquisition and transmission (such as reduced signal modulation in foggy weather and thermal noise introduced by low illumination), existing video processing units often lack integrated signal conditioning mechanisms. Traditional transmission architectures separate "signal noise reduction" and "frame rate conversion" into independent serial modules, causing frame interpolation units to be unable to distinguish between effective high-frequency components and channel noise. This easily leads to noise being misidentified as motion vectors, thus introducing severe block artifacts and artifacts into the reconstructed stream. Furthermore, in distributed display scenarios, the latency of traditional software processing prevents display terminals from achieving precise frame-level synchronization across screens, severely affecting the temporal consistency of review footage and failing to meet the real-time retrieval requirements of professional-grade monitoring systems for minute details.

[0004] In summary, the technical problem to be solved by this invention is: how to achieve real-time slow-motion video replay with high signal-to-noise ratio and no artifacts for high-speed moving targets in complex acquisition environments containing noise interference, low illumination or smoke obstruction, and solve the problem that motion detail analysis fails due to the degradation of video source image quality in existing systems.

[0005] To address this, a video slow-motion frame interpolation playback system based on generative adversarial networks is proposed. Summary of the Invention

[0006] The purpose of this invention is to provide a video slow-motion frame interpolation and playback system based on generative adversarial networks. By constructing parallel signal processing paths, waveform correction and denoising and geometric structure feature extraction are performed simultaneously. By utilizing generative adversarial networks, a real-time slow-motion replay system with high signal-to-noise ratio and no artifacts can be achieved for high-speed moving targets under harsh working conditions.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A video slow-motion frame interpolation and playback system based on generative adversarial networks includes:

[0009] The video signal acquisition module receives the main video signal through a multi-standard video communication interface, performs time-domain slicing on the main video signal using a programmable logic array and writes it into a circular buffer, outputting a continuous sequence of raw time-series video frames.

[0010] The signal processing module receives the original time-series video frame sequence and performs parallel signal processing. The first path performs waveform correction for denoising and dehazing of the signal and separates the video detail component signal. The second path synchronously separates the edge gradient features in the main video signal and outputs a geometric structure reference signal.

[0011] The frame rate conversion module loads the geometric structure reference signal as a boundary locking condition into the generative video synthesis unit, guides the video detail component signal to perform nonlinear motion compensation and intermediate frame signal reconstruction between adjacent frames, and outputs the reconstructed video data stream after spatiotemporal consistency verification.

[0012] The playback encoding output module receives the reconstructed video data stream, responds to the playback control command input via the human-computer interaction terminal, performs time-base remapping processing, and encodes and transmits the slow-motion replay signal to the display terminal through a distributed display control device.

[0013] Preferably, the multi-standard video communication interface is integrated with an SDI interface, an HDMI interface, and an optical fiber video interface, which uniformly converts the incoming main video signal into a parallel digital signal and transmits it to the programmable logic array. The method by which the programmable logic array performs time-domain slicing operation on the parallel digital signal includes: stamping each frame signal with a hardware timestamp, writing the data into a circular buffer according to a first-in-first-out buffer logic, wherein the circular buffer is mapped to the physical address space of DDR SDRAM, performing a loop-over addressing with the beginning and end connected using a write pointer to construct a sliding time window, and extracting the corresponding video frame from the sliding time window using the hardware timestamp as an index and integrating it to output the original time-series video frame sequence.

[0014] Preferably, the method for receiving the original temporal video frame sequence and performing parallel signal processing includes: splitting the original temporal video frame sequence into two synchronously running processing paths; after the original temporal video frame sequence enters the first path, performing a dehazing operation and using an image restoration algorithm to restore the scene radiance; based on the restored scene radiance, performing a bilateral filtering denoising operation on the original temporal video frame sequence, performing a sliding window convolution on each video frame to extract video detail component signals, wherein the video detail component signals include surface high-frequency texture components, motion features, and contour edge information of moving targets in the video image; after the original temporal video frame sequence synchronously enters the second path, using horizontal and vertical convolution templates, performing a weighted difference operation on the pixel grayscale of each video frame to obtain the horizontal gradient component and the vertical gradient component and fusing them into a gradient vector magnitude as the edge gradient feature; and generating a geometric structure reference signal after binarizing the edge gradient feature.

[0015] Preferably, the generative video synthesis unit employs a conditional generative adversarial network. The overall architecture comprises a deep convolutional neural network consisting of a dual-stream encoder and a cascaded decoder. The hierarchical structure includes: a structural feature encoding branch, composed of multiple downsampling convolutional layers, used to extract spatial topological features of video frames; a texture motion encoding branch, set in parallel with the structural feature encoding branch, internally integrating a feature inverse twisting layer, used to extract texture features of video frames and perform feature alignment; and a spatial feature modulation decoding layer, located in the network's decoding stage, serving as the data aggregation node for the structural feature encoding branch and the texture motion encoding branch. Internally, it includes multiple cascaded upsampling deconvolutional layers. These upsampling deconvolutional layers receive the aggregated feature data, utilize structural features to perform pixel-by-pixel normalization modulation of texture features, and perform resolution restoration operations layer by layer.

[0016] Preferably, the specific method for performing nonlinear motion compensation and intermediate frame signal reconstruction includes: the structural feature encoding branch performs multi-scale feature extraction on the received geometric structure reference signal, generates affine transformation parameters containing spatial semantic layout information, and constructs boundary locking conditions that restrict the spatial degrees of freedom of the synthesis process; the texture motion encoding branch receives the video detail component signal, drives the feature inverse distortion layer to perform nonlinear motion compensation based on the boundary locking conditions, remaps the texture feature pixels of the video frame to the predicted position of the current intermediate frame, and generates a motion-calibrated texture feature tensor; the spatial feature modulation decoding layer performs parameterized modulation on the texture feature tensor to recover the video texture details, and reconstructs the intermediate video frame after multi-level cascaded upsampling; the intermediate video frame is embedded into the original temporal video frame sequence in timestamp order to form an interpolated video stream to be verified.

[0017] Preferably, the step of outputting the reconstructed video data stream after spatiotemporal consistency verification includes: extracting intermediate video frames from the interpolated video stream, calculating the luminance histogram difference value and structural similarity index between the intermediate video frame and adjacent video frames, and determining the signal-to-noise ratio and structural integrity of the generated video frame; performing spatiotemporal consistency verification, and if the difference value corresponding to the signal-to-noise ratio is lower than a preset safety threshold and the structural similarity index corresponding to the structural integrity is higher than a preset standard, the verification is deemed successful, and the interpolated video stream is marked and output as the reconstructed video data stream; if the difference value corresponding to the signal-to-noise ratio is not lower than a preset safety threshold, then pixel backfilling based on the nearest neighbor in the temporal domain is triggered, and based on the position of the current intermediate frame on the time axis, the video frame closest to the intermediate frame is locked as the target reference frame, the pixel data of the spatially corresponding position in the target reference frame is extracted, and directly migrated and covered to the abnormal area of ​​the intermediate video frame, and the modified reconstructed video data stream is output.

[0018] Preferably, the method for performing time base remapping processing in response to playback control commands input via a human-computer interaction terminal includes: constructing a virtual display time base, stripping the original acquisition timestamps from the reconstructed video data stream, and reallocating monotonically increasing virtual display timestamps for each video frame based on the slow motion ratio set by the playback control command, with the first frame as the reference point; establishing a rate adaptation buffer pool, writing the generated reconstructed video data stream into the rate adaptation buffer pool, controlling the reading rate based on the virtual display timestamps, and integrating the video frame data with the virtual display timestamps before inputting them into a video compression encoder to generate slow motion replay signal encoding.

[0019] Preferably, the distributed display and control device integrates a signal distribution unit and is connected to a display terminal array, comprising: the signal distribution unit receiving slow-motion replay signal encoding, encapsulating the slow-motion replay signal encoding into a data stream adapted to the transmission format of the current physical communication link, wherein the physical communication link includes an Ethernet network and an optical fiber transmission link; distributing the data stream to each node of the display terminal array using the physical communication link; each node of the display terminal array independently receiving the data stream, performing local decoding on the bitstream in the data stream, and synchronously controlling the screen refresh timing according to the virtual display timestamp to maintain the timing synchronization of the slow-motion replay images.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0021] 1. This invention employs a dual-stream coding architecture for generative adversarial networks (GANs). Addressing the motion artifacts caused by the disconnect between image enhancement and frame interpolation in background techniques, it utilizes a spatial feature modulation decoding layer to perform pixel-by-pixel normalization modulation of texture features using structural features, thus forcibly constraining the texture generation range. Combined with nonlinear motion compensation from a feature inverse distortion layer, it eliminates ghosting and tearing at the edges of high-speed moving objects, resolving the visual artifacts that accompany traditional methods when improving sharpness, and enhancing the detail fidelity of high-speed maneuvering targets against complex backgrounds.

[0022] 2. This invention synchronously performs denoising, dehazing, waveform correction, and edge gradient feature extraction through a parallel signal processing path. Addressing the shortcomings of motion feature extraction failure caused by severe weather and insufficient lighting, as mentioned in the background technology, this invention utilizes the insensitivity of gradient modulus to light intensity fluctuations. This overcomes the optical flow calculation deviations caused by extremely low contrast in smoke, strong light, or low-light environments in existing technologies, ensuring effective identification of minute details even under harsh acquisition conditions, and outputting high signal-to-noise ratio, clear outline replay images.

[0023] 3. This invention utilizes a field-programmable logic array (FPGA) to perform hardware-level time-domain slicing of multi-standard signals, and performs cyclic overwrite addressing within a circular buffer mapped to a double-rate synchronous dynamic random access memory (DRAM) by writing pointers. Addressing the frame loss and timing misalignment bottlenecks that easily occur during real-time acquisition of high-resolution video streams in the background technology, this mechanism solves the frame loss problem caused by insufficient bus throughput bandwidth in traditional software architectures through a heterogeneous hierarchical storage strategy at the hardware level. This provides precise timing anchors for subsequent algorithms, ensuring the authenticity and continuity of the analysis foundation.

[0024] 4. This invention establishes a real-time quality inspection and motion-aligned pixel backfilling mechanism based on structural similarity. Addressing the shortcomings of traditional software processing delays in distributed display scenarios, which prevent display terminals from achieving precise frame-level synchronization across screens and severely affect the timing consistency of playback footage, this invention effectively suppresses false textures generated by the generative model when features are missing by performing motion vector-corrected reference frame pixel backfilling on abnormally reconstructed areas. Combined with virtual display time base and distributed synchronization control, it fundamentally solves the frame synchronization deviation problem in existing systems during multi-terminal playback. Attached Figure Description

[0025] Figure 1 This is an overall architecture diagram of the video slow-motion frame interpolation playback system based on generative adversarial networks provided in Embodiment 1 of the present invention;

[0026] Figure 2 This is a schematic diagram of the entire process of video signal acquisition, parallel signal processing, and distributed synchronous display and control provided in this embodiment.

[0027] Figure 3This is a schematic diagram of the real-time inference acceleration and data interaction logic of the FPGA and GPU computing architecture provided in this embodiment 2. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Please see Figures 1 to 3 This invention provides a video slow-motion frame interpolation playback system based on generative adversarial networks, the technical solution of which is as follows:

[0030] A video slow-motion frame interpolation and playback system based on generative adversarial networks includes:

[0031] The video signal acquisition module receives the main video signal through a multi-standard video communication interface, performs time-domain slicing on the main video signal using a programmable logic array and writes it into a circular buffer, outputting a continuous sequence of raw time-series video frames.

[0032] The signal processing module receives the original time-series video frame sequence and performs parallel signal processing. The first path performs waveform correction for denoising and dehazing of the signal and separates the video detail component signal. The second path synchronously separates the edge gradient features in the main video signal and outputs a geometric structure reference signal.

[0033] The frame rate conversion module loads the geometric structure reference signal as a boundary locking condition into the generative video synthesis unit, guides the video detail component signal to perform nonlinear motion compensation and intermediate frame signal reconstruction between adjacent frames, and outputs the reconstructed video data stream after spatiotemporal consistency verification.

[0034] The playback encoding output module receives the reconstructed video data stream, responds to the playback control command input via the human-computer interaction terminal, performs time-base remapping processing, and encodes and transmits the slow-motion replay signal to the display terminal through a distributed display control device.

[0035] Example 1:

[0036] This embodiment demonstrates the specific application of the video slow-motion frame interpolation and playback system based on generative adversarial networks provided by this invention in multi-format video stream acquisition and real-time slow-motion review scenarios. (See also...) Figures 1 to 2 The implementation steps are as follows:

[0037] Furthermore, the multi-standard video communication interface is integrated with an SDI interface, an HDMI interface, and an optical fiber video interface, which uniformly converts the incoming main video signal into a parallel digital signal and transmits it to the programmable logic array. The method by which the programmable logic array performs time-domain slicing operation on the parallel digital signal includes: stamping each frame signal with a hardware timestamp, writing the data into a circular buffer according to a first-in-first-out buffer logic, wherein the circular buffer is mapped to the physical address space of DDR SDRAM, performing a loop-over addressing with the beginning and end connected using a write pointer to construct a sliding time window, and extracting the corresponding video frame from the sliding time window using the hardware timestamp as an index and integrating it to output the original time-series video frame sequence.

[0038] Specifically, in this embodiment, the multi-standard video communication interface is integrated with an SDI interface, an HDMI interface, and an optical fiber video interface, and adopts an FPGA mezzanine card architecture. This mezzanine card architecture is based on a modular hardware design concept, supporting independent upgrades and replacements of interface daughter cards. The SDI interface is configured as a 12G-SDI interface conforming to the SMPTE ST 2082 standard, supporting uncompressed main channel video signal access at 60fps in 4K resolution. The HDMI interface follows the HDMI 2.1 protocol specification, possessing dynamic HDR metadata parsing and 48Gbps high-bandwidth transmission capability. The optical fiber video interface integrates an SFP+ optical transceiver module for carrying high-speed optical signals. The differential signal received by the physical interface is first input to the high-speed serial transceiver inside the programmable logic array. A phase-locked loop circuit is used to recover the clock signal from the bitstream, and serial-to-parallel conversion is performed to reconstruct the data words. To accommodate the frequency difference between the front-end recovery clock domain and the back-end programmable logic array system's main clock domain, an asynchronous first-in-first-out queue with a depth of 1024 is deployed between the deserialization module and the core logic to ensure that the storage process follows a first-in-first-out logical order. The asynchronous first-in-first-out queue's write port is driven by the recovery clock, and the read port is driven by the system master clock. A full / empty flag monitoring mechanism prevents data overflow or empty reads, and its depth configuration is sufficient to smooth clock jitter and instantaneous data rate fluctuations. Valid pixel data synchronized to the master clock domain then undergoes protocol parsing logic to remove audio packets and auxiliary timecodes during line and field blanking. It is then converted into parallel digital signals in RGB format via a color space conversion matrix and a chroma upsampling filter before being transmitted to the subsequent processing unit of the programmable logic array.

[0039] Specifically, the programmable logic array (PLA) reserves a contiguous physical address space in the DDR SDRAM as a circular buffer through the memory controller to ensure the efficiency of burst access. The write control logic is configured for burst transmission mode on the AXI4 bus, packaging parallel digital signals into 256-byte burst transactions and writing them to the circular buffer in the physical address space of the DDR SDRAM. The write pointer maintains an address counter, which is incremented after each data packet write. When the counter value reaches the end address of the circular buffer, the logic circuit resets it to the start address, thereby using the write pointer to perform a loop-over addressing that connects the beginning and end, constructing a sliding time window for temporary data storage. In addition, the PLA logic captures the rising edge of the field synchronization signal of the video stream in real time. Once a frame end marker is detected, the write pointer address is forcibly aligned to the preset start offset address of the next frame, ensuring that each frame of data follows a fixed boundary alignment rule within the circular buffer. The read logic maintains a linked list structure with 5 nodes, each node recording the starting physical address, data length, and hardware timestamp of a single frame image. The system reads pointers, uses hardware timestamps as indexes, compares the current system clock with the hardware timestamps in the frame header metadata, calculates the time difference, searches the linked list, locks the most matching frame node, and then continuously reads 5 video frames from the sliding time window. Finally, it integrates them and outputs the original time-series video frame sequence.

[0040] In a preferred embodiment, the ring buffer employs a heterogeneous hierarchical storage architecture consisting of a DDR SDRAM primary cache and an NVMe solid-state storage array secondary persistent storage area. The programmable logic array integrates a PCIe-based NVMe controller IP core, hierarchical scheduling logic, and a direct memory access controller. The hierarchical scheduling logic is configured to monitor the data level of the primary cache in real time, and when the level exceeds a preset threshold, trigger the DMA controller to initiate a background data migration task. This task batch-transfers the earliest timestamped data blocks in the primary cache to the secondary storage area in raw data block mode and updates the global logical address mapping table. The read pointer is configured to prioritize querying the primary cache, and if a cache miss occurs, retrieve data from the secondary storage area based on the global logical address mapping table. This embodiment utilizes heterogeneous hierarchical storage to significantly extend the effective time window while ensuring nanosecond-level write response, supporting long-term historical backtracking frame insertion, and effectively preventing the loss of critical historical data due to power outages.

[0041] This invention integrates multi-standard interfaces with an FPGA circular buffer to achieve unified access and parallel processing of signals from different physical protocols such as SDI, HDMI, and fiber optics. By utilizing hardware timestamp indexing and a loop-overlay addressing mechanism that connects the beginning and end of the loop, it ensures high-speed throughput and zero-frame-dropping acquisition of massive high-bandwidth video data in limited physical memory, providing an absolutely accurate time-series data source for subsequent algorithms.

[0042] Further, the method for receiving the original temporal video frame sequence and performing parallel signal processing includes: splitting the original temporal video frame sequence into two synchronously running processing paths; after the original temporal video frame sequence enters the first path, performing a dehazing operation and using an image restoration algorithm to restore the scene radiance; based on the restored scene radiance, performing a bilateral filtering denoising operation on the original temporal video frame sequence, performing a sliding window convolution on each video frame to extract video detail component signals, wherein the video detail component signals include surface high-frequency texture components, motion features, and contour edge information of moving targets in the video image; after the original temporal video frame sequence synchronously enters the second path, using horizontal and vertical convolution templates, performing a weighted difference operation on the pixel grayscale of each video frame to obtain the horizontal gradient component and the vertical gradient component and fuse them into a gradient vector magnitude as the edge gradient feature; and generating a geometric structure reference signal after binarizing the edge gradient feature.

[0043] Specifically, the method for receiving the original temporal video frame sequence and performing parallel signal processing includes the following steps: The system uses data exchange logic to split the original temporal video frame sequence into two synchronously running processing paths to achieve parallel feature extraction. After the original temporal video frame sequence enters the first path, a dehazing operation is first performed, using an image restoration algorithm to restore the scene's radiance. This process is specifically manifested as follows: The dehazing module extracts the minimum values ​​of the red, green, and blue channels by traversing the pixels within a local window to construct a dark channel map. The system selects the average value of the original pixels corresponding to the top 0.1% of the pixels in the dark channel map as the global atmospheric light value, and estimates and refines the transmittance map using a guided filtering algorithm. Subsequently, an inverse operation is performed based on the physical atmospheric scattering model, subtracting the atmospheric light value from the original pixel value and dividing by the transmittance value, thereby eliminating signal attenuation caused by smoke.

[0044] Specifically, based on the restored scene radiance, the system performs bilateral filtering denoising on the original temporal video frame sequence, and performs sliding window convolution on each video frame to extract video detail component signals containing surface high-frequency texture components of moving targets in the video image, microscopic features reflecting high-speed motion details, and contour edge information of moving targets. During sliding window convolution, the system traverses image pixels with a preset sliding stride, using the currently traversed target pixel as the geometric center, and delineates a corresponding rectangular neighboring pixel set in a two-dimensional spatial coordinate system according to preset window size parameters, thereby establishing a local support domain window. Within the local support domain window, the system calculates the spatial geometric distance and grayscale statistical difference between neighboring pixels and the center pixel in real time, and maps both to Gaussian distribution curves to obtain corresponding spatial proximity weights and pixel similarity weights. By performing pixel-by-pixel multiplication and normalization on the two types of weights, an adaptive convolution kernel with edge-preserving properties is constructed.

[0045] Specifically, the system uses an adaptive convolution kernel to perform a weighted summation of pixel values ​​within a window, thereby obtaining a smooth image component after filtering out shot noise. Finally, the system subtracts the smoothed image component tensor from the original pixel tensor before processing to obtain a non-stationary detail residual tensor, and further decouples multi-dimensional feature information based on this non-stationary detail residual tensor. The Laplacian operator performs high-pass filtering on the residual tensor to obtain high-frequency texture components reflecting the surface roughness and micro-particle distribution of the object. The absolute value of the difference between the residual tensors of two adjacent frames is calculated, and combined with spatial neighborhood expansion processing, motion features reflecting the temporal afterimage range and motion activity of high-speed moving targets are obtained. Specifically, the spatial neighborhood expansion processing uses a search window of a preset size sliding across the absolute difference tensor and the Canny edge detection operator to locate linearly distributed pixel response points in the residual tensor, resulting in binarized contour edge information reflecting the geometric boundaries of the object. Subsequently, the high-frequency texture components, motion features, and contour edge information of the moving target are each constructed into a single-channel feature map, and dimensional stacking is performed along the channel axis to fuse and generate the video detail component signal.

[0046] Specifically, after the original temporal video frame sequence is synchronously entered into the second path, the system uses horizontal and vertical convolution templates to perform weighted difference operations on the grayscale of each video frame pixel, obtaining the horizontal and vertical gradient components and fusing them into a gradient vector magnitude, which serves as the edge gradient feature. In practical implementation, the system converts the color video frames into single-channel grayscale images using a weighted averaging formula, and employs... The system performs convolution on all pixels of the image using a Sobel operator of varying scale. It calculates the gradient values ​​in the horizontal and vertical directions separately, and then takes the square root of the sum of the squares of these two values ​​to obtain the gradient vector magnitude representing the edge intensity. Finally, the system binarizes the edge gradient features to generate a geometric reference signal. This process uses Otsu's method to traverse the grayscale histogram, determining the optimal segmentation threshold by calculating the maximum inter-class variance between the foreground and background. Pixels with gradient magnitude values ​​greater than this threshold are set to logic 1, and the rest are set to logic 0, thus generating a signal containing only the contour topology information of the moving target.

[0047] This invention employs a dual-path processing strategy that combines restoration followed by denoising and gradient feature extraction to effectively address smoke obstruction and low-light noise interference. The first path actively restores the scene's radiance and filters out shot noise, improving the signal-to-noise ratio; the second path extracts clean geometric structures through weighted differential extraction. The two paths work together to eliminate the impact of harsh working conditions on feature extraction accuracy.

[0048] Furthermore, the generative video synthesis unit employs a conditional generative adversarial network. Its overall architecture comprises a deep convolutional neural network consisting of a dual-stream encoder and a cascaded decoder. The hierarchical structure includes: a structural feature encoding branch, composed of multiple downsampling convolutional layers, used to extract spatial topological features of video frames; a texture motion encoding branch, set in parallel with the structural feature encoding branch, internally integrating a feature inverse warping layer, used to extract texture features of video frames and perform feature alignment; and a spatial feature modulation decoding layer, located in the network's decoding stage, serving as the data aggregation node for the structural feature encoding branch and the texture motion encoding branch. Internally, it includes multiple cascaded upsampling deconvolutional layers. These upsampling deconvolutional layers receive the aggregated feature data, utilize structural features to perform pixel-by-pixel normalization modulation of texture features, and perform resolution restoration operations layer by layer.

[0049] Specifically, the conditional generative adversarial network (GAN) model used in the generative video synthesis unit employs a hybrid dataset co-training strategy. To ensure the stability of the GAN model under harsh conditions, the training process involves updating the overall weights of the two-stream encoder (composed of a structural feature coding branch and a texture motion coding branch) and the cascaded decoder (composed of a spatial feature modulation decoding layer). The base dataset uses Vimeo-90K video sequences, and online degradation enhancement is performed on the original frames using an atmospheric scattering model and a Gaussian noise model to generate simulated smoke (concentration range...). Synthetic samples of signals and random thermal noise enable the conditional generative adversarial network (GAN) model to extract features from degraded signals. During training, the optimizer is Adam, and the initial learning rate is set to... The cosine annealing algorithm is used to gradually decay the learning rate over 500 epochs. The system employs a weighted total loss function, balancing the signal-to-noise ratio and texture detail of the output image from the generative adversarial network model by adjusting the weight ratios of reconstruction loss, perceptual loss, and adversarial loss. The perceptual loss extracts features from a pre-trained VGG-19 network, selecting the feature map output from the fourth convolutional layer of the fourth convolutional group to calculate the Euclidean distance between the generated intermediate frame and the real reference frame in the feature space. This loss calculation allows the model to ignore pixel-level noise shifts and instead learn the geometric contours and texture consistency of high-speed moving targets. In the initial training phase, the system first performs 50 epochs of pre-training on the texture motion encoding branch to obtain stable optical flow guidance capabilities, followed by end-to-end joint fine-tuning of the entire network parameters.

[0050] Specifically, the structural feature encoding branch in the dual-stream encoder is constructed as a five-level cascaded convolutional neural network architecture. Each encoding module contains a downsampling convolutional layer with a stride of 2, and the kernel size is set to... The data stream processing sequentially passes through a spectral normalization convolutional layer, an instance normalization layer, and a ReLU activation function (with a negative slope set to 0.2). The spectral normalization convolutional layer uses a power-law iteration algorithm (number of iterations...). Dynamically maintain the weight matrix as the learning parameters of the convolutional layer. spectral norm And update the weights to Stable GAN training. The spectral norm mentioned above. Weight matrix The maximum singular value is used to characterize the upper bound of the induced norm of the matrix operator; its specific derivation process is as follows: the system first selects a randomly initialized vector and repeatedly combines it with the convolution kernel weight matrix. and its transpose matrix Perform matrix multiplication iterative operations until the vector values ​​converge to approximate the weight matrix. The maximum singular value as the spectral norm After five downsampling iterations, the feature map resolution is reduced to a fraction of the original image resolution. To achieve cross-resolution feature alignment, the system deploys a parameter generation head after each level of downsampling, utilizing... Convolutional layers map the spatial features of the current layer to scaling factors corresponding to the resolution. With bias coefficient For spatial feature modulation decoding layers with different resolutions (such as...) , The system upsamples the lowest-level coefficient tensor using a bilinear interpolation algorithm and then weights and fuses it with the local features output by the corresponding level parameter generator to generate modulation parameters that are completely consistent with the spatial dimensions of each layer of the decoder.

[0051] Specifically, the texture motion coding branch integrates a feature inverse twisting layer at its end. The feature inverse twisting layer invokes a simplified optical flow estimation sub-network based on a pyramid optical flow network architecture. This optical flow estimation sub-network employs a streamlined pyramid optical flow network architecture, comprising a five-layer feature pyramid structure. From the first to the fifth pyramid layer, the convolutional kernel size is uniformly set to [size missing]. The number of output channels was set to 32, 64, 96, 128, and 196 respectively, with a stride of 2 for downsampling in each convolutional layer. At each level, a local cost convolution was constructed using relevant layers, with a fixed search radius of 4 pixels. Subsequently, an optical flow predictor consisting of six tightly connected convolutional layers was used to decode the local cost convolution, regressing the optical flow residual for that level. A coarse-to-fine cascading strategy was employed, using the upsampled optical flow from the previous level as the initial estimate and superimposing it onto the current layer to generate the final optical flow vector for the current level. Each layer used a leaky linear rectified function as the activation function. This leaky linear rectified function maintained its original output value when the feature output value of the previous convolutional layer was greater than zero; when the feature output value was less than or equal to zero, it multiplied the feature output value by a leakage coefficient for linear compression. The leakage coefficient was set to the reciprocal of the product of the convolutional kernel size and the search radius to prevent neuron inactivation during optical flow estimation when processing low-contrast or dark features. The feature inverse distortion layer performs spatial alignment on the feature map of the previous frame after acquiring the optical flow vector. To further eliminate ghosting under large-scale motion, a spatiotemporal self-attention module is embedded in the bottleneck layer at the end of the encoder. This module adopts a multi-head attention mechanism with 8 heads, and the embedding dimension is consistent with the number of feature channels in this layer. During the calculation, the current frame feature is used as the query vector, and the features of adjacent frames and their corresponding spatial position codes are used as the key vector and numerical vector, respectively. Spatiotemporal correlation weights are calculated through matrix multiplication to achieve feature compensation for pixels in occluded areas.

[0052] Specifically, the spatial feature modulation decoding layer in the cascaded decoder performs specific parameterized modulation operations. First, the system performs channel-level normalization on the input texture feature tensor, making its mean zero and standard deviation one. Then, the system uses the scaling and bias coefficients to perform a pixel-wise affine transformation on the normalized texture features: that is, multiplying the normalized pixel value by the corresponding scaling coefficient and adding the corresponding bias coefficient. This step ensures that texture generation is strictly controlled within the geometric boundaries defined by the scaling and bias coefficients. To facilitate generator training, the system simultaneously constructs a multi-scale discriminator, which consists of three sub-networks with identical structures but progressively halved input resolutions. Each sub-network achieves downsampling through average pooling. Each sub-network employs a 4-layer fully convolutional architecture with a kernel size of [missing information]. With a step size of 2, its receptive field covers the original image respectively. , and The system employs a weighted total loss function during training. This function is composed of pixel reconstruction loss, texture-aware loss, and adversarial generation loss, weighted by preset hyperparameter weights. The pixel reconstruction loss is obtained by calculating the sum of the absolute differences in pixel values ​​between the generated intermediate video frames and the real reference intermediate frames, used to constrain the accuracy of the image in low-frequency structures. To address the image blurring caused by simple pixel comparison, the system utilizes the VGG-19 model to extract high-level feature maps and constructs a texture-aware loss function. The system calculates the cumulative sum of points between different channels of the feature map, generating a Gram matrix representing the statistical distribution of texture. To eliminate the amplification effect of spatial resolution and channel depth of the feature map on the numerical magnitude and prevent excessively large gradient values ​​or texture weights during model training, the following normalization process is performed: the sum of the squares of the differences between the generated frame and the reference frame's Gram matrix is ​​divided by the square of the product of the number of channels, height, and width of the feature map at that layer. After this normalization process, the numerical magnitude of the texture loss is constrained to a range similar to that of the pixel reconstruction loss, enabling the model to learn the main structure restoration and fine texture generation in a balanced manner with smaller weight coefficients.

[0053] In a preferred embodiment, the generative video synthesis unit embeds a spatiotemporal self-attention module in series between the outputs of the structural feature encoding branch and the texture motion encoding branch and the input of the spatial feature modulation and decoding layer. The spatiotemporal self-attention module receives feature tensors from the encoding branches and expands them into sequence vectors in both the temporal and spatial dimensions. Internally, the module calculates the dot product similarity matrix between the query vector, key vector, and value vector, generates a global attention map using the Softmax function, and performs a weighted summation on the input feature sequence using the global attention map to capture long-distance spatiotemporal dependencies in the video frame sequence. The attention-weighted feature tensor is then transmitted to the spatial feature modulation and decoding layer for subsequent modulation operations. This embodiment overcomes the limitations of the local receptive field of convolution operations, enabling the model to capture long-distance feature dependencies spanning multiple time steps, effectively suppressing texture flickering and temporal discontinuities that easily occur in the generation of long slow-motion sequences.

[0054] This invention employs a dual-stream network architecture that includes a spatial feature modulation layer, using parameters generated by the structural feature encoding branch to perform pixel-by-pixel normalized modulation of texture features. This deep learning topology, which "constrains texture with structure," ensures from the network's underlying logic that the generated texture is strictly limited to the range defined by the object's skeleton, fundamentally eliminating the edge overflow and structural collapse phenomena commonly found in traditional frame interpolation techniques.

[0055] Furthermore, the specific method for performing nonlinear motion compensation and intermediate frame signal reconstruction includes: the structural feature encoding branch performs multi-scale feature extraction on the received geometric structure reference signal, generates affine transformation parameters containing spatial semantic layout information, and constructs boundary locking conditions that restrict the spatial degrees of freedom of the synthesis process; the texture motion encoding branch receives the video detail component signal, drives the feature inverse distortion layer to perform nonlinear motion compensation based on the boundary locking conditions, remaps the texture feature pixels of the video frame to the predicted position of the current intermediate frame, and generates a motion-calibrated texture feature tensor; the spatial feature modulation decoding layer performs parameterized modulation on the texture feature tensor to recover the video texture details, and reconstructs the intermediate video frame after multi-level cascaded upsampling; the intermediate video frame is embedded into the original temporal video frame sequence in timestamp order to form an interpolated video stream to be verified.

[0056] Specifically, the structural feature encoding branch performs multi-scale feature extraction through five cascaded convolutional stages. In each encoding stage, the spectral normalization convolutional layer dynamically maintains the spectral norm of the weight matrix using a power-law iteration algorithm. First, a randomly initialized vector is selected, and iterative matrix multiplication operations are repeatedly performed with the weight matrix and its transpose until the vector values ​​converge to approximate the maximum singular value of the weight matrix as the spectral norm. Then, each element in the weight matrix is ​​divided by the spectral norm to generate a numerically constrained normalized weight matrix for convolution. Next, the instance normalization layer, for each independent feature channel within a single feature map, first calculates the pixel mean of all spatially located pixels within that channel and then calculates the average of the sum of squares of the differences between each pixel and the mean as the pixel variance. Then, each pixel value is subtracted from the pixel mean and divided by the arithmetic square root of the sum of the pixel variance and the minimum constant to complete the standardization process. Finally, it is multiplied by a learnable scaling parameter and an offset parameter is added before outputting the result. Finally, the encoded features are mapped to spatially aligned scaling and bias coefficients, i.e., the affine transformation parameters, using a mapping convolutional layer. This creates a boundary-locking condition within the network that restricts the spatial degrees of freedom of the synthesis process, forcing the spatial distribution of the generated texture to strictly follow the boundary defined by the geometric structure signal.

[0057] Specifically, the texture motion coding branch drives the feature inverse twisting layer to perform nonlinear motion compensation. The system first calls the optical flow subnetwork to calculate the pixel displacement vector field between two adjacent frames of the original video, thereby determining the predicted position of each texture feature pixel in the intermediate frame. For each non-integer coordinate point in the predicted position, the system performs the following remapping sampling calculation: locate the four adjacent integer pixels surrounding the non-integer coordinate point in the original feature map, and calculate the four corresponding weighting coefficients based on the distance differences between the non-integer coordinate and these four integer points in the horizontal and vertical directions; then multiply the feature values ​​of these four integer pixels with their corresponding weighting coefficients and sum them to obtain the feature vector at the non-integer coordinate, thereby generating a spatially continuous motion-calibrated texture feature tensor.

[0058] Specifically, the spatial feature modulation decoding layer performs parametric modulation on the texture feature tensor to recover video texture details. The calculation process is as follows: First, synchronous batch normalization is performed across computation nodes on the texture features, calculating the global mean and global variance of the current batch of data and standardizing the feature values. Then, each standardized pixel value is multiplied by the scaling factor at the corresponding position in the aforementioned affine transformation parameters, and the product is added to the bias factor at the corresponding position, realizing pixel-level guidance of texture generation based on structural information as a boundary locking condition. The modulated features are then processed through a multi-level cascaded upsampling architecture containing deconvolution operators, performing resolution recovery operations layer by layer to reconstruct intermediate video frames, which are then embedded into the original temporal video frame sequence in timestamp order to form the interpolated video stream to be verified.

[0059] This invention utilizes feature inverse distortion technology to perform nonlinear motion compensation, combined with a multi-scale feature pyramid, to effectively solve the texture alignment problem under high-speed, large-displacement motion. At the same time, the inverse mapping mechanism avoids pixel holes that may be generated by forward mapping, ensuring that the texture of the synthesized intermediate frames is dense and spatially continuous, significantly improving the visual smoothness of the image.

[0060] Further, the output of the reconstructed video data stream after spatiotemporal consistency verification includes: extracting intermediate video frames from the interpolated video stream, calculating the luminance histogram difference value and structural similarity index between the intermediate video frame and adjacent video frames (the structural similarity index is... Figure 2The SSIM (Signal-to-Noise Ratio) of the generated video frame is determined, and the structural integrity is assessed. Spatiotemporal consistency is checked. If the difference value corresponding to the SNR is lower than a preset safety threshold and the structural similarity index corresponding to the structural integrity is higher than a preset standard, the check is deemed successful, and the interpolated video stream is marked and output as a reconstructed video data stream. If the difference value corresponding to the SNR is not lower than the preset safety threshold, pixel backfilling based on the nearest neighbor in the temporal domain is triggered. Based on the position of the current intermediate frame on the time axis, the video frame closest to the intermediate frame is locked as the target reference frame. The pixel data of the spatially corresponding position in the target reference frame is extracted, directly migrated and covered to the abnormal area of ​​the intermediate video frame, and the modified reconstructed video data stream is output.

[0061] Specifically, the system extracts the intermediate video frames from the interpolated video stream. By calculating the luminance histogram difference between the intermediate video frames and adjacent video frames, and using a structural similarity index, the system measures the signal-to-noise ratio and structural integrity of the generated video frames. The detailed calculation process for the structural similarity index is as follows: The system utilizes... A Gaussian sliding window with a size and standard deviation of 1.5 is used to generate video frames along with adjacent video frames. Within each local window, a luminance contrast component reflecting the difference in mean, a contrast contrast component reflecting the difference in standard deviation, and a structural contrast component reflecting the pixel covariance are calculated. The final index value is the weighted average of the product of these three components over the entire image. The luminance histogram difference value is calculated by constructing a normalized histogram vector by statistically analyzing the pixel frequency distribution from gray levels 0 to 255 within the two frames, and the Bach distance between the two vectors is calculated as the structural similarity index.

[0062] Specifically, the system performs a spatiotemporal consistency check. If the difference value of the brightness histogram corresponding to the signal-to-noise ratio is lower than the preset safety threshold (the safety threshold is 0.05 in this embodiment), and the structural similarity index corresponding to the structural integrity is higher than the preset standard (the standard is 0.85 in this embodiment), the check is deemed successful, and the system marks the interpolated video stream and outputs it as a reconstructed video data stream. The specific method for determining the preset safety threshold and preset standard is as follows: Based on the Bach distance statistical quantization and comparison analysis of the brightness histograms between adjacent frames in the original video sequence, it is determined that when the difference value is controlled within 0.05, the deviation between the intermediate frame and the original scene in the global brightness distribution can effectively suppress the screen flicker caused by signal degradation, and the preset safety threshold is set to 0.05; Based on the performance test and index optimization of the structural similarity index for the video dataset during the model verification stage, 0.85 is determined as the lower limit benchmark to ensure that the generated image has rigorous geometric topology and no edge structural distortion, which can ensure that the generated frame strictly follows the edge contour of the original moving target and prevent the model from producing misleading illusions, and the preset standard is set to 0.85. If the difference in signal-to-noise ratio is not lower than a preset safety threshold, i.e., the structural similarity index is lower than 0.85, a pixel backfilling process based on the nearest neighbor in the temporal domain is triggered. First, based on the position of the current intermediate frame on the time axis, the time difference between it and its preceding and following video frames is calculated, and the frame with the smallest time difference is locked as the source frame. Subsequently, the system performs a linear weighted fusion operation based on a soft mask: a smooth transition soft mask is generated using the calculated local structural similarity index. When the local structural similarity index is lower than a preset standard, the weight value of the mask decreases linearly as the index decreases. For each pixel in the image, the output pixel value is equal to the pixel value output by the generation network multiplied by the mask weight, plus the complement of the pixel value at the corresponding position in the source frame multiplied by the mask weight (i.e., 1 minus the mask weight). During backfilling, a motion-guided resampling mechanism is adopted. Specifically, the system uses the coordinates to be filled in the current intermediate frame as a reference and searches for the corresponding physical pixel coordinates in the nearest neighbor frames along the opposite direction of the motion vector. If the retrieved coordinates are at a non-integer position, the hardware texture memory is invoked to perform a four-point bilinear interpolation operation to obtain accurate sub-pixel level color components. In this way, it is ensured that the backfilled pixels are strictly aligned with the current moving target on the physical trajectory, eliminating double ghosting and edge tearing caused by rapid displacement of the object's center of gravity during weighted blending.

[0063] Specifically, based on the current intermediate frame's position on the timeline, the system calculates the absolute value of the difference between the timestamp of the current intermediate frame and the timestamps of its immediate and adjacent video frames, and locks the video frame closest to the intermediate frame as the target reference frame. The locking operation selects the frame with the smaller difference as the target reference frame. Using motion vectors generated by the optical flow estimation sub-network, pixels in the target reference frame are mapped to their predicted positions at the intermediate frame's time, resulting in motion-compensated reference pixels. Subsequently, linear interpolation hybrid logic is executed to directly migrate and overlay the pixel data into the abnormal regions of the intermediate video frame. For each pixel within the image, a pixel value is output. ,in For the motion-compensated reference pixel, To generate frame pixels, The soft mask weights are calculated as follows: First, the local structural similarity index of pixel blocks within the abnormal region of the current intermediate frame is obtained, and the difference between the structural similarity index and a preset lower bound mapping threshold is calculated. Next, the interval difference between the preset upper bound mapping threshold and the lower bound mapping threshold is calculated. Then, the aforementioned index difference is divided by the interval difference to obtain an initial quotient. Finally, the initial quotient is constrained to a value between zero and one as the final soft mask weight. The lower bound mapping threshold is preset to 0.75, and the upper bound mapping threshold is preset to 0.85. The upper bound mapping threshold is numerically consistent with the aforementioned preset standard used to determine if the verification passes. The basis for setting these two thresholds is: through prior testing of the edge reconstruction stability of high-speed maneuvering targets, it was determined that when the structural similarity index is between 0.85 and 0.75, the generated pixels begin to show slightly visible distortion, requiring linear intervention by introducing pixels from the target reference frame; while when the index is below 0.75, the structural topology of the generated pixels has severely failed, requiring complete switching to physical pixels from the target reference frame. In the computational logic, if the initial quotient is less than 0, the soft mask weight is forcibly assigned to 0; if the initial quotient is greater than 1, the soft mask weight is forcibly assigned to 1. After smoothing the abnormal region of the intermediate frame through the above calculation, the timestamp of the intermediate frame is kept unchanged, and it is re-encapsulated into the video sequence, thus forming the final reconstructed video data stream.

[0064] This invention establishes a spatiotemporal consistency verification mechanism based on structural similarity and brightness histogram differences, and designs pixel backfilling as a fallback strategy for engineering. When abnormalities or "illusions" are detected in AI-generated indicators, the system can automatically switch to physical pixel coverage based on the nearest neighbor in the temporal domain, ensuring the authenticity and credibility of the output image in extreme scenarios and meeting the rigorous requirements of task review.

[0065] Furthermore, the method for performing time base remapping processing in response to playback control commands input via a human-computer interaction terminal includes: constructing a virtual display time base, stripping the original acquisition timestamps from the reconstructed video data stream, and reallocating monotonically increasing virtual display timestamps for each video frame based on the slow motion ratio set by the playback control command, with the first frame as the reference point; establishing a rate adaptation buffer pool, writing the generated reconstructed video data stream into the rate adaptation buffer pool, controlling the reading rate based on the virtual display timestamps, and integrating the video frame data with the virtual display timestamps before inputting them into a video compression encoder to generate slow motion replay signal encoding.

[0066] Specifically, the system responds to playback control commands input via the human-machine interface terminal, reads the slow-motion ratio parameter set by the command by accessing registers, and thus initiates time base remapping processing. The system first constructs a virtual display time base. This process involves stripping the original acquisition timestamp from the header of the received reconstructed video data packet, thereby eliminating the constraints of physical acquisition time. To calculate the new timeline, the system first calculates the output frame interval time between each frame based on the slow-motion ratio set by the command and the system's preset baseline display frame rate. The calculation method is to multiply the reciprocal of the baseline display frame rate by the slow-motion ratio value.

[0067] Specifically, the system selects the first frame in the processing sequence and initializes the time zero point using the first frame as the reference point. Based on this, the system reallocates a monotonically increasing virtual display timestamp for each video frame. The specific calculation logic is as follows: the system adds the cumulative time increment before the current video frame to the calculated output frame interval, ensuring that the allocated timestamp is strictly monotonically increasing on the time axis. This reallocated virtual display timestamp is rewritten into the data packet header as the timing reference for subsequent screen refreshes by the display device.

[0068] Specifically, the system establishes a rate adaptation buffer pool, which, at the physical storage level, consists of a first-in, first-out queue with a fixed address space within DDR SDRAM. The write controller is configured to write the generated reconstructed video data stream into the rate adaptation buffer pool at the full-speed clock frequency after frame interpolation algorithm processing. To achieve a uniform slow-motion effect, the system controls the read rate based on the virtual display timestamp. This process relies on a hardware timer integrated within the read controller, operating at the system's main clock frequency. The hardware timer continuously increments its value based on the main clock pulses. Whenever the accumulated value of the hardware timer reaches the number of clock cycles corresponding to the difference between the current frame's virtual display timestamp and the previous frame's virtual display timestamp, the system triggers a read operation, retrieving one frame of data from the rate adaptation buffer pool queue. The number of clock cycles is calculated by multiplying the time difference between the two virtual display timestamps by the operating frequency of the hardware timer. The system integrates the extracted video frame data with the virtual display timestamp into a complete bitstream packet. The integrated data packet is then input into the video compression encoder, which uses the encoder's hardware circuitry to perform high-efficiency video encoding operations, ultimately generating a slow-motion replay signal encoding for network transmission and playback on the display terminal.

[0069] This invention constructs a virtual display time base decoupled from physical acquisition time, and, in conjunction with a rate-adaptive buffer, achieves timeline reconstruction at any slow-motion magnification. This mechanism dynamically matches the generation rate with the playback rate, eliminating the stuttering effect caused by traditional slow-motion methods that simply repeat or drop frames, thus achieving smooth, uniform, stepless speed-adjustable playback.

[0070] Furthermore, the distributed display and control device integrates a signal distribution unit and is connected to a display terminal array, comprising: the signal distribution unit receiving slow-motion replay signal encoding, encapsulating the slow-motion replay signal encoding into a data stream adapted to the transmission format of the current physical communication link, wherein the physical communication link includes an Ethernet network and an optical fiber transmission link; distributing the data stream to each node of the display terminal array using the physical communication link; each node of the display terminal array independently receiving the data stream, performing local decoding on the bitstream in the data stream, and synchronously controlling the screen refresh timing according to the virtual display timestamp to maintain the timing synchronization of the slow-motion replay images.

[0071] Specifically, the distributed display and control device integrates a signal distribution unit and is connected to the display terminal array. The signal distribution unit receives slow-motion replay signal encoding and encapsulates the encoded network abstraction layer (NAT) data packets into an RTP / UDP / IP network transport protocol stack, thus encapsulating the slow-motion replay signal encoding into a data stream adapted to the current physical communication link transmission format. The signal distribution unit configures the target IP address as a multicast group address, thereby distributing the data stream to each node of the display terminal array using the physical communication link, which includes an Ethernet network and a fiber optic transmission link.

[0072] Specifically, each node in the display terminal array independently receives data streams. To maintain timing synchronization of slow-motion playback, all nodes run a precise time protocol. Each node obtains four precise timestamps—the master clock's transmission time, the slave clock's reception time, the slave clock's transmission of the delay request, and the master clock's reception of the delay request—by exchanging synchronization messages, follow messages, delay request messages, and delay response messages. The system calculates the master-slave clock deviation using the average of the master-slave time differences and calculates the network transmission path delay by subtracting the internal processing time of each node from the total round-trip path time. Based on these calculations, each node adjusts the frequency and phase of its local clock to ensure that the local clocks of all terminal nodes are synchronized with the network master clock at the microsecond level.

[0073] Specifically, the decoder inside the node performs local decoding on the bitstream in the data stream, storing the restored pixel data into a dual-buffered display memory. This dual-buffered structure includes a back buffer and a front buffer, where the back buffer is used to receive new decoded frames, and the front buffer is used for the current display scan. The display controller continuously polls the currently calibrated system time, and sends a vertical synchronization signal to the display driver circuit only when the system time is greater than or equal to the virtual display timestamp of the current frame. The vertical synchronization signal triggers the hardware to perform a front and back buffer swap operation, mapping the image in the back buffer to the front buffer for scanning output.

[0074] In a preferred embodiment, the distributed display control device includes a master control node and multiple display sub-nodes cascaded via Ethernet. The master control node is configured to distribute a global system clock to all display sub-nodes based on the IEEE 1588 Precision Time Protocol. Each display sub-node integrates a vertical synchronization controller, which is configured to calculate the phase deviation between the rising edge of the vertical synchronization signal of the local physical screen and the target display time, and dynamically adjust the duration parameter of the vertical blanking interval of the current frame using the phase deviation to maintain phase lock between the refresh cycle of the local physical screen and the virtual display time base. This embodiment utilizes soft phase-locked loop technology, which eliminates the need for additional physical synchronization cables, to solve the microsecond-level asynchrony problem caused by crystal oscillator drift in distributed splicing screen arrays. This ensures that all screen nodes maintain frame-level synchronization even during extremely slow playback, eliminating dynamic tearing of cross-screen images.

[0075] This invention utilizes a precise time protocol and a frame buffer exchange triggering mechanism based on a global clock to solve the impact of network transmission latency jitter on distributed display terminals. This design ensures that in large-scale video wall arrays, all screen nodes can ignore physical link differences and accurately refresh the same frame at the same absolute moment, achieving a tear-free synchronous display effect.

[0076] Example 2:

[0077] This embodiment demonstrates the hardware architecture and computational acceleration method employed by the present invention to address the computational demands of generative adversarial networks in scenarios involving high-speed moving target image acquisition and real-time slow-motion replay. (Refer to...) Figure 3 The specific steps are as follows:

[0078] The system employs a heterogeneous computing architecture combining FPGA preprocessing and GPU deep acceleration. The signal processing and frame rate conversion modules are physically deployed within an edge computing server, internally integrating an FPGA accelerator card interconnected via a PCIe bus and a high-performance graphics processing unit. The FPGA accelerator card handles pixel-level preprocessing tasks such as video signal acquisition, ring buffer management, and waveform correction; the graphics processing unit handles deep learning inference tasks for the generative video synthesis unit. The system utilizes the PCIe bus's base address register mapping mechanism to directly map the graphics processing unit's memory space to the FPGA's direct memory access addressing space. The FPGA accelerator card obtains the GPU's base address register parameters by enumerating the PCIe configuration space and establishes a mapping relationship between virtual addresses and physical memory addresses in the FPGA's internal address translation table. The DMA controller is configured in a scatter-cluster mode, using a descriptor linked list containing contiguous physical page addresses of the GPU memory to directly push the waveform-corrected pixel data to the target memory buffer in TLP packets. After the FPGA completes waveform correction and geometric reference signal generation for a frame, it bursts the video detail component signal and geometric reference signal into the input tensor buffer in the video memory via a direct memory access controller, without going through the host CPU's memory. This point-to-point transmission mechanism reduces data transfer latency to the microsecond level, reserving a time window for network inference.

[0079] For generative adversarial networks (GANs), the system employs an operator fusion and half-precision quantization acceleration strategy highly coupled with the network topology at the software level. During the compilation phase, the inference engine deeply scans the structural feature encoding branches. For each downsampling module in the feature pyramid, it identifies a fixed computational paradigm of "spectral normalized convolution - instance normalization - activation function" and compiles it into a single fusion operator kernel. This kernel loads data once in the L1 cache of the graphics processing unit and continuously performs convolution multiplication and addition, normalization statistics calculation, and nonlinear activation, avoiding bandwidth loss caused by repeatedly writing intermediate feature maps to GPU memory. In the FPGA preprocessing stage, the system utilizes internal block memory to construct a row cache queue with depth matching the image width. Scrolling window access of image rows is achieved through circular mapping of read and write pointers. The convolution kernel weights are solidified as constant multipliers and directly mapped to the cascaded links of digital signal processing slices, thus achieving single-clock-cycle pipelined output without waiting for the full frame buffer. During the GPU deep learning inference stage, for feature extraction and inverse warp operators, the system executes customized thread mapping, directly and linearly mapping the two-dimensional spatial coordinates of the image to the two-dimensional indices of the CUDA computing grid and thread blocks. This ensures that adjacent threads process adjacent pixels to meet the requirements of memory merging access and maximizes memory bandwidth utilization. In addition, for the frequent non-integer coordinate sampling operations in the frame interpolation algorithm, the system establishes a dedicated texture mapping mechanism. Through the driver interface, the memory pointer storing the optical flow vector is bound to a read-only texture object. The dedicated texture picking unit inside the GPU directly performs hardware-based bilinear interpolation calculations, thereby compressing the original multi-step memory read and software interpolation logic into a single hardware sampling instruction.

[0080] Modulation decoding layer for spatial features (i.e. Figure 3To address the dense pixel-wise affine transformation operations in the SPADE layer, the system implements an inter-layer operator fusion strategy, applying two independent convolutional layers that generate scaling and bias coefficients to the normalized texture feature map. In the GPU computation kernel implementation, the system deploys shared memory to slice and tile the texture feature data, ensuring that data resides near the L1 cache during scaling and bias calculations for each thread, eliminating intermediate tensor read / write operations to GPU memory. During feature inverse warping, the system creates a texture object through the CUDA driver interface, binds the texture feature tensor in GPU memory to the hardware texture memory address, and utilizes the GPU's internal texture pickup unit to perform hard-wired interpolation calculations, thus completing sub-pixel position sampling within a single instruction cycle. During computation, after the structural features are read into the register file, the modulation parameters are immediately calculated and directly applied to the texture features, eliminating the temporary storage overhead of scaling and bias coefficient tensors in global GPU memory. For the frequent bilinear interpolation sampling in the feature inverse warping layer, the system calls a custom computation kernel function. This function utilizes the graphics processor's driver interface to bind the texture feature tensor to be distorted as a read-only texture object, configuring the addressing mode to boundary fill mode and the filtering mode to linear filtering mode. During kernel function execution, the system directly calls the hardware-built-in texture picking instruction. This instruction bypasses the general-purpose computing core and directly utilizes the dedicated hard-wired interpolation circuitry within the graphics processor's texture unit to complete four-point bilinear interpolation calculations at floating-point coordinates within a single clock cycle.

[0081] The system constructs a multi-stream concurrent execution pipeline within the graphics processing unit that strictly corresponds to the two-stream network topology, opening three CUDA streams with independent instruction queues. The first stream is dedicated to the structural feature encoding branch, loading the waveform-corrected geometric reference signal into video memory and driving the feature pyramid network to extract multi-scale spatial topological features. Due to the high computational density of this branch, the first stream is allocated high-priority computational resources. The second stream is dedicated to the texture motion encoding branch, processing video detail component signals in parallel and driving the optical flow estimation sub-network and the feature inverse warp layer. Utilizing its independence, optical flow computation and inverse warp operations can mask the latency of some convolutional computations in the first stream. The optical flow estimation sub-network contains three sets of residual convolutional blocks, each consisting of two layers with 64 channels and a convolutional kernel of [missing information]. The system consists of convolutional layers. The optical flow estimation subnetwork establishes a five-layer feature pyramid and constructs a partial cost volume at each layer, setting the search radius to 4 pixels to capture motion information. The third stream is responsible for executing the spatial feature modulation decoding layer, which is configured with an event synchronization mechanism, only starting execution after detecting that both the first and second streams have emitted computation completion signals. During the decoding process, the third stream sequentially calls the fusion operators at each resolution level, modulating the structural feature parameters generated by the first stream onto the calibration texture features generated by the second stream in real time. Through the above-mentioned heterogeneous computing links and deep optimization of specific algorithm layers, the single-frame end-to-end inference time of the generative model, including spatial feature modulation and inverse warping, is stably compressed to less than 16 milliseconds, meeting the stringent timing requirements of broadcast-level real-time processing of 60 frames per second.

[0082] This invention utilizes an FPGA circular buffer to achieve zero-frame-loss acquisition of massive amounts of data, providing precise timing anchors for the algorithm. Parallel signal processing synchronously performs denoising, dehazing, and gradient feature extraction, leveraging the insensitivity of gradient magnitude to illumination to address feature extraction biases under harsh conditions. A dual-stream adversarial network with geometrically constrained texture generation, combined with feature inverse distortion for nonlinear motion compensation, eliminates interpolation artifacts and structural breakdowns in high-speed moving targets. Combined with distributed soft phase-locked loop control, it achieves high signal-to-noise ratio, tear-free real-time slow-motion replay in complex backgrounds.

[0083] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A video slow-motion frame interpolation playback system based on generative adversarial networks, characterized in that, include: The video signal acquisition module receives the main video signal through a multi-standard video communication interface, performs time-domain slicing on the main video signal using a programmable logic array and writes it into a circular buffer, outputting a continuous sequence of raw time-series video frames. The signal processing module receives the original time-series video frame sequence and performs parallel signal processing. The first path performs waveform correction for denoising and dehazing of the signal and separates the video detail component signal. The second path synchronously separates the edge gradient features in the main video signal and outputs a geometric structure reference signal. The frame rate conversion module loads the geometric structure reference signal as a boundary locking condition into the generative video synthesis unit, guides the video detail component signal to perform nonlinear motion compensation and intermediate frame signal reconstruction between adjacent frames, and outputs the reconstructed video data stream after spatiotemporal consistency verification. The generative video synthesis unit employs a conditional generative adversarial network. Its overall architecture comprises a deep convolutional neural network consisting of a dual-stream encoder and a cascaded decoder. The hierarchical structure includes: a structural feature encoding branch, composed of multiple downsampling convolutional layers, used to extract spatial topological features of video frames; a texture motion encoding branch, set in parallel with the structural feature encoding branch, internally integrating a feature inverse warping layer, used to extract texture features of video frames and perform feature alignment; and a spatial feature modulation decoding layer, located in the network's decoding stage, serving as the data aggregation node for the structural feature encoding branch and the texture motion encoding branch. Internally, it includes multiple cascaded upsampling deconvolutional layers. These upsampling deconvolutional layers receive the aggregated feature data, utilize structural features to perform pixel-by-pixel normalization modulation of texture features, and perform resolution restoration operations layer by layer. The playback encoding output module receives the reconstructed video data stream, responds to playback control commands input via a human-computer interaction terminal, and performs time base remapping processing, including: constructing a virtual display time base, stripping the original acquisition timestamps from the reconstructed video data stream, and reallocating monotonically increasing virtual display timestamps for each video frame based on the slow motion ratio set by the playback control command, with the first frame as the reference point; establishing a rate adaptation buffer pool, writing the generated reconstructed video data stream into the rate adaptation buffer pool, controlling the reading rate based on the virtual display timestamps, integrating the video frame data with the virtual display timestamps, and inputting the result into a video compression encoder to generate a slow motion replay signal encoding; and transmitting the slow motion replay signal encoding to the display terminal via a distributed display control device.

2. The video slow-motion frame interpolation and playback system based on generative adversarial networks according to claim 1, characterized in that: The multi-standard video communication interface is integrated with SDI interface, HDMI interface and fiber optic video interface, which converts the incoming main video signal into a parallel digital signal and transmits it to the programmable logic array. The method for the programmable logic array to perform time-domain slicing operation on the parallel digital signal includes: stamping each frame of signal with a hardware timestamp, writing data into a circular buffer according to a first-in-first-out buffer logic, wherein the circular buffer is mapped to the physical address space of DDR SDRAM, performing a loop-over addressing with the beginning and end connected using a write pointer to construct a sliding time window, and extracting the corresponding video frame from the sliding time window using the hardware timestamp as an index and integrating it through a read pointer to output the original time-series video frame sequence.

3. The video slow-motion frame interpolation and playback system based on generative adversarial networks according to claim 1, characterized in that: The method for receiving the original temporal video frame sequence and performing parallel signal processing includes: splitting the original temporal video frame sequence into two synchronously running processing paths; after the original temporal video frame sequence enters the first path, performing a dehazing operation and using an image restoration algorithm to restore the scene radiance; based on the restored scene radiance, performing a bilateral filtering denoising operation on the original temporal video frame sequence, performing a sliding window convolution on each video frame to extract video detail component signals, wherein the video detail component signals include surface high-frequency texture components, motion features, and contour edge information of moving targets in the video image; after the original temporal video frame sequence synchronously enters the second path, using horizontal and vertical convolution templates, performing a weighted difference operation on the pixel grayscale of each video frame to obtain the horizontal gradient component and the vertical gradient component and fusing them into a gradient vector magnitude as the edge gradient feature; and generating a geometric structure reference signal after binarizing the edge gradient feature.

4. The video slow-motion frame interpolation and playback system based on generative adversarial networks according to claim 1, characterized in that: The specific method for performing nonlinear motion compensation and intermediate frame signal reconstruction includes: a structural feature encoding branch performing multi-scale feature extraction on the received geometric structure reference signal to generate affine transformation parameters containing spatial semantic layout information, and constructing boundary locking conditions that restrict the spatial degrees of freedom of the synthesis process; a texture motion encoding branch receiving video detail component signals, driving the feature inverse distortion layer to perform nonlinear motion compensation based on the boundary locking conditions, remapping the texture feature pixels of the video frame to the predicted position of the current intermediate frame, and generating a motion-calibrated texture feature tensor; a spatial feature modulation decoding layer performing parameterized modulation on the texture feature tensor to recover video texture details, and reconstructing the intermediate video frame after multi-level cascaded upsampling; and embedding the intermediate video frame into the original temporal video frame sequence in timestamp order to form an interpolated video stream to be verified.

5. The video slow-motion frame interpolation and playback system based on generative adversarial networks according to claim 4, characterized in that: The process of outputting the reconstructed video data stream after spatiotemporal consistency verification includes: extracting intermediate video frames from the interpolated video stream, calculating the luminance histogram difference value and structural similarity index between the intermediate video frame and adjacent video frames, and determining the signal-to-noise ratio and structural integrity of the generated video frame; performing spatiotemporal consistency verification, and if the difference value corresponding to the signal-to-noise ratio is lower than a preset safety threshold and the structural similarity index corresponding to the structural integrity is higher than a preset standard, the verification is deemed successful, the interpolated video stream is marked and output as the reconstructed video data stream; if the difference value corresponding to the signal-to-noise ratio is not lower than a preset safety threshold, then pixel backfilling based on the nearest neighbor in the temporal domain is triggered, and based on the position of the current intermediate frame on the time axis, the video frame closest to the intermediate frame is locked as the target reference frame, the pixel data of the spatially corresponding position in the target reference frame is extracted, and directly migrated and covered to the abnormal area of ​​the intermediate video frame, and the modified reconstructed video data stream is output.

6. The video slow-motion frame interpolation and playback system based on generative adversarial networks according to claim 1, characterized in that: The distributed display and control device integrates a signal distribution unit and is connected to a display terminal array. The signal distribution unit receives slow-motion replay signal encoding, encapsulates the slow-motion replay signal encoding into a data stream adapted to the current physical communication link transmission format, where the physical communication link includes an Ethernet network and an optical fiber transmission link; it then distributes the data stream to each node of the display terminal array using the physical communication link; each node of the display terminal array independently receives the data stream, performs local decoding on the bitstream in the data stream, and synchronously controls the screen refresh timing based on the virtual display timestamp to maintain the timing synchronization of the slow-motion replay images.

Images