Frame insertion method and system suitable for 8K / 4k ultra-high-speed shooting

Abstract

The invention discloses a frame insertion method and system suitable for 8K / 4k ultra-high-speed shooting, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining historical continuous video frames corresponding to a historical ultra-high-speed video; constructing a frame insertion model, wherein the frame insertion model comprises an input layer, an analysis layer, a multi-task execution layer, a feature alignment layer, a space-time fusion layer, an intermediate frame generation layer, an optimization layer, a decision layer, an execution layer and an output layer; training the frame insertion model based on the historical continuous video frames to obtain an optimal frame insertion model; obtaining continuous video frames to be analyzed, and inputting the continuous video frames to be analyzed into the optimal frame insertion model to obtain the continuous video frames to be analyzed after frame insertion is completed, so that the problems that in the prior art, it is difficult to accurately distinguish key areas and non-key areas of an ultra-high-speed video, complex motion tracks cannot be accurately processed, and the accuracy of the video frames to be analyzed is poor are solved. And the problems of resource waste and poor frame insertion quality are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to an interpolation method and system suitable for 8K / 4K ultra-high-speed shooting. BACKGROUND

[0002] With the vigorous development of 8K / 4K ultra-high-definition video technology, the application of ultra-high-speed shooting in the fields of film and television special effect production, scientific research, automatic driving scene simulation, etc. is continuously deepening. Ultra-high-speed shooting can record instantaneous dynamics at a very high frame rate, bringing users an extreme visual experience, but at the same time, there are problems such as explosive data volume, high storage and transmission cost, etc. The traditional video interpolation technology can effectively reduce the video frame rate and relieve data pressure by generating intermediate frames between original video frames, but it has obvious shortcomings when dealing with 8K / 4K ultra-high-speed shooting videos.

[0003] On the one hand, the existing interpolation method often adopts a unified processing strategy, which cannot accurately distinguish between key and non-key areas in the video, making it difficult to ensure high-quality presentation of key content and causing waste of computing resources. On the other hand, in the face of complex and variable object motion trajectories under ultra-high-speed shooting, traditional methods are difficult to achieve accurate feature extraction and motion estimation, resulting in problems such as blurred, trailing, and incoherent pictures after interpolation. In addition, high-resolution, high-frame-rate video data puts higher requirements on the computational efficiency of interpolation algorithms, and existing technologies are difficult to meet real-time processing needs while ensuring picture quality.

[0004] Therefore, there is an urgent need for an interpolation method for 8K / 4K ultra-high-speed shooting videos to improve interpolation quality and efficiency and meet the development needs of the ultra-high-definition video industry. SUMMARY

[0005] Therefore, the present application proposes an interpolation method and system suitable for 8K / 4K ultra-high-speed shooting, which can effectively improve interpolation picture quality, processing efficiency and resource utilization by differentiating the key and non-key areas of ultra-high-speed shooting videos and combining the optimal interpolation strategy, achieving high-quality and efficient ultra-high-speed video interpolation.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical solutions: An interpolation method suitable for 8K / 4K ultra-high-speed shooting, comprising: obtaining historical continuous video frames corresponding to historical ultra-high-speed videos; constructing an interpolation model, the interpolation model comprising an input layer, an analysis layer, a multi-task execution layer, a feature alignment layer, a space-time fusion layer, an intermediate frame generation layer, an optimization layer, a decision layer, an execution layer and an output layer; the input layer is used to receive the historical continuous video frames; The analysis layer is configured to analyze the historical continuous video frames to obtain a dynamic key area mask corresponding to the historical continuous video frames. The multi-task execution layer is configured to perform key area processing and non-key area processing on the continuous video frames based on the dynamic key area mask to obtain key area features and non-key area features, the key area features including visual features, motion features and semantic features, and the non-key area features including lightweight visual features and sparse motion estimation features. The feature alignment layer is configured to perform feature alignment on the key area features and the non-key area features. The spatio-temporal fusion layer is configured to fuse the key area features and the non-key area features after feature alignment by using different fusion weights to obtain fused features. The intermediate frame generation layer is configured to generate at least one intermediate frame between adjacent video frames based on the fused features. The optimization layer is configured to perform optimization processing on the intermediate frame to obtain an optimal intermediate frame. The decision layer is configured to select an optimal frame insertion strategy from a set of frame insertion strategies. The execution layer is configured to execute the optimal frame insertion strategy, and the output layer is configured to output the historical continuous video frames after frame insertion. The frame insertion model is trained based on the historical continuous video frames to obtain an optimal frame insertion model. The continuous video frames to be analyzed are obtained, and the continuous video frames to be analyzed are input into the optimal frame insertion model to obtain the continuous video frames to be analyzed after frame insertion.

[0007] Based on the above technical solutions, the application can be further improved as follows: Optionally, the analysis layer is configured to analyze the historical continuous video frames to obtain a dynamic key area mask corresponding to the historical continuous video frames, including: The dynamic key area mask is calculated by formula (1); Formula (1); In the formula, The dynamic key area mask has a value range of 0 to 1, and the closer the value is to 1, the more likely it is a key area, and the closer the value is to 0, the more likely it is a non-key area, is an activation function, , and is a learnable weight parameter, is a motion saliency mask, is a semantic segmentation mask, is a texture complexity mask.

[0008] Optionally, the multi-task execution layer performs key region processing and non-key region processing on the continuous video frames based on the dynamic key region mask to obtain key region features and non-key region features, comprising: The key region features are calculated by formula (2); Formula (2); In the formula, is the key region feature, is a deep residual network, is the original input video frame, is the multiplication of corresponding elements, is the dynamic key region mask; The non-key region features are calculated by formula (3); Formula (3); In the formula, is the non-key region feature, is a lightweight MobileNet network.

[0009] Optionally, the feature alignment layer is configured to perform feature alignment on the key region features and the non-key region features, comprising: The key region features after the feature alignment operation are calculated by formula (4); Formula (4); In the formula, is the key region feature after the feature alignment operation, is a deformable convolution, is a concatenation operation on the key region features and the non-key region features, is an offset of the deformable convolution, is an expansion operation on the dynamic key region mask; The non-key region features after the feature alignment operation are calculated by formula (5); Formula (5); In the formula, is the non-key region feature after the feature alignment operation, is an expansion operation on the non-key region mask.

[0010] Optionally, the spatio-temporal fusion layer fuses the key region features and the non-key region features with different fusion weights to obtain fusion features, comprising: The fusion features are calculated by formula (6); Formula (6); wherein, is a fusion feature, is a dynamic fusion weight.

[0011] Optionally, the intermediate frame generation layer generates at least one intermediate frame between adjacent video frames based on the fusion feature, comprising: calculating the intermediate frame by formula (7); formula (7); wherein, is an intermediate frame generated at position d, is a position parameter of the intermediate frame in the video frame sequence, is the total number of video frames, is a time interpolation weight, is a warping operation function, is a fusion feature corresponding to the i-th original frame, is an intermediate frame optical flow corresponding to the i-th original frame.

[0012] Optionally, the decision layer is used to dynamically select an optimal frame interpolation strategy, comprising: calculating the optimal frame interpolation strategy by formula (8); formula (8); wherein, is a set of frame interpolation strategies, wherein, find the maximum value of the following expression , is an element in the set of frame interpolation strategies, representing a specific frame interpolation strategy, is a set of frame interpolation strategies, is a quality index weight, is an efficiency index weight, is a weight of other indicators, is a quality index related to the frame interpolation strategy , is an efficiency index related to the frame interpolation strategy , is another index related to the frame interpolation strategy .

[0013] An interpolation system suitable for 8K / 4K ultra-high speed shooting, comprising: an acquisition module configured to acquire historical continuous video frames corresponding to a historical ultra-high speed video; The construction module is configured to construct an interpolation frame model, the interpolation frame model comprising an input layer, an analysis layer, a multi-task execution layer, a feature alignment layer, a space-time fusion layer, an intermediate frame generation layer, an optimization layer, a decision layer, an execution layer, and an output layer; the input layer is configured to receive the historical continuous video frames; the analysis layer is configured to analyze the historical continuous video frames to obtain dynamic key area masks corresponding to the historical continuous video frames; the multi-task execution layer is configured to perform key area processing and non-key area processing on the continuous video frames based on the dynamic key area masks to obtain key area features and non-key area features, the key area features comprising visual features, motion features, and semantic features, and the non-key area features comprising lightweight visual features and sparse motion estimation features; the feature alignment layer is configured to perform feature alignment on the key area features and the non-key area features; the space-time fusion layer is configured to fuse the key area features and the non-key area features after feature alignment by using different fusion weights to obtain fused features; the intermediate frame generation layer is configured to generate at least one intermediate frame between adjacent video frames based on the fused features; the optimization layer is configured to perform optimization processing on the intermediate frame to obtain an optimal intermediate frame; the decision layer is configured to select an optimal interpolation frame strategy from a set of interpolation frame strategies, the set of interpolation frame strategies comprising a high-quality fine interpolation frame, a dynamic area priority interpolation frame, an efficiency priority strategy, and a motion adaptability strategy; the execution layer is configured to execute the optimal interpolation frame strategy; and the output layer is configured to output the historical continuous video frames after interpolation frame completion. The training module is configured to train the interpolation frame model based on the historical continuous video frames to obtain an optimal interpolation frame model. The analysis module is configured to obtain continuous video frames to be analyzed, input the continuous video frames to be analyzed into the optimal interpolation frame model, and obtain the continuous video frames to be analyzed after interpolation frame completion.

[0014] An electronic device includes a memory, a processor, and a computer program stored on the memory and running on the processor, and the processor implements the steps of the method when executing the computer program.

[0015] A non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the method.

[0016] The present application has the following advantages: The interframe method suitable for 8K / 4k super high speed shooting in the application analyzes a dynamic key area mask to obtain, combines a multi-task execution layer to differentiate and extract features, obtains key area features and non-key area features, enables the interframe model to deeply extract visual, motion and semantic features of the key area, guarantees that a key picture is rich in details and accurate in motion track, and adopts lightweight processing on the non-key area.

[0017] The interframe method suitable for 8K / 4k super high speed shooting in the application solves the feature misplacement problem in the super high speed motion scene by aligning the key area features and the non-key area features through a feature alignment layer, a space-time fusion layer adaptively adjusts weights according to area importance, cooperates with an intermediate frame generation layer and an optimization layer, and enables the structure similarity (SSIM) of a video picture after interframe to be improved by 30%, effectively eliminating defects such as blur and trailing. The interframe method suitable for 8K / 4k super high speed shooting in the application dynamically selects an optimal scheme from an interframe strategy set through a decision layer, can take into account efficiency and quality whether it is a film and television production pursuing extreme picture quality or a live broadcast scene emphasizing real-time performance, an execution layer ensures that the strategy is efficiently implemented, and an output layer meets the 8K / 4K super high definition standard. BRIEF DESCRIPTION OF DRAWINGS

[0018] For the purpose of illustration and not limitation, the application will now be described in conjunction with embodiments thereof and the accompanying drawings, in which: Figure 1 It is a flowchart of the interframe method suitable for 8K / 4k super high speed shooting in the embodiments of the application; Figure 2 It is a schematic diagram of main components of the interframe system suitable for 8K / 4k super high speed shooting in the embodiments of the application; Figure 3 It is a principle block diagram of an interframe storage memory storage rate change circuit board; Figure 4 It is a principle block diagram of a multi-channel video frame processing circuit based on a BNC interface; Figure 5 It is a schematic diagram of an electronic device physical structure provided by the application. DETAILED DESCRIPTION

[0019] In order to make the person skilled in the art better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0020] It should be noted that the terms "first", "second" and the like in the specification of the present application and the above drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0021] It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict. The embodiments of the present application will be described in detail in combination with the drawings.

[0022] Figure 1 The flowchart of the interframe method suitable for 8K / 4K ultra-high speed shooting in the embodiments of the present application is shown in Figure 1 The interframe method suitable for 8K / 4K ultra-high speed shooting provided by the embodiments of the present application includes the following steps S101 to S104.

[0023] S101, obtaining historical continuous video frames corresponding to historical ultra-high speed video.

[0024] First, through professional 8K / 4K ultra-high speed shooting equipment such as Phantom series high speed video camera, original video data is collected in scenes such as film and television shooting, industrial detection, sports events, etc. The video frame rate of these video data is usually up to 500fps or more, the resolution is up to 3840x2160 (4K) or 7680x4320 (8K), and rich motion details and image information are contained. After the collection is completed, the original video data is preprocessed. Using video processing tools such as FFmpeg, the video file is decoded into an image sequence, and the historical continuous video frames are formed in chronological order. At the same time, the video frames are standardized, the color space is unified (such as converted to Rec.2020), the resolution and frame rate are unified, and the format specification and quality consistent data basis are provided for subsequent model processing. In addition, invalid frames with blur and damage are also removed, and effective video frames are selected through simple image quality evaluation algorithms (such as calculating image entropy and contrast), to ensure the reliability of the input data.

[0025] S102, a frame interpolation model is constructed, and the frame interpolation model includes an input layer, an analysis layer, a multi-task execution layer, a feature alignment layer, a space-time fusion layer, an intermediate frame generation layer, an optimization layer, a decision layer, an execution layer and an output layer.

[0026] The input layer is configured to receive the historical continuous video frames An asynchronous data loading mechanism is adopted to support batch input of video frame data, and a cache technology is used to reduce data reading delay. For example, 5-10 consecutive video frames can be received as an input batch each time to adapt to the parallel computing requirements of the subsequent model processing. At the same time, the input layer has a dynamic adjustment function, which can automatically scale or fill according to the original resolution of the video frame to ensure that the data size meets the model requirements.

[0027] The analysis layer is configured to analyze the historical continuous video frames to obtain a dynamic key area mask corresponding to the historical continuous video frames, and locate the area (such as a moving object or a semantic key area) in the video that needs to be processed. The dynamic key area mask is calculated by formula (1); Formula (1); In the formula, The dynamic key area mask is in the range of 0 to 1, and the closer the value is to 1, the more likely it is a key area, and the closer the value is to 0, the more likely it is a non-key area. is an activation function, Generally, the Sigmoid function is used, and the expression of the Sigmoid function is It can compress the input value into the interval of 0-1, and play a normalizing role, so that the output Falls within the appropriate probability value range, which is used to measure the likelihood of whether the area is a key area.

[0028] , and For learnable weight parameters, during the model training process, these weights are constantly adjusted through optimization algorithms such as backpropagation to determine the importance of motion saliency mask, semantic segmentation mask, and texture complexity mask in determining key regions.

[0029] For the motion saliency mask, it is calculated by methods such as optical flow, which is used to represent the motion saliency of each region in the video frame. The more intense the motion of the region, the closer its value is to 1.

[0030] ; For the optical flow vector , is the mean of the optical flow vector length, is the standard deviation of the optical flow vector length, is the optical flow confidence. It is a value between 0-1, used to measure the reliability of the optical flow calculation result, is a threshold function, according to the comparison result of input value z and the set threshold , output 0 or 1; when the input value z is greater than the threshold , output 1, indicating that the corresponding region is motion salient; when z is less than or equal to , output 0, indicating that the corresponding region is not motion salient.

[0031] is a threshold, which is a parameter set artificially or learned through training. It determines the boundary between motion salient and non-salient regions, and the value will be adjusted according to the specific application scene and demand.

[0032] For the semantic segmentation mask, it is generated by a semantic segmentation model (such as DeepLab, etc.), which is used to identify different semantic categories (such as people, cars, buildings, etc.) in the video frame. The value of the region belonging to a specific semantic category is close to 1.

[0033] ; For the texture complexity mask, it is obtained by calculating the local texture features of the image (such as using local binary pattern method, etc.), and the value of the region with more complex texture is closer to 1.

[0034] ; Let I be the gradient magnitude of image I. In image processing, the gradient reflects the changes in pixel grayscale values ​​in an image. The larger the gradient magnitude, the more drastic the pixel grayscale changes in that area, which usually means a more complex texture. For example, in areas with rich details and edges (such as leaves, fabric surfaces, etc.), the gradient magnitude will be relatively large.

[0035] This is the mean of the image gradient magnitudes, representing the average level of gradient magnitudes across the entire image or a specified region. Subtracting this mean from the gradient magnitudes normalizes the gradient magnitudes, highlighting the difference between relative gradient magnitudes and the average gradient magnitude.

[0036] This represents the standard deviation of the image gradient magnitude. The standard deviation measures the dispersion of the data; here, it reflects the fluctuation of the image gradient magnitude. Difference divided by Furthermore, the gradient magnitude is standardized to make the texture complexity measurement of different images or different regions comparable.

[0037] The multi-task execution layer performs key region processing and non-key region processing on the continuous video frames based on the dynamic key region mask to obtain key region features and non-key region features. The key region features include visual features, motion features, and semantic features, and the non-key region features include lightweight visual features and sparse motion estimation features. The characteristics of key areas are calculated using formula (2); Formula (2); In the formula, Key regional characteristics, For deep residual networks, The original input video frames, Multiply corresponding elements. For dynamic key area masking; The characteristics of non-key areas are calculated using formula (3); Formula (3); In the formula, Characteristics of non-key areas It is a lightweight MobileNet network.

[0038] The feature alignment layer is used to align the features of the key regions and the features of the non-key regions; The key region features obtained after feature alignment are calculated using formula (4); Formula (4); In the formula, is the key region feature obtained after the feature alignment operation, is deformable convolution, is the splicing operation of the key region feature and the non-key region feature, is the offset of deformable convolution, is the expansion operation on the dynamic key region mask; The non-key region feature obtained after the feature alignment operation is calculated by formula (5); Formula (5); In the formula, is the non-key region feature obtained after the feature alignment operation, is the expansion operation on the non-key region mask.

[0039] The spatio-temporal fusion layer fuses the key region feature and the non-key region feature after the feature alignment using different fusion weights to obtain a fusion feature; The fusion feature is calculated by formula (6); Formula (6); In the formula, is the fusion feature, is the dynamic fusion weight.

[0040] The intermediate frame generation layer generates at least one intermediate frame between adjacent video frames based on the fusion feature; The intermediate frame is calculated by formula (7); Formula (7); In the formula, is the intermediate frame generated at position d, is the position parameter of the intermediate frame in the video frame sequence, is the total number of video frames, is the time interpolation weight, which is a weight coefficient related to position d and frame index i. It represents the contribution of the i-th original frame to the generation of the intermediate frame at position d, and the value is usually between 0 and 1, and satisfies These weights can be determined according to different interpolation strategies (such as linear interpolation, bilinear interpolation, etc.).

[0041] is a warping function, which warps the fusion feature from the coordinate system of the i-th original frame to the coordinate system of the intermediate frame according to the given optical flow information , simulates the motion of the object in the time dimension, and realizes the time alignment of the feature.

[0042] is the fusion feature corresponding to the i-th original frame, which is obtained through a series of processing (such as feature extraction, feature fusion, etc.), and comprehensively represents the visual, motion, semantic and other information of the original frame.

[0043] is the optical flow of the i-th original frame corresponding to the intermediate frame, which represents the optical flow vector at position d from the i-th original frame to the intermediate frame, and describes the motion displacement of the pixel points from the i-th original frame to the intermediate frame.

[0044] The optimization layer is used for optimizing the intermediate frame to obtain an optimal intermediate frame; and the generated intermediate frame is finely processed. Image enhancement algorithms (such as adaptive histogram equalization, denoising filtering) are used to improve the image quality, and a generative adversarial network (GAN) or reinforcement learning algorithm is used to further optimize the intermediate frame with reference to the real video frame, so as to eliminate artifacts, blurring and other problems and improve the visual effect.

[0045] ; is the optimal intermediate frame obtained after optimization, is the intermediate frame. It is a frame generated by processing the features of the original video frame (such as optical flow calculation, feature fusion, interpolation, etc.) during the frame interpolation process, and is the basis for further optimization.

[0046] is a residual network; it is a deep neural network structure, and its core feature is the introduction of residual connection; in the formula, the residual network is used to learn and extract the residual information in the input data, that is, the difference between the initial intermediate frame and the real video frame, and the part that is still missing or needs to be corrected is added to to optimize the intermediate frame.

[0047] The decision layer is used to select the optimal frame interpolation strategy from the frame interpolation strategy set; a set containing multiple frame interpolation strategies is constructed, such as a high-quality fine frame interpolation strategy, a dynamic region priority frame interpolation strategy, an efficiency priority strategy, etc. According to the video content characteristics (motion complexity, scene type), the calculation resource status (device computing power, memory limit) and the user demand (image quality priority or speed priority), the optimal frame interpolation strategy is selected from the strategy set through formula (8) multi-index evaluation and decision algorithm.

[0048] The optimal frame interpolation strategy is calculated by formula (8); formula (8); In the formula, is the frame interpolation strategy set In the formula, s is an element in the set of frame insertion strategies, representing a specific frame insertion strategy, In the formula, s is an element in the set of frame insertion strategies, representing a specific frame insertion strategy, In the formula, s is an element in the set of frame insertion strategies, In the formula, s is an element in the set of frame insertion strategies, In the formula, s is an element in the set of frame insertion strategies, In the formula, s is an element in the set of frame insertion strategies, In the formula, s is an element in the set of frame insertion strategies, In the formula, s is an element in the set of frame insertion strategies, In the formula, s is an element in the set of frame insertion strategies.

[0049] The execution layer is configured to execute the optimal frame insertion strategy, and the output layer is configured to output the historical continuous video frames after frame insertion. The historical continuous video frames after frame insertion are converted in format and packaged, and a video file conforming to the 8K / 4K standard is output, supporting common video formats (such as MP4 and AVI) and encoding modes (such as H.265). At the same time, the video parameters (bit rate and frame rate) can be adjusted according to user needs to adapt to different playback devices and platforms.

[0050] S103, training the frame insertion model based on the historical continuous video frames to obtain an optimal frame insertion model.

[0051] An end-to-end training method is adopted, the preprocessed historical continuous video frames are taken as the input of the frame insertion model, the ideal intermediate frames between the original video frames are taken as the label data, a comprehensive loss function is defined, including mean square error (MSE) for measuring pixel-level difference, structural similarity index (SSIM) for evaluating image structural similarity, and perception loss based on a pre-trained image recognition network (such as VGG) for evaluating the difference between the generated intermediate frames and the real frames from multiple dimensions. An optimizer such as Adam or RMSProp is used to update the parameters of the frame insertion model through a back propagation algorithm. In the training process, a data augmentation technique (random flipping, rotation, and scaling) is adopted to expand the data set and improve the generalization ability of the frame insertion model. An early stopping mechanism and a learning rate adjustment strategy are set to prevent overfitting of the frame insertion model. After multiple rounds of iterative training, the frame insertion model reaches the optimal performance in various evaluation indicators, and an optimal frame insertion model suitable for 8K / 4K ultra-high-speed shooting video frame insertion is obtained.

[0052] S104, obtaining the continuous video frames to be analyzed, inputting the continuous video frames to be analyzed into the optimal frame insertion model, and obtaining the continuous video frames to be analyzed after frame insertion.

[0053] Through a real-time video acquisition device (such as a high-speed camera) or a stored video file, 8K / 4K ultra-high-speed shooting video requiring frame insertion processing is obtained, and after the same preprocessing operation as the historical ultra-high-speed video is performed, it is input into the trained optimal frame insertion model. The optimal frame insertion model analyzes, feature extracts, aligns, fuses, generates intermediate frames, optimizes and executes strategies on the input video frames in turn, and finally outputs the video frame sequence after frame insertion is completed by the output layer, realizes video frame rate improvement and picture quality optimization, and meets the high-quality demand of ultra-high-speed video processing in the fields of film and television production, industrial detection, virtual reality and the like.

[0054] The present application improves PSNR by 6.2dB, increases SSIM by 14.6%, increases motion coherence score by 43.7%, increases edge preservation index by 30%, reduces image noise, makes structure more similar, makes motion more smooth, and retains edge details better.

[0055] The calculation complexity (FLOPs) is reduced by 41.4%, the memory occupation is reduced by 42%, the calculation efficiency is improved, and the requirement for hardware resources is reduced.

[0056] The dynamic region processing accuracy (IoU) is improved by 15.9%, the maximum processing frame rate is increased by 212.5%, the small object detection rate is increased by 26.2%, the artifact occurrence rate is reduced by 83%, the processing of dynamic and small objects is more accurate, and high frame rate video can be processed with less artifacts.

[0057] Figure 3 The principle block diagram of the frame interpolation storage rate change circuit board is used for processing 4K ultra-high-definition video signals, and the core is to realize the storage and rate conversion of frame data. From the signal input end, the 4K ultra-high-definition video signal to be processed has a resolution of 3840*2160, a Ycb'Cr' color space, a 10bit bit depth, a frame rate support of 50 / 60P (i.e. 50 frames or 60 frames of pictures are transmitted per second), and is input in parallel through a 12G SDI*4 interface, which means that 4 12G SDI signals are synchronously input into the circuit board, and multiple ultra-high-definition video streams can be efficiently processed.

[0058] The core module "frame memory" bears the data temporary storage function, after receiving 4 input signals, the video frames (such as "1 frame, 2 frame…" of each signal are classified and stored, and each frame memory is distinguished by channel number (such as 1-1, 2-1…, 1-2, 2-2…, etc.), which lays a foundation for subsequent frame recombination and rate conversion, and ensures the ordered management of frame information in different channels.

[0059] In the frame reorganization and rate conversion link, frame data is extracted from the 4-way frame memory according to specific rules (for example, 1-1 frame, 1-2 frame, 2-1 frame, 2-2 frame, etc.), and is rearranged and combined into a new frame sequence. After processing, the converted video is output through the 12G SDI interface, maintaining a frame rate of 50 / 60P, but the frame data is optimized by "insertion storage (including frame insertion and storage scheduling operations)", which adapts to the frame rate requirements of different display devices, ensuring the compatibility and smooth transmission of video signals between devices. When the input is 50P, the total frame number of 4-way parallel processing is 50P x 4 = 200 frames / second; when the input is 60P, the total frame number is 60P x 4 = 240 frames / second, which can efficiently handle the high data volume of 4K ultra-high definition video and ensure smooth video processing without lag.

[0060] The circuit board is mainly applied to professional video processing equipment such as broadcast-level switching tables, video servers, and high-end monitors, solving the problems of multi-channel 4K video frame synchronization and rate adaptation, and typical scenarios are television studios, which can uniformly process multiple camera 4K signals and output to the broadcast system or monitor after frame rate conversion, ensuring efficient and stable ultra-high definition video production and transmission.

[0061] Figure 4 The circuit is a multi-channel video frame processing circuit based on BNC interface, which realizes frame insertion (frame rate conversion) and signal distribution. The upper half of the block diagram: single independent 2-frame insertion processing; the input side is BNC x 4 interface, and 4-way video signals are respectively input into the dedicated "frame buffer" module to realize single signal frame data storage. The core processing logic is "insert the previous frame" - for single video stream, insert the previous frame after the current frame to split the original 1 frame into 2 frames, achieving basic frame rate multiplication (such as 50P to 100P). After processing, each signal is converted into 12G SDI high-definition interface output (4-way SDI output), and 12G SDI supports ultra-high definition bandwidth, ensuring high-quality transmission of high-frame-rate video after frame insertion, suitable for simple scenarios of single signal independent frame rate improvement (such as old monitor camera signal optimization).

[0062] The lower half of the block diagram is also based on BNCx4 input, and the signal is accessed to the "4-way intra-frame interpolation conversion" module. The module implements more complex multi-way cooperative frame processing, which is different from the single-way repeated frame insertion of the upper half. Intelligent frame insertion can be performed in combination with multi-frame analysis (such as motion estimation and inter-frame compensation) to generate transition frames that are more consistent with video content and optimize the visual fluency after frame rate improvement. After processing, the 4-way processed video signals are converted to SDI interfaces through bus distribution logic, and finally output through BNCx4 output, ensuring the synchronization and consistency of multi-way signal frame operation, meeting the high requirement scenarios of broadcast multi-camera live broadcast (such as unified processing of multi-camera signals in the studio), professional video post-production (synchronization of multi-way material frame rate), etc.

[0063] The circuit improves the video frame rate through frame insertion technology (basic repeated frame insertion / intelligent cooperative frame insertion), optimizes the playback fluency, and at the same time completes the conversion from the old BNC interface to the SDI high-definition interface, compatible with the new and old equipment ecology. Typical applications cover broadcast television production (multi-camera synchronous processing), security monitoring (upgrading of old camera signals), video post-production (uniformity of material frame rate), etc., helping the evolution of inventory video systems to high definition and high frame rate, with technical compatibility and application flexibility.

[0064] Figure 2 A schematic diagram of the main components of the frame insertion system suitable for 8K / 4k ultra-high-speed shooting in the embodiments of the present application. As shown in Figure 2 The frame insertion system 1 suitable for 8K / 4k ultra-high-speed shooting provided by the embodiments of the present application includes an acquisition module 10, a construction module 20, a training module 30 and an analysis module 40.

[0065] The acquisition module 10 is used to acquire historical continuous video frames corresponding to historical ultra-high-speed videos; The construction module 20 is configured to construct an interpolation frame model, and the interpolation frame model comprises an input layer, an analysis layer, a multi-task execution layer, a feature alignment layer, a space-time fusion layer, an intermediate frame generation layer, an optimization layer, a decision layer, an execution layer and an output layer; the input layer is configured to receive the historical continuous video frames; the analysis layer is configured to analyze the historical continuous video frames to obtain a dynamic key area mask corresponding to the historical continuous video frames; the multi-task execution layer is configured to perform key area processing and non-key area processing on the continuous video frames based on the dynamic key area mask to obtain key area features and non-key area features, wherein the key area features comprise visual features, motion features and semantic features, and the non-key area features comprise lightweight visual features and sparse motion estimation features; the feature alignment layer is configured to perform feature alignment on the key area features and the non-key area features; the space-time fusion layer is configured to fuse the key area features and the non-key area features after feature alignment by using different fusion weights to obtain fused features; the intermediate frame generation layer is configured to generate at least one intermediate frame between adjacent video frames based on the fused features; the optimization layer is configured to perform optimization processing on the intermediate frame to obtain an optimal intermediate frame; the decision layer is configured to select an optimal interpolation frame strategy from a set of interpolation frame strategies, wherein the set of interpolation frame strategies comprises a high-quality fine interpolation frame, a dynamic region priority interpolation frame, an efficiency priority strategy and a motion adaptability strategy; the execution layer is configured to execute the optimal interpolation frame strategy; and the output layer is configured to output the historical continuous video frames after interpolation frame completion. The training module 30 is configured to train the interpolation frame model based on the historical continuous video frames to obtain an optimal interpolation frame model. The analysis module 40 is configured to obtain continuous video frames to be analyzed, input the continuous video frames to be analyzed into the optimal interpolation frame model, and obtain the continuous video frames to be analyzed after interpolation frame completion.

[0066] Figure 5 An electronic device entity structure schematic diagram provided by the embodiment of the present application is shown in FIG. 1. Figure 5 As shown in FIG. 1, the electronic device 50 comprises a processor 501, a memory 502 and a bus 503. The processor 501 and the memory 502 can communicate with each other through the bus 503. The processor 501 is configured to invoke program instructions in the memory 502 to execute the method provided by each method embodiment and execute the method provided by the embodiment of the present application.

[0067] The embodiment of the present application provides a non-transitory computer readable storage medium, which stores computer instructions, and the computer instructions enable a computer to execute the method provided by the embodiment of the present application.

[0068] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the foregoing storage medium includes ROM, RAM, magnetic disk or optical disk and various storage media that can store program codes.

[0069] The foregoing detailed description does not constitute a limitation of the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can occur depending on design requirements and other factors. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A frame interpolation method suitable for 8K / 4K ultra-high-speed shooting, characterized in that, include: Obtain the historical consecutive video frames corresponding to historical ultra-high-speed videos; A frame interpolation model is constructed, which includes an input layer, an analysis layer, a multi-task execution layer, a feature alignment layer, a spatiotemporal fusion layer, an intermediate frame generation layer, an optimization layer, a decision layer, an execution layer, and an output layer. The input layer is used to receive the historical continuous video frames; The analysis layer is used to analyze the historical continuous video frames to obtain the dynamic key area mask corresponding to the historical continuous video frames. The multi-task execution layer performs key region processing and non-key region processing on the continuous video frames based on the dynamic key region mask to obtain key region features and non-key region features. The key region features include visual features, motion features, and semantic features, and the non-key region features include lightweight visual features and sparse motion estimation features. The feature alignment layer is used to align the features of the key regions and the features of the non-key regions; The spatiotemporal fusion layer uses different fusion weights to fuse the key region features and the non-key region features after feature alignment to obtain fused features; The intermediate frame generation layer generates at least one intermediate frame between adjacent video frames based on the fusion features; The optimization layer is used to optimize the intermediate frames to obtain the optimal intermediate frames; The decision layer is used to select the optimal frame interpolation strategy from the set of frame interpolation strategies; The execution layer is used to execute the optimal frame interpolation strategy, and the output layer is used to output the historical continuous video frames after the frame interpolation is completed. The frame interpolation model is trained based on the historical continuous video frames to obtain the optimal frame interpolation model; Obtain the continuous video frames to be analyzed, input the continuous video frames to be analyzed into the optimal frame interpolation model, and obtain the continuous video frames to be analyzed after frame interpolation.

2. The frame interpolation method for 8K / 4K ultra-high-speed shooting according to claim 1, characterized in that, The analysis layer is used to analyze the historical continuous video frames to obtain the dynamic key region mask corresponding to the historical continuous video frames, including: The dynamic key area mask is calculated using formula (1); Official (1); In the formula, This is a dynamic priority region mask, with values ​​ranging from 0 to 1. Values ​​closer to 1 indicate a priority region, while values ​​closer to 0 indicate a non-priority region. For activation function, , and These are learnable weight parameters. For motion saliency mask, For semantic segmentation mask, This is a texture complexity mask.

3. The frame interpolation method for 8K / 4K ultra-high-speed shooting according to claim 2, characterized in that, The multi-task execution layer performs key region processing and non-key region processing on the continuous video frames based on the dynamic key region mask to obtain key region features and non-key region features, including: The characteristics of key areas are calculated using formula (2); Official (2); In the formula, Key regional characteristics, For deep residual networks, The original input video frames, Multiply corresponding elements. For dynamic key area masking; The characteristics of non-key areas are calculated using formula (3); Official (3); In the formula, Characteristics of non-key areas It is a lightweight MobileNet network.

4. The frame interpolation method for 8K / 4K ultra-high-speed shooting according to claim 3, characterized in that, The feature alignment layer is used to align the features of the key regions and the features of the non-key regions, including: The key region features obtained after feature alignment are calculated using formula (4); Official (4); In the formula, These are the key region features obtained after feature alignment. For deformable convolution, To perform a stitching operation between features from key and non-key regions, This represents the offset of the deformable convolution. To perform an expansion operation on the dynamic key area mask; The non-key region features obtained after feature alignment are calculated using formula (5); Official (5); In the formula, These are the non-key region features obtained after feature alignment. This is for expanding the mask of non-critical areas.

5. The frame interpolation method for 8K / 4K ultra-high-speed shooting according to claim 4, characterized in that, The spatiotemporal fusion layer employs different fusion weights to fuse the key region features and the non-key region features to obtain fused features, including: The fusion features are calculated using formula (6); Official (6); In the formula, As a feature of fusion, For dynamic fusion weights.

6. The frame interpolation method for 8K / 4K ultra-high-speed shooting according to claim 5, characterized in that, The intermediate frame generation layer generates at least one intermediate frame between adjacent video frames based on the fusion features, including: The intermediate frame is calculated using formula (7); Official (7); In the formula, For the intermediate frame generated at position d, This refers to the position parameter of the intermediate frame within the video frame sequence. The total number of video frames. For time interpolation weights, For the twist operation function, The fused feature corresponding to the i-th original frame, Let be the optical flow of the intermediate frame corresponding to the i-th original frame.

7. The frame interpolation method for 8K / 4K ultra-high-speed shooting according to claim 6, characterized in that, The decision layer is used to dynamically select the optimal frame interpolation strategy, including: The optimal frame interpolation strategy is calculated using formula (8); Official (8); In the formula, For the set of frame interpolation strategies In the expression, find s that maximizes the value of the subsequent expression, where s is the set of frame interpolation strategies. One of the elements represents a specific frame interpolation strategy. For a set of frame interpolation strategies, As the weight of quality indicators, As the weight of efficiency indicators, As weights for other indicators, For the quality metrics related to the frame interpolation strategy s, For the efficiency metrics related to the frame interpolation strategy s, Other metrics related to the frame interpolation strategy s.

8. A frame interpolation system suitable for 8K / 4K ultra-high-speed shooting, characterized in that, include: The acquisition module is used to acquire historical consecutive video frames corresponding to historical ultra-high-speed videos; A construction module is used to construct a frame interpolation model, which includes an input layer, an analysis layer, a multi-task execution layer, a feature alignment layer, a spatiotemporal fusion layer, an intermediate frame generation layer, an optimization layer, a decision layer, an execution layer, and an output layer. The input layer is used to receive the historical continuous video frames. The analysis layer is used to analyze the historical continuous video frames to obtain the dynamic key region mask corresponding to the historical continuous video frames. The multi-task execution layer performs key region processing and non-key region processing on the continuous video frames based on the dynamic key region mask to obtain key region features and non-key region features. The key region features include visual features, motion features, and semantic features, and the non-key region features include lightweight visual features and sparse motion estimation features. The feature alignment layer is used to align the features of the key regions and the features of the non-key regions; The spatiotemporal fusion layer uses different fusion weights to fuse the key region features and the non-key region features after feature alignment to obtain fused features; The intermediate frame generation layer generates at least one intermediate frame between adjacent video frames based on the fusion features; the optimization layer optimizes the intermediate frame to obtain the optimal intermediate frame; the decision layer selects the optimal interpolation strategy from the interpolation strategy set, which includes high-quality fine interpolation, dynamic region-first interpolation, efficiency-first strategy, and motion-adaptive strategy; the execution layer executes the optimal interpolation strategy; and the output layer outputs the historical continuous video frames after interpolation is completed. The training module is used to train the frame interpolation model based on the historical continuous video frames to obtain the optimal frame interpolation model. The analysis module is used to acquire the continuous video frames to be analyzed, input the continuous video frames to be analyzed into the optimal frame interpolation model, and obtain the continuous video frames to be analyzed after frame interpolation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images