Adaptive real-time video processing system based on AI automatic interpolation

Abstract

The invention discloses a self-adaptive real-time video processing system based on AI automatic interpolation, and belongs to the technical field of video signal processing, and the embodiment of the invention adopts the following technical scheme that the self-adaptive real-time video processing system based on AI automatic interpolation comprises an input module used for inputting a data stream into the system; the analysis processing module is used for analyzing the input data flow information and judging the missing condition of the data frame; according to the invention, algorithms such as AI image generation dynamic interpolation and the like are flexibly applied to make up for system instability caused by external noise, and when input and output equipment has irreconcilable clock skew, the system can adjust the preset cache amount to ensure that the system is not stuck in live broadcast time to cause instability of an output end; when multiple devices are synchronized, the system has a natural solution that clocks of different devices are not consistent in a self-adaptive mode, and even if the system is used in cooperation with the device without a clock synchronization signal, the data consistency of output streams can be effectively guaranteed.

Description

Technical Field

[0001] The present application belongs to the field of video signal processing technology, and specifically, to an adaptive real-time video processing system based on AI automatic interpolation. Background Art

[0002] In the process of video stream transmission and broadcasting through different devices, there are generally two ways. The first is that all devices have synchronization signals, such as broadcasting and television. The second is that the devices lack a unified synchronization signal, and software and hardware cooperation are required to ensure that the transmitted video does not lose frames. Now that live video and VR camera equipment are widely used, this demand is becoming more and more urgent. Among them, the second method of transmission and broadcasting is widely used due to its advantages of fast response and high clarity.

[0003] However, the above-mentioned transmission broadcasting method still has certain shortcomings in daily use. During the transmission process, the input data stream enters the system, and then is provided to the output device after being processed by the system cache. However, there may be delays between the input devices, and audio and video may be out of sync, freeze, repeated frames, etc.; at the same time, during VR multicast, the VR video data stream is transferred to PCIE through the SDI interface card on the computer to establish a video transmission channel. During the transmission process of the above-mentioned transmission method, due to the large amount of transmitted data, the upstream and downstream devices do not have accurate clock synchronization signals when routing, and time consistency may not be achieved. Summary of the invention

[0004] In order to solve the above problems and technical defects, the embodiment of the present application adopts the following technical solution, an adaptive real-time video processing system based on AI automatic interpolation, including:

[0005] Input module: used to input data flow into the system;

[0006] Analysis and processing module: used to analyze the input data stream information and determine the missing status of data frames according to the input data stream information;

[0007] Cache processing module: used for receiving cache input data stream and providing cache processed data stream to output device;

[0008] Missing frame generation module: used to generate missing frames of images;

[0009] Insertion module: Insert the generated missing frames into the cache processing module.

[0010] Furthermore, the analysis and processing results of the analysis and processing module include: a normal state of input-output synchronization and a delayed state of input-output asynchronization.

[0011] Preferably, the normal state of the input-output synchronization is specifically: the data flow information of the input end is synchronized with the data flow information of the output end, and the buffer amount consumed by the output within the system buffer range is consistent with the number of input frames.

[0012] Furthermore, the delay state of the input-output asynchronism is specifically: there is a delay in the input and a white blank frame appears.

[0013] Furthermore, when the input is delayed and white blank frames appear, the insertion module inserts missing frames into the input data to offset them.

[0014] Furthermore, the specific production steps of the missing frame generation module include:

[0015] S1, collect image frames in the cache;

[0016] S2, detecting the similarity of the image frames on both sides of the interpolation time point, and comparing it with the threshold;

[0017] S3, when the similarity of the image frames on both sides of the time point is greater than the threshold, multiple data frames on the left and right sides of the time point to be interpolated are used;

[0018] S4, when the similarity of the image frames on both sides of the time point is less than the threshold, the subsequent data frames after the specified time point are used;

[0019] S5, generating a video frame using an interpolation algorithm;

[0020] S6, inputting the data frame generated by interpolation into the AI ​​diffusion model as a seed image;

[0021] S7. After being processed by the model, an interpolated frame with noise removed is output.

[0022] Compared with the prior art, the beneficial effects of the embodiments of the present application are:

[0023] (1) This application compensates for the system instability caused by external noise by flexibly using algorithms such as AI image generation and dynamic interpolation. When the input and output devices have irreconcilable clock deviations, the system can dynamically adjust the pre-set cache amount to ensure that the system will not freeze during the live broadcast time, causing instability at the output end; when multiple devices are synchronized, the system has a natural solution to adaptively adapt to the inconsistent clocks of different devices, and can effectively ensure the data consistency of the output stream even when used in conjunction with devices without clock synchronization signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In the attached picture:

[0025] Figure 1 A video system topology diagram of an embodiment of the present application;

[0026] Figure 2 This is a schematic diagram of the data frame cached by the system in the embodiment of the present application. Figure 1 ;

[0027] Figure 3 This is a schematic diagram of the data frame cached by the system in the embodiment of the present application. Figure 2 ;

[0028] Figure 4 This is a schematic diagram of the data frame cached by the system in the embodiment of the present application. Figure 3 ;

[0029] Figure 5 This is a schematic diagram of the cache situation between different devices in the embodiment of the present application;

[0030] Figure 6 This is a flowchart of generating image missing frames in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0032] Example:

[0033] like Figure 1 As shown, the adaptive real-time video processing system based on AI automatic interpolation includes:

[0034] Input module: used to input data flow into the system;

[0035] Analysis and processing module: used to analyze the input data stream information and determine the missing status of data frames according to the input data stream information;

[0036] Cache processing module: used for receiving cache input data stream and providing cache processed data stream to output device;

[0037] Missing frame generation module: used to generate missing frames of images;

[0038] Insertion module: Insert the generated missing frames into the cache processing module.

[0039] Reference Figure 2 , Figure 3 , Figure 4 ,The analysis and processing results of the analysis and processing module include: a normal state of input and output synchronization, and a delayed state of input and output asynchrony.

[0040] Reference Figure 2 The normal state of input-output synchronization is as follows: the data stream information of the input end is synchronized with the data stream information of the output end, and the amount of cache consumed by the output within the system cache range is consistent with the number of input frames.

[0041] Reference Figure 3 ,The delay state of input and output asynchrony is as follows: there is a delay in the input and a white blank frame appears.

[0042] When the input is delayed and white blank frames appear, the insertion module inserts the missing frames into the input data to offset them.

[0043] There are two ways to insert missing frames:

[0044] One is that there are relatively few white blank interval frames. The system will use the AI ​​image generation algorithm to insert a similar frame based on the two closest familiar images in the cached data, thereby offsetting the clock inconsistency introduced by environmental noise.

[0045] It should be noted that the AI ​​image algorithm is any one of GLIDE, DALL-E, and GAN.

[0046] GLIDE is an image generation algorithm that is based on diffusion modeling rather than generative adversarial networks or VQ-VAE-like models; diffusion modeling is a generative model paradigm that corrupts data by gradually adding noise and then learns the inverse process to recover the original data from the noise.

[0047] DALL-E is a deep learning-based image generation model that can generate corresponding images based on the text description provided by the user;

[0048] Generative Adversarial Network (GAN) is a deep learning algorithm that consists of two main components: a generator and a discriminator; the task of the generator is to generate data samples that are as realistic as possible, while the task of the discriminator is to distinguish the generated samples from real data; these two components continue to evolve in the process of mutual confrontation, ultimately enabling the generator to generate highly realistic images.

[0049] The other is that the interval between white blank frames is too long, which may be caused by some special reasons, such as large clock difference, occasional freeze of input signal, etc.

[0050] When this happens, we need to leave enough cache margin at the beginning of the system. The amount of this cache margin is set by the starting interface. The expected frame rate of each input device, the actual frame rate of the video system, and the expected output frame rate of the output device are set and automatically calculated.

[0051] Reference Figure 4,When the input device is a multiple of the frame number of the output device, the adaptive system will dynamically sample the frame number that meets the output device requirements according to the expected clock interval and finally output it.

[0052] Since the system will either get the input frame from the input device or automatically generate a blank data frame at a fixed time, multiple different devices can be synchronized even in the absence of a clock synchronization signal.

[0053] Reference Figure 5 , the input frame data in device 1 is later than the input frame of device 2 and there are blank data frames caused by noise. At this time, the system can automatically select time point A as the starting point of the entire output stream according to the cache input situation. This avoids the problem that all devices must synchronize the clock line, because the system will generate data frames according to the set time. If special processing of data and cache is required, you can refer to Figure 2-Figure 4 The single device processing method is used to ensure the stable operation of the system.

[0054] Reference Figure 6 ,The specific production steps of the missing frame generation module include:

[0055] S1, collect image frames in the cache;

[0056] S2, detecting the similarity of the image frames on both sides of the interpolation time point, and comparing it with the threshold;

[0057] S3, when the similarity of the image frames on both sides of the time point is greater than the threshold, multiple data frames on the left and right sides of the time point to be interpolated are used;

[0058] S4, when the similarity of the image frames on both sides of the time point is less than the threshold, the subsequent data frames after the specified time point are used;

[0059] S5, generating a video frame using an interpolation algorithm;

[0060] S6, inputting the data frame generated by interpolation into the AI ​​diffusion model as a seed image;

[0061] S7. After being processed by the model, an interpolated frame with noise removed is output.

[0062] This system flexibly uses AI image generation, dynamic interpolation and other algorithms to compensate for system instability caused by external noise. When the input and output devices have irreconcilable clock deviations, the system can dynamically adjust the pre-set cache amount to ensure that the system will not be stuck during the live broadcast time, causing instability on the output end; when multiple devices are synchronized, the system has a natural solution to adaptively adapt to the inconsistent clocks of different devices, and can effectively ensure the data consistency of the output stream even when used in conjunction with devices that do not have clock synchronization signals.

[0063] The above-mentioned embodiments only express the preferred implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for ordinary technicians in this field, several modifications, improvements and substitutions can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. Adaptive real-time video processing system based on AI automatic interpolation, characterized by: include: Input module: used to input data flow into the system; Analysis and processing module: used to analyze the input data stream information and determine the missing status of data frames according to the input data stream information; Cache processing module: used for receiving cache input data stream and providing cache processed data stream to output device; Missing frame generation module: used to generate missing frames of images; Insertion module: Insert the generated missing frames into the cache processing module.

2. The adaptive real-time video processing system based on AI automatic interpolation according to claim 1 is characterized in that: The analysis and processing results of the analysis and processing module include: a normal state in which input and output are synchronized, and a delayed state in which input and output are not synchronized.

3. The adaptive real-time video processing system based on AI automatic interpolation according to claim 2 is characterized in that: The normal state of the input-output synchronization is specifically: the data flow information of the input end is synchronized with the data flow information of the output end, and the buffer amount consumed by the output within the system buffer range is consistent with the number of input frames.

4. The adaptive real-time video processing system based on AI automatic interpolation according to claim 2 is characterized in that: The delay state of the input-output asynchronism is specifically: there is a delay in the input and a white blank frame appears.

5. The adaptive real-time video processing system based on AI automatic interpolation according to claim 4 is characterized in that: When the input is delayed and white blank frames appear, the insertion module inserts the missing frames into the input data to offset them.

6. The adaptive real-time video processing system based on AI automatic interpolation according to claim 1, characterized in that: The specific production steps of the missing frame generation module include: S1, collect image frames in the cache; S2, detecting the similarity of the image frames on both sides of the interpolation time point, and comparing it with the threshold; S3, when the similarity of the image frames on both sides of the time point is greater than the threshold, multiple data frames on the left and right sides of the time point to be interpolated are used; S4, when the similarity of the image frames on both sides of the time point is less than the threshold, the subsequent data frames after the specified time point are used; S5, generating a video frame using an interpolation algorithm; S6, inputting the data frame generated by interpolation into the AI ​​diffusion model as a seed image; S7. After being processed by the model, an interpolated frame with noise removed is output.

Images