A universal video temporal alignment method based on neural network

Abstract

The present invention provides a general video time domain alignment method based on a neural network, comprising the steps of: collecting all original video image frames of the current video; processing the original video image frames through an image processing neural network model to obtain processed image frames; constructing a deep convolutional neural network that can be used to align the time domain between video image frames; using the original video image frames and the processed image frames as input, and obtaining output time domain aligned video image frames through the deep convolutional neural network; synthesizing the output time domain aligned video image frames to obtain a complete video with final time domain alignment. The present invention solves the problem of video time domain inconsistency caused by directly applying an image processing model to a video task, and has the advantage of using a general algorithm to solve the time domain inconsistency problem of multiple different video tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of video image processing, and in particular to a universal video time domain alignment method based on a neural network. Background Art

[0002] In recent years, with the rapid development of computer vision, various neural network-based image processing tasks have made major breakthroughs, and many image processing algorithms have shown excellent performance in single image processing tasks. For example, in image denoising tasks, a noise-filled image can be converted into a clear, noise-free image through a carefully designed neural network, achieving excellent visual effects. In real applications, video scenes are more in line with actual needs. Currently, most video tasks are processed by extracting the video into several image frames, and then processing them frame by frame using an image processing neural network model, and finally synthesizing the processed image frames to obtain the processed video. The video quality obtained by this processing method is generally poor because the frame-by-frame processing method causes the time domain between the generated video image frames to be uneven, which can easily lead to problems such as video flickering and incoherent video details, affecting the overall visual effect of the video.

[0003] In order to ensure the temporal consistency of videos, some researchers have designed special algorithms for different video processing tasks, such as video colorization, video denoising, and video enhancement. Although task-specific video processing algorithms can improve the temporal consistency of videos processed by specific tasks to a certain extent, similar strategies cannot be directly applied to other tasks, which has great limitations. The need to design different processing algorithms for different video processing tasks also increases the R&D cost and implementation cost in practical applications. Summary of the invention

[0004] In view of this, the purpose of the present invention is to propose a universal video time domain alignment method based on neural network, which solves the problem of video time domain inconsistency caused by directly applying image processing models to video tasks, and has the advantage of using a universal algorithm to solve the time domain inconsistency problem of multiple different video tasks.

[0005] The present invention is implemented by the following scheme: a general video time domain alignment method based on neural network, specifically comprising the following steps:

[0006] Collect all original video image frames of the current video;

[0007] Processing the original video image frame through an image processing neural network model to obtain a processed image frame;

[0008] Build a deep convolutional neural network that can be used to align the temporal domains of video image frames;

[0009] Using the original video image frame and the processed image frame as input, and obtaining an output time-domain aligned video image frame through the deep convolutional neural network;

[0010] The output time-domain aligned video image frames are synthesized to obtain a final time-domain aligned complete video.

[0011] Furthermore, the image processing neural network model includes an image enhancement model, an image denoising model, an image defogging model, and an image colorization model.

[0012] Furthermore, the deep convolutional neural network used to align the time domain between video image frames is a U-Net image transformation network that integrates a ConvLSTM convolutional long short-term memory unit layer.

[0013] Furthermore, the U-Net image transformation network is an encoder-decoder architecture, including four downsampling and four upsampling operations, forming a U-shaped structure; after the fourth downsampling operation, a ConvLSTM convolutional long short-term memory unit layer is connected, and then an upsampling operation is performed.

[0014] Furthermore, the ConvLSTM convolutional long short-term memory unit layer includes a forget gate, an input gate, and an output gate. The forget gate is based on the current input And the output of the previous moment Determine which part needs to be forgotten; the input gate is based on the current input And the output of the previous moment Decide what information to add to the previous state Generate a new state The output gate is based on the latest state Output at the last moment and the current input To determine the output at that moment

[0015] Furthermore, the method of using the original video image frame and the processed image frame as input and obtaining the output time-domain aligned video image frame through the deep convolutional neural network is specifically:

[0016] When t is the first frame, set t-1=t, t-2=t; when t is the second frame, set t-1=t; when t is the last frame, set t+1=t;

[0017] For the tth original video image frame, taking the t-1th original video image frame, the t-1th processed image frame, the tth original video image frame and the tth processed image frame as input at the same time, obtaining the tth aligned image frame that is initially stable in the time domain through the deep convolutional neural network; calculating the supervised learning training function of the tth processed image frame and the tth aligned image frame;

[0018] For the t-1th original video image frame, the t-2th original video image frame, the t-2th processed image frame, the t-1th original video image frame and the t-1th processed image frame are taken as input at the same time, and the t-1th aligned image frame with preliminary time domain stability is obtained through the deep convolutional neural network, and the supervised learning training function of the t-1th processed image frame and the t-1th aligned image frame is calculated;

[0019] For the t+1th original video image frame, the tth processed image frame, the t+1th original video image frame and the t+1th processed image frame are taken as input at the same time, a t+1th aligned image frame with preliminary temporal stability is obtained through the deep convolutional neural network, and a supervised learning training function of the t+1th processed image frame and the t+1th aligned image frame is calculated;

[0020] The calculated supervised learning training function of the t-1th processed image frame and the t-1th aligned image frame, the supervised learning training function of the tth processed image frame and the tth aligned image frame, and the supervised learning training function of the t+1th processed image frame and the t+1th aligned image frame are added in a ratio of 1:1:1 to obtain the total supervised learning training function of the deep convolutional neural network finally used to optimize the processing of the tth original video image frame.

[0021] Further, the supervised learning training function includes a perceptual loss function, a structural similarity loss function and a total training function;

[0022] The perceptual loss function is as follows:

[0023]

[0024] In the formula, the subscript j represents the jth layer of the network, C j H j W j represents the size of the feature map of the jth layer; the loss network uses the VGG-19 pre-trained network, and φ represents the network; y represents the processed image frame, represents the aligned image frame;

[0025] The structural similarity loss function is as follows:

[0026]

[0027] In the formula, x, y represent the processed image frame and the aligned image frame respectively, μ x and μ y Represent the average values ​​of x and y respectively. and Represents the variance of x and y, σ xy represents the covariance of x and y, c1 and c2 are constants;

[0028] The total training function is:

[0029] L=L t娰1 +L t +L t+1 ;

[0030] Among them, L t represents the loss function between the t-th processed image frame and the t-th aligned image frame, expressed as follows:

[0031]

[0032] In the formula, represents the perceptual loss function, represents the structural similarity loss function shown, represents the tth aligned image frame, y t represents the tth processed image frame;

[0033] L is finally used to optimize the overall supervised training function of the deep convolutional neural network.

[0034] The present invention also provides a universal video time domain alignment system based on a neural network, comprising a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0035] The present invention also provides a computer-readable storage medium on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] First, the present invention proposes for the first time to use a universal algorithm to solve the problem of video time domain inconsistency in different video processing tasks, making it possible to directly apply the current image processing neural network model with excellent performance to the corresponding video processing tasks, which can meet engineering needs and greatly reduce the traditional R&D and application costs of using different processing algorithms according to different video tasks.

[0038] Second, the present invention adopts a novel neural network structure, which combines a convolutional neural network and a recurrent neural network. The neural network has excellent image processing capabilities and the ability to capture spatiotemporal correlations, so that the video image frames processed by the neural network structure have spatiotemporal consistency.

[0039] Third, the present invention designs a sophisticated supervised training function, which combines the perceptual loss function and the structural similarity loss function, so that the final output aligned image frame can achieve the expected processing effect.

[0040] Fourth, the method provided by the present invention can satisfy the need to directly apply image processing models to video processing tasks in the field of computer vision without causing sensory problems such as video flickering and discontinuous video details, thereby satisfying the real experience of users in real life and gaining recognition from users. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The figure is a schematic diagram of a method flow of an embodiment of the present invention.

[0042] Figure 2 The network structure and training function of the neural network-based universal video time-domain alignment method according to an embodiment of the present invention.

[0043] Figure 3 Schematic diagram of the structure of the ConvLSTM long short-term memory unit layer in the network structure of the general video time domain alignment method based on neural network in an embodiment of the present invention.

[0044] Figure 4 This is a comparison chart of the effects of applying the neural network-based universal video time-domain alignment method according to an embodiment of the present invention to a video colorization task.

[0045] Figure 5 This is a comparison diagram of the effects of applying the neural network-based universal video time-domain alignment method according to an embodiment of the present invention to a video denoising task. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0047] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0048] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0049] like Figure 1 As shown, this embodiment provides a general video time domain alignment method based on a neural network, which specifically includes the following steps:

[0050] Step 1: Collect all original video image frames of the current video; specifically, the original video image frames may be black-and-white video image frames collected from a black-and-white video or noisy video image frames collected from a noisy video.

[0051] Step 2: Processing the original video image frame through an image processing neural network model to obtain a processed image frame;

[0052] Step 3: Construct a deep convolutional neural network that can be used to align the temporal domain between video image frames;

[0053] Step 4: using the original video image frame and the processed image frame as input, and obtaining an output time-domain aligned video image frame through the deep convolutional neural network;

[0054] Step 5: synthesize the output time-domain aligned video image frames to obtain a final time-domain aligned complete video.

[0055] In this embodiment, the image processing neural network model includes neural network models for image processing in the field of computer vision such as image enhancement model, image denoising model, image defogging model, image coloring model, etc. Specifically, the processed image frame can be a color video image frame after black and white video is processed by the image coloring model or a noise-free video image frame after noisy video is processed by the image denoising model.

[0056] In this embodiment, the deep convolutional neural network used to align the time domain between video image frames is a U-Net image transformation network that integrates a ConvLSTM convolutional long short-term memory unit layer.

[0057] In this embodiment, if Figure 2 As shown, the U-Net image transformation network is an encoder-decoder architecture, which includes four downsampling and four upsampling operations to form a U-shaped structure; after the fourth downsampling operation, a ConvLSTM convolutional long short-term memory unit layer is connected, and then an upsampling operation is performed.

[0058] Among them, the downsampling operation is implemented by the maximum pooling layer, and the upsampling operation is implemented by deconvolution.

[0059] In this embodiment, if Figure 3 As shown in the figure, the ConvLSTM convolutional long short-term memory unit layer is mainly composed of three "gate" structures, including a forget gate, an input gate, and an output gate. Each "gate" structure is an operation using a sigmoid neural network and a bitwise multiplication.

[0060] The forget gate is based on the current input And the output of the previous moment Determine which part needs to be forgotten; the input gate is based on the current input And the output of the previous moment Decide what information to add to the previous state Generate a new state The output gate is based on the latest state Output at the last moment and the current input To determine the output at that moment

[0061] The specific implementation formula of the ConvLSTM convolutional long short-term memory unit layer is as follows:

[0062]

[0063]

[0064]

[0065]

[0066]

[0067] Among them, i t represents the input gate, f t represents the forget gate, Represents the updated status, o t represents the output gate, Represents the output, is the current input. xi represents the input weight of the input gate, W hi represents the output weight of the input gate, W ci represents the updated state weight of the input gate, b i represents the bias term of the input gate, W hf represents the output weight of the forget gate, W cf represents the updated state weight of the forget gate, b fRepresents the bias term of the forget gate, W xc represents the input weight of the updated state, W hc represents the output weight of the updated state, b c represents the bias term of the updated state, W So Represents the input weight of the output gate, W ho represents the output weight of the output gate, W co represents the updated state weight of the output gate, b o Represents the bias term of the output gate.

[0068] In this embodiment, the original video image frame and the processed image frame are used as input, and the output time-domain aligned video image frame is obtained through the deep convolutional neural network as follows:

[0069] When t is the first frame, set t-1=t, t-2=t; when t is the second frame, set t-1=t; when t is the last frame, set t+1=t;

[0070] For the tth original video image frame, taking the t-1th original video image frame, the t-1th processed image frame, the tth original video image frame and the tth processed image frame as input at the same time, obtaining the tth aligned image frame that is initially stable in the time domain through the deep convolutional neural network; calculating the supervised learning training function of the tth processed image frame and the tth aligned image frame;

[0071] For the t-1th original video image frame, the t-2th original video image frame, the t-2th processed image frame, the t-1th original video image frame and the t-1th processed image frame are taken as input at the same time, and the t-1th aligned image frame with preliminary time domain stability is obtained through the deep convolutional neural network, and the supervised learning training function of the t-1th processed image frame and the t-1th aligned image frame is calculated;

[0072] For the t+1th original video image frame, the tth processed image frame, the t+1th original video image frame and the t+1th processed image frame are taken as input at the same time, a t+1th aligned image frame with preliminary temporal stability is obtained through the deep convolutional neural network, and a supervised learning training function of the t+1th processed image frame and the t+1th aligned image frame is calculated;

[0073] The calculated supervised learning training function of the t-1th processed image frame and the t-1th aligned image frame, the supervised learning training function of the tth processed image frame and the tth aligned image frame, and the supervised learning training function of the t+1th processed image frame and the t+1th aligned image frame are added in a ratio of 1:1:1 to obtain the total supervised learning training function of the deep convolutional neural network finally used to optimize the processing of the tth original video image frame.

[0074] In this embodiment, the supervised learning training function includes a perceptual loss function, a structural similarity loss function, and a total training function;

[0075] The perceptual loss function is to compare the aligned image frame processed by the neural network with the processed image frame to calculate the perceptual loss function. The formula of the perceptual function is as follows:

[0076]

[0077] In the formula, the subscript j represents the jth layer of the network, C j H j W j represents the size of the feature map of the jth layer; the loss network uses the VGG-19 pre-trained network, and φ represents the network; y represents the processed image frame, represents the aligned image frame;

[0078] The structural similarity loss function compares the aligned image frame processed by the deep convolutional neural network with the processed image frame to calculate the structural similarity loss function. The formula of the structural similarity loss function is as follows:

[0079]

[0080] In the formula, x, y represent the processed image frame and the aligned image frame respectively, μ x and μ y Represent the average values ​​of x and y respectively. and Represents the variance of x and y, σ xy represents the covariance of x and y, C1 and c2 are constants;

[0081] The total training function compares the t-1th aligned image frame and the t-1th processed image frame, the tth aligned image frame and the tth processed image frame, and the t+1th aligned image frame and the t+1th processed image frame after being processed by the deep convolutional neural network, calculates the total training function, and continuously optimizes it. The total training function is specifically:

[0082] L=L t娰1 +L t +L t+1 ;

[0083] Among them, L t represents the loss function between the t-th processed image frame and the t-th aligned image frame, expressed as follows:

[0084]

[0085] In the formula, represents the perceptual loss function, represents the structural similarity loss function shown, represents the tth aligned image frame, y t represents the tth processed image frame;

[0086] L is finally used to optimize the overall supervised training function of the deep convolutional neural network.

[0087] Finally, the output time-domain aligned video image frames are synthesized to obtain a complete video with consistent time domain. Specifically, the complete video can be a time-domain aligned color video or a time-domain aligned noise-free video. Figure 4 and Figure 5 As shown in the result comparison diagram of the video colorization task and the result comparison diagram of the video denoising task, it can be seen that there are obvious flickering and abrupt detail changes between the processed image frames with inconsistent time domains, which affect the visual effect. The general video time domain alignment method proposed in the present invention can very effectively perform time domain alignment between video image frames while ensuring the processing effect of video image frames, thereby achieving the purpose of video flicker removal and video detail coherence, and finally obtaining a video with excellent visual effects.

[0088] This embodiment also provides a general video time domain alignment system based on a neural network, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0089] This embodiment further provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0090] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0091] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0094] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A general video temporal alignment method based on neural network, characterized in that: The following steps are involved: Collect all original video image frames of the current video; Processing the original video image frame through an image processing neural network model to obtain a processed image frame; Build a deep convolutional neural network that can be used to align the temporal domains of video image frames; Using the original video image frame and the processed image frame as input, and obtaining an output time-domain aligned video image frame through the deep convolutional neural network; Synthesizing the output time-domain aligned video image frames to obtain a final time-domain aligned complete video; The deep convolutional neural network used to align the temporal domain between video image frames is a U-Net image transformation network that integrates a ConvLSTM convolutional long short-term memory unit layer; The U-Net image transformation network is an encoder-decoder architecture, including four downsampling and four upsampling operations, forming a U-shaped structure; after the fourth downsampling operation, a ConvLSTM convolutional long short-term memory unit layer is connected, and then an upsampling operation is performed; The method of using the original video image frame and the processed image frame as input and obtaining the output time-domain aligned video image frame through the deep convolutional neural network is specifically as follows: When t is the first frame, set t-1=t, t-2=t; when t is the second frame, set t-1=t; when t is the last frame, set t+1=t; For the tth original video image frame, taking the t-1th original video image frame, the t-1th processed image frame, the tth original video image frame and the tth processed image frame as input at the same time, obtaining the tth aligned image frame that is initially stable in the time domain through the deep convolutional neural network; calculating the supervised learning training function of the tth processed image frame and the tth aligned image frame; For the t-1th original video image frame, the t-2th original video image frame, the t-2th processed image frame, the t-1th original video image frame and the t-1th processed image frame are taken as input at the same time, and the t-1th aligned image frame with preliminary time domain stability is obtained through the deep convolutional neural network, and the supervised learning training function of the t-1th processed image frame and the t-1th aligned image frame is calculated; For the t+1th original video image frame, the tth processed image frame, the t+1th original video image frame and the t+1th processed image frame are taken as input at the same time, a t+1th aligned image frame with preliminary temporal stability is obtained through the deep convolutional neural network, and a supervised learning training function of the t+1th processed image frame and the t+1th aligned image frame is calculated; The calculated supervised learning training function of the t-1th processed image frame and the t-1th aligned image frame, the supervised learning training function of the tth processed image frame and the tth aligned image frame, and the supervised learning training function of the t+1th processed image frame and the t+1th aligned image frame are added in a ratio of 1:1:1 to obtain a total supervised learning training function of the deep convolutional neural network for optimizing the processing of the tth original video image frame; The supervised learning training function includes a perceptual loss function, a structural similarity loss function and a total training function; The perceptual loss function is as follows: In the formula, the subscript j represents the jth layer of the network, C j H j W j represents the size of the feature map of the jth layer; the loss network uses the VGG-19 pre-trained network, and φ represents the network; y represents the processed image frame, represents the aligned image frame; The structural similarity loss function is as follows: In the formula, x, y represent the processed image frame and the aligned image frame respectively, μ x and μ y Represent the average values ​​of x and y respectively. and Represents the variance of x and y respectively, σ xy represents the covariance of x and y, c1 and c2 are constants; The total training function is: L=L t-1 +L t +L t+1 ; Among them, L t represents the loss function between the t-th processed image frame and the t-th aligned image frame, expressed as follows: In the formula, represents the perceptual loss function, represents the structural similarity loss function, represents the tth aligned image frame, y t represents the tth processed image frame; L is finally used to optimize the overall supervised training function of the deep convolutional neural network.

2. The neural network-based universal video temporal alignment method according to claim 1, characterized in that: The image processing neural network model includes an image enhancement model, an image denoising model, an image defogging model or an image coloring model.

3. The neural network-based universal video temporal alignment method according to claim 1, characterized in that: The ConvLSTM convolutional long short-term memory unit layer includes a forget gate, an input gate and an output gate. The forget gate is based on the current input And the output of the previous moment Determine which part needs to be forgotten; the input gate is based on the current input And the output of the previous moment Decide what information to add to the previous state Generate a new state The output gate is based on the latest state Output at the last moment and the current input To determine the output at that moment 4. A general video temporal alignment system based on neural network, characterized in that: The method comprises a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the method according to any one of claims 1 to 3 can be implemented.

5. A computer-readable storage medium storing computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, the method according to any one of claims 1 to 3 can be implemented.

Images