Processing method, training method, device, electronic equipment and medium

Abstract

The application discloses a video frame image processing method and device, electronic equipment and storage medium. The processing method comprises the following steps: obtaining a to-be-processed video frame image and a neighboring video frame image of the to-be-processed video frame image, and processing the to-be-processed video frame image and the neighboring video frame image by using an image processing algorithm model to obtain a target video frame image, wherein the image processing algorithm model comprises an optical flow network and an enhancement network, and the image processing algorithm model is obtained by training in multiple training stages by using multiple loss functions in sequence. In the processing method, the to-be-processed video frame and the neighboring video frame image are processed by using the image processing algorithm model composed of the optical flow network and the enhancement network, so that the compression picture of the current frame can be repaired by using the neighboring frame information, the problems of video frame image blurring, artifacts, color blocks, noise and the like are effectively improved, and the display effect is improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a method for processing video frame images, a processing device, a training method for image processing algorithm models, a training device, an electronic device, and a storage medium. Background Technology

[0002] To improve video file transmission efficiency and reduce file size, video files are typically compressed. However, compression can easily cause problems such as blurring, artifacts, and color banding, affecting the display quality of the video file. Summary of the Invention

[0003] This application aims to at least solve one of the technical problems existing in the prior art. To this end, this application provides a method for processing video frame images, a method for training an image processing algorithm model, a device for processing video frame images, a device for training an image processing algorithm model, an electronic device, and a storage medium.

[0004] The video frame image processing method of the embodiments of this application includes:

[0005] Obtain the video frame image to be processed and the adjacent video frame images of the video frame image to be processed;

[0006] The target video frame image is obtained by processing the video frame image to be processed and the adjacent video frame images using an image processing algorithm model.

[0007] The image processing algorithm model includes an optical flow network and an enhancement network, and is obtained by training the model sequentially using multiple loss functions in multiple training stages.

[0008] In some embodiments, the step of processing the video frame image to be processed and the adjacent video frame images using an image processing algorithm model to obtain the target video frame image includes:

[0009] Optical flow information is obtained by processing the video frame image to be processed and the adjacent video frame images through the optical flow network;

[0010] The optical flow information is used to process the adjacent video frame images to obtain the predicted video frame images;

[0011] The optical flow information, the predicted video frame image, and the video frame image to be processed are combined to obtain the enhanced input video frame image;

[0012] The target video frame image is obtained by processing the enhanced input video frame image through the enhancement network.

[0013] In some embodiments, processing the video frame image to be processed and the adjacent video frame images through the optical flow network to obtain optical flow information includes:

[0014] Multi-level concatenation data is obtained based on the video frame image to be processed and the adjacent video frame images. The first level of concatenation data is obtained by connecting the video frame image to be processed and the adjacent video frame images.

[0015] The multi-level connection data is processed to obtain multi-level sub-optical flow information. Each level of data processing includes downsampling processing, first convolution processing, and upsampling processing. The upsampling processing factor of each level is twice the downsampling processing factor. The downsampling processing factor of the previous level is twice the downsampling processing factor of the next level, and the upsampling processing factor of the previous level is twice the downsampling processing factor of the next level.

[0016] The video frame image to be processed and the sub-optical flow information of the multiple levels are estimated to obtain multi-level sub-prediction images. The concatenation data of each level after the first level is obtained by concatenating the sub-prediction image of the previous level and the adjacent video frame image.

[0017] The optical flow information is obtained by adding the sub-optical flow information of the multiple levels together.

[0018] In some embodiments, processing the enhanced input video frame image through the enhancement network to obtain the target video frame image includes:

[0019] The predicted video frame image and the video frame image to be processed are concatenated and processed by a second convolution to obtain a first feature map.

[0020] The adjacent video frame images are processed by a third convolution and then subjected to an image affine transformation with the optical flow information to obtain a second feature map and a third feature map.

[0021] The first feature map, the second feature map, and the third feature map are concatenated and processed by a fourth convolution to obtain the fourth feature map;

[0022] The target video frame image is obtained by adaptive attention processing of the fourth feature map and then by fifth convolution processing.

[0023] In some embodiments, acquiring the video frame image to be processed and its adjacent video frame images includes:

[0024] The video to be processed is segmented to obtain at least one sub-video, and the sub-video includes multiple consecutive video frame images with the same scene;

[0025] The first and last frames of the sub-video are copied and supplemented by frame copying;

[0026] The video frame image to be processed and two adjacent video frame images adjacent to the video frame image to be processed are obtained from the sub-video.

[0027] In some embodiments, segmenting the video to be processed to obtain at least one sub-video includes:

[0028] Convert two adjacent frames of the video to be processed to a preset color space;

[0029] Obtain the histogram of the preset color channels of two adjacent frames of the video to be processed in the preset color space;

[0030] If the histograms of two adjacent frames of the video to be processed corresponding to the preset color channels meet the preset conditions, then the two adjacent frames of the video to be processed are determined to be consecutive video frame images of the same sub-video.

[0031] In some implementations, the video to be processed includes a compressed and remastered video.

[0032] In the training method of the image processing algorithm model in the embodiments of this application, the image processing algorithm model includes an optical flow network and an enhancement network, and the training method includes:

[0033] Obtain the training video frame image and the adjacent training video frame images of the training video frame image;

[0034] The image processing algorithm model is constructed to process the training video frame image and the adjacent training video frame image in multiple training stages to obtain the training output video frame image.

[0035] In multiple training phases, based on the training output video frame images, the loss value of the image processing algorithm model is calculated using multiple loss functions respectively;

[0036] The parameters of the image processing algorithm model in the corresponding training stage are corrected based on the loss value of the image processing algorithm model.

[0037] In some implementations, the step of processing the training video frame images and the adjacent training video frame images through the constructed image processing algorithm model in multiple training stages to obtain training output video frame images includes:

[0038] In the multiple training phases, based on the training output video frame images, the loss value of the image processing algorithm model is calculated using multiple loss functions, including:

[0039] In the first training phase, based on the first training output video frame image, the loss value of the image processing algorithm model is calculated using a first loss function;

[0040] The step of correcting the parameters of the image processing algorithm model in the corresponding training stage based on the loss value of the image processing algorithm model includes:

[0041] The parameters of the image processing algorithm model in the first training phase are corrected based on the loss value of the image processing algorithm model calculated by the first loss function, so as to obtain the image processing algorithm model trained in the first training phase.

[0042] In some implementations, the first loss function is expressed as:

[0043]

[0044] in, The image is the ground truth image, and W, H, and C are the height, width, and number of channels of the first training output video frame image, respectively.

[0045] In some implementations, the step of processing the training video frame images and the adjacent training video frame images through the constructed image processing algorithm model in multiple training stages to obtain training output video frame images includes:

[0046] The image processing algorithm model trained in the first training phase is used to process the training video frame image and the adjacent training video frame image in the second training phase to obtain the second training output video frame image.

[0047] The discrimination network processes the second training output video frame image to obtain the discrimination output value;

[0048] In the multiple training phases, based on the training output video frame images, the loss value of the image processing algorithm model is calculated using multiple loss functions, including:

[0049] In the second training phase, based on the discrimination output value, the loss value of the image processing algorithm model is calculated using the second loss function;

[0050] The step of correcting the parameters of the image processing algorithm model in the corresponding training stage based on the loss value of the image processing algorithm model includes:

[0051] The parameters of the image processing algorithm model in the second training stage are corrected based on the loss value of the image processing algorithm model calculated by the second loss function, so as to obtain the image processing algorithm model trained in the second training stage.

[0052] In some implementations, the second loss function is expressed as:

[0053]

[0054] in, For the true value image, D(Y) is the discrimination output value of the ground truth image, and D(Y) is the discrimination output value of the second training output video frame image.

[0055] In some implementations, the step of processing the training video frame images and the adjacent training video frame images through the constructed image processing algorithm model in multiple training stages to obtain training output video frame images includes:

[0056] The image processing algorithm model trained in the second training phase is used to process the training video frame image and the adjacent training video frame image in the third training phase to obtain the third training output video frame image.

[0057] In the multiple training phases, based on the training output video frame images, the loss value of the image processing algorithm model is calculated using multiple loss functions, including:

[0058] In the third training phase, based on the third training output video frame image, the loss value of the image processing algorithm model is calculated using the third loss function;

[0059] The step of correcting the parameters of the image processing algorithm model in the corresponding training stage based on the loss value of the image processing algorithm model includes:

[0060] The parameters of the image processing algorithm model in the third training stage are corrected based on the loss value of the image processing algorithm model calculated by the third loss function, so as to obtain the trained image processing algorithm model.

[0061] In some implementations, the third loss function includes a Fast Fourier Transform loss function, which is expressed as:

[0062]

[0063]

[0064] Among them, X f Let X be the spectrum obtained by fast Fourier transform, and Im(X) be the spectrum obtained by fast Fourier transform. f ) is X f The imaginary part of , Re(X f ) is X f The real part of .

[0065] In some implementations, during the first training phase, the learning rate of the image processing algorithm model is a first preset value. During the second training phase, the learning rate of the image processing algorithm model is a second preset value, and the learning rate of the discrimination network is a third preset value, wherein the second preset value is less than the first preset value, and the second preset value is less than the third preset value.

[0066] In some implementations, acquiring the training video frame image and its adjacent training video frame images includes:

[0067] The original video is compressed and remade once or multiple times to obtain the training video;

[0068] The training video is segmented to obtain at least one training sub-video, and the training sub-video includes multiple consecutive video frame images with the same scene;

[0069] The first and last frames of the training sub-video are copied and padded using frame duplication.

[0070] In the training sub-video, obtain two adjacent training video frame images that are adjacent to the training video frame image.

[0071] The video frame image processing apparatus according to the embodiments of this application includes:

[0072] The first acquisition module is used to acquire the video frame image to be processed and the adjacent video frame images of the video frame image to be processed;

[0073] The processing module is used to process the video frame image to be processed and the adjacent video frame images through an image processing algorithm model to obtain the target video frame image;

[0074] The image processing algorithm model includes an optical flow network and an enhancement network, and is obtained by training the model sequentially using multiple loss functions in multiple training stages.

[0075] The training apparatus for the image processing algorithm model according to the embodiments of this application includes:

[0076] The second acquisition module is used to acquire training video frame images and adjacent training video frame images of the training video frame images;

[0077] The construction module is used to process the training video frame images and the adjacent training video frame images in multiple training stages through the constructed image processing algorithm model to obtain the training output video frame images;

[0078] The calculation module is used to calculate the loss value of the image processing algorithm model based on the training output video frame images in multiple training stages, respectively, through multiple loss functions;

[0079] The correction module is used to correct the parameters of the image processing algorithm model in the corresponding training stage based on the loss value of the image processing algorithm model.

[0080] The electronic device according to the embodiments of this application includes a processor, a memory, and a computer program, wherein the computer program is stored in the memory, and when the computer program is executed by the processor, the processor performs the above-described video frame image processing method and the above-described image processing algorithm model training method.

[0081] The non-volatile computer-readable storage medium of this application includes a computer program that, when executed by a processor, causes the processor to perform the above-described video frame image processing method and the above-described image processing algorithm model training method.

[0082] The video frame image processing method, processing apparatus, electronic device, and computer storage medium of this application utilize an image processing algorithm model composed of an optical flow network and an enhancement network to process the video frame to be processed and its adjacent video frame images. On one hand, by leveraging the similarity in features between adjacent video frame images and the current video frame image, a lightweight optical flow network can quickly calculate motion estimation between adjacent frames, extracting more reference information from adjacent video frame images to repair the current video frame image. This effectively improves problems such as blurring, artifacts, color blocks, and noise in video frame images, ensuring the accuracy of the video frame image content and enhancing the display effect. On the other hand, a generative adversarial network is used to construct an adversarial loss function, which, combined with other loss functions, trains the image processing algorithm model in stages, ensuring the accuracy of the image processing algorithm model and further improving image clarity.

[0083] Additional aspects and advantages of embodiments of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of embodiments of this application. Attached Figure Description

[0084] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0085] Figure 1 This is a flowchart illustrating a method for processing video frame images according to certain embodiments of this application;

[0086] Figure 2This is a schematic diagram of a video frame image processing apparatus according to certain embodiments of this application;

[0087] Figure 3 This is a schematic diagram of a video frame image processing method according to certain embodiments of this application;

[0088] Figure 4 This is a flowchart illustrating a method for processing video frame images according to certain embodiments of this application;

[0089] Figure 5 This is a schematic diagram of a video frame image processing method according to certain embodiments of this application;

[0090] Figure 6 This is a flowchart illustrating a method for processing video frame images according to certain embodiments of this application;

[0091] Figure 7 This is a schematic diagram of a video frame image processing method according to certain embodiments of this application;

[0092] Figure 8 This is a flowchart illustrating a method for processing video frame images according to certain embodiments of this application;

[0093] Figure 9 This is a schematic diagram of a video frame image processing method according to certain embodiments of this application;

[0094] Figure 10 This is a flowchart illustrating a method for processing video frame images according to certain embodiments of this application;

[0095] Figure 11 This is a schematic diagram of a video frame image processing method according to certain embodiments of this application;

[0096] Figure 12 This is a flowchart illustrating a method for processing video frame images according to certain embodiments of this application;

[0097] Figure 13 This is a flowchart illustrating the training method of an image processing algorithm model according to certain embodiments of this application;

[0098] Figure 14 This is a schematic diagram of a module of a training apparatus for an image processing algorithm model according to certain embodiments of this application;

[0099] Figure 15-17 This is a flowchart illustrating the training method of an image processing algorithm model according to certain embodiments of this application;

[0100] Figure 18 This is a schematic diagram of a scenario for training an image processing algorithm model according to certain embodiments of this application;

[0101] Figure 19 This is a flowchart illustrating the training method of an image processing algorithm model according to certain embodiments of this application. Detailed Implementation

[0102] The embodiments of the present application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present application, and should not be construed as limiting the implementation of the present application.

[0103] Please see Figure 1 This application provides a method for processing video frame images, the method including the following steps:

[0104] 01. Obtain the video frame image to be processed and its adjacent video frame images;

[0105] 02. The target video frame image is obtained by processing the video frame image to be processed and adjacent video frame images through the image processing algorithm model. The image processing algorithm model includes an optical flow network and an enhancement network. The image processing algorithm model is obtained by training multiple loss functions in multiple training stages.

[0106] Please see Figure 2 This application provides a video frame image processing device 10, which includes a first acquisition module 11 and a processing module 12.

[0107] Step 01 can be implemented by the first acquisition module 11, and step 02 can be implemented by the processing module 12. Alternatively, the first acquisition module 11 can be used to acquire the video frame image to be processed and the adjacent video frame images; the processing module 12 can be used to process the video frame image to be processed and the adjacent video frame images through an image processing algorithm model to obtain the target video frame image.

[0108] The electronic device according to the embodiments of this application includes a processor, a memory, and a computer program. When the computer program is executed by the processor, the processor is configured to acquire a video frame image to be processed and adjacent video frame images of the video frame image to be processed, and to process the video frame image to be processed and adjacent video frame images through an image processing algorithm model to obtain a target video frame image.

[0109] In the processing method, processing apparatus 10, and electronic device of this application, an image processing algorithm model composed of an optical flow network and an enhancement network processes the video frame to be processed and its adjacent video frame images. On the one hand, by utilizing the similarity in features between adjacent video frame images and the current video frame image, a lightweight optical flow network can quickly calculate motion estimation between adjacent frames, extracting more reference information from adjacent video frame images to repair the current video frame image. This effectively improves problems such as blurring, artifacts, color blocks, and noise in video frame images, ensuring the accuracy of video frame image content and improving the display effect of video frame images. On the other hand, an adversarial loss function is constructed using an enhancement network and a generative adversarial network, combined with various other loss functions, to train the image processing algorithm model in stages, further improving image clarity.

[0110] In some embodiments, the electronic device can be a terminal device such as a mobile phone, tablet, or personal computer. For example, in this application, a mobile phone can be used as an example for illustration. That is, the embodiments of this application can implement the above-described video frame image processing method using a mobile phone. In this way, the electronic device can implement the processing method of the embodiments of this application.

[0111] In some embodiments, the processing device 10 may be part of an electronic device. In other words, the electronic device includes the processing device 10.

[0112] In some embodiments, the processing device 10 may be a discrete component assembled in a certain way to have the aforementioned functions, or a chip having the aforementioned functions in the form of an integrated circuit, or a computer software code segment that enables the computer to have the aforementioned functions when run on a computer.

[0113] A video frame image can be a video frame image obtained after decoding a compressed video. Understandably, when an electronic device downloads a video, the video file is usually very large, causing the download time to be too long and occupying space on the electronic device. Therefore, video files are often compressed and re-processed multiple times to obtain a compressed video before being transmitted to the electronic device.

[0114] An image processing algorithm model is a mathematical model used to process and repair video frame images. The algorithm can be established based on preset logic and mathematical algorithms. The preset logic is business logic, which refers to the rules and processes that an entity should follow in order to provide services to another entity. The mathematical algorithm can be a deep learning network algorithm based on optical flow networks (FlowNet) and enhancement networks (EnhanceNet).

[0115] It should be noted that optical flow networks are a type of neural network that uses the temporal changes of pixels in an image sequence and the correlation between adjacent frames to find the correspondence between the previous frame and the current frame, thereby calculating the motion information of objects between adjacent frames.

[0116] The enhanced network uses a fully convolutional approach, allowing the input image to be of any size. All convolutional kernels are 3x3 in size. The network input is a low-resolution image, and the nearest neighbor method is used at the end of the network to upsample to achieve the size of a high-resolution image.

[0117] A loss function is used to estimate the degree of inconsistency between the predicted value f(x) and the true value Y of an image processing algorithm model. It is a non-negative real-valued function, usually denoted by L(Y,f(x)). The smaller the loss function, the better the robustness of the model. In this application, multiple loss functions are used. During the image processing algorithm model training, a staged training approach is adopted, and each training stage uses a different loss function to train the image processing algorithm model.

[0118] Please combine Figure 3 The adjacent video frames of the video frame to be processed refer to the two video frames before and after the video frame to be processed. That is, the image processing algorithm model processes the video frame to be processed and its two adjacent frames (a total of three frames) to obtain the target frame image. Understandably, when the video frame to be processed can be the first or last video frame, it only has one adjacent frame. Therefore, the gap can be filled by copying the video frame to be processed. For example, when the video frame to be processed is the first video frame, the first video frame is copied, and the copied image, along with the second video frame, serves as the adjacent frame.

[0119] Please see Figure 4 In some implementations, step 01 includes the following sub-steps:

[0120] 011. The video to be processed is segmented to obtain at least one sub-video, and the sub-video includes multiple consecutive video frame images with the same scene.

[0121] 012, The first and last frames of the sub-video are copied and padded using frame copying;

[0122] 013. Obtain the video frame image to be processed and the two video frame images adjacent to the video frame image to be processed from the sub-video.

[0123] In some implementations, sub-steps 011-013 can be implemented by the first acquisition module 11. In other words, the first acquisition module 11 is also used to segment the video to be processed to obtain at least one sub-video. The sub-video includes multiple consecutive video frame images with the same scene. The first frame and the last frame of the sub-video are copied and supplemented by frame copying. The sub-video also acquires the video frame image to be processed and two video frame images adjacent to the video frame image to be processed in the sub-video.

[0124] In some implementations, the processor can be used to segment the video to be processed to obtain at least one sub-video, the sub-video comprising multiple consecutive video frame images with the same scene, and to copy and fill in the first and last frames of the sub-video by frame copying, and to obtain the video frame image to be processed and two adjacent video frame images of the video frame image to be processed in the sub-video.

[0125] Understandably, a video contains multiple scenes. Since image processing algorithms use optical flow networks to calculate motion estimates between adjacent frames, if images are not from the same scene, invalid optical flow information will be introduced, potentially interfering with the output. Therefore, it is necessary to segment and process the scenes in the video, grouping video frames with the same scene into the same sub-video. This ensures that adjacent frames within a sub-video are similar in content.

[0126] Specifically, please combine Figure 5 The video to be processed is decrypted to obtain a frame sequence image. Then, the similarity between adjacent video frame images is used to determine whether to perform segmentation processing. When the similarity between adjacent frame images meets the preset conditions, segmentation processing is performed between these adjacent video frame images. In this way, multiple scene sub-videos can be obtained.

[0127] Furthermore, since in each sub-video, the first video frame only includes the adjacent frame image of the next frame, and the last video frame only includes the adjacent frame image of the previous frame, in sub-step 012, it is necessary to copy the first and last frames of the sub-video by frame duplication to complete the first video frame image with the adjacent frame image of the previous frame, and the last video frame image with the adjacent frame image of the next frame. This ensures that both the first and last video frame images include the corresponding two adjacent video frame images. Thus, each video frame image to be processed includes two adjacent video frame images.

[0128] Please see Figure 6 In some implementations, sub-step 011 includes:

[0129] 0111, convert two adjacent frames of the video to be processed to the preset color space;

[0130] 0112, Obtain the histogram of the preset color channels of two adjacent frames of the video to be processed in the preset color space;

[0131] 0113. If the histograms of the preset color channels of two adjacent frames of the video to be processed meet the preset conditions, the two adjacent frames of the video to be processed are determined to be consecutive video frame images of the same sub-video.

[0132] In some implementations, sub-steps 0111-0113 can be implemented by the first acquisition module 11. In other words, the first acquisition module 11 is further used to convert two adjacent frames of the video to be processed to a preset color space, and acquire the histogram of the preset color channel of the two adjacent frames of the video to be processed in the preset color space. When the histogram of the preset color channel of the two adjacent frames of the video to be processed satisfies the preset conditions, the two adjacent frames of the video to be processed are determined to be consecutive video frame images of the same sub-video.

[0133] In some implementations, the processor can be used to convert two adjacent frames of the video to be processed to a preset color space, and obtain the histogram of the preset color channel of the two adjacent frames of the video to be processed in the preset color space, and determine that the two adjacent frames of the video to be processed are consecutive video frame images of the same sub-video when the histogram of the preset color channel of the two adjacent frames of the video to be processed satisfies a preset condition.

[0134] For example, please combine Figure 7 In some examples, during the scene segmentation process in a video, the video frame images of the video to be processed are first color-converted, thereby converting the video frame images from the RGB color space to the YUM color space. Then, each time, two adjacent video frame images converted to the YUM color space are obtained and processed to obtain two histograms in the Y channel. The correlation between the two adjacent video frame images is then determined based on the histograms. If the correlation between the two histograms is greater than or equal to a certain threshold (e.g., 80%), the two adjacent video frame images are determined to be images of the same scene. If the correlation between the two histograms is less than a certain threshold (e.g., 80%), the two adjacent video frame images are determined to be images of different scenes, and scene segmentation is then performed.

[0135] Please see Figure 8 In some implementations, step 02 includes the following sub-steps:

[0136] 021, Optical flow information is obtained by processing the video frame image to be processed and adjacent video frame images through an optical flow network;

[0137] 022, Optical flow information is used to process adjacent video frame images to obtain the predicted video frame image;

[0138] 023, by concatenating optical flow information, predicted video frame images, and video frame images to be processed, an enhanced input video frame image is obtained;

[0139] 024, The target video frame image is obtained by processing the enhanced input video frame image through the enhancement network.

[0140] Please combine further Figure 2 In some embodiments, sub-steps 021-024 can be implemented by processing module 12. That is, processing module 12 can be used to process the video frame image to be processed and adjacent video frame images through an optical flow network to obtain optical flow information, and use the optical flow information to process adjacent video frame images to obtain a predicted video frame image; processing module 12 can also be used to connect the optical flow information, the predicted video frame image and the video frame image to be processed to obtain an enhanced input video frame image, and process the enhanced input video frame image through an enhancement network to obtain a target video frame image.

[0141] In some implementations, the processor can be used to process the video frame image to be processed and adjacent video frame images through an optical flow network to obtain optical flow information, and use the optical flow information to process adjacent video frame images to obtain a predicted video frame image; the processor can also be used to connect the optical flow information, the predicted video frame image and the video frame image to be processed to obtain an enhanced input video frame image, and process the enhanced input video frame image through an enhancement network to obtain a target video frame image.

[0142] The processor can calculate the optical flow information from each adjacent video frame to the video frame to be processed using an optical flow network. Then, it performs an affine (Warp) transformation on the adjacent video frames based on the optical flow information to obtain the predicted video frame image. As those skilled in the art will understand, an affine transformation refers to a linear transformation followed by a translation in geometry, transforming a vector space into another vector space. Furthermore, since adjacent video frames include two consecutive frames, the resulting predicted video frame image also includes two frames.

[0143] Furthermore, after obtaining the optical flow information and the predicted video frame image, the preceding adjacent frame image, the predicted video frame image obtained by processing the preceding adjacent frame image, the video frame image to be processed, the predicted video frame image obtained by processing the following adjacent frame image, the following adjacent video frame image, and the optical flow information are sequentially concatenated to obtain the enhanced input video frame image. Then, the obtained enhanced input video frame image is used as the input to the enhancement network, and processed by the enhancement network to obtain the target video frame image.

[0144] For example, please combine Figure 9 In some examples, image X n-1 X n Xn+1 Given three video frames in the same scene with time points n-1, n, and n+1, where X... n For the current video frame image, X n-1 and X n+1 These are the adjacent frames of the current video frame. After inputting the three video frame images into the image processing algorithm model, the optical flow network first calculates X respectively. n-1 To X n and X n+1 To X n Optical flow information F 0→1 F 2→1 Image X n-1 Reference optical flow F 0→1 Perform an affine transformation on the image to obtain the predicted video frame image W(X) at time node n. n-1 Similarly, image X n+1 According to optical flow F 2→1 Obtain the predicted video frame image W(X) at time node n. n+1 ), sequentially X n-1 W(X) n-1 ), X n W(X) n+1 ), X n+1 F 0→1 F 2→1 A concatenation operation is performed to obtain the enhanced input video frame image. This concatenated data is then used as input to the EnhanceNet network to obtain the repaired target video frame image Y. n .

[0145] Please see Figure 10 In some implementations, step 021 includes the following sub-steps:

[0146] 0211, Multi-level concatenation data is obtained based on the video frame image to be processed and adjacent video frame images. The first-level concatenation data is obtained by connecting the video frame image to be processed and adjacent video frame images.

[0147] 0212, multi-level sub-optical flow information is obtained by processing the multi-level connected data separately. Each level of data processing includes downsampling processing, first convolution processing and upsampling processing. The upsampling processing factor of each level is twice the downsampling processing factor. The downsampling processing factor of the previous level is twice the downsampling processing factor of the next level, and the upsampling processing factor of the previous level is twice the downsampling processing factor of the next level.

[0148] 0213, the multi-level sub-prediction images are estimated by estimating the video frame images to be processed and the multi-level sub-optical flow information respectively. The concatenation data of each level after the first level is obtained by concatenating the previous level sub-prediction image and the adjacent video frame images.

[0149] 0214, the optical flow information is obtained by adding the multi-level sub-optical flow information.

[0150] Please combine further Figure 2 In some embodiments, sub-steps 0211-0214 can be implemented by processing module 12. That is, processing module 12 can be used to obtain multi-level concatenation data based on the video frame image to be processed and adjacent video frame images. The first-level concatenation data is obtained by concatenating the video frame image to be processed and adjacent video frame images. The multi-level concatenation data is then processed to obtain multi-level sub-optical flow information. Each level of data processing includes downsampling processing, first convolution processing, and upsampling processing. The upsampling processing factor is twice the downsampling processing factor, the downsampling processing factor of the previous level is twice the downsampling processing factor of the next level, and the upsampling processing factor of the previous level is twice the downsampling processing factor of the next level. Processing module 12 can also be used to estimate multi-level sub-prediction images from the video frame image to be processed and the multi-level sub-optical flow information. The concatenation data of each level after the first level is obtained by concatenating the previous level sub-prediction image and adjacent video frame images. The multi-level sub-optical flow information is then added together to obtain optical flow information.

[0151] In some implementations, the processor can be used to obtain multi-level concatenation data based on the video frame image to be processed and adjacent video frame images. The first-level concatenation data is obtained by concatenating the video frame image to be processed and adjacent video frame images. The processor can also perform data processing on the multi-level concatenation data to obtain multi-level sub-optical flow information. Each level of data processing includes downsampling processing, first convolution processing, and upsampling processing. The upsampling processing factor of each level is twice the downsampling processing factor. The downsampling processing factor of the previous level is twice the downsampling processing factor of the next level, and the upsampling processing factor of the previous level is twice the downsampling processing factor of the next level. The processor can also be used to estimate multi-level sub-prediction images from the video frame image to be processed and the multi-level sub-optical flow information. The concatenation data of each level after the first level is obtained by concatenating the previous level sub-prediction image and adjacent video frame images. The processor can also add the multi-level sub-optical flow information to obtain optical flow information.

[0152] For example, please combine Figure 11 Two images, X0 and X1, are given as H×W×C frames, where H is the height, W is the width, and C is the number of channels (C is 3 for RGB color images). First, the two input images are concatenated to obtain H×W×2C concatenated data. This data is then downsampled by 8 times and input into a ConvBlock convolutional block structure for convolution processing. The output is the first optical flow data F between the two downsampled images (8 times). 8↓ The first optical flow data F 8↓ The sample is upsampled by 16 times and then warped with X0. X0 is based on the first optical flow data F.8↓ Estimate

[0153] Furthermore, the results estimated in the first step... The image is then concatenated with X1, downsampled by 4x, and convolved within the input convolutional block structure ConvBlock. The output is the second optical flow data F between the two images. 4↓ Then the second optical flow data F 4↓ Upsampled by 8 times, compared with the first upsampled optical flow data F obtained in the previous step. 8↓ Add them together, and then perform an affine transformation (warp) operation with X0 to estimate from X0.

[0154] Furthermore, the result obtained in the previous step After being concatenated with X1 and downsampled by 2x, the data is input into a ConvBlock convolutional block structure for convolution, outputting the third optical flow data F between the two images. 2↓ , and the third optical flow data F 2↓ Upsampled by 4 times, and compared with the upsampled second optical flow data F obtained in the previous step. 4↓ Add them together, then perform a warp operation with X0 to estimate from X0.

[0155] Finally, take the result from the previous step The image is concatenated with X1 and then fed into a ConvBlock structure for convolution. The output is the fourth optical flow data F between the two images. The fourth optical flow data F is upsampled by a factor of 2 and then compared with the upsampled third optical flow data F obtained in the previous step. 2↓ The summation yields the final optical flow information F output by the optical flow network. 0→1 .

[0156] Please see Figure 12 In some implementations, step 024 includes the following sub-steps:

[0157] 0241, The predicted video frame image and the video frame image to be processed are concatenated and processed by the second convolution to obtain the first feature map;

[0158] 0242, adjacent video frame images are processed by the third convolution and then subjected to an affine transformation with optical flow information to obtain the second feature map and the third feature map;

[0159] 0243, the first feature map, the second feature map, and the third feature map are concatenated and processed by the fourth convolution to obtain the fourth feature map;

[0160] 0244, adaptive attention processing of the fourth feature map and the fifth convolution processing are used to obtain the target video frame image.

[0161] Please combine further Figure 2 In some embodiments, sub-steps 0241-0244 can be implemented by processing module 12. That is, processing module 12 can be used to concatenate the predicted video frame image and the video frame image to be processed and perform a second convolution process to obtain a first feature map, and then perform an image affine transformation process on adjacent video frame images after a third convolution process and optical flow information to obtain a second feature map and a third feature map; processing module 12 can also be used to concatenate the first feature map, the second feature map and the third feature map and perform a fourth convolution process to obtain a fourth feature map, and perform adaptive attention processing on the fourth feature map and perform a fifth convolution process to obtain the target video frame image.

[0162] In some implementations, the processor can be used to concatenate the predicted video frame image and the video frame image to be processed and perform a second convolution process to obtain a first feature map, and then perform an image affine transformation process on adjacent video frame images after a third convolution process and optical flow information to obtain a second feature map and a third feature map; the processor can also be used to concatenate the first feature map, the second feature map and the third feature map and perform a fourth convolution process to obtain a fourth feature map, and perform adaptive attention processing on the fourth feature map and perform a fifth convolution process to obtain the target video frame image.

[0163] For example, the input to an augmentation network includes F 0→1 Xn-1, W (Xn-1) 、Xn、W (Xn+1) 、Xn+1、F 2→1 The meanings of each symbol are as follows:

[0164] F 0→1 The optical flow estimation network calculates optical flow information using Xn-1 and Xn as inputs;

[0165] F 2→1 The optical flow estimation network calculates optical flow information using Xn and Xn+1 as inputs;

[0166] X n-1 The previous frame image adjacent to the current frame;

[0167] Xn: The frame to be processed;

[0168] Xn+1: The image of the next frame adjacent to the current frame;

[0169] W (Xn-1) :Xn-1 according to F 0→1 The estimated image obtained by performing a warp operation;

[0170] W (Xn+1) :Xn+1 according to F 2→ 1. The estimated image obtained by performing a warp operation;

[0171] W (Xn-1) 、Xn、W (Xn+1) After concatenating the three images, they are input into the first convolutional layer, ConvLayer, to obtain the first feature map of the images.

[0172] Xn-1 is input into the second convolutional layer ConvLayer to obtain the feature map of the image. Then, a warp operation is performed with F0→1 to obtain the estimated second feature map.

[0173] Xn+1 is input into the third convolutional layer ConvLayer to obtain the feature map of the image. Then, a warp operation is performed with F2→1 to obtain the estimated third feature map.

[0174] The first feature map Second feature map Third Feature Map After performing concat processing, the data is fed into the fourth convolutional layer (ConvLayer) to obtain the fourth feature map of the deeper layer.

[0175] The fourth feature map The input is processed through four cascaded adaptive attention modules, with the processing flow of each module as follows:

[0176] Input the fourth feature map After passing through two convolutional layers (ConvLayer), a deeper fifth feature map is obtained. Fifth Feature Map Calculated by the channel attention module The output of the adaptive attention module is

[0177] The output of the fourth adaptive attention module is processed by the fourth convolutional layer ConvLayer to obtain the target video frame image of the augmented network.

[0178] Please combine Figure 13 This application provides a method for training an image processing algorithm model, wherein the image processing algorithm model includes an optical flow network and an enhancement network. The method for training the image processing algorithm model is as follows:

[0179] 001, Obtain the training video frame image and the adjacent training video frame images;

[0180] 002. By constructing an image processing algorithm model, the training video frame images and adjacent training video frame images are processed in multiple training stages to obtain the training output video frame images.

[0181] 003. In multiple training phases, based on the training output video frame images, the loss value of the image processing algorithm model is calculated using multiple loss functions respectively.

[0182] 004. Based on the loss value of the image processing algorithm model, the parameters of the image processing algorithm model in the corresponding training stage are corrected.

[0183] Please combine Figure 14 This application provides a training device 20 for an image processing algorithm model. The training device 20 includes a second acquisition module 21, a construction module 22, a calculation module 23, and a correction module 24.

[0184] Step 001 can be implemented by the second acquisition module 21, step 002 by the construction module 22, step 003 by the calculation module 23, and step 004 by the correction module 24. Alternatively, the second acquisition module 21 can be used to acquire training video frame images and adjacent training video frame images; the construction module 22 can be used to process the training video frame images and adjacent training video frame images in multiple training stages using the constructed image processing algorithm model to obtain training output video frame images; the calculation module 23 can be used to calculate the loss value of the image processing algorithm model based on the training output video frame images in multiple training stages using multiple loss functions; and the correction module 24 can be used to correct the parameters of the image processing algorithm model in the corresponding training stage based on the loss value of the image processing algorithm model.

[0185] This application also provides an electronic device, which includes a processor, a memory, and a computer program. When the computer program is executed by the processor, the processor is configured to: acquire training video frame images and adjacent training video frame images, and process the training video frame images and adjacent training video frame images in multiple training stages using a constructed image processing algorithm model to obtain training output video frame images; in multiple training stages, calculate the loss value of the image processing algorithm model using multiple loss functions based on the training output video frame images, and correct the parameters of the image processing algorithm model in the corresponding training stage according to the loss value of the image processing algorithm model.

[0186] In the image processing algorithm model training method, training device 20, and electronic device of this application, the constructed image processing algorithm model is processed on training video frame images and adjacent training video frame images in multiple training stages to obtain training output video frame images. Furthermore, in each of the multiple training stages, the loss value of the image processing algorithm model is calculated based on the training output video frame images using corresponding loss functions. The parameters of the image processing algorithm model in the corresponding training stages are then corrected based on the loss value of the image processing algorithm model. This ensures the accuracy of the trained algorithm processing model. Therefore, when the trained algorithm processing model is used to repair video frame images, problems such as blurriness, artifacts, color blocks, and noise in the video frame images can be effectively improved, ensuring the accuracy of the video frame image content and enhancing the display effect of the video frame images.

[0187] Please see Figure 15 In some implementations, step 001 includes the following sub-steps:

[0188] 0011, The original video is compressed and remade once or multiple times to obtain the training video;

[0189] 0012, The training video is segmented to obtain at least one training sub-video, and the training sub-video includes multiple consecutive video frame images with the same scene;

[0190] 0013, The first and last frames of the training sub-video are copied and padded using frame copying;

[0191] 0014. Obtain the training video frame image and two adjacent training video frame images from the training sub-video.

[0192] In some implementations, sub-steps 0011-0014 can be implemented by the second acquisition module 21. In other words, the second acquisition module 21 is used to compress and re-encode the original video one or more times to obtain a training video, and to acquire training video frame images and two adjacent training video frame images from the training sub-video. The second acquisition module 21 can also be used to copy and pad the first and last frames of the training sub-video using frame duplication, and to acquire training video frame images and two adjacent training video frame images from the training sub-video using the image processing algorithm model trained in the second training stage.

[0193] In some implementations, the processor can also be used to compress and re-encode the original video once or multiple times to obtain a training video, and to obtain training video frame images and two adjacent training video frame images in the training sub-video. The processor can also be used to copy and fill in the first and last frames of the training sub-video by frame copying, and to obtain training video frame images and two adjacent training video frame images in the training sub-video by using the image processing algorithm model trained in the second training stage.

[0194] Thus, it's understandable that a video contains multiple scenes. Since image processing algorithms use optical flow networks to calculate motion estimates between adjacent frames, if images are from different scenes, invalid optical flow information will be introduced, potentially interfering with the output. Therefore, it's necessary to segment and process the scenes in the training video to obtain multiple training sub-videos. This ensures that adjacent training frames within the training sub-videos are similar in content.

[0195] Furthermore, since in the training sub-video, the first training video frame only includes the adjacent training video frame image of the next frame, and the last training video frame only includes the adjacent training video frame image of the previous frame, in sub-step 0013, it is necessary to copy the first and last training video frame images of the training sub-video by frame copying to complete the adjacent training video frame images of the first training video frame image in the previous frame and the adjacent training video frame images of the last training video frame image in the next frame. This ensures that both the first and last training video frame images include the corresponding adjacent training video frame images of the two frames before and after. Thus, each training video frame image includes two adjacent training video frame images.

[0196] Please see Figure 16 In some implementations, step 002 includes the following sub-steps:

[0197] 0021. By constructing an image processing algorithm model, the first training output video frame image is obtained by processing the training video frame image and adjacent training video frame images in the first training stage.

[0198] Step 003 includes the following sub-steps:

[0199] 0031. In the first training phase, based on the first training output video frame image, the loss value of the image processing algorithm model is calculated using the first loss function;

[0200] Step 004 includes the following sub-steps:

[0201] 0041: The parameters of the image processing algorithm model in the first training stage are corrected based on the loss value of the image processing algorithm model calculated by the first loss function, so as to obtain the image processing algorithm model trained in the first training stage.

[0202] In some implementations, sub-step 0021 can be implemented by the construction module 22, sub-step 0031 can be implemented by the calculation module 23, and sub-step 0041 can be implemented by the correction module 24. Alternatively, the construction module 22 is further configured to process the training video frame image and adjacent training video frame images using the constructed image processing algorithm model in the first training phase to obtain the first training output video frame image. The calculation module 23 can also be configured to calculate the loss value of the image processing algorithm model based on the first training output video frame image using a first loss function in the first training phase; the correction module 24 is further configured to correct the parameters of the image processing algorithm model in the first training phase according to the loss value of the image processing algorithm model calculated by the first loss function, thereby obtaining the trained image processing algorithm model in the first training phase.

[0203] In some implementations, the processor can be used to process training video frame images and adjacent training video frame images in a first training phase using a constructed image processing algorithm model to obtain a first training output video frame image. In the first training phase, based on the first training output video frame image, a loss value of the image processing algorithm model is calculated using a first loss function. The processor is also used to correct the parameters of the image processing algorithm model in the first training phase based on the loss value of the image processing algorithm model calculated using the first loss function, to obtain the trained image processing algorithm model in the first training phase.

[0204] It should be noted that the first loss function is expressed as:

[0205]

[0206] in, The image is the ground truth image, and W, H, and C are the height, width, and number of channels of the first training output video frame image, respectively.

[0207] Specifically, in the first training phase, the constructed image processing algorithm model first processes the training video frame image and its adjacent video frame images to obtain the first training output video frame image. Then, the first training output video frame image is input into the first loss function, which calculates the loss value of the image processing algorithm model based on the first training output video frame image and the ground truth image. Subsequently, the parameters of the constructed image processing algorithm model are corrected based on the loss value to obtain the image processing algorithm model trained in the first training phase.

[0208] In the first training phase, the learning rate of the image processing algorithm model can be a first preset value. This first preset value can be set relatively large; for example, it can be set to 1×10⁻⁶. -3 Understandably, using a larger learning rate can make the loss value of the image processing algorithm model converge quickly. At this time, the image processing algorithm model is affected by the first loss function and pays more attention to reconstructing low-frequency content in the image that contains less detail.

[0209] Please see Figure 17 In some implementations, step 002 includes the following sub-steps:

[0210] 0022, The image processing algorithm model trained in the first training stage is used to process the training video frame image and the adjacent training video frame image in the second training stage to obtain the second training output video frame image;

[0211] 0023, The second training output video frame image is processed by the discrimination network to obtain the discrimination output value;

[0212] Step 003 includes the following sub-steps:

[0213] 0032, In the second training phase, based on the discriminative output value, the loss value of the image processing algorithm model is calculated using the second loss function;

[0214] Step 004 includes the following sub-steps:

[0215] 0042: The parameters of the image processing algorithm model in the second training stage are corrected based on the loss value of the image processing algorithm model calculated by the second loss function, so as to obtain the trained image processing algorithm model in the second training stage.

[0216] In some implementations, sub-steps 0022-0023 can be implemented by the construction module 22, sub-step 0032 can be implemented by the calculation module 23, and sub-step 0042 can be implemented by the correction module 24. Alternatively, the construction module 22 is further configured to process the training video frame image and adjacent training video frame images in the second training phase using the image processing algorithm model trained in the first training phase to obtain a second training output video frame image, and to process the second training output video frame image using a discrimination network to obtain a discrimination output value; the calculation module 23 is further configured to calculate the loss value of the image processing algorithm model based on the discrimination output value using a second loss function in the second training phase; and the correction module 24 is further configured to correct the parameters of the image processing algorithm model in the second training phase according to the loss value of the image processing algorithm model calculated by the second loss function, thereby obtaining the image processing algorithm model trained in the second training phase.

[0217] In some implementations, the processor can be used to process the training video frame image and adjacent training video frame images to obtain a second training output video frame image in a second training phase, and to process the second training output video frame image through a discrimination network to obtain a discrimination output value. The processor can also be used in the second training phase to calculate the loss value of the image processing algorithm model based on the discrimination output value using a second loss function; and to correct the parameters of the image processing algorithm model in the second training phase according to the loss value of the image processing algorithm model calculated by the second loss function, to obtain the image processing algorithm model trained in the second training phase.

[0218] It should be noted that the second loss function is expressed as:

[0219]

[0220] in, For the true value image, D(Y) is the discrimination output value of the ground truth image, and D(Y) is the discrimination output value of the second training output video frame image.

[0221] Specifically, in the second training phase, the image processing algorithm model trained in the first training phase is used as a generator network, and a discriminator network is then set up to form a generative adversarial network (GAN). The generator network processes the training video frame images and adjacent training video frame images to obtain the second training output video frame images. Then, the discriminator network processes the second training output video frame images generated by the generator network to obtain discriminator output values. Subsequently, a second loss function is used to calculate the loss value of the image processing algorithm model based on the discriminator output value and the ground truth image. The parameters of the image processing algorithm model in the second training phase are then corrected based on the loss value to obtain the image processing algorithm model trained in the second training phase.

[0222] Furthermore, in the second training phase, the learning rate of the image processing algorithm model is a second preset value, and the learning rate of the discrimination network is a third preset value. The second preset value is less than the first preset value, and the second preset value is less than the third preset value. For example, in some examples, the second preset value can be set to 1×10. -4 The third preset value can be set to 1×10. -3 Understandably, in the second training phase, setting a smaller learning rate allows the image processing algorithm model to focus on reconstructing high-frequency details in the image, improving image clarity and enhancing visual perception. Conversely, setting a larger learning rate for the discrimination network can improve the efficiency of adversarial training.

[0223] Thus, in the second training phase, by setting up a generative adversarial network, the image processing algorithm model can enhance the restoration of details and textures in the image. As a result, when the image processing algorithm model processes the video frame to be processed and its adjacent video frames to obtain the target video frame image, the image quality of the target video frame image can be further improved.

[0224] For example, please combine Figure 18 In some scenarios, Figure 17 For comparison, (a) is a compressed image, (b) is the ground truth image (original image), and (c) is the image generated by the image processing algorithm model after the second training phase. It can be seen that after adversarial training, the generated image possesses high-frequency detail information, thus improving image clarity and enhancing visual perception.

[0225] Please see Figure 19 In some implementations, step 002 includes the following sub-steps:

[0226] 0024. The image processing algorithm model trained in the second training stage is used to process the training video frame image and adjacent training video frame images in the third training stage to obtain the third training output video frame image.

[0227] Step 003 includes the following sub-steps:

[0228] 0033, In the third training phase, based on the third training output video frame image, the loss value of the image processing algorithm model is calculated through the third loss function;

[0229] Step 004 includes the following sub-steps:

[0230] 0043: The parameters of the image processing algorithm model in the third training stage are corrected based on the loss value of the image processing algorithm model calculated by the third loss function, so as to obtain the trained image processing algorithm model.

[0231] In some implementations, sub-step 0024 can be implemented by the construction module 22, sub-step 0033 can be implemented by the calculation module 23, and sub-step 0043 can be implemented by the correction module 24. Alternatively, the construction module 22 is further configured to process the training video frame image and adjacent training video frame images in the third training stage using the image processing algorithm model trained in the second training stage to obtain a third training output video frame image; the calculation module 23 is further configured to calculate the loss value of the image processing algorithm model based on the third training output video frame image using a third loss function in the third training stage; and the correction module 24 is further configured to correct the parameters of the image processing algorithm model in the third training stage based on the loss value of the image processing algorithm model calculated using the third loss function, thereby obtaining a trained image processing algorithm model.

[0232] In some implementations, the processor can be used to process the training video frame image and adjacent training video frame images in the third training stage to obtain the third training output video frame image; the processor can also be used in the third training stage to calculate the loss value of the image processing algorithm model based on the third training output video frame image using a third loss function, and to correct the parameters of the image processing algorithm model in the third training stage based on the loss value of the image processing algorithm model calculated by the third loss function, so as to obtain the trained image processing algorithm model.

[0233] It should be noted that the third loss function is expressed as:

[0234]

[0235] Where A(X) is the amplitude of the image and the ground truth image in the frequency domain, P(X) is the Euclidean distance between the phases, and L... FFT For the third loss function, X f Let X be the spectrum obtained by fast Fourier transform, and Im(X) be the spectrum obtained by fast Fourier transform. f ) is X f The imaginary part of , Re(X f ) is X f The real part of .

[0236] Specifically, in the third training phase, the image processing algorithm model trained and corrected in the second training phase is used to process the training video frame images and adjacent training video frame images to obtain the third training output video frame image. Then, the third training output video frame image is input into the third loss function. The third loss function calculates the Euclidean distance between the third training output video frame image and the ground truth image in the frequency domain, representing the amplitude and phase of the two images. Then, based on the Euclidean distance in the frequency domain, the image processing algorithm model corrected in the second training phase is corrected for the third time, resulting in the image processing algorithm model trained in the third training phase.

[0237] It should be noted that in the third training stage, by adjusting the training time and parameter learning rate of the image processing algorithm model, image processing models with different degrees of correction can be obtained. Finally, the model parameters with the best performance are selected. In this way, by cross-training with multiple different loss functions, the balance between fidelity of low-frequency content and high-frequency details in the restored image can be controlled.

[0238] This application also provides a non-volatile computer-readable storage medium containing a computer program, which, when executed by a processor, causes the processor to perform the above-described video frame image processing method and the above-described image processing algorithm model training method.

[0239] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any other combination. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0240] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0241] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0242] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0243] The above description is merely a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the protection scope of the present application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims

1. A method for processing video frame images, characterized in that, The processing method includes: Obtain the video frame image to be processed and the adjacent video frame images of the video frame image to be processed, wherein the adjacent video frame images are the two video frame images before and after the video frame image to be processed; The target video frame image is obtained by processing the video frame image to be processed and the adjacent video frame images using an image processing algorithm model. The image processing algorithm model includes an optical flow network and an enhancement network. The model is obtained by sequentially training multiple loss functions in multiple training stages, including: The image processing algorithm model is constructed, and the training video frame image and adjacent training video frame images are processed in the first training stage to obtain the first training output video frame image. In the first training phase, based on the first training output video frame image, the loss value of the image processing algorithm model is calculated using a first loss function; The parameters of the image processing algorithm model in the first training stage are corrected based on the loss value of the image processing algorithm model calculated by the first loss function, so as to obtain the image processing algorithm model trained in the first training stage. The image processing algorithm model trained in the first training phase is used to process the training video frame image and the adjacent training video frame image in the second training phase to obtain the second training output video frame image. The second training output video frame image is processed by a discrimination network to obtain discrimination output values; In the second training phase, based on the discrimination output value, the loss value of the image processing algorithm model is calculated using the second loss function; The parameters of the image processing algorithm model in the second training stage are corrected based on the loss value of the image processing algorithm model calculated by the second loss function, so as to obtain the image processing algorithm model trained in the second training stage.

2. The processing method according to claim 1, characterized in that, The step of processing the video frame image to be processed and the adjacent video frame images using an image processing algorithm model to obtain the target video frame image includes: Optical flow information is obtained by processing the video frame image to be processed and the adjacent video frame images through the optical flow network; The optical flow information is used to process the adjacent video frame images to obtain the predicted video frame images; The optical flow information, the predicted video frame image, and the video frame image to be processed are combined to obtain the enhanced input video frame image; The target video frame image is obtained by processing the enhanced input video frame image through the enhancement network.

3. The processing method according to claim 2, characterized in that, The step of processing the video frame image to be processed and the adjacent video frame images through the optical flow network to obtain optical flow information includes: Multi-level concatenation data is obtained based on the video frame image to be processed and the adjacent video frame images. The first level of concatenation data is obtained by connecting the video frame image to be processed and the adjacent video frame images. The multi-level connection data is processed to obtain multi-level sub-optical flow information. Each level of data processing includes downsampling processing, first convolution processing, and upsampling processing. The upsampling processing factor of each level is twice the downsampling processing factor. The downsampling processing factor of the previous level is twice the downsampling processing factor of the next level, and the upsampling processing factor of the previous level is twice the downsampling processing factor of the next level. The video frame image to be processed and the sub-optical flow information of the multiple levels are estimated to obtain multi-level sub-prediction images. The concatenation data of each level after the first level is obtained by concatenating the sub-prediction image of the previous level and the adjacent video frame image. The optical flow information is obtained by adding the sub-optical flow information of the multiple levels together.

4. The processing method according to claim 2, characterized in that, The step of processing the enhanced input video frame image through the enhanced network to obtain the target video frame image includes: The predicted video frame image and the video frame image to be processed are concatenated and processed by a second convolution to obtain a first feature map. The adjacent video frame images are processed by a third convolution and then subjected to an image affine transformation with the optical flow information to obtain a second feature map and a third feature map. The first feature map, the second feature map, and the third feature map are concatenated and processed by a fourth convolution to obtain the fourth feature map; The target video frame image is obtained by adaptive attention processing of the fourth feature map and then by fifth convolution processing.

5. The processing method according to claim 1, characterized in that, The process of acquiring the video frame image to be processed and its adjacent video frame images includes: The video to be processed is segmented to obtain at least one sub-video, and the sub-video includes multiple consecutive video frame images with the same scene; The first and last frames of the sub-video are copied and supplemented by frame copying; The video frame image to be processed and two adjacent video frame images adjacent to the video frame image to be processed are obtained from the sub-video.

6. The processing method according to claim 5, characterized in that, The video to be processed is segmented to obtain at least one sub-video, including: Convert two adjacent frames of the video to be processed to a preset color space; Obtain the histogram of the preset color channels of two adjacent frames of the video to be processed in the preset color space; If the histograms of two adjacent frames of the video to be processed corresponding to the preset color channels meet the preset conditions, then the two adjacent frames of the video to be processed are determined to be consecutive video frame images of the same sub-video.

7. The processing method according to claim 6, characterized in that, The video to be processed includes the video obtained after compression and reprocessing.

8. A training method for an image processing algorithm model, characterized in that, The image processing algorithm model includes an optical flow network and an enhancement network, and the training method includes: Obtain the training video frame image and the adjacent training video frame images of the training video frame image; The constructed image processing algorithm model processes the training video frame images and adjacent training video frame images in multiple training stages to obtain training output video frame images. The adjacent training video frame images are the two video frame images preceding and following the training video frame images. The process includes: processing the training video frame images and adjacent training video frame images in a first training stage using the constructed image processing algorithm model to obtain a first training output video frame image; processing the training video frame images and adjacent training video frame images in a second training stage using the image processing algorithm model trained in the first training stage to obtain a second training output video frame image; and processing the second training output video frame image using a discrimination network to obtain a discrimination output value. In multiple training phases, based on the training output video frame images, the loss value of the image processing algorithm model is calculated using multiple loss functions, including: in the first training phase, based on the first training output video frame images, the loss value of the image processing algorithm model is calculated using a first loss function; in the second training phase, based on the discrimination output value, the loss value of the image processing algorithm model is calculated using a second loss function. The parameters of the image processing algorithm model in the corresponding training stage are corrected based on the loss value of the image processing algorithm model, including: correcting the parameters of the image processing algorithm model in the first training stage based on the loss value of the image processing algorithm model calculated by the first loss function, to obtain the image processing algorithm model trained in the first training stage; and correcting the parameters of the image processing algorithm model in the second training stage based on the loss value of the image processing algorithm model calculated by the second loss function, to obtain the image processing algorithm model trained in the second training stage.

9. The training method according to claim 8, characterized in that, The first loss function is expressed as: in, The image is the ground truth image, and W, H, and C are the height, width, and number of channels of the first training output video frame image, respectively.

10. The training method according to claim 8, characterized in that, The second loss function is expressed as: in, For the true value image, The output value for discriminating the truth image. This is the discrimination output value of the second training output video frame image.

11. The training method according to claim 8, characterized in that, The image processing algorithm model, through which the training video frame images and adjacent training video frame images are processed in multiple training stages to obtain training output video frame images, includes: The image processing algorithm model trained in the second training phase is used to process the training video frame image and the adjacent training video frame image in the third training phase to obtain the third training output video frame image. In the multiple training phases, based on the training output video frame images, the loss value of the image processing algorithm model is calculated using multiple loss functions, including: In the third training phase, based on the third training output video frame image, the loss value of the image processing algorithm model is calculated using the third loss function; The step of correcting the parameters of the image processing algorithm model in the corresponding training stage based on the loss value of the image processing algorithm model includes: The parameters of the image processing algorithm model in the third training stage are corrected based on the loss value of the image processing algorithm model calculated by the third loss function, so as to obtain the trained image processing algorithm model.

12. The training method according to claim 11, characterized in that, The third loss function includes the Fast Fourier Transform loss function, which is expressed as follows: in, The spectrum of X obtained by fast Fourier transform is shown below. for The imaginary part of , Re( )for The real part of .

13. The training method according to any one of claims 8-12, characterized in that, In the first training phase, the learning rate of the image processing algorithm model is a first preset value. In the second training phase, the learning rate of the image processing algorithm model is a second preset value, and the learning rate of the discrimination network is a third preset value. The second preset value is less than the first preset value, and the second preset value is less than the third preset value.

14. The training method according to claim 8, characterized in that, The step of acquiring the training video frame image and the adjacent training video frame images includes: The original video is compressed and remade once or multiple times to obtain the training video; The training video is segmented to obtain at least one training sub-video, and the training sub-video includes multiple consecutive video frame images with the same scene. The first and last frames of the training sub-video are copied and padded using frame duplication. The training video frame image and two adjacent training video frame images adjacent to the training video frame image are obtained from the training sub-video.

15. An electronic device, characterized in that, The device includes a processor, a memory, and a computer program, wherein the computer program is stored in the memory, and when the computer program is executed by the processor, the processor performs the video frame image processing method of any one of claims 1-7 or the image processing algorithm model training method of any one of claims 8-14.

16. A non-volatile computer-readable storage medium containing a computer program, characterized in that, When the computer program is executed by the processor, the processor performs the video frame image processing method according to any one of claims 1-7 or the image processing algorithm model training method according to any one of claims 8-14.

Images