Video processing method, device, computer equipment and medium

Abstract

The present disclosure relates to a video processing method, device, computer equipment and medium, and belongs to the field of Internet technology. In the embodiment of the present disclosure, by obtaining the motion information of an image pair in a sample video, the motion information of the image pair is used to obtain an image block pair whose motion information meets the motion condition in the image pair. In this way, by setting the motion condition, the motion information of two frames of the image pair is used to determine whether the motion information of the two frames of the image pair meets the motion condition, so as to obtain image blocks containing more motion information, and then perform model training based on the obtained image block pairs, so that the super-resolution model can focus on the motion changes between two adjacent frames of images, enrich the information referenced by the model training, and can train a high-accuracy super-resolution model, thereby improving the accuracy of super-resolution reconstruction, and thus improving the accuracy of video processing.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to a video processing method, device, computer equipment and medium. Background Art

[0002] In the field of video technology, super-resolution reconstruction technology has a wide range of applications and research significance. Super-resolution reconstruction technology uses an image to reconstruct a high-resolution image with higher pixel density and more complete details at the corresponding moment. With the development of deep learning, super-resolution reconstruction technology based on neural networks has achieved rapid development.

[0003] At present, before super-resolution reconstruction of a video is performed based on a neural network, the neural network needs to be trained first to obtain a super-resolution model with super-resolution reconstruction function. The training data of the neural network usually needs to be randomly intercepted from multiple sample videos to obtain image blocks in the sample video. These image blocks are used as training data for the neural network to perform model training.

[0004] However, due to the uncertainty of random capture, image blocks containing little information are often captured. Therefore, the accuracy of the super-resolution model trained using these image blocks is low, which in turn reduces the accuracy of super-resolution reconstruction and the accuracy of video processing. Summary of the invention

[0005] The present disclosure provides a video processing method, apparatus, computer equipment and medium, which can train a high-accuracy super-resolution model, thereby improving the accuracy of super-resolution reconstruction, and thus improving the accuracy of video processing. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a video processing method is provided, the method comprising:

[0007] Acquire motion information of an image pair in a sample video, where the image pair includes a pair of adjacent images in the sample video, and the motion information indicates pixel motion between two frames of the image pair;

[0008] From the image pair, a sample image block pair whose motion information satisfies a motion condition is obtained, wherein two sample image blocks in the sample image block pair are respectively located in two frames of the image pair;

[0009] Based on the multiple pairs of sample image blocks in the sample video, a model is trained to obtain a super-resolution model, and the super-resolution model is used to perform super-resolution reconstruction on the video.

[0010] In the disclosed embodiments, motion information of an image pair in a sample video is obtained and used to obtain a pair of image blocks in the image pair whose motion information satisfies a motion condition. Thus, the motion condition is set and the motion information of two frames of the image pair is used to determine whether the motion information of the two frames of the image pair satisfies the motion condition, so as to obtain image blocks containing more motion information. Model training is then performed based on the obtained image block pairs, so that the super-resolution model can focus on the motion changes between two adjacent frames of the image, enriching the information referenced by the model training and being able to train a super-resolution model with high accuracy, thereby improving the accuracy of super-resolution reconstruction and thus improving the accuracy of video processing.

[0011] In some embodiments, obtaining motion information of image pairs in a sample video includes any of the following:

[0012] Obtain optical flow information of the image pair in the sample video, where the optical flow information indicates the amount of movement of pixels from a previous frame image to a next frame image in the image pair; or obtain temporal difference information of the image pair in the sample video, where the temporal difference information indicates the motion change of two frames of the image pair in time sequence.

[0013] In the embodiments of the present disclosure, two methods of acquiring motion information of an image pair, namely an optical flow method and a time difference method, are provided, both of which can quickly acquire motion information of an image pair, thereby improving the flexibility of acquiring motion information. Specifically, by acquiring the optical flow information of an image pair, since the optical flow information can represent the amount of movement of pixels from a previous frame image to a next frame image in the image pair, it can also represent the pixel motion between the two frames of the image pair, so that the optical flow information can be subsequently used to perform a motion condition judgment process. By acquiring the time difference information of an image pair, since the time difference information can represent the motion change of the two frames of the image pair in time sequence, it can also represent the pixel motion between the two frames of the image pair, so that the time difference information can be subsequently used to perform a motion condition judgment process.

[0014] In some embodiments, the motion information includes a motion parameter, which indicates the degree of pixel motion between two frames of the image pair; and obtaining a sample image block pair whose motion information satisfies the motion condition from the image pair includes:

[0015] From the image pair, a motion image block pair whose motion parameter is greater than or equal to a first threshold and a still image block pair whose motion parameter is less than a second threshold are obtained, and the first threshold is greater than or equal to the second threshold.

[0016] In the disclosed embodiment, by setting a motion parameter so that the motion parameter can be used later to determine whether the motion information of two frames of an image pair meets the motion condition, the efficiency of determining the motion condition is improved, and the efficiency of obtaining sample image block pairs is also improved. Moreover, by setting a motion condition in which the motion parameter is greater than or equal to a first threshold value to obtain a motion image block pair, since the motion image block pair includes pixels in motion, when the motion image block pair is subsequently used for model training, the super-resolution model based on the motion image block pair can focus on the motion changes between two adjacent frames of images. By setting a motion condition in which the motion parameter is less than a second threshold value to obtain a still image block pair, since the still image block pair includes pixels in a still state, when the still image block pair is subsequently used for model training, the super-resolution model based on the still image block pair can focus on the structural information of the image itself. In this way, the information referenced by the model training is enriched, and a super-resolution model with high accuracy can be trained, thereby improving the accuracy of super-resolution reconstruction, and thus improving the accuracy of video processing.

[0017] In some embodiments, obtaining a motion image block pair whose motion parameter is greater than or equal to a first threshold and a still image block pair whose motion parameter is less than a second threshold from the image pair includes: determining, from the image pair, a first pixel whose motion parameter is greater than or equal to the first threshold, and taking the first pixel as a starting point to intercept an image block pair of a target size to obtain the motion image block pair; determining a second pixel whose motion parameter is less than the second threshold, and taking the second pixel as a starting point to intercept an image block pair of a target size to obtain the still image block pair.

[0018] In the disclosed embodiment, for the process of acquiring a motion image block pair, by first determining a first pixel and then taking the first pixel as a starting point to intercept an image block pair of a target size, the motion image block pair can be quickly acquired, thereby improving the efficiency of acquiring the motion image block pair; for the process of acquiring a still image block pair, by first determining a second pixel and then taking the second pixel as a starting point to intercept an image block pair of a target size, the still image block pair can be quickly acquired, thereby improving the efficiency of acquiring the still image block pair.

[0019] In some embodiments, the first threshold and the second threshold are determined based on motion parameters of a plurality of pixels included in any frame image in the image pair.

[0020] In the embodiment of the present disclosure, a first threshold corresponding to a moving image block pair and a second threshold corresponding to a still image block pair are determined based on motion parameters of a plurality of pixels included in any frame image in an image pair, so that, based on the first threshold and the second threshold, it is determined whether a pixel in the image is a moving pixel or a still pixel, thereby improving the accuracy of determining motion conditions.

[0021] In some embodiments, the number of the moving image block pairs is the same as the number of the still image block pairs.

[0022] In the disclosed embodiment, an equal number of motion image block pairs and still image block pairs are obtained so that the equal number of motion image block pairs and still image block pairs can be used as training data for model training to obtain a super-resolution model. Since the motion image block pairs contain pixels in motion and the still image block pairs contain pixels in a still state, the model is trained based on the motion image block pairs and the still image block pairs, so that the super-resolution model can focus on the motion changes between two adjacent frames of images based on the motion image block pairs, and can focus on the structural information of the image itself based on the still image block pairs, thereby enriching the information referenced by the model training and being able to train a highly accurate super-resolution model, thereby improving the accuracy of super-resolution reconstruction, and thus improving the accuracy of video processing.

[0023] In some embodiments, the sample video includes a sample video of a first resolution and a sample video of a second resolution, the first resolution being smaller than the second resolution; and obtaining, from the image pair, a sample image block pair whose motion information satisfies the motion condition includes:

[0024] From the image pair of the sample video of the first resolution, obtain a first sample image block pair whose motion information satisfies the motion condition; from the image pair of the sample video of the second resolution, obtain a second sample image block pair whose motion information satisfies the motion condition, wherein the first sample image block pair is the sample image block pair of the first resolution, and the second sample image block pair is the sample image block pair of the second resolution;

[0025] Based on multiple pairs of sample image blocks in the sample video, model training is performed to obtain a super-resolution model including:

[0026] Model training is performed based on multiple pairs of first sample image block pairs in the sample video of the first resolution and multiple pairs of second sample image block pairs in the sample video of the second resolution to obtain the super-resolution model.

[0027] In the disclosed embodiment, a sample image block pair of a first resolution is obtained in a sample video of a first resolution, and a sample image block pair of a second resolution is obtained in a sample video of a second resolution, so that the sample image block pair of the first resolution and the sample image block pair of the second resolution can be used as training data for model training to obtain a super-resolution model.

[0028] In some embodiments, obtaining, from the image pair of the sample video of the first resolution, a first sample image block pair whose motion information satisfies the motion condition, and obtaining, from the image pair of the sample video of the second resolution, a second sample image block pair whose motion information satisfies the motion condition comprises any of the following:

[0029] Based on the motion information of the image pair in the sample video of the first resolution, obtain the first sample image block pair from the image pair of the sample video of the first resolution, and based on the first sample image block pair, obtain the second sample image block pair from the image pair of the sample video of the second resolution;

[0030] Based on the motion information of the image pair in the sample video of the second resolution, obtain the second sample image block pair from the image pair of the sample video of the second resolution, and based on the second sample image block pair, obtain the first sample image block pair from the image pair of the sample video of the first resolution;

[0031] Based on the motion information of the image pair in the sample video of the first resolution, upsampling processing is performed to obtain the motion information of the image pair in the sample video of the second resolution; based on the motion information of the image pair in the sample video of the second resolution, the second sample image block pair is obtained from the image pair of the sample video of the second resolution; based on the second sample image block pair, the first sample image block pair is obtained from the image pair of the sample video of the first resolution.

[0032] In the embodiment of the present disclosure, three methods for acquiring the first sample image block pair and the second sample image block pair are provided, thereby improving the flexibility of acquiring the first sample image block pair and the second sample image block pair.

[0033] In some embodiments, based on the first sample image block pair, acquiring the second sample image block pair from the image pair of the sample video of the second resolution includes: acquiring the second sample image block pair from the image pair of the sample video of the second resolution based on a resolution magnification and the first sample image block pair, the resolution magnification indicating a resolution ratio of the second resolution to the first resolution;

[0034] Based on the second sample image block pair, obtaining the first sample image block pair from the image pair of the sample video of the first resolution includes: based on the resolution magnification and the second sample image block pair, obtaining the first sample image block pair from the image pair of the sample video of the first resolution.

[0035] In the disclosed embodiment, after acquiring the first sample image block pair, based on the resolution magnification, the first sample image block pair can be converted from the first resolution to the second resolution, that is, the second sample image block pair is obtained; or, after acquiring the second sample image block pair, based on the resolution magnification, the second sample image block pair can be converted from the second resolution to the first resolution, that is, the first sample image block pair is obtained, thereby improving the efficiency of acquiring the sample image block pair.

[0036] In some embodiments, performing model training based on multiple pairs of first sample image block pairs in the sample video of the first resolution and multiple pairs of second sample image block pairs in the sample video of the second resolution to obtain the super-resolution model includes:

[0037] In the i-th iteration process of model training, the first sample image block pairs are input into the super-resolution model determined in the i-1-th iteration process to obtain the image training result of the i-th iteration process, where i is a positive integer greater than 1;

[0038] Based on the image training result of the i-th iterative process and the second sample image block pairs, the model parameters of the super-resolution model determined in the i-1-th iterative process are adjusted, and the i+1-th iterative process is performed based on the adjusted model parameters, and the iterative process of the above training is repeated until the training meets the target conditions.

[0039] In the disclosed embodiment, a network model with better model parameters is obtained as a super-resolution model through iterative training, so as to obtain a super-resolution model with better super-resolution reconstruction capability, thereby improving the accuracy of the super-resolution model.

[0040] In some embodiments, before performing model training based on multiple pairs of sample image blocks in the sample video to obtain a super-resolution model, the method further includes any of the following:

[0041] Performing mirror-inversion on the pixels at the inner edge position of the sample image block pair, and adding the mirror-inverted pixels to the outer edge position of the sample image block pair;

[0042] Copying pixels at the inner edge position of the sample image block pair, and adding the copied pixels to the outer edge position of the sample image block pair;

[0043] Zero pixels are added at the outer edge positions of the sample image block pair.

[0044] In the disclosed embodiment, pixel expansion is performed at the outer edge of the sample image block pair by means of pixel expansion methods such as mirror inversion or copying pixels or adding zero pixels, thereby increasing the pixel information of the image edge. Subsequently, model training is performed based on the sample image block pairs after pixel expansion, so that the super-resolution model pays more attention to the reconstruction of the image edge, avoiding the problem of noise at the image edge, and can train a super-resolution model with high accuracy, thereby improving the accuracy of super-resolution reconstruction, and thus improving the accuracy of video processing.

[0045] In some embodiments, after model training is performed based on multiple pairs of sample image blocks in the sample video to obtain a super-resolution model, the method further includes: super-resolution reconstruction of the video through the super-resolution model to obtain an initial super-resolution video; for any frame image in the super-resolution video, pixels at the inner edge position of the image are cut, and a target super-resolution video of the video is generated based on the cut image.

[0046] In the disclosed embodiment, after super-resolution reconstruction is performed on a video, pixels at inner edge positions of an image in the output initial super-resolution video are cut to remove redundant pixels, thereby obtaining a target super-resolution video.

[0047] According to a second aspect of an embodiment of the present disclosure, a video processing device is provided, the device comprising:

[0048] An information acquisition unit is configured to acquire motion information of an image pair in a sample video, the image pair comprising a pair of adjacent images in the sample video, the motion information representing pixel motion between two frames of the image pair;

[0049] An image acquisition unit is configured to acquire, from the image pair, a pair of sample image blocks whose motion information satisfies a motion condition, wherein two sample image blocks in the pair of sample image blocks are respectively located in two frames of the image pair;

[0050] The training unit is configured to perform model training based on multiple pairs of sample image blocks in the sample video to obtain a super-resolution model, and the super-resolution model is used to perform super-resolution reconstruction on the video.

[0051] In some embodiments, the information acquisition unit is configured to perform any of the following: obtaining optical flow information of the image pair in the sample video, the optical flow information indicating the amount of movement of pixels from a previous frame image to a next frame image in the image pair; or obtaining timing difference information of the image pair in the sample video, the timing difference information indicating the motion change of two frames of the image pair in time sequence.

[0052] In some embodiments, the motion information includes motion parameters, which represent the degree of pixel motion between two frames of the image pair; the image acquisition unit is configured to execute acquisition of motion image block pairs whose motion parameters are greater than or equal to a first threshold and still image block pairs whose motion parameters are less than a second threshold from the image pair, and the first threshold is greater than or equal to the second threshold.

[0053] In some embodiments, the image acquisition unit is configured to perform: determining, from the image pair, the first pixel whose motion parameter is greater than or equal to the first threshold, and intercepting the image block pair of the target size with the first pixel as the starting point to obtain the motion image block pair; determining the second pixel whose motion parameter is less than the second threshold, and intercepting the image block pair of the target size with the second pixel as the starting point to obtain the still image block pair.

[0054] In some embodiments, the first threshold and the second threshold are determined based on motion parameters of a plurality of pixels included in any frame image in the image pair.

[0055] In some embodiments, the number of the moving image block pairs is the same as the number of the still image block pairs.

[0056] In some embodiments, the sample video includes a sample video of a first resolution and a sample video of a second resolution, the first resolution being smaller than the second resolution; the image acquisition unit is configured to execute, from the image pair of the sample video of the first resolution, acquiring a first sample image block pair whose motion information satisfies the motion condition, and from the image pair of the sample video of the second resolution, acquiring a second sample image block pair whose motion information satisfies the motion condition, the first sample image block pair being a sample image block pair of the first resolution, and the second sample image block pair being a sample image block pair of the second resolution; the training unit is configured to execute model training based on multiple pairs of first sample image block pairs in the sample video of the first resolution and multiple pairs of second sample image block pairs in the sample video of the second resolution to obtain the super-resolution model.

[0057] In some embodiments, the image acquisition unit includes any of the following items: a first image acquisition subunit, configured to execute based on the motion information of the image pair in the sample video of the first resolution, to acquire the first sample image block pair from the image pair of the sample video of the first resolution, and based on the first sample image block pair, to acquire the second sample image block pair from the image pair of the sample video of the second resolution; a second image acquisition subunit, configured to execute based on the motion information of the image pair in the sample video of the second resolution, to acquire the second sample image block pair from the image pair of the sample video of the second resolution, and based on the second sample image block pair, to acquire the first sample image block pair from the image pair of the sample video of the first resolution; a third image acquisition subunit, configured to execute based on the motion information of the image pair in the sample video of the first resolution, to perform upsampling processing to obtain the motion information of the image pair in the sample video of the second resolution, to acquire the second sample image block pair from the image pair of the sample video of the second resolution based on the motion information of the image pair in the sample video of the second resolution, and to acquire the first sample image block pair from the image pair of the sample video of the first resolution based on the second sample image block pair.

[0058] In some embodiments, the first image acquisition subunit is configured to execute, based on a resolution magnification and the first sample image block pair, to acquire the second sample image block pair from the image pair of the sample video of the second resolution, wherein the resolution magnification represents a resolution ratio between the second resolution and the first resolution; and the second image acquisition subunit is configured to execute, based on the resolution magnification and the second sample image block pair, to acquire the first sample image block pair from the image pair of the sample video of the first resolution.

[0059] In some embodiments, the training unit is configured to perform: in the i-th iteration process of model training, multiple pairs of first sample image blocks are input into the super-resolution model determined by the i-1-th iteration process to obtain the image training result of the i-th iteration process, where i is a positive integer greater than 1; based on the image training result of the i-th iteration process and multiple pairs of second sample image blocks, the model parameters of the super-resolution model determined by the i-1-th iteration process are adjusted, and the i+1-th iteration process is performed based on the adjusted model parameters, and the above-mentioned training iteration process is repeated until the training meets the target conditions.

[0060] In some embodiments, the device further includes any of the following: a mirror inversion unit, configured to perform mirror inversion on pixels at the inner edge position of the sample image block pair, and add the mirror-inverted pixels to the outer edge position of the sample image block pair; a copying unit, configured to copy the pixels at the inner edge position of the sample image block pair, and add the copied pixels to the outer edge position of the sample image block pair; an adding unit, configured to add zero pixels at the outer edge position of the sample image block pair.

[0061] In some embodiments, the device also includes: a super-resolution unit, configured to perform super-resolution reconstruction on the video through the super-resolution model to obtain an initial super-resolution video; a cutting unit, configured to perform cutting of pixels at the inner edge position of any frame image in the super-resolution video, and generate a target super-resolution video of the video based on the cut image.

[0062] According to a third aspect of an embodiment of the present disclosure, a computer device is provided, the computer device comprising:

[0063] one or more processors;

[0064] A memory for storing program codes executable by the processor;

[0065] The processor is configured to execute the program code to implement the above-mentioned video processing method.

[0066] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, which includes: when the program code in the computer-readable storage medium is executed by a processor of a computer device, the computer device is enabled to execute the above-mentioned video processing method.

[0067] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the above-mentioned video processing method when executed by a processor.

[0068] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0070] Figure 1 is a schematic diagram of an implementation environment of a video processing method according to an exemplary embodiment;

[0071] Figure 2 is a flow chart of a video processing method according to an exemplary embodiment;

[0072] Figure 3 is a flow chart of a video processing method according to an exemplary embodiment;

[0073] Figure 4 is a schematic diagram of an optical flow graph according to an exemplary embodiment;

[0074] Figure 5 is a schematic diagram of an image block according to an exemplary embodiment;

[0075] Figure 6 is a schematic diagram showing a super-resolution test result according to an exemplary embodiment;

[0076] Figure 7 is a block diagram of a video processing device according to an exemplary embodiment;

[0077] Figure 8 is a block diagram of a terminal according to an exemplary embodiment;

[0078] Fig. 9 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0079] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0080] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0081] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of the present disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the motion information involved in the embodiments of the present disclosure is obtained with full authorization. In some embodiments, a permission inquiry page is provided in the embodiments of the present disclosure, and the permission inquiry page is used to inquire whether to grant permission to obtain the above information. In the permission inquiry page, an authorization consent control and an authorization rejection control are displayed. When a trigger operation of the authorization consent control is detected, the video processing method provided in the embodiments of the present disclosure is used to obtain the above information, thereby realizing the training of the super-resolution model.

[0082] The video processing method provided in the embodiments of the present disclosure can be applied to the super-resolution reconstruction scenario of the video, for example, the super-resolution reconstruction of the live video, the super-resolution reconstruction of the game video, and so on. In some embodiments, the video processing method provided in the embodiments of the present disclosure can be applied to the training stage of the super-resolution model to train a super-resolution model with super-resolution reconstruction capability. Among them, super-resolution reconstruction is to use an image of a first resolution to reconstruct a second-resolution image with a higher pixel density and more complete details at the corresponding moment, and the first resolution is smaller than the second resolution. Correspondingly, super-resolution reconstruction of the video is to use a video of the first resolution to reconstruct a video of the second resolution with a higher pixel density and more complete details at the corresponding moment. It should be understood that a high-resolution image can provide richer information, and it is easier to further mine and process the information therein than a low-resolution image.

[0083] In some embodiments, in the video transmission scenario, the video of the first resolution may be used for transmission, and after receiving the video of the first resolution, the super-resolution model trained by the embodiment of the present disclosure may be used for super-resolution reconstruction to obtain a video of higher resolution, thereby effectively reducing the resources consumed by video transmission. In some other embodiments, in the video rendering scenario, the video of the first resolution may be used for rendering, and then when displaying the video of the first resolution, the super-resolution model trained by the embodiment of the present disclosure may be used for super-resolution reconstruction to obtain a video of higher resolution, thereby effectively reducing the resources consumed by video rendering.

[0084] Figure 1 is a schematic diagram of an implementation environment of a video processing method according to an exemplary embodiment. Figure 1 , the implementation environment includes: a terminal 101 and a server 102.

[0085] The terminal 101 may be at least one of a smart phone, a smart watch, a desktop computer, a laptop, a virtual reality terminal, an augmented reality terminal, a wireless terminal, and a laptop computer. The terminal 101 has a communication function and can access a wired network or a wireless network. The terminal 101 may generally refer to one of a plurality of terminals. This embodiment is only illustrated by taking the terminal 101 as an example. Those skilled in the art may know that the number of the above terminals may be more or less.

[0086] The server 102 may be an independent physical server, or a server cluster or distributed file system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the number of the above-mentioned servers 102 may be more or less, and the embodiments of the present disclosure are not limited to this. Of course, the server 102 may also include other functional servers to provide more comprehensive and diversified services.

[0087] In some embodiments, the video processing method provided by the embodiments of the present disclosure is executed by terminal 101. For example, after terminal 101 detects the processing operation on the sample video, the video processing method provided by the embodiments of the present disclosure is used to obtain sample image block pairs in the image pairs of the sample video whose motion information meets the motion conditions, and then based on multiple pairs of sample image block pairs in the sample video, model training is performed to obtain a super-resolution model.

[0088] In other embodiments, the video processing method provided by the embodiments of the present disclosure is executed by the server 102. For example, after the server 102 receives a processing request for a sample video, the video processing method provided by the embodiments of the present disclosure is used to obtain sample image block pairs in image pairs of the sample video whose motion information satisfies the motion condition, and then perform model training based on the multiple pairs of sample image block pairs in the sample video to obtain a super-resolution model; or, the server 102 periodically obtains sample image block pairs in image pairs of the sample video whose motion information satisfies the motion condition, and then performs model training based on the multiple pairs of sample image block pairs in the sample video to obtain a super-resolution model.

[0089] In other embodiments, the server 102 and the terminal 101 are directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present disclosure. Accordingly, in some embodiments, if the terminal 101 detects a processing operation on a sample video, it sends a processing request for the sample video to the server 102 to request the server 102 to use the video processing method provided in the embodiments of the present disclosure to obtain sample image block pairs in the image pairs of the sample video whose motion information meets the motion conditions, and then perform model training based on multiple pairs of sample image block pairs in the sample video to obtain a super-resolution model. The embodiments of the present disclosure will subsequently refer to the terminal 101 or the server 102 as a computer device.

[0090] Figure 2 is a flowchart of a video processing method according to an exemplary embodiment. Figure 2 As shown, the method is executed by a computer device, which can be provided as the above Figure 1 The terminal or server shown schematically comprises the following steps:

[0091] In step 201, a computer device obtains motion information of an image pair in a sample video, where the image pair includes a pair of adjacent images in the sample video, and the motion information represents pixel motion between two frames of the image pair.

[0092] In step 202, the computer device obtains a sample image block pair whose motion information satisfies the motion condition from the image pair, wherein two sample image blocks in the sample image block pair are respectively located in two frames of the image pair.

[0093] In step 203, the computer device performs model training based on the multiple pairs of sample image blocks in the sample video to obtain a super-resolution model, and the super-resolution model is used to perform super-resolution reconstruction on the video.

[0094] The technical solution provided by the embodiments of the present disclosure obtains motion information of an image pair in a sample video, and uses the motion information of the image pair to obtain an image block pair in the image pair whose motion information satisfies a motion condition. In this way, by setting the motion condition, the motion information of two frames of the image pair is used to determine whether the motion information of the two frames of the image pair satisfies the motion condition, so as to obtain image blocks containing more motion information, and then perform model training based on the obtained image block pairs, so that the super-resolution model can be based on the motion changes between two adjacent frames of the image, enriching the information referenced by the model training, and being able to train a super-resolution model with high accuracy, thereby improving the accuracy of super-resolution reconstruction, and thus improving the accuracy of video processing.

[0095] In some embodiments, obtaining motion information of image pairs in a sample video includes any of the following:

[0096] Obtain optical flow information of the image pair in the sample video, where the optical flow information indicates the amount of movement of pixels from a previous frame image to a next frame image in the image pair; or obtain temporal difference information of the image pair in the sample video, where the temporal difference information indicates the motion change of two frames of the image pair in time sequence.

[0097] In some embodiments, the motion information includes a motion parameter, which indicates the degree of pixel motion between two frames of the image pair; and obtaining a sample image block pair whose motion information satisfies the motion condition from the image pair includes:

[0098] From the image pair, a motion image block pair whose motion parameter is greater than or equal to a first threshold and a still image block pair whose motion parameter is less than a second threshold are obtained, and the first threshold is greater than or equal to the second threshold.

[0099] In some embodiments, obtaining a motion image block pair whose motion parameter is greater than or equal to a first threshold and a still image block pair whose motion parameter is less than a second threshold from the image pair includes: determining, from the image pair, a first pixel whose motion parameter is greater than or equal to the first threshold, and taking the first pixel as a starting point to intercept an image block pair of a target size to obtain the motion image block pair; determining a second pixel whose motion parameter is less than the second threshold, and taking the second pixel as a starting point to intercept an image block pair of a target size to obtain the still image block pair.

[0100] In some embodiments, the first threshold and the second threshold are determined based on motion parameters of a plurality of pixels included in any frame image in the image pair.

[0101] In some embodiments, the number of the moving image block pairs is the same as the number of the still image block pairs.

[0102] In some embodiments, the sample video includes a sample video of a first resolution and a sample video of a second resolution, the first resolution being smaller than the second resolution; and obtaining, from the image pair, a sample image block pair whose motion information satisfies the motion condition includes:

[0103] From the image pair of the sample video of the first resolution, obtain a first sample image block pair whose motion information satisfies the motion condition; from the image pair of the sample video of the second resolution, obtain a second sample image block pair whose motion information satisfies the motion condition, wherein the first sample image block pair is the sample image block pair of the first resolution, and the second sample image block pair is the sample image block pair of the second resolution;

[0104] Based on multiple pairs of sample image blocks in the sample video, model training is performed to obtain a super-resolution model including:

[0105] Model training is performed based on multiple pairs of first sample image block pairs in the sample video of the first resolution and multiple pairs of second sample image block pairs in the sample video of the second resolution to obtain the super-resolution model.

[0106] In some embodiments, obtaining, from the image pair of the sample video of the first resolution, a first sample image block pair whose motion information satisfies the motion condition, and obtaining, from the image pair of the sample video of the second resolution, a second sample image block pair whose motion information satisfies the motion condition comprises any of the following:

[0107] Based on the motion information of the image pair in the sample video of the first resolution, obtain the first sample image block pair from the image pair of the sample video of the first resolution, and based on the first sample image block pair, obtain the second sample image block pair from the image pair of the sample video of the second resolution;

[0108] Based on the motion information of the image pair in the sample video of the second resolution, obtain the second sample image block pair from the image pair of the sample video of the second resolution, and based on the second sample image block pair, obtain the first sample image block pair from the image pair of the sample video of the first resolution;

[0109] Based on the motion information of the image pair in the sample video of the first resolution, upsampling processing is performed to obtain the motion information of the image pair in the sample video of the second resolution; based on the motion information of the image pair in the sample video of the second resolution, the second sample image block pair is obtained from the image pair of the sample video of the second resolution; based on the second sample image block pair, the first sample image block pair is obtained from the image pair of the sample video of the first resolution.

[0110] In some embodiments, based on the first sample image block pair, acquiring the second sample image block pair from the image pair of the sample video of the second resolution includes: acquiring the second sample image block pair from the image pair of the sample video of the second resolution based on a resolution magnification and the first sample image block pair, the resolution magnification indicating a resolution ratio of the second resolution to the first resolution;

[0111] Based on the second sample image block pair, obtaining the first sample image block pair from the image pair of the sample video of the first resolution includes: based on the resolution magnification and the second sample image block pair, obtaining the first sample image block pair from the image pair of the sample video of the first resolution.

[0112] In some embodiments, performing model training based on multiple pairs of first sample image block pairs in the sample video of the first resolution and multiple pairs of second sample image block pairs in the sample video of the second resolution to obtain the super-resolution model includes:

[0113] In the i-th iteration process of model training, the first sample image block pairs are input into the super-resolution model determined in the i-1-th iteration process to obtain the image training result of the i-th iteration process, where i is a positive integer greater than 1;

[0114] Based on the image training result of the i-th iterative process and the second sample image block pairs, the model parameters of the super-resolution model determined in the i-1-th iterative process are adjusted, and the i+1-th iterative process is performed based on the adjusted model parameters, and the iterative process of the above training is repeated until the training meets the target conditions.

[0115] In some embodiments, before performing model training based on multiple pairs of sample image blocks in the sample video to obtain a super-resolution model, the method further includes any of the following:

[0116] Performing mirror-inversion on the pixels at the inner edge position of the sample image block pair, and adding the mirror-inverted pixels to the outer edge position of the sample image block pair;

[0117] Copying pixels at the inner edge position of the sample image block pair, and adding the copied pixels to the outer edge position of the sample image block pair;

[0118] Zero pixels are added at the outer edge positions of the sample image block pair.

[0119] In some embodiments, after model training is performed based on multiple pairs of sample image blocks in the sample video to obtain a super-resolution model, the method further includes: super-resolution reconstruction of the video through the super-resolution model to obtain an initial super-resolution video; for any frame image in the super-resolution video, pixels at the inner edge position of the image are cut, and a target super-resolution video of the video is generated based on the cut image.

[0120] Above Figure 2 The following is only a basic process of the present disclosure. The solution provided by the present disclosure is further described based on a specific implementation method. Figure 3 is a flowchart of a video processing method according to an exemplary embodiment. Figure 3 , the method comprising:

[0121] In step 301, a computer device obtains motion information of an image pair in a sample video, where the image pair includes a pair of adjacent images in the sample video, and the motion information represents pixel motion between two frames of the image pair.

[0122] In an embodiment of the present disclosure, a computer device may be provided as a terminal or a server. The computer device provides a function of constructing training data for a super-resolution model and a function of training a model based on the training data. The super-resolution model is used to perform super-resolution reconstruction on a video, wherein super-resolution reconstruction is to use an image of a first resolution to reconstruct an image of a second resolution with a higher pixel density and more complete details at a corresponding moment, and the first resolution is smaller than the second resolution. Accordingly, super-resolution reconstruction of a video is to use a video of the first resolution to reconstruct a video of the second resolution with a higher pixel density and more complete details at a corresponding moment.

[0123] The sample video refers to a training video used to train the super-resolution model. In some embodiments, the number of sample videos is multiple. In some embodiments, the sample video is a video stored locally in the terminal, or the sample video is a video stored in a server, or the sample video is a video stored in a video library associated with the server, etc., which is not limited in the embodiments of the present disclosure.

[0124] The pixel motion refers to the temporal motion of the pixels in the image sequence, such as the displacement of the motion, the speed of the motion, etc. Correspondingly, the motion information is the motion of the pixels in the previous frame image to the next frame image in the image pair. In some embodiments, the motion information includes any one of the optical flow information and the temporal difference information. Based on the following (1) and (2), the process of obtaining the motion information of the image pair in the sample video by the computer device is described below:

[0125] (1) The computer device obtains the optical flow information of the image pair in the sample video, and the optical flow information indicates the movement amount of the pixels of the previous frame image to the next frame image in the image pair. In this embodiment, by obtaining the optical flow information of the image pair, since the optical flow information can indicate the movement amount of the pixels of the previous frame image to the next frame image in the image pair, it can also indicate the pixel motion between the two frames of the image pair, so that the optical flow information can be used to perform the motion condition judgment process later.

[0126] The movement amount is the displacement of the pixel movement. In some embodiments, the optical flow information includes the movement amount of the pixels of the previous frame image to the next frame image in the horizontal direction and the movement amount of the pixels of the previous frame image to the next frame image in the vertical direction. In some embodiments, the optical flow information is in the form of an optical flow map, such as an optical flow map corresponding to the horizontal direction (such as the x-axis direction) and an optical flow map corresponding to the vertical direction (such as the y-axis direction).

[0127] In some embodiments, a computer device obtains optical flow information of image pairs in the sample video through an optical flow network. The optical flow network is used to predict optical flow information of two adjacent frames of images. The corresponding process is: the computer device inputs the image sequence included in the sample video into the optical flow network, processes the image pairs in the image sequence through the optical flow network, and obtains the optical flow information of the image pairs in the sample video.

[0128] In some embodiments, the optical flow network is obtained by training based on a sample image pair and the optical flow information of the sample image pair. Accordingly, the training process of the optical flow network is: the computer device performs model training based on the sample image pair and the optical flow information of the sample image pair to obtain the optical flow network. Further, in some embodiments, during the mth iteration of model training, the server inputs the sample image pair into the optical flow network determined by the m-1th iteration process to obtain the optical flow training result predicted by the mth iteration process; based on the optical flow training result predicted by the mth iteration process and the optical flow information of the sample image pair, the model parameters of the optical flow network determined by the m-1th iteration process are adjusted, and the m+1th iteration process is performed based on the adjusted model parameters, and the above training iteration process is repeated until the training meets the target condition. Wherein, m is a positive integer greater than 1. In some embodiments, the target condition satisfied by the training is that the number of training iterations of the model reaches the target number, which is a preset number of training iterations, such as 1000 times; or, the target condition satisfied by the training is that the loss value meets the target threshold condition, such as the loss value is less than 0.00001. The present disclosure embodiment does not limit the setting of the target condition.

[0129] In this way, through iterative training, the network model with better model parameters is obtained as an optical flow network, so as to obtain an optical flow network with better prediction ability, thereby improving the prediction accuracy of the optical flow network.

[0130] It should be noted that the above embodiments take the use of an optical flow network to obtain the optical flow information of an image pair as an example, and in other embodiments, the computer device can also use other methods to obtain the optical flow information of the image pair, for example, the computer device uses an optical flow prediction algorithm to predict the optical flow information of the image pair, such as a sparse optical flow prediction algorithm, a dense optical flow prediction algorithm, etc. The embodiments of the present disclosure are not limited to this.

[0131] (2) The computer device obtains the time difference information of the image pair in the sample video, and the time difference information indicates the motion change of the two frames of the image pair in time. In this embodiment, by obtaining the time difference information of the image pair, since the time difference information can indicate the motion change of the two frames of the image pair in time, it can also indicate the pixel motion between the two frames of the image pair, so that the time difference information can be used to perform the motion condition judgment process later.

[0132] The motion change is the speed of the pixel motion. In some embodiments, the time difference information is in the form of a time difference graph. For example, the horizontal axis of the time difference graph represents the timestamps of the two frames of images, and the vertical axis of the time difference graph represents the motion change of the two frames of images in time sequence.

[0133] In some embodiments, a computer device obtains the timing difference information of the image pairs in the sample video through a timing difference network. The timing difference network is used to predict the timing difference information of two adjacent frames of images. The corresponding process is: the computer device inputs the image sequence included in the sample video into the timing difference network, processes the image pairs in the image sequence through the timing difference network, and obtains the timing difference information of the image pairs in the sample video.

[0134] In some embodiments, the temporal difference network is trained based on a sample image pair and the temporal difference information of the sample image pair. Accordingly, the training process of the temporal difference network is: the computer device performs model training based on the sample image pair and the temporal difference information of the sample image pair to obtain the temporal difference network. Further, in some embodiments, during the nth iteration of model training, the server inputs the sample image pair into the temporal difference network determined by the n-1th iteration process to obtain the temporal difference training result predicted by the nth iteration process; based on the temporal difference training result predicted by the nth iteration process and the temporal difference information of the sample image pair, the model parameters of the temporal difference network determined by the n-1th iteration process are adjusted, and the n+1th iteration process is performed based on the adjusted model parameters, and the above training iteration process is repeated until the training meets the target condition. Wherein, n is a positive integer greater than 1. In some embodiments, the target condition satisfied by the training is that the number of training iterations of the model reaches the target number, which is a preset number of training iterations, such as 1000 times; or, the target condition satisfied by the training is that the loss value meets the target threshold condition, such as the loss value is less than 0.00001. The present disclosure embodiment does not limit the setting of the target condition.

[0135] In this way, through iterative training, the network model with better model parameters is obtained as a temporal difference network, so as to obtain a temporal difference network with better prediction ability, thereby improving the prediction accuracy of the temporal difference network.

[0136] It should be noted that the above embodiment uses the temporal difference network to obtain the temporal difference information of the image pair as an example, and in other embodiments, the computer device can also use other methods to obtain the temporal difference information of the image pair, for example, the computer device uses a temporal difference algorithm to predict the temporal difference information of the image pair, such as a TD (Temporal Difference) algorithm. The embodiments of the present disclosure are not limited to this.

[0137] In the above embodiments, two methods of acquiring motion information of an image pair, namely, an optical flow method and a temporal difference method, are provided, both of which can quickly acquire motion information of an image pair, thereby improving the flexibility of acquiring motion information.

[0138] In step 302, the computer device obtains a sample image block pair whose motion information satisfies the motion condition from the image pair, wherein two sample image blocks in the sample image block pair are respectively located in two frames of the image pair.

[0139] The process of obtaining an image block pair from an image pair in step 302 is to obtain a sample image block whose motion information satisfies the motion condition from the previous frame image of the image pair, and to obtain a sample image block whose motion information satisfies the motion condition from the next frame image of the image pair, thereby obtaining a sample image block pair.

[0140] In some embodiments, the motion information includes a motion parameter, which indicates the degree of pixel motion between the two frames of the image pair. It should be understood that the larger the value of the motion parameter, the more intense the pixel motion between the two frames, and the smaller the value of the motion parameter, the more gentle the pixel motion between the two frames. Based on the two types of motion information shown in step 301, in some embodiments, the motion parameter is the amount of movement provided by the optical flow information (such as the optical flow map), or the motion parameter is the motion change value provided by the time difference information (time difference map). It should be understood that the motion parameter is obtained by subtracting the pixel position of the next frame of the image pair from the pixel position of the previous frame. In this way, by setting the motion parameter, the motion parameter can be used later to determine whether the motion information of the two frames of the image pair meets the motion condition, thereby improving the efficiency of judging the motion condition and thus improving the efficiency of obtaining the sample image block pair.

[0141] In some embodiments, the motion condition is that the motion parameter is greater than or equal to a first threshold value, and accordingly, the process by which the computer device obtains sample image block pairs whose motion information satisfies the motion condition is: from the image pair, obtain a motion image block pair whose motion parameter is greater than or equal to the first threshold value; in some further embodiments, the motion condition is that the motion parameter is less than a second threshold value, and accordingly, the process by which the computer device obtains sample image block pairs whose motion information satisfies the motion condition is: from the image pair, obtain a still image block pair whose motion parameter is less than the second threshold value; in other embodiments, the motion condition is that the motion parameter is greater than or equal to the first threshold value and the motion parameter is less than the second threshold value, and accordingly, the process by which the computer device obtains sample image block pairs whose motion information satisfies the motion condition is: from the image pair, obtain a motion image block pair whose motion parameter is greater than or equal to the first threshold value and a still image block pair whose motion parameter is less than the second threshold value.

[0142] Wherein, the first threshold is greater than or equal to the second threshold. In some embodiments, the first threshold and the second threshold are determined based on the motion parameters of multiple pixels included in any frame image in the image pair. Schematically, taking the first threshold being equal to the second threshold as an example, the mean of the motion parameters of multiple pixels included in any frame image in the image pair is determined as the first threshold and the second threshold; taking the first threshold being greater than the second threshold as an example, the mean of the motion parameters of multiple pixels included in any frame image in the image pair is determined, and the target value is added to the mean to obtain the first threshold, and the target value is subtracted from the mean to obtain the second threshold. Wherein, the target value is a pre-set value, such as 3 or 5 or other values. In this way, based on the motion parameters of multiple pixels included in any frame image in the image pair, the first threshold corresponding to the motion image block pair and the second threshold corresponding to the static image block pair are determined, so as to judge whether the pixel in the image is a motion pixel or a static pixel based on the first threshold and the second threshold, thereby improving the accuracy of judging the motion condition.

[0143] In this embodiment, a motion condition in which a motion parameter is greater than or equal to a first threshold is set to obtain a motion image block pair. Since the motion image block pair includes pixels in a motion state, when the motion image block pair is subsequently used for model training, the super-resolution model based on the motion image block pair can focus on the motion changes between two adjacent frames of images. A motion condition in which a motion parameter is less than a second threshold is set to obtain a still image block pair. Since the still image block pair includes pixels in a still state, when the still image block pair is subsequently used for model training, the super-resolution model based on the still image block pair can focus on the structural information of the image itself. In this way, the information referenced by the model training is enriched, and a highly accurate super-resolution model can be trained, thereby improving the accuracy of super-resolution reconstruction, and thus improving the accuracy of video processing.

[0144] Regarding the above-mentioned process of obtaining the motion image block pair, in some embodiments, the computer device determines the first pixel whose motion parameter is greater than or equal to the first threshold from the image pair, and captures the image block pair of the target size with the first pixel as the starting point to obtain the motion image block pair.

[0145] The first pixel is a pixel whose motion parameter is greater than or equal to the first threshold, that is, a pixel in motion. The target size is a pre-set size, such as P×P size, where P is a positive integer greater than 1. In some embodiments, the computer device determines the position of the first pixel whose motion parameter is greater than or equal to the first threshold from the image pair, and intercepts the image block pair of the target size with the position of the first pixel as the starting point to obtain the motion image block pair. In some embodiments, the position of the first pixel is represented by the coordinates of the first pixel.

[0146] In some embodiments, after the computer device determines the first pixel, the image block pair of the target size is intercepted with the first pixel as the upper left corner vertex; or, the image block pair of the target size is intercepted with the first pixel as the upper right corner vertex; or, the image block pair of the target size is intercepted with the first pixel as the lower left corner vertex; or, the image block pair of the target size is intercepted with the first pixel as the lower right corner vertex. The embodiments of the present disclosure are not limited to this.

[0147] In the above embodiment, in the process of acquiring motion image block pairs, by first determining the first pixel and then taking the first pixel as the starting point to capture the image block pairs of the target size, the motion image block pairs can be quickly acquired, thereby improving the efficiency of acquiring the motion image block pairs.

[0148] Regarding the above process of obtaining a still image block pair, in some embodiments, the computer device determines from the image pair a second pixel whose motion parameter is less than the second threshold, and captures an image block pair of a target size with the second pixel as a starting point to obtain the still image block pair.

[0149] The second pixel is a pixel whose motion parameter is less than the second threshold, that is, a pixel in a static state. In some embodiments, the computer device determines the position of the second pixel whose motion parameter is less than the second threshold from the image pair, and intercepts the image block pair of the target size with the position of the second pixel as the starting point to obtain the static image block pair. In some embodiments, the position of the second pixel is represented by the coordinates of the second pixel.

[0150] In some embodiments, after the computer device determines the second pixel, the second pixel is used as the upper left corner vertex to intercept the image block pair of the target size; or the second pixel is used as the upper right corner vertex to intercept the image block pair of the target size; or the second pixel is used as the lower left corner vertex to intercept the image block pair of the target size; or the second pixel is used as the lower right corner vertex to intercept the image block pair of the target size. The embodiments of the present disclosure are not limited to this.

[0151] In the above embodiment, in the process of acquiring a still image block pair, by first determining the second pixel and then using the second pixel as a starting point to capture an image block pair of a target size, the still image block pair can be quickly acquired, thereby improving the efficiency of acquiring the still image block pair.

[0152] With respect to the above-mentioned embodiment of simultaneously acquiring motion image block pairs and still image block pairs, in some embodiments, the number of the motion image block pairs is the same as the number of the still image block pairs. In this way, by acquiring an equal number of motion image block pairs and still image block pairs, so that the equal number of motion image block pairs and still image block pairs can be used as training data for model training in the future, a super-resolution model can be obtained. Since the motion image block pairs contain pixels in motion and the still image block pairs contain pixels in a still state, the model training is performed based on the motion image block pairs and the still image block pairs, so that the super-resolution model can focus on the motion changes between two adjacent frames of images based on the motion image block pairs, and can focus on the structural information of the image itself based on the still image block pairs, thereby enriching the information referenced by the model training, and being able to train a highly accurate super-resolution model, thereby improving the accuracy of super-resolution reconstruction, and thus improving the accuracy of video processing.

[0153] For example, Figure 4 is a schematic diagram of an optical flow graph according to an exemplary embodiment, see Figure 4 , Figure 4 The images shown on the left are image pairs in a sample video. It should be noted that: Figure 4 The left side presents the image pair by taking the overlap of two frames of the image pair as an example. Figure 4 The image shown on the right is the optical flow diagram of the image pair. It should be noted that Figure 4 The optical flow diagram shown uses color brightness as an example to present the degree of pixel movement. That is, the brighter the color, the more violent the pixel movement, and the darker the color, the more gentle the pixel movement. Figure 4 The optical flow graph shown is used as an exemplary illustration. In other embodiments, the optical flow graph can also be represented in other forms, such as using a two-dimensional vector to present the degree of pixel movement, such as the amount of pixel movement in the horizontal direction (x-axis direction) and the amount of pixel movement in the vertical direction (y-axis direction).

[0154] based on Figure 4 The optical flow diagram shown on the right is from Figure 4 In the image pair shown on the left, obtain a sample image block pair whose motion information (movement amount) satisfies the motion condition. For example, Figure 5 is a schematic diagram of an image block according to an exemplary embodiment. Figure 5 , Figure 5 The image blocks (a), (b), and (c) shown are image blocks in the optical flow map, wherein the image blocks (a) and (b) are image blocks whose motion parameters are greater than or equal to a first threshold, that is, motion image blocks, and the image block (c) is an image block whose motion parameter is less than a second threshold, that is, a static image block. Accordingly, Figure 5 The sample image blocks corresponding to the above-mentioned image blocks in the sample video are also shown, namely, sample image block (d), sample image block (e) and sample image block (f). It should be understood that after the embodiment of the present disclosure determines the image blocks whose motion information meets the motion condition based on the optical flow map, it is necessary to obtain the corresponding sample image blocks from the original image of the sample video.

[0155] In the disclosed embodiment, the sample video includes a sample video of a first resolution and a sample video of a second resolution, the first resolution being smaller than the second resolution. Accordingly, the above step 302 may be replaced by: the computer device obtains a first sample image block pair whose motion information satisfies the motion condition from the image pair of the sample video of the first resolution, and obtains a second sample image block pair whose motion information satisfies the motion condition from the image pair of the sample video of the second resolution.

[0156] The sample video of the first resolution and the sample video of the second resolution are both acquired based on a shooting device (such as a camera). The first sample image block pair is a sample image block pair in the sample video of the first resolution, that is, the first sample image block pair is a sample image block pair of the first resolution. The second sample image block pair is a sample image block pair in the sample video of the second resolution, that is, the second sample image block pair is a sample image block pair of the second resolution.

[0157] In this embodiment, a sample image block pair of a first resolution is obtained in a sample video of a first resolution, and a sample image block pair of a second resolution is obtained in a sample video of a second resolution, so that the sample image block pair of the first resolution and the sample image block pair of the second resolution are subsequently used as training data for model training to obtain a super-resolution model.

[0158] It should be noted that, based on the difference in resolution, the size of the first sample image block pair is different from the size of the second sample image block pair. In some embodiments, the target size includes a first size and a second size, wherein the first sample image block pair is the first size, and the second sample image block pair is the second size, the first size is smaller than the second size, and the ratio of the second size to the first size is a resolution magnification, and the resolution magnification represents the resolution ratio of the second resolution to the first resolution, such as 4. For example, assuming that the first size is P×P, the second size is 4P×4P, and P is a positive integer greater than 1.

[0159] In some embodiments, the process of the computer device acquiring the first sample image block pair and the second sample image block pair is as follows:

[0160] The first method: the computer device obtains the first sample image block pair from the image pair of the sample video of the first resolution based on the motion information of the image pair in the sample video of the first resolution, and obtains the second sample image block pair from the image pair of the sample video of the second resolution based on the first sample image block pair.

[0161] In some embodiments, the computer device obtains the first sample image block pair of the first size from the image pair of the sample video of the first resolution based on the motion information of the image pair in the sample video of the first resolution, and obtains the second sample image block pair of the second size from the image pair of the sample video of the second resolution based on the first sample image block pair.

[0162] Regarding the motion information of the image pair in the sample video of the first resolution, the process of obtaining the first sample image block pair from the image pair of the sample video of the first resolution refers to the process of obtaining the sample image block pair based on the motion parameters and motion conditions, which will not be repeated here.

[0163] The following describes a process of obtaining the second sample image block pair from the image pair of the sample video of the second resolution based on the first sample image block pair. In some embodiments, a computer device obtains the second sample image block pair from the image pair of the sample video of the second resolution based on a resolution magnification and the first sample image block pair.

[0164] In some embodiments, the computer device determines the position of the second sample image block pair based on the resolution magnification and the position of the first sample image block pair, and obtains the second sample image block pair from the image pair of the sample video of the second resolution based on the position of the second sample image block pair.

[0165] The position of the first sample image block pair is represented by the vertex coordinates of the first sample image block pair, and the position of the second sample image block pair is represented by the vertex coordinates of the second sample image block pair, and the vertex coordinates are the upper left vertex coordinates, the lower left vertex coordinates, the upper right vertex coordinates, or the lower right vertex coordinates. In an optional embodiment, the computer device performs a product operation based on the resolution magnification and the vertex coordinates of the first sample image block pair to obtain the vertex coordinates of the second sample image block pair, and in the image pair of the sample video of the second resolution, based on the vertex coordinates of the second sample image block pair, cuts out the image block pair of the second size to obtain the second sample image block pair.

[0166] Illustratively, taking the resolution magnification of 4 as an example and the vertex coordinates of the first sample image block pair as the upper left corner vertex coordinates (x, y) as an example, the computer device performs a multiplication operation based on the resolution magnification of 4 and the upper left corner vertex coordinates (x, y) of the first sample image block pair to obtain the upper left corner vertex coordinates (4x, 4y) of the second sample image block pair. Furthermore, based on the upper left corner vertex coordinates (4x, 4y) of the second sample image block pair, an image block pair of a second size is cut out from the image pair of the sample video of the second resolution with (4x, 4y) as the upper left corner vertex to obtain the second sample image block pair.

[0167] In this embodiment, after the first sample image block pair is acquired, the first sample image block pair can be converted from the first resolution to the second resolution based on the resolution magnification, that is, the second sample image block pair is obtained, thereby improving the efficiency of acquiring the sample image block pair.

[0168] The second type: the computer device obtains the second sample image block pair from the image pair of the sample video of the second resolution based on the motion information of the image pair in the sample video of the second resolution, and obtains the first sample image block pair from the image pair of the sample video of the first resolution based on the second sample image block pair.

[0169] In some embodiments, the computer device obtains the second sample image block pair of the second size from the image pair of the sample video of the second resolution based on the motion information of the image pair in the sample video of the second resolution, and obtains the first sample image block pair of the first size from the image pair of the sample video of the first resolution based on the second sample image block pair.

[0170] Regarding the process of obtaining the second sample image block pair from the image pair in the sample video of the second resolution based on the motion information of the image pair in the sample video of the second resolution, refer to the process of obtaining the sample image block pair based on the motion parameters and motion conditions, which will not be repeated here.

[0171] The following describes a process of obtaining the first sample image block pair from the image pair of the sample video of the first resolution based on the second sample image block pair. In some embodiments, a computer device obtains the first sample image block pair from the image pair of the sample video of the first resolution based on the resolution magnification and the second sample image block pair.

[0172] In some embodiments, the computer device determines the position of the first sample image block pair based on the resolution magnification and the position of the second sample image block pair, and obtains the first sample image block pair from the image pair of the sample video of the first resolution based on the position of the first sample image block pair.

[0173] In an optional embodiment, the computer device performs a division operation based on the resolution ratio and the vertex coordinates of the second sample image block pair to obtain the vertex coordinates of the first sample image block pair, and in the image pair of the sample video of the first resolution, based on the vertex coordinates of the first sample image block pair, cuts out an image block pair of a first size to obtain the first sample image block pair.

[0174] Illustratively, taking the resolution magnification of 4 as an example and the vertex coordinates of the second sample image block pair as the upper left vertex coordinates (x, y) as an example, the computer device performs a division operation based on the resolution magnification of 4 and the upper left vertex coordinates (x, y) of the second sample image block pair to obtain the upper left vertex coordinates (x / 4, y / 4) of the first sample image block pair. Furthermore, based on the upper left vertex coordinates (x / 4, y / 4) of the first sample image block pair, an image block pair of a first size is cut out from the image pair of the sample video of the first resolution with (x / 4, y / 4) as the upper left vertex to obtain the first sample image block pair.

[0175] In this embodiment, after the second sample image block pair is acquired, the second sample image block pair can be converted from the second resolution to the first resolution based on the resolution magnification, that is, the first sample image block pair is obtained, thereby improving the efficiency of acquiring the sample image block pair.

[0176] The third method: the computer device performs upsampling processing based on the motion information of the image pair in the sample video of the first resolution to obtain the motion information of the image pair in the sample video of the second resolution; based on the motion information of the image pair in the sample video of the second resolution, obtains the second sample image block pair from the image pair of the sample video of the second resolution; based on the second sample image block pair, obtains the first sample image block pair from the image pair of the sample video of the first resolution.

[0177] In the process of performing upsampling processing on the motion information of the image pair in the sample video based on the first resolution to obtain the motion information of the image pair in the sample video of the second resolution, in some embodiments, the computer device performs the upsampling processing based on a linear interpolation upsampling method, or the computer device performs the upsampling processing based on a deep learning upsampling method (such as deconvolution). The disclosed embodiments do not limit the content of the upsampling processing.

[0178] Regarding the motion information of the image pair in the sample video with the second resolution, the second sample image block pair is obtained from the image pair of the sample video with the second resolution, and based on the second sample image block pair, the first sample image block pair is obtained from the image pair of the sample video with the first resolution. Please refer to the content of the second implementation method mentioned above and will not be repeated here.

[0179] In the above embodiment, three methods for acquiring the first sample image block pair and the second sample image block pair are provided, which improves the flexibility of acquiring the first sample image block pair and the second sample image block pair.

[0180] It should be noted that the above step 301 is to input the image sequence included in the sample video into the optical flow network or the temporal difference network to obtain the motion information of the image pairs in the sample video, that is, to obtain the motion information of all the image pairs in the sample video, and then perform step 302 to obtain the image block pairs of all the image pairs in the sample video. In other embodiments, the computer device can also obtain the motion information of the image pair based on any image pair (any two adjacent frames of images) in the sample video, and perform step 302 based on the motion information of the image pair to obtain the sample image block pair in the image pair, and then, based on the position of the sample image block pair in the image pair, sequentially obtain the sample image blocks in the remaining images in the sample video, so as to obtain all the image blocks in the sample video. In this way, the computer device only needs to perform the above steps on a pair of images, which greatly reduces the processing content of the computer device and improves the processing efficiency of the computer device.

[0181] In step 303, the computer device performs pixel expansion at the outer edge position of the sample image block pair, and executes step 304 based on the sample image block pair after pixel expansion.

[0182] The outer edge position includes the outer edge positions of the upper, lower, left and right sides of the sample image block. In some embodiments, for any image block of the sample image block pair, the computer device performs pixel expansion at the outer edge positions of the upper, lower, left and right sides of the image block, and executes step 304 based on the sample image block pair after pixel expansion.

[0183] In some embodiments, the process of pixel expansion performed by the computer device includes any of the following:

[0184] In some embodiments, the computer device performs mirror inversion on the pixels at the inner edge position of the sample image block pair, and adds the mirror-inverted pixels to the outer edge position of the sample image block pair. In some embodiments, for any image block of the sample image block pair, the computer device performs mirror inversion on the pixels at the inner edge position of the sample image block, and adds the mirror-inverted pixels to the outer edge position of the sample image block.

[0185] Mirror inversion is also symmetrical flipping. In some embodiments, for any image block of the sample image block pair, the computer device mirror-inverts the pixels at the inner edge positions of the upper, lower, left, and right sides of the sample image block, and adds the mirror-inverted pixels to the outer edge positions of the upper, lower, left, and right sides of the sample image block.

[0186] In some further embodiments, the computer device copies the pixels at the inner edge position of the sample image block pair, and adds the copied pixels to the outer edge position of the sample image block pair. In some embodiments, for any image block of the sample image block pair, the computer device copies the pixels at the inner edge position of the sample image block, and adds the copied pixels to the outer edge position of the sample image block.

[0187] In some embodiments, for any image block of the sample image block pair, the computer device copies the pixels at the inner edge positions of the upper, lower, left and right sides of the sample image block, and adds the copied pixels to the outer edge positions of the upper, lower, left and right sides of the sample image block, respectively.

[0188] In other embodiments, the computer device adds zero pixels at the outer edge of the sample image block pair. In some embodiments, for any image block of the sample image block pair, the computer device adds zero pixels at the outer edge of the sample image block. A zero pixel refers to a pixel with a pixel value of 0, which is used to characterize the brightness (or grayscale) of the pixel location. Usually, a pixel with a pixel value of 0 is also a black pixel.

[0189] In some embodiments, for any image block of the sample image block pair, the computer device adds zero pixels to the outer edge positions of the top, bottom, left and right sides of the sample image block respectively.

[0190] In some embodiments, based on the first sample image block pair and the second sample image block pair shown in step 302, based on the difference in resolution, the pixel expansion number of the first sample image block pair is different from the pixel expansion number of the second sample image block pair. In some embodiments, the pixel expansion number of the first sample image block pair is the first expansion number, and the pixel expansion number of the second sample image block pair is the second expansion number, wherein the first expansion number is less than the second expansion number, and the ratio of the second expansion number to the first expansion number is the resolution magnification. For example, taking the resolution magnification as 4 as an example, assuming that the first expansion number is s, the second expansion number is 4s, and s is a positive integer greater than 1.

[0191] In the above embodiment, pixel expansion is performed at the outer edge of the sample image block pair by means of pixel expansion methods such as mirror inversion or copying pixels or adding zero pixels, thereby increasing the pixel information of the image edge. Subsequently, model training is performed based on the sample image block pairs after pixel expansion, so that the super-resolution model pays more attention to the reconstruction of the image edge, avoiding the problem of noise at the image edge, and can train a super-resolution model with high accuracy, thereby improving the accuracy of super-resolution reconstruction, and thus improving the accuracy of video processing.

[0192] It should be noted that the above step 303 is an optional step. In other embodiments, after obtaining the sample image block pairs whose motion information meets the motion condition based on step 302, step 304 is executed without executing step 303.

[0193] In step 304, the computer device performs model training based on the multiple pairs of sample image blocks in the sample video to obtain a super-resolution model, and the super-resolution model is used to perform super-resolution reconstruction on the video.

[0194] Based on the sample video of the first resolution and the sample video of the second resolution shown in step 302, in some embodiments, the computer device performs model training based on multiple pairs of first sample image block pairs in the sample video of the first resolution and multiple pairs of second sample image block pairs in the sample video of the second resolution to obtain the super-resolution model.

[0195] Furthermore, in some embodiments, during the i-th iteration of model training, the computer device inputs multiple pairs of first sample image blocks into the super-resolution model determined by the i-1-th iteration to obtain an image training result of the i-th iteration; based on the image training result of the i-th iteration and the multiple pairs of second sample image blocks, the model parameters of the super-resolution model determined by the i-1-th iteration are adjusted, and the i+1-th iteration is performed based on the adjusted model parameters, and the above training iteration process is repeated until the training meets the target conditions.

[0196] In some embodiments, during the first iteration of model training, the computer device inputs multiple pairs of first sample image blocks into the initial model to obtain image training results of the first iteration, adjusts model parameters of the initial model based on the image training results of the first iteration and multiple pairs of second sample image blocks, performs the next iteration based on the adjusted model parameters, and repeats the above training iteration process until the training meets the target conditions.

[0197] In the above embodiment, the network model with better model parameters is obtained as a super-resolution model through iterative training, so as to obtain a super-resolution model with better super-resolution reconstruction capability, thereby improving the accuracy of the super-resolution model.

[0198] In some embodiments, after the super-resolution model is obtained through training based on the above steps 301 to 304, in the testing phase of the super-resolution model, the computer device also uses the super-resolution model to perform super-resolution reconstruction on the video to obtain an initial super-resolution video, and for any frame image in the super-resolution video, the pixels at the inner edge position of the image are cut, and based on the cut image, a target super-resolution video of the video is generated. In this embodiment, after the video is super-resolution reconstructed, the pixels at the inner edge position of the image in the output initial super-resolution video are cut to remove redundant pixels, thereby obtaining the target super-resolution video of the video.

[0199] Schematically, assume that before the video is super-resolution reconstructed, the outer edge position of the video is expanded by q pixels, where q is a positive integer greater than 1. After the initial super-resolution video is obtained based on the super-resolution model, for any frame image of the initial super-resolution video, 4q pixels are cut from the inner edge position of the image, and the cut super-resolution video is used to determine the target super-resolution video.

[0200] The technical solution provided by the embodiments of the present disclosure obtains motion information of an image pair in a sample video, and uses the motion information of the image pair to obtain an image block pair in the image pair whose motion information satisfies a motion condition. In this way, by setting the motion condition, the motion information of two frames of the image pair is used to determine whether the motion information of the two frames of the image pair satisfies the motion condition, so as to obtain image blocks containing more motion information, and then perform model training based on the obtained image block pairs, so that the super-resolution model can be based on the motion changes between two adjacent frames of the image, enriching the information referenced by the model training, and being able to train a super-resolution model with high accuracy, thereby improving the accuracy of super-resolution reconstruction, and thus improving the accuracy of video processing.

[0201] In the disclosed embodiment, Figure 6is a schematic diagram showing a super-resolution test result according to an exemplary embodiment. Figure 6 Among the three images shown, the image on the left is the original image, the image in the middle is the image after super-resolution reconstruction using the solution of randomly intercepting training data in the related art, and the image on the right is the image after super-resolution reconstruction using the embodiment of the present disclosure. It can be found that the image obtained using the related art is still not clear enough in the restoration of the license plate numbers, and the digital blur phenomenon occurs. The images obtained using the embodiment of the present disclosure are clear in the restoration of the license plate numbers. Moreover, in the related art, only the random interception method is used to obtain image blocks, and the super-resolution model trained will pay more attention to the reconstruction inside the image, while ignoring the reconstruction of the image edge, which is easy to generate noise at the image edge. The model training using the solution provided by the embodiment of the present disclosure can effectively avoid the problem of noise at the image edge.

[0202] Figure 7 is a block diagram of a video processing device according to an exemplary embodiment. Figure 7 The device includes an information acquisition unit 701, an image acquisition unit 702 and a training unit 703.

[0203] The information acquisition unit 701 is configured to acquire motion information of an image pair in a sample video, where the image pair includes a pair of adjacent images in the sample video, and the motion information indicates pixel motion between two frames of the image pair;

[0204] The image acquisition unit 702 is configured to acquire, from the image pair, a pair of sample image blocks whose motion information satisfies a motion condition, wherein two sample image blocks in the pair of sample image blocks are respectively located in two frames of the image pair;

[0205] The training unit 703 is configured to perform model training based on multiple pairs of sample image blocks in the sample video to obtain a super-resolution model, and the super-resolution model is used to perform super-resolution reconstruction on the video.

[0206] The technical solution provided by the embodiments of the present disclosure obtains motion information of an image pair in a sample video, and uses the motion information of the image pair to obtain an image block pair in the image pair whose motion information satisfies a motion condition. In this way, by setting the motion condition, the motion information of two frames of the image pair is used to determine whether the motion information of the two frames of the image pair satisfies the motion condition, so as to obtain image blocks containing more motion information, and then perform model training based on the obtained image block pairs, so that the super-resolution model can be based on the motion changes between two adjacent frames of the image, enriching the information referenced by the model training, and being able to train a super-resolution model with high accuracy, thereby improving the accuracy of super-resolution reconstruction, and thus improving the accuracy of video processing.

[0207] In some embodiments, the information acquisition unit 701 is configured to perform any of the following: obtaining optical flow information of the image pair in the sample video, the optical flow information indicating the amount of movement of pixels from a previous frame image to a next frame image in the image pair; or obtaining temporal difference information of the image pair in the sample video, the temporal difference information indicating the motion change of two frames of the image pair in time sequence.

[0208] In some embodiments, the motion information includes motion parameters, which represent the degree of pixel motion between two frames of the image pair; the image acquisition unit 702 is configured to execute, from the image pair, acquiring a motion image block pair whose motion parameters are greater than or equal to a first threshold and a still image block pair whose motion parameters are less than a second threshold, wherein the first threshold is greater than or equal to the second threshold.

[0209] In some embodiments, the image acquisition unit 702 is configured to perform: determining, from the image pair, the first pixel whose motion parameter is greater than or equal to the first threshold, and intercepting the image block pair of the target size with the first pixel as the starting point to obtain the motion image block pair; determining the second pixel whose motion parameter is less than the second threshold, and intercepting the image block pair of the target size with the second pixel as the starting point to obtain the still image block pair.

[0210] In some embodiments, the first threshold and the second threshold are determined based on motion parameters of a plurality of pixels included in any frame image in the image pair.

[0211] In some embodiments, the number of the moving image block pairs is the same as the number of the still image block pairs.

[0212] In some embodiments, the sample video includes a sample video of a first resolution and a sample video of a second resolution, the first resolution being smaller than the second resolution; the image acquisition unit 702 is configured to execute, from the image pair of the sample video of the first resolution, acquiring a first sample image block pair whose motion information satisfies the motion condition, and from the image pair of the sample video of the second resolution, acquiring a second sample image block pair whose motion information satisfies the motion condition, the first sample image block pair being a sample image block pair of the first resolution, and the second sample image block pair being a sample image block pair of the second resolution; the training unit 703 is configured to execute model training based on multiple pairs of first sample image block pairs in the sample video of the first resolution and multiple pairs of second sample image block pairs in the sample video of the second resolution to obtain the super-resolution model.

[0213] In some embodiments, the image acquisition unit 702 includes any of the following: a first image acquisition subunit, configured to execute based on the motion information of the image pair in the sample video of the first resolution, to acquire the first sample image block pair from the image pair of the sample video of the first resolution, and based on the first sample image block pair, to acquire the second sample image block pair from the image pair of the sample video of the second resolution; a second image acquisition subunit, configured to execute based on the motion information of the image pair in the sample video of the second resolution, to acquire the second sample image block pair from the image pair of the sample video of the second resolution, and based on the second sample image block pair, to acquire the first sample image block pair from the image pair of the sample video of the first resolution; a third image acquisition subunit, configured to execute based on the motion information of the image pair in the sample video of the first resolution, to perform upsampling processing to obtain the motion information of the image pair in the sample video of the second resolution, to acquire the second sample image block pair from the image pair of the sample video of the second resolution based on the motion information of the image pair in the sample video of the second resolution, and to acquire the first sample image block pair from the image pair of the sample video of the first resolution based on the second sample image block pair.

[0214] In some embodiments, the first image acquisition subunit is configured to execute, based on a resolution magnification and the first sample image block pair, to acquire the second sample image block pair from the image pair of the sample video of the second resolution, wherein the resolution magnification represents a resolution ratio between the second resolution and the first resolution; and the second image acquisition subunit is configured to execute, based on the resolution magnification and the second sample image block pair, to acquire the first sample image block pair from the image pair of the sample video of the first resolution.

[0215] In some embodiments, the training unit 703 is configured to perform: in the i-th iteration process of model training, multiple pairs of first sample image blocks are input into the super-resolution model determined by the i-1-th iteration process to obtain the image training result of the i-th iteration process, where i is a positive integer greater than 1; based on the image training result of the i-th iteration process and multiple pairs of second sample image blocks, the model parameters of the super-resolution model determined by the i-1-th iteration process are adjusted, and the i+1-th iteration process is performed based on the adjusted model parameters, and the above-mentioned iterative process of training is repeated until the training meets the target conditions.

[0216] In some embodiments, the device further includes any of the following: a mirror inversion unit, configured to perform mirror inversion on pixels at the inner edge position of the sample image block pair, and add the mirror-inverted pixels to the outer edge position of the sample image block pair; a copying unit, configured to copy the pixels at the inner edge position of the sample image block pair, and add the copied pixels to the outer edge position of the sample image block pair; an adding unit, configured to add zero pixels at the outer edge position of the sample image block pair.

[0217] In some embodiments, the device also includes: a super-resolution unit, configured to perform super-resolution reconstruction on the video through the super-resolution model to obtain an initial super-resolution video; a cutting unit, configured to perform cutting of pixels at the inner edge position of any frame image in the super-resolution video, and generate a target super-resolution video of the video based on the cut image.

[0218] It should be noted that: the video processing device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate the video processing. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the video processing device provided in the above embodiment and the video processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0219] The computer device mentioned in the embodiments of the present disclosure may be provided as a terminal. Figure 8 800 is a block diagram of a terminal 800 according to an exemplary embodiment. The terminal 800 may be a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 800 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0220] Typically, the terminal 800 includes a processor 801 and a memory 802 .

[0221] The processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 801 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0222] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one program code, which is used to be executed by the processor 801 to implement the process executed by the terminal in the video processing method provided in the method embodiment of the present disclosure.

[0223] In some embodiments, the terminal 800 may further optionally include: a peripheral device interface 803 and at least one peripheral device. The processor 801, the memory 802 and the peripheral device interface 803 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 803 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807, a positioning assembly 808 and a power supply 809.

[0224] The peripheral device interface 803 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 801 and the memory 802. In some embodiments, the processor 801, the memory 802, and the peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 801, the memory 802, and the peripheral device interface 803 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0225] The radio frequency circuit 804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 804 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 804 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 804 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0226] The display screen 805 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 805 is a touch display screen, the display screen 805 also has the ability to collect touch signals on the surface or above the surface of the display screen 805. The touch signal can be input to the processor 801 as a control signal for processing. At this time, the display screen 805 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 805 can be one, set on the front panel of the terminal 800; in other embodiments, the display screen 805 can be at least two, respectively set on different surfaces of the terminal 800 or in a folding design; in other embodiments, the display screen 805 can be a flexible display screen, set on the curved surface or folding surface of the terminal 800. Even, the display screen 805 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 805 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0227] The camera assembly 806 is used to capture images or videos. Optionally, the camera assembly 806 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 806 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0228] The audio circuit 807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 801 for processing, or input them into the radio frequency circuit 804 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 800. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 801 or the radio frequency circuit 804 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 807 may also include a headphone jack.

[0229] The positioning component 808 is used to locate the current geographical location of the terminal 800 to implement navigation or LBS (Location Based Service).

[0230] The power supply 809 is used to power various components in the terminal 800. The power supply 809 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 809 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0231] In some embodiments, the terminal 800 further includes one or more sensors 810 , including but not limited to: an acceleration sensor 811 , a gyroscope sensor 812 , a pressure sensor 813 , a fingerprint sensor 814 , an optical sensor 815 , and a proximity sensor 816 .

[0232] The acceleration sensor 811 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 800. For example, the acceleration sensor 811 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 801 can control the display screen 805 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 811. The acceleration sensor 811 can also be used to collect game or user motion data.

[0233] The gyroscope sensor 812 can detect the body direction and rotation angle of the terminal 800, and the gyroscope sensor 812 can cooperate with the acceleration sensor 811 to collect the user's 3D actions on the terminal 800. The processor 801 can implement the following functions based on the data collected by the gyroscope sensor 812: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0234] The pressure sensor 813 can be set on the side frame of the terminal 800 and / or the lower layer of the display screen 805. When the pressure sensor 813 is set on the side frame of the terminal 800, it can detect the user's holding signal of the terminal 800, and the processor 801 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 813. When the pressure sensor 813 is set on the lower layer of the display screen 805, the processor 801 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 805. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0235] The optical sensor 814 is used to collect the ambient light intensity. In one embodiment, the processor 801 can control the display brightness of the display screen 805 according to the ambient light intensity collected by the optical sensor 814. Specifically, when the ambient light intensity is high, the display brightness of the display screen 805 is increased; when the ambient light intensity is low, the display brightness of the display screen 805 is reduced. In another embodiment, the processor 801 can also dynamically adjust the shooting parameters of the camera assembly 806 according to the ambient light intensity collected by the optical sensor 814.

[0236] The proximity sensor 815, also called a distance sensor, is usually arranged on the front panel of the terminal 800. The proximity sensor 815 is used to collect the distance between the user and the front of the terminal 800. In one embodiment, when the proximity sensor 815 detects that the distance between the user and the front of the terminal 800 is gradually decreasing, the processor 801 controls the display screen 805 to switch from the screen-on state to the screen-off state; when the proximity sensor 815 detects that the distance between the user and the front of the terminal 800 is gradually increasing, the processor 801 controls the display screen 805 to switch from the screen-off state to the screen-on state.

[0237] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the terminal 800, and the terminal 800 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0238] The computer device mentioned in the embodiments of the present disclosure may be provided as a server. Fig. 9It is a block diagram of a server according to an exemplary embodiment. The server 900 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 901 and one or more memories 902, wherein the one or more memories 902 store at least one program code, and the at least one program code is loaded and executed by the one or more processors 901 to implement the process executed by the server in the video processing method provided by the above-mentioned various method embodiments. Of course, the server 900 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 900 may also include other components for implementing device functions, which will not be described in detail here.

[0239] In an exemplary embodiment, a computer-readable storage medium including a program code is also provided, such as a memory 902 including the program code, and the program code can be executed by a processor 901 of a server 900 to complete the above-mentioned video processing method. Optionally, the computer-readable storage medium can be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact-Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0240] In an exemplary embodiment, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the above-mentioned video processing method is implemented.

[0241] In some embodiments, the computer program involved in the embodiments of the present disclosure may be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected through a communication network. Multiple computer devices distributed at multiple locations and interconnected through a communication network may constitute a blockchain system.

[0242] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0243] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video processing method, characterized in that: The method comprises: Acquire motion information of an image pair in a sample video, the image pair comprising a pair of adjacent images in the sample video, the motion information representing pixel motion between two frames of the image pair, the sample video comprising a sample video of a first resolution and a sample video of a second resolution, the first resolution being smaller than the second resolution; From an image pair of a sample video of the first resolution, a first sample image block pair whose motion information satisfies a motion condition is obtained; and from an image pair of a sample video of the second resolution, a second sample image block pair whose motion information satisfies a motion condition is obtained, wherein the first sample image block pair is a sample image block pair of the first resolution, and the second sample image block pair is a sample image block pair of the second resolution, and two sample image blocks in the sample image block pair are respectively located in two frames of the image pair; In the i-th iteration process of model training, multiple pairs of the first sample image blocks are input into the super-resolution model determined in the i-1-th iteration process to obtain the image training result of the i-th iteration process, where i is a positive integer greater than 1; Based on the image training result of the i-th iteration process and a plurality of pairs of the second sample image block pairs, adjusting the model parameters of the super-resolution model determined in the i-1-th iteration process, performing the i+1-th iteration process based on the adjusted model parameters, repeating the above training iteration process until the training meets the target condition, and the super-resolution model is used to perform super-resolution reconstruction on the video; The motion information includes motion parameters, and the motion parameters represent the degree of pixel motion between two frames of the image pair; obtaining the image block pair whose motion information satisfies the motion condition includes: obtaining, from the image pair, a motion image block pair whose motion parameters are greater than or equal to a first threshold and a still image block pair whose motion parameters are less than a second threshold, wherein the first threshold is greater than or equal to the second threshold.

2. The video processing method according to claim 1, characterized in that: The obtaining of motion information of image pairs in the sample video includes any of the following: Obtaining optical flow information of an image pair in the sample video, wherein the optical flow information indicates a movement amount of pixels from a previous frame image to a next frame image in the image pair; or, The time sequence difference information of the image pair in the sample video is obtained, where the time sequence difference information represents the motion change of two frames of the image pair in time sequence.

3. The video processing method according to claim 1, characterized in that: The acquiring, from the image pair, a pair of motion image blocks having a motion parameter greater than or equal to a first threshold and a pair of still image blocks having a motion parameter less than a second threshold comprises: Determine, from the image pair, a first pixel whose motion parameter is greater than or equal to the first threshold, and intercept an image block pair of a target size with the first pixel as a starting point to obtain the motion image block pair; A second pixel whose motion parameter is smaller than the second threshold is determined, and an image block pair of a target size is intercepted with the second pixel as a starting point to obtain the still image block pair.

4. The video processing method according to claim 1 or 3, characterized in that: The first threshold and the second threshold are determined based on motion parameters of a plurality of pixels included in any frame image in the image pair.

5. The video processing method according to claim 1 or 3, characterized in that: The number of the moving image block pairs is the same as the number of the still image block pairs.

6. The video processing method according to claim 1, characterized in that: The step of obtaining, from the image pair of the sample video of the first resolution, a first sample image block pair whose motion information satisfies the motion condition, and obtaining, from the image pair of the sample video of the second resolution, a second sample image block pair whose motion information satisfies the motion condition comprises any of the following: Based on motion information of image pairs in the sample video of the first resolution, obtaining the first sample image block pair from the image pairs of the sample video of the first resolution, and based on the first sample image block pair, obtaining the second sample image block pair from the image pairs of the sample video of the second resolution; Based on the motion information of the image pair in the sample video of the second resolution, obtaining the second sample image block pair from the image pair of the sample video of the second resolution, and based on the second sample image block pair, obtaining the first sample image block pair from the image pair of the sample video of the first resolution; Based on the motion information of the image pairs in the sample video of the first resolution, upsampling processing is performed to obtain the motion information of the image pairs in the sample video of the second resolution; based on the motion information of the image pairs in the sample video of the second resolution, the second sample image block pairs are obtained from the image pairs of the sample video of the second resolution; based on the second sample image block pairs, the first sample image block pairs are obtained from the image pairs of the sample video of the first resolution.

7. The video processing method according to claim 6, characterized in that: The acquiring the second sample image block pair from the image pair of the sample video of the second resolution based on the first sample image block pair comprises: Based on a resolution ratio and the first sample image block pair, obtaining the second sample image block pair from an image pair of a sample video of the second resolution, the resolution ratio indicating a resolution ratio of the second resolution to the first resolution; The acquiring the first sample image block pair from the image pair of the sample video of the first resolution based on the second sample image block pair comprises: Based on the resolution magnification and the second sample image block pair, the first sample image block pair is obtained from an image pair of the sample video of the first resolution.

8. The video processing method according to claim 1, characterized in that: Before performing model training based on the multiple pairs of sample image block pairs in the sample video to obtain the super-resolution model, the method further includes any of the following: Performing mirror inversion on pixels at the inner edge position of the sample image block pair, and adding the mirror inverted pixels to the outer edge position of the sample image block pair; Copying pixels at the inner edge position of the sample image block pair, and adding the copied pixels to the outer edge position of the sample image block pair; Zero pixels are added at the outer edge positions of the sample image block pairs.

9. The video processing method according to claim 8, characterized in that: After the model training is performed based on the multiple pairs of sample image block pairs in the sample video to obtain the super-resolution model, the method further includes: Performing super-resolution reconstruction on the video using the super-resolution model to obtain an initial super-resolution video; For any frame image in the super-resolution video, pixels at the inner edge position of the image are cut, and based on the cut image, a target super-resolution video of the video is generated.

10. A video processing device, characterized in that: The device comprises: an information acquisition unit, configured to acquire motion information of an image pair in a sample video, the image pair comprising a pair of adjacent images in the sample video, the motion information representing pixel motion between two frames of the image pair, the sample video comprising a sample video of a first resolution and a sample video of a second resolution, the first resolution being smaller than the second resolution; an image acquisition unit, configured to acquire, from an image pair of a sample video of the first resolution, a first sample image block pair whose motion information satisfies a motion condition, and acquire, from an image pair of a sample video of the second resolution, a second sample image block pair whose motion information satisfies a motion condition, wherein the first sample image block pair is a sample image block pair of the first resolution, and the second sample image block pair is a sample image block pair of the second resolution, and two sample image blocks in the sample image block pair are respectively located in two frames of the image pair; The training unit is configured to perform, in an i-th iteration process of model training, inputting a plurality of pairs of the first sample image blocks into the super-resolution model determined in the i-1th iteration process to obtain an image training result of the i-th iteration process, where i is a positive integer greater than 1; adjusting model parameters of the super-resolution model determined in the i-1th iteration process based on the image training result of the i-th iteration process and a plurality of pairs of the second sample image blocks, performing an i+1th iteration process based on the adjusted model parameters, and repeating the above-mentioned iteration process of training until the training meets the target condition, and the super-resolution model is used to perform super-resolution reconstruction on the video; The motion information includes motion parameters, which represent the degree of pixel motion between two frames of the image pair; the image acquisition unit is also configured to execute, from the image pair, acquiring a pair of motion image blocks whose motion parameters are greater than or equal to a first threshold and a pair of still image blocks whose motion parameters are less than a second threshold, wherein the first threshold is greater than or equal to the second threshold.

11. A computer device, characterized in that: The computer device comprises: one or more processors; a memory for storing program code executable by the processor; The processor is configured to execute the program code to implement the video processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: When the program code in the computer-readable storage medium is executed by a processor of a computer device, the computer device is enabled to execute the video processing method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the video processing method according to any one of claims 1 to 9 is implemented.

Images