Video processing method, device and equipment

Abstract

The present application discloses a video processing method, device and equipment, which are applied to the field of image technology. The technical solution provided by the embodiment of the present application can decouple different types of motion areas in the front and rear adjacent video frames of the video frame based on the motion situation in the video frame when super-resolution processing is required, and perform corresponding video feature fusion and other processing based on different types of motion areas, omitting the motion estimation and compensation steps based on pixel-by-pixel operations, simplifying the spatial complexity of the network, saving video processing time, and because the features of different types of motion areas are extracted separately, the extracted features are more accurate, which can effectively avoid the same processing of all areas, and there will be no inaccurate motion estimation of fast-moving areas or occluded areas, thereby avoiding the problem of artifacts in the video and greatly improving the clarity of the video.

Description

Technical Field

[0001] The present application relates to the field of image technology, and in particular to a video processing method, device and equipment. Background Art

[0002] In video processing tasks, video super-resolution technology is very important. Its purpose is to convert low-resolution videos into high-resolution videos. In the video processing process, it is necessary to fill in the missing information during resolution conversion. The current mainstream approach is to use convolutional neural networks to perform motion estimation and compensation on videos to achieve video super-resolution.

[0003] The video processing method in the related art uses the optical flow between the front and rear adjacent video frames to perform motion estimation and compensation on the video. For a certain video frame, firstly, a certain adjacent video frame and the video frame are spliced ​​in the color channel, and the spliced ​​image is upsampled twice to obtain low-resolution, medium-resolution and high-resolution images. The three resolution images are input into the convolutional neural network, and the optical flow map between the adjacent video frame and the video frame is extracted through the convolutional neural network. Then, based on the optical flow map, the adjacent video frame is aligned to the video frame, and the above operation is performed on the previous and next adjacent video frames of the video frame. Finally, the aligned front and back adjacent video frames and the video frame are fused to obtain a video frame with improved resolution. The above operation is performed on multiple video frames to obtain the processed video.

[0004] However, for the above video processing method, the extraction of optical flow maps and alignment based on optical flow maps are performed pixel by pixel, which makes the whole process time-consuming. When there is occlusion or large movement between adjacent frames, the optical flow estimation is inaccurate, and the distortion of pixels based on inaccurate optical flow will produce artifacts, resulting in unclear video images. Summary of the invention

[0005] The embodiments of the present application provide a video processing method, apparatus, and device, which can save video processing time, avoid artifact problems in the video, and greatly improve the clarity of the video. The technical solution is as follows:

[0006] In one aspect, a video processing method is provided, the method comprising:

[0007] Based on a video frame of a video and a previous video frame of the video frame, a first image and a second image of the previous video frame are determined, wherein the first image includes an area where the motion condition meets the target condition, and the second image includes an area where the motion condition does not meet the target condition.

[0008] Based on a video frame of the video and a subsequent video frame of the video frame, determining a first image and a second image of the subsequent video frame;

[0009] Based on a first feature map and a second feature map of the video frame, obtaining a fused feature map of the video frame, wherein the first feature map is determined based on the video frame, a first image of the previous video frame, and a first image of the next video frame, and the second feature map is determined based on the video frame, a second image of the previous video frame, and a second image of the next video frame;

[0010] Based on the fused feature map of the multiple video frames of the video and the multiple video frames, a target video of the video is generated.

[0011] In some embodiments, the video frame based on the video and the previous video frame of the video frame, determining the first image and the second image of the previous video frame includes: obtaining a first residual map of the video frame, the first residual map is the residual between the video frame and the previous video frame of the video frame; based on the first residual map, determining a pixel value threshold corresponding to the previous video frame, the pixel value threshold is the mean pixel value of the pixel points in the first residual map; based on the pixel value threshold corresponding to the previous video frame and the first residual map, generating the first image and the second image of the previous video frame.

[0012] In some embodiments, generating the first image and the second image of the previous video frame based on the pixel value threshold corresponding to the previous video frame and the first residual map includes: determining, in the first residual map, an area whose pixel value is greater than or equal to the pixel value threshold based on the pixel value threshold; generating a mask of the previous video frame based on an area in the previous video frame corresponding to the determined area; generating the first image and the second image of the previous video frame based on the mask of the previous video frame and the first image of the previous video frame.

[0013] In some embodiments, generating the first image and the second image of the previous video frame based on the mask of the previous video frame and the first image of the previous video frame includes: multiplying the mask of the previous video frame by the previous video frame to obtain the first image of the previous video frame; and subtracting the first image of the previous video frame from the previous video frame to obtain the second image of the previous video frame.

[0014] In some embodiments, the step of obtaining a fused feature map of the video frame based on the first feature map and the second feature map of the video frame includes: obtaining the first feature map and the second feature map of the video frame; and fusing the first feature map and the second feature map to obtain a fused feature map of the video frame.

[0015] In some embodiments, the process of acquiring the first feature map of the video frame includes: splicing the video frame, the first image of the previous video frame and the first image of the next video frame to obtain a first spliced ​​video frame of the video frame; inputting the first spliced ​​video frame of the video frame into a first convolutional neural network, performing feature extraction on the first spliced ​​video frame of the video frame through the first convolutional neural network to obtain a first feature map, and the first convolutional neural network is trained using a first sample video frame, and the first sample video frame includes an area whose motion meets the target condition.

[0016] In some embodiments, the process of acquiring the second feature map of the video frame includes: splicing the video frame, the second image of the previous video frame and the second image of the subsequent video frame to obtain a second spliced ​​video frame of the video frame; inputting the second spliced ​​video frame of the video frame into a second convolutional neural network, and performing feature extraction on the second spliced ​​video frame of the video frame through the second convolutional neural network to obtain a second feature map, and the second convolutional neural network is trained using a second sample video frame, and the second sample video frame includes an area where the motion does not meet the target condition.

[0017] In some embodiments, the fused feature map of multiple video frames of the video and the multiple video frames to generate a target video of the video include: for each video frame in the video, upsampling the fused feature map of the video frame to obtain a target feature map; upsampling the video frame to obtain an upsampled video frame corresponding to the video frame; fusing the target feature map with the upsampled video frame corresponding to the video frame to obtain a target video frame of the video frame; encoding the target video frames corresponding to the multiple video frames to generate a target video.

[0018] In some embodiments, the video processing method further includes: performing scene switching detection based on the acquired video frame, and if it is detected that no scene switching occurs in the video frame, performing processing on the video frame.

[0019] In one aspect, a video processing device is provided, the device comprising:

[0020] A determining unit is configured to determine, based on a video frame of the video and a previous video frame of the video frame, a first image and a second image of the previous video frame, wherein the first image includes an area where the motion condition meets the target condition, and the second image includes an area where the motion condition does not meet the target condition;

[0021] The determining unit is further configured to determine the first image and the second image of the subsequent video frame based on the video frame of the video and the subsequent video frame of the video frame;

[0022] a fusion unit configured to execute, based on a first feature map and a second feature map of the video frame, to obtain a fused feature map of the video frame, wherein the first feature map is determined based on the video frame, a first image of the previous video frame, and a first image of the next video frame, and the second feature map is determined based on the video frame, a second image of the previous video frame, and a second image of the next video frame;

[0023] A generating unit is configured to generate a target video of the video based on the fused feature map of the multiple video frames of the video and the multiple video frames.

[0024] In some embodiments, the determination unit includes: a first acquisition subunit, configured to execute acquisition of a first residual map of the video frame, the first residual map being the residual between the video frame and a previous video frame of the video frame; a determination subunit, configured to execute determination of a pixel value threshold corresponding to the previous video frame based on the first residual map, the pixel value threshold being the mean pixel value of the pixel points in the first residual map; and a generation subunit, configured to execute generation of a first image and a second image of the previous video frame based on the pixel value threshold corresponding to the previous video frame and the first residual map.

[0025] In some embodiments, the generating subunit includes: a first determining subunit, configured to determine, based on the pixel value threshold, an area in the first residual map whose pixel value is greater than or equal to the pixel value threshold; a first generating subunit, configured to generate a mask of the previous video frame based on an area in the previous video frame corresponding to the determined area; and a second generating subunit, configured to generate a first image and a second image of the previous video frame based on the mask of the previous video frame and the first image of the previous video frame.

[0026] In some embodiments, the second generating subunit is configured to execute: multiplying the mask of the previous video frame by the previous video frame to obtain a first image of the previous video frame; and subtracting the first image of the previous video frame from the previous video frame to obtain a second image of the previous video frame.

[0027] In some embodiments, the fusion unit includes: a second acquisition subunit, configured to execute acquisition of a first feature map and a second feature map of the video frame; and a fusion subunit, configured to execute fusion of the first feature map and the second feature map to obtain a fused feature map of the video frame.

[0028] In some embodiments, the second acquisition subunit is configured to execute: splicing the video frame, the first image of the previous video frame, and the first image of the next video frame to obtain a first spliced ​​video frame of the video frame; inputting the first spliced ​​video frame of the video frame into a first convolutional neural network, and performing feature extraction on the first spliced ​​video frame of the video frame through the first convolutional neural network to obtain a first feature map, and the first convolutional neural network is trained using a first sample video frame, and the first sample video frame includes an area whose motion meets the target condition.

[0029] In some embodiments, the second acquisition subunit is further configured to execute: splicing the video frame, the second image of the previous video frame, and the second image of the subsequent video frame to obtain a second spliced ​​video frame of the video frame; inputting the second spliced ​​video frame of the video frame into a second convolutional neural network, and performing feature extraction on the second spliced ​​video frame of the video frame through the second convolutional neural network to obtain a second feature map, and the second convolutional neural network is trained using a second sample video frame, and the second sample video frame includes an area where the motion does not meet the target condition.

[0030] In some embodiments, the generating unit is configured to perform, for each video frame in the video, upsampling the fused feature map of the video frame to obtain a target feature map; upsampling the video frame to obtain an upsampled video frame corresponding to the video frame; fusing the target feature map with the upsampled video frame corresponding to the video frame to obtain a target video frame of the video frame; encoding the target video frames corresponding to the multiple video frames to generate a target video.

[0031] In some embodiments, the video processing device performs scene switching detection based on the acquired video frame and, if it is detected that no scene switching occurs in the video frame, executes the processing of the video frame by the determination unit, the fusion unit and the generation unit.

[0032] In one aspect, a computer device is provided, the computer device comprising:

[0033] one or more processors;

[0034] A memory for storing program codes executable by the processor;

[0035] The processor is configured to execute the program code to implement the above-mentioned video processing method.

[0036] On the one hand, a computer-readable storage medium is provided. When the program code in the computer-readable storage medium is executed by a processor of a server, the server can execute the above-mentioned video processing method.

[0037] In one aspect, a computer program product is provided, comprising a computer program, which implements the above-mentioned video processing method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 It is a schematic diagram of an implementation environment of a video processing method provided in an embodiment of the present application;

[0040] Figure 2 is a flow chart of a video processing method provided by an embodiment of the present application;

[0041] Figure 3 is a flow chart of a video processing method provided by an embodiment of the present application;

[0042] Figure 4 It is a schematic diagram of a first convolutional neural network structure provided in an embodiment of the present application;

[0043] Figure 5 This is a schematic diagram of the structure of an upsampling module provided in an embodiment of the present application;

[0044] Figure 6 It is a motion area division validity verification diagram provided by an embodiment of the present application;

[0045] Figure 7 This is a comparison diagram of video processing effects provided by an embodiment of the present application;

[0046] Figure 8 is a structural schematic diagram of a video processing device provided in an embodiment of the present application;

[0047] Fig. 9 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0048] Fig.10 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0050] In the present application, the terms "first", "second", etc. are used to distinguish between identical or similar items having substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is the quantity and execution order limited. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of various examples, a first image can be referred to as a second image, and similarly, a second image can be referred to as a first image. Both the first image and the second image can be images, and in some cases, can be separate and different images.

[0051] The term "at least one" in this application means one or more, and the term "plurality" in this application means two or more, for example, a plurality of data packets means two or more data packets.

[0052] It should be understood that the terms used in the description of various examples herein are only for describing specific examples and are not intended to be limiting. As used in the description of various examples and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0053] It should also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. The term "and / or" is a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.

[0054] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0055] It should also be understood that determining B based on A does not mean determining B only based on A. B can also be determined based on A and / or other information.

[0056] It should also be understood that the term “comprising” (also known as “inCludes”, “inCluding”, “Comprises” and / or “Comprising”) when used in this specification specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0057] It should also be understood that the term "if" may be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined that ..." or "if [a stated condition or event] is detected" may be interpreted to mean "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0058] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the videos involved in this application are all obtained with full authorization.

[0059] Figure 1 is a schematic diagram of an implementation environment of a video processing method provided in an embodiment of the present application, see Figure 1 , the implementation environment may include a terminal 101 and a server 102.

[0060] The terminal 101 is connected to the server 102 via a wireless network or a wired network. Optionally, the terminal 101 is a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, etc., but is not limited thereto. The terminal 101 installs and runs an application that supports video processing, for example, the application is a plug-in application, a small program, or other types of applications.

[0061] The server 102 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The server 102 provides background services for applications running on the terminal 101. Optionally, the server 102 undertakes the main processing work and the terminal 101 undertakes the secondary processing work; or, the server 102 undertakes the secondary processing work and the terminal 101 undertakes the main processing work; or, the server 102 or the terminal 101 can each undertake the processing work independently; or, the server 102 and the terminal 101 use a distributed computing architecture for collaborative computing.

[0062] Those skilled in the art will appreciate that the number of the above-mentioned terminals 101 and servers 102 can be more or less. For example, the above-mentioned terminal 101 and server 102 can be only one, or the above-mentioned terminals 101 and servers 102 can be dozens or hundreds, or more. The embodiment of the present application does not limit the number and device type of the terminals or servers.

[0063] After introducing the implementation environment of the embodiment of the present application, the application scenario of the embodiment of the present application will be introduced in combination with the above-mentioned implementation environment. In the following description, the terminal is the terminal 101 in the above-mentioned implementation environment, and the server is the server 102 in the above-mentioned implementation environment.

[0064] The technical solution provided in the embodiments of the present application can be applied in the scenario of super-resolution processing of videos. For example, the super-resolution processing can be performed offline or online through a server.

[0065] In the scenario of performing super-resolution processing on videos offline, the application can apply the video processing method to perform super-resolution processing on the uploaded videos to obtain super-resolution videos, thereby achieving the purpose of improving the video resolution.

[0066] In the scenario where super-resolution processing is performed on a video online through a server, the uploaded video is sent to the server, and the server applies the video processing method to perform super-resolution processing on the uploaded video to obtain a super-resolution video, and returns the super-resolution video to the terminal, thereby achieving the purpose of improving the video resolution.

[0067] Of course, in addition to being applicable to the above-mentioned scenarios, the technical solution provided in the embodiments of the present application can also be applied to other scenarios of super-resolution processing of videos, and the embodiments of the present application are not limited to this.

[0068] Next, the technical solution provided by the present disclosure is introduced. Figure 2 is a flow chart of a video processing method provided by an embodiment of the present application, such as Figure 2 As shown, taking the method used in the server 102 as an example, the method includes the following steps:

[0069] In step 201, based on a video frame of a video and a previous video frame of the video frame, a first image and a second image of the previous video frame are determined, wherein the first image includes an area whose motion meets the target condition, and the second image includes an area whose motion does not meet the target condition.

[0070] In step 202, based on a video frame of a video and a subsequent video frame of the video frame, a first image and a second image of the subsequent video frame are determined.

[0071] In step 203, based on the first feature map and the second feature map of the video frame, a fused feature map of the video frame is obtained, wherein the first feature map is determined based on the video frame, the first image of the previous video frame, and the first image of the next video frame, and the second feature map is determined based on the video frame, the second image of the previous video frame, and the second image of the next video frame.

[0072] In step 204, a target video of the video is generated based on the fused feature map of the multiple video frames of the video and the video frame.

[0073] The technical solution provided by the embodiment of the present application can decouple different types of motion regions in the adjacent video frames before and after the video frame based on the motion situation in the video frame when super-resolution processing is required, and perform corresponding video feature fusion and other processing based on different types of motion regions to obtain the final super-resolution result of the video frame. Video processing time is saved, and because feature extraction is performed on different types of motion regions separately, the extracted features are more accurate, which can effectively avoid the problem of artifacts in the video caused by the same processing of all regions, and greatly improve the clarity of the video.

[0074] After introducing the implementation environment and application scenarios of the embodiments of the present application, the technical solution provided by the embodiments of the present application is introduced below, taking the execution subject as a server as an example. Figure 3 is a flowchart of a video processing method provided in an embodiment of the present application, see Figure 3 , the video processing method comprises:

[0075] In step 301, the video is decoded to obtain a plurality of video frames.

[0076] In the embodiment of the present application, the video is a video to be subjected to super-resolution processing. In some embodiments, the video can be any type of video, which is not limited in the embodiment of the present application.

[0077] It can be understood that the above step 301 is an implementation method of obtaining multiple video frames of a video, and the embodiment of the present application does not limit this process.

[0078] In some embodiments, scene switching detection is performed based on the acquired video frame, and if it is detected that the video frame does not undergo scene switching, step 302 is executed. Since the video processing method decouples the motion area and extracts features of continuous content in the same scene based on the adjacent video frames before and after the video frame, it is more suitable for continuous videos in which the scene does not switch. By performing scene detection on the acquired video and performing video processing on the video frame in which the scene switching is detected not to have occurred, the video processing method can also be applied to videos containing multiple scenes.

[0079] In step 302, a first residual image of the i-th video frame is obtained, where the first residual image is a residual between the i-th video frame and the (i-1)-th video frame, wherein i is a positive integer greater than 1.

[0080] Among them, each pixel point in the first residual image is the difference in pixel value between each pixel point at the corresponding position in the i-th video frame and the i-1-th video frame, and the calculation formula is shown in the following formula (1).

[0081] Ri-1=|Ii-Ii-1|(1)

[0082] Among them, I i is the pixel value of the pixel in the i-th video frame, I i-1 is the pixel value of the corresponding position in the i-1th video frame, R i-1 is the difference in pixel value between the pixel points at the corresponding positions in the i-th video frame and the i-1-th video frame, where i is a positive integer greater than 1.

[0083] In step 303, based on the first residual image, a pixel value threshold corresponding to the (i-1)th video frame is determined.

[0084] In some embodiments, the pixel value threshold is the average pixel value of the pixel points in the first residual image.

[0085] In step 304, based on the pixel value threshold corresponding to the i-1th video frame and the first residual map, a first image and a second image of the i-1th video frame are generated, the first image including areas in the i-1th video frame where the motion meets the target conditions, and the second image including areas in the i-1th video frame where the motion does not meet the target conditions.

[0086] The motion situation refers to the change of pixel points.

[0087] In some embodiments, based on the pixel value threshold, the areas in the first residual image where the pixel values ​​are greater than or equal to the pixel value threshold are determined to be areas where the motion conditions meet the target conditions, and the areas in the first residual image where the pixel values ​​are less than the pixel value threshold are determined to be areas where the motion conditions do not meet the target conditions.

[0088] Taking the example of first determining an area in the first residual image whose pixel value is greater than or equal to the pixel value threshold based on the pixel value threshold, determining an area in the first residual image whose pixel value is greater than or equal to the pixel value threshold, and generating a mask of the i-1th video frame based on the area corresponding to the determined area in the i-1th video frame, and generating a first image and a second image of the i-1th video frame based on the mask of the i-1th video frame and the first image of the i-1th video frame.

[0089] In some embodiments, the mask of the i-1th video frame is multiplied with the i-1th video frame to obtain the first image of the i-1th video frame, and the first image of the i-1th video frame is subtracted from the i-1th video frame to obtain the second image of the i-1th video frame. Through mask-based processing, processing efficiency can be greatly improved.

[0090] In step 305, a second residual image of the i-th video frame is obtained, where the second residual image is a residual between the video frame and the (i+1)-th video frame, where i is a positive integer greater than 1.

[0091] Among them, each pixel point in the second residual image is the difference in pixel value between each pixel point at the corresponding position in the i-th video frame and the i+1-th video frame, and the calculation formula is shown in the following formula (2).

[0092] Ri+1=|Ii-Ii+1|(2)

[0093] Among them, I i is the pixel value of the pixel in the i-th video frame, I i+1 is the pixel value of the corresponding position in the i+1th video frame, R i+1 is the difference in pixel value between the pixel points at the corresponding positions in the i-th video frame and the i+1-th video frame, where i is a positive integer greater than 1.

[0094] In step 306, a pixel value threshold corresponding to the (i+1)th video frame is determined based on the second residual image.

[0095] The process of determining the pixel value threshold in step 306 is the same as that in step 303, and will not be described in detail here.

[0096] In step 307, based on the pixel value threshold corresponding to the i+1th video frame and the second residual map, a first image and a second image of the i+1th video frame are generated, wherein the first image includes areas in the i+1th video frame where the motion meets the target conditions, and the second image includes areas in the i+1th video frame where the motion does not meet the target conditions.

[0097] The process of generating the first image and the second image in step 307 is the same as that in step 304, and will not be described in detail here.

[0098] It should be noted that the above steps 302 to 304 and 305 to 307 may be executed in parallel or in the current order. Of course, steps 302 to 304 may be executed first and then steps 305 to 307. This is not limited in the present embodiment.

[0099] In step 308 , the ith video frame, the first image of the i-1th video frame, and the first image of the i+1th video frame are spliced ​​to obtain a first spliced ​​video frame of the ith video frame.

[0100] The server splices the first image of the i-th video frame, the first image of the i-1-th video frame, and the first image of the i+1-th video frame in RGB channels to obtain a first spliced ​​video frame of nine channels.

[0101] In step 309, the first spliced ​​video frame of the i-th video frame is input into a first convolutional neural network, and features of the first spliced ​​video frame of the i-th video frame are extracted by the first convolutional neural network to obtain a first feature map.

[0102] In some embodiments, Figure 4 This is a schematic diagram of a first convolutional neural network structure provided in an embodiment of the present application, see Figure 4 The convolutional neural network is composed of multiple residual modules, which include two two-dimensional convolutional layers 401 and 403, activation layers 402 and 404, and a cross-layer direct edge 405. Optionally, the two-dimensional convolutional layer uses a convolution kernel, and the activation layer uses a ReLU activation function.

[0103] Taking the first residual module as an example, the server inputs the first spliced ​​video frame of the i-th video frame to the two-dimensional convolution layer 401, the two-dimensional convolution layer 401 extracts features from the input video frame, and the obtained feature activity value is input to the activation layer 402, the activation layer 402 outputs the feature, and the output of the activation layer 402 is the input of the two-dimensional convolution layer 403. The two-dimensional convolution layer 403 extracts features from the input features, and the server adds the obtained feature activity value to the input of the two-dimensional convolution layer 401 directly connected through 405, and inputs the result of the addition into the activation layer 404, which processes and outputs the features of the residual module.

[0104] Taking the residual module located between the head and the tail of multiple residual modules as an example, the server inputs the features output by the previous residual module to the two-dimensional convolution layer 401, the two-dimensional convolution layer 401 extracts features from the input features, and the obtained feature activity values ​​are input to the activation layer 402, the activation layer 402 outputs new features, and the output of the activation layer 402 is the input of the two-dimensional convolution layer 403, the two-dimensional convolution layer 403 extracts features from the input features, and the server adds the obtained feature activity values ​​to the input of the two-dimensional convolution layer 401 directly connected through 405, and inputs the added result to the activation layer 404, which processes and outputs the features of the residual module.

[0105] Taking the last residual module as an example, the server inputs the features output by the previous residual module to the two-dimensional convolution layer 401. The two-dimensional convolution layer 401 extracts features from the input features, and the obtained feature activity values ​​are input to the activation layer 402. The activation layer 402 outputs new features. The output of the activation layer 402 is the input of the two-dimensional convolution layer 403. The two-dimensional convolution layer 403 extracts features from the input features. The server adds the obtained feature activity values ​​to the input of the two-dimensional convolution layer 401 directly connected through 405, and inputs the input to the activation layer 404, which processes and outputs the first feature map.

[0106] Of course, in the embodiment of the present application, the first convolutional neural network can adopt the above-mentioned convolutional neural network, or can adopt other network structures for feature extraction, and the embodiment of the present application is not limited to this.

[0107] In step 310, the i-th video frame, the second image of the i-1-th video frame and the second image of the i+1-th video frame are spliced ​​to obtain a second spliced ​​video frame of the i-th video frame.

[0108] The server splices the second image of the i-th video frame, the i-1th video frame, and the second image of the i+1th video frame in RGB channels to obtain a second spliced ​​video frame of nine channels.

[0109] In step 311, the second spliced ​​video frame of the i-th video frame is input into a second convolutional neural network, and features of the second spliced ​​video frame of the i-th video frame are extracted by the second convolutional neural network to obtain a second feature map.

[0110] In some embodiments, the second convolutional neural network adopts the same network structure as the first convolutional neural network mentioned above, which is not described here.

[0111] The first convolutional neural network and the second convolutional neural network are trained separately, wherein the first convolutional neural network is trained using a first sample video frame, the first sample video frame includes an area whose motion meets the target conditions, and the second convolutional neural network is trained using a second sample video frame, the second sample video frame includes an area whose motion does not meet the target conditions. Through separate deep learning, the first convolutional neural network and the second convolutional neural network can learn different network processing capabilities, so that the first convolutional neural network is more suitable for extracting regional features in video frames whose motion meets the target conditions, and the second convolutional neural network is more suitable for extracting regional features in video frames whose motion does not meet the target conditions.

[0112] In continuous video frames, the area where the motion meets the target conditions has a large change, and the area where the motion does not meet the target conditions has a small change. Considering the information redundancy problem in the area with a small change, by using convolutional neural networks with different capabilities, different types of motion areas are processed separately, which can effectively extract more accurate regional features, thereby making the super-resolution result of the video frame better. Among them, the deep learning method can be supervised learning, unsupervised learning, reinforcement learning, etc., which is not limited in the embodiments of the present application.

[0113] In some embodiments, the convolutional layer in the second convolutional neural network can be a sparse convolutional layer. When performing feature extraction, the sparse convolutional layer can skip certain areas in the slow motion area. By reducing the processed area, the computational complexity of the network can be effectively reduced and the video processing time can be reduced.

[0114] In step 312, the first feature map and the second feature map are fused to obtain a fused feature map of the video frame.

[0115] The above-mentioned fusion can be performed in any fusion method. For example, in some embodiments, the first feature map and the second feature map are input into an adder to fuse the first feature map and the second feature map. The adder is used to add the corresponding pixels in the first feature map and the second feature map to obtain a fused feature map.

[0116] In some embodiments, the first feature map and the second feature map are input into a convolutional neural network to fuse the first feature map and the second feature map. The convolutional neural network is used to fuse the first feature map and the second feature map based on their contribution to obtain a fused feature map that can achieve better super-resolution effect.

[0117] The above steps 308 to 312 are descriptions of the process of obtaining a fused feature map of the video frame based on the first feature map and the second feature map of the video frame.

[0118] In step 313, the fused feature map of the i-th video frame is upsampled to obtain a target feature map.

[0119] In an embodiment of the present application, the fused feature map is upsampled to obtain a target feature map with a larger size, which is used as the basis for super-resolution images.

[0120] In some embodiments, the fused feature map is upsampled by an upsampling module. In some embodiments, Figure 5 It is a schematic diagram of the structure of an upsampling module provided in an embodiment of the present application. The upsampling module is a convolutional neural network, which includes an input layer 501, a convolutional layer 502, a deconvolution layer 503 and an output layer 504.

[0121] The server inputs the fused feature map of the video frame through the input layer 501, the convolution layer 502 extracts features from the input fused feature map to obtain new features, the deconvolution layer 503 is an upsampling layer, performs transposed convolution on the features output by the convolution layer 502 to obtain the target feature map of the video frame, and the output layer 504 outputs the target feature map of the video frame.

[0122] Of course, in the embodiment of the present application, the upsampling step can use the above-mentioned convolutional neural network, or other network structures can be used for upsampling, and the embodiment of the present application is not limited to this.

[0123] In step 314, the i-th video frame is up-sampled to obtain an up-sampled video frame corresponding to the i-th video frame.

[0124] In some embodiments, the i-th video frame is upsampled by an image interpolation method, which may be a nearest neighbor interpolation method, a bilinear interpolation method, a bicubic interpolation method, etc., to obtain an upsampled video frame corresponding to the i-th video frame.

[0125] Of course, the above-mentioned image interpolation method can be used in this embodiment, and other upsampling methods can also be used, such as upsampling through a convolutional neural network, which is not limited in this embodiment of the present application.

[0126] In step 315, the target feature map and the upsampled video frame corresponding to the i-th video frame are fused to obtain a target video frame of the i-th video frame.

[0127] The above-mentioned fusion can be performed in any fusion method. For example, in some embodiments, the target feature map and the upsampled video frame are input into an adder to fuse the two. The pixel points in the first feature map and the second feature map are added correspondingly by the adder to obtain the target video frame of the i-th video frame.

[0128] In some embodiments, the target feature map and the upsampled video frame are input into a convolutional neural network to fuse the two. The target feature map and the upsampled video frame are fused based on the contribution of the target feature map and the upsampled video frame through the convolutional neural network to obtain a target video frame of the i-th video frame that can make the final super-resolution effect better.

[0129] The above steps 302 to 315 are only an illustration of the processing process of one video frame. For each video frame in the video, the server executes a corresponding process to obtain a target video frame corresponding to each video frame.

[0130] In step 316, target video frames corresponding to the multiple video frames are encoded to generate a target video.

[0131] The target video frames corresponding to the multiple video frames are encoded to obtain the target video. It can be understood that the above step 316 is an implementation method for generating the target video, and the embodiment of the present application does not limit the process.

[0132] Since the target video has been decoupled into different types of motion areas and different types of motion areas are input into different convolutional neural networks for feature extraction, the regional features obtained are more accurate, which makes the final video clearer and greatly improves its playback effect.

[0133] The technical solution provided by the embodiment of the present application can decouple different types of motion regions in the adjacent video frames before and after the video frame based on the motion situation in the video frame when super-resolution processing is required, and perform corresponding video feature fusion and other processing based on different types of motion regions to obtain the final super-resolution result of the video frame. In the whole process, since the motion region is directly classified by using the pixel residuals of adjacent video frames, the motion estimation and compensation steps based on pixel-by-pixel operations are omitted, the spatial complexity of the network is simplified, and the video processing time is saved. Moreover, since the features of different types of motion regions are extracted separately, the extracted features are more accurate, which can effectively avoid the same processing of all regions, and there will be no inaccurate motion estimation of fast-moving regions or occluded regions, thereby avoiding the problem of artifacts in the video and greatly improving the clarity of the video.

[0134] Figure 6 This is a motion area division validity verification diagram provided by the embodiment of the present application, see Figure 6 In the figure, 601 is the original video frame, and 602 is the result of the video processing method dividing the corresponding original video frame into motion areas. In the figure, 601A represents the area where the motion does not meet the target conditions, and 602B represents the area where the motion meets the target conditions. It can be seen that the video processing method can effectively divide different types of motion areas in the original video frame.

[0135] Figure 7 This is a comparison chart of video processing effects provided by the embodiment of the present application, see Figure 7 , 701 is a real high-resolution video image, 702 to 707 are the results of other video processing methods, and 708 is the result of this video processing method. It can be seen that compared with other methods, this video processing method produces more details.

[0136] Figure 8 is a schematic diagram of the structure of a video processing device provided in an embodiment of the present application. Figure 8 The device includes: a determining unit 801, a fusing unit 802 and a generating unit 803.

[0137] The determination unit 801 is configured to execute a first image and a second image of a previous video frame based on a video frame of the video and the previous video frame of the video frame, wherein the first image includes an area where the motion meets the target condition, and the second image includes an area where the motion does not meet the target condition.

[0138] The determining unit 801 is further configured to determine the first image and the second image of the subsequent video frame based on the video frame of the video and the subsequent video frame of the video frame.

[0139] The fusion unit 802 is configured to execute a first feature map and a second feature map based on the video frame to obtain a fused feature map of the video frame, wherein the first feature map is determined based on the video frame, the first image of the previous video frame, and the first image of the next video frame, and the second feature map is determined based on the video frame, the second image of the previous video frame, and the second image of the next video frame.

[0140] The generating unit 803 is configured to execute the fusion feature map based on multiple video frames of the video and the video frame to generate a target video of the video.

[0141] In some embodiments, the determination unit 801 includes: a first acquisition subunit, configured to execute acquisition of a first residual map of the video frame, the first residual map being the residual between the video frame and a previous video frame of the video frame; a determination subunit, configured to execute determination of a pixel value threshold corresponding to the previous video frame based on the first residual map, the pixel value threshold being the mean pixel value of the pixel points in the first residual map; and a generation subunit, configured to execute generation of a first image and a second image of the previous video frame based on the pixel value threshold corresponding to the previous video frame and the first residual map.

[0142] In some embodiments, the generating subunit includes: a first determining subunit, configured to determine, based on the pixel value threshold, an area in the first residual map whose pixel value is greater than or equal to the pixel value threshold; a first generating subunit, configured to generate a mask of the previous video frame based on an area in the previous video frame corresponding to the determined area; and a second generating subunit, configured to generate a first image and a second image of the previous video frame based on the mask of the previous video frame and the first image of the previous video frame.

[0143] In some embodiments, the second generating subunit is configured to execute: multiplying the mask of the previous video frame by the previous video frame to obtain a first image of the previous video frame; and subtracting the first image of the previous video frame from the previous video frame to obtain a second image of the previous video frame.

[0144] In some embodiments, the fusion unit 802 includes: a second acquisition subunit, configured to execute acquisition of a first feature map and a second feature map of the video frame; and a fusion subunit, configured to execute fusion of the first feature map and the second feature map to obtain a fused feature map of the video frame.

[0145] In some embodiments, the second acquisition subunit is configured to execute: splicing the video frame, the first image of the previous video frame, and the first image of the next video frame to obtain a first spliced ​​video frame of the video frame; inputting the first spliced ​​video frame of the video frame into a first convolutional neural network, and performing feature extraction on the first spliced ​​video frame of the video frame through the first convolutional neural network to obtain a first feature map, and the first convolutional neural network is trained using a first sample video frame, and the first sample video frame includes an area whose motion meets the target condition.

[0146] In some embodiments, the second acquisition subunit is further configured to execute: splicing the video frame, the second image of the previous video frame, and the second image of the subsequent video frame to obtain a second spliced ​​video frame of the video frame; inputting the second spliced ​​video frame of the video frame into a second convolutional neural network, and performing feature extraction on the second spliced ​​video frame of the video frame through the second convolutional neural network to obtain a second feature map, and the second convolutional neural network is trained using a second sample video frame, and the second sample video frame includes an area where the motion does not meet the target condition.

[0147] In some embodiments, the generating unit 803 is configured to perform, for each video frame in the video, upsampling the fused feature map of the video frame to obtain a target feature map; upsampling the video frame to obtain an upsampled video frame corresponding to the video frame; fusing the target feature map with the upsampled video frame corresponding to the video frame to obtain a target video frame of the video frame; encoding the target video frames corresponding to the multiple video frames to generate a target video.

[0148] In some embodiments, the video processing device performs scene switching detection based on the acquired video frame and, if it is detected that no scene switching occurs in the video frame, executes the processing of the video frame by the determination unit 801, the fusion unit 802 and the generation unit 803.

[0149] It should be noted that: when the video processing device provided in the above embodiment processes a video, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the video processing method provided in the above embodiment and the video processing device embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0150] The embodiment of the present application provides a computer device for executing the above method. The computer device can be implemented as a terminal or a server. The structure of the terminal is first introduced below:

[0151] Fig. 9 It is a structural diagram of a terminal provided in an embodiment of the present application.

[0152] Typically, the terminal 900 includes: one or more processors 901 and one or more memories 902 .

[0153] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0154] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one computer program, which is used to be executed by the processor 901 to implement the video processing method provided in the method embodiment of the present application.

[0155] In some embodiments, the terminal 900 may further optionally include: a peripheral device interface 903 and at least one peripheral device. The processor 901, the memory 902 and the peripheral device interface 903 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 903 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907 and a power supply 908.

[0156] The peripheral device interface 903 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0157] The radio frequency circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 904 communicates with a communication network and other communication devices via electromagnetic signals. The radio frequency circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like.

[0158] The display screen 905 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 905 is a touch display screen, the display screen 905 also has the ability to collect touch signals on the surface or above the surface of the display screen 905. The touch signal may be input as a control signal to the processor 901 for processing. At this time, the display screen 905 may also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards.

[0159] The camera assembly 906 is used to collect images or videos. Optionally, the camera assembly 906 includes a front camera and a rear camera. Usually, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal.

[0160] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them to the processor 901 for processing, or input them to the radio frequency circuit 904 to achieve voice communication.

[0161] The power supply 908 is used to supply power to various components in the terminal 900. The power supply 908 may be alternating current, direct current, a disposable battery, or a rechargeable battery.

[0162] In some embodiments, the terminal 900 further includes one or more sensors 909 . The one or more sensors 909 include, but are not limited to: an acceleration sensor 910 , a gyroscope sensor 911 , a pressure sensor 912 , an optical sensor 913 , and a proximity sensor 914 .

[0163] The acceleration sensor 910 can detect the magnitude of acceleration on three coordinate axes of the coordinate system established with the terminal 900 .

[0164] The gyro sensor 911 can detect the body direction and rotation angle of the terminal 900 . The gyro sensor 911 can cooperate with the acceleration sensor 910 to collect the user's 3D actions on the terminal 900 .

[0165] The pressure sensor 912 may be provided at the side frame of the terminal 900 and / or the lower layer of the display screen 905. When the pressure sensor 912 is provided at the side frame of the terminal 900, it may detect the user's holding signal of the terminal 900, and the processor 901 may perform left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 912. When the pressure sensor 912 is provided at the lower layer of the display screen 905, the processor 901 may control the operability controls on the UI interface according to the user's pressure operation on the display screen 905.

[0166] The optical sensor 913 is used to collect the ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the display screen 905 according to the ambient light intensity collected by the optical sensor 913.

[0167] The proximity sensor 914 is used to collect the distance between the user and the front of the terminal 900 .

[0168] Those skilled in the art will understand that Fig. 9 The structure shown in the figure does not constitute a limitation on the terminal 900, and the terminal 900 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0169] The above-mentioned computer device can also be implemented as a server. The structure of the server is introduced below:

[0170] Fig.10It is a structural diagram of a server provided in an embodiment of the present application. The server 1000 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1001 and one or more memories 1002, wherein the one or more memories 1002 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 1001 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server 1000 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 1000 may also include other components for implementing device functions, which will not be described in detail here.

[0171] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one computer program is stored, and the computer program is loaded and executed by a processor to implement the video processing method in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0172] In an exemplary embodiment, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the video processing method is implemented.

[0173] In some embodiments, the computer program involved in the embodiments of the present application may be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain system.

[0174] A person skilled in the art will understand that all or part of the steps of implementing the above embodiments may be accomplished by hardware, or by controlling the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0175] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A video processing method, characterized in that: The method comprises: Determine, based on a video frame of a video and a previous video frame of the video frame, a first image and a second image of the previous video frame, wherein the first image of the previous video frame includes an area whose motion condition meets the target condition, and the second image of the previous video frame includes an area whose motion condition does not meet the target condition; Determine, based on a video frame of a video and a subsequent video frame of the video frame, a first image and a second image of the subsequent video frame, wherein the first image of the subsequent video frame includes an area whose motion condition meets the target condition, and the second image of the subsequent video frame includes an area whose motion condition does not meet the target condition; Based on a first feature map and a second feature map of the video frame, obtaining a fused feature map of the video frame, wherein the first feature map is determined based on the video frame, a first image of the previous video frame, and a first image of the next video frame, and the second feature map is determined based on the video frame, a second image of the previous video frame, and a second image of the next video frame; A target video of the video is generated based on the fused feature maps of the multiple video frames of the video and the multiple video frames.

2. The video processing method according to claim 1, characterized in that: The determining, based on a video frame of a video and a previous video frame of the video frame, the first image and the second image of the previous video frame comprises: Acquire a first residual image of the video frame, where the first residual image is a residual between the video frame and a previous video frame of the video frame; Based on the first residual map, determining a pixel value threshold corresponding to the previous video frame, the pixel value threshold being a mean pixel value of pixel points in the first residual map; A first image and a second image of the previous video frame are generated based on a pixel value threshold corresponding to the previous video frame and the first residual map.

3. The video processing method according to claim 2, characterized in that: The generating the first image and the second image of the previous video frame based on the pixel value threshold corresponding to the previous video frame and the first residual map comprises: Based on the pixel value threshold, determining in the first residual image an area where the pixel value is greater than or equal to the pixel value threshold; generating a mask of the previous video frame based on an area in the previous video frame corresponding to the determined area; Based on the mask of the previous video frame and the first image of the previous video frame, a first image and a second image of the previous video frame are generated.

4. The video processing method according to claim 3, characterized in that: The step of generating the first image and the second image of the previous video frame based on the mask of the previous video frame and the first image of the previous video frame comprises: Multiplying the mask of the previous video frame by the previous video frame to obtain a first image of the previous video frame; The first image of the previous video frame is subtracted from the previous video frame to obtain the second image of the previous video frame.

5. The video processing method according to claim 1, characterized in that: Based on the first feature map and the second feature map of the video frame, obtaining a fused feature map of the video frame includes: Obtaining a first feature map and a second feature map of the video frame; The first feature map and the second feature map are fused to obtain a fused feature map of the video frame.

6. The video processing method according to claim 5, characterized in that: The process of acquiring the first feature map of the video frame includes: splicing the video frame, the first image of the previous video frame, and the first image of the next video frame to obtain a first spliced ​​video frame of the video frame; The first spliced ​​video frame of the video frames is input into a first convolutional neural network, and features are extracted from the first spliced ​​video frame of the video frames by the first convolutional neural network to obtain a first feature map, wherein the first convolutional neural network is trained using a first sample video frame, and the first sample video frame includes an area whose motion meets the target condition.

7. The video processing method according to claim 5, characterized in that: The process of acquiring the second feature map of the video frame includes: splicing the video frame, the second image of the previous video frame, and the second image of the next video frame to obtain a second spliced ​​video frame of the video frame; The second spliced ​​video frame of the video frames is input into a second convolutional neural network, and features are extracted from the second spliced ​​video frame of the video frames by the second convolutional neural network to obtain a second feature map, wherein the second convolutional neural network is trained using second sample video frames, and the second sample video frames include areas where the motion does not meet the target conditions.

8. The video processing method according to claim 1, characterized in that: The generating a target video of the video based on the fused feature map of the multiple video frames of the video and the multiple video frames comprises: For each video frame in the video, upsampling the fused feature map of the video frame to obtain a target feature map; Upsampling the video frame to obtain an upsampled video frame corresponding to the video frame; Fusion of the target feature map and the upsampled video frame corresponding to the video frame to obtain a target video frame of the video frame; The target video frames corresponding to the multiple video frames are encoded to generate a target video.

9. The video processing method according to claim 1, characterized in that: The method further comprises: A scene switching detection is performed based on the acquired video frame, and if it is detected that no scene switching occurs in the video frame, processing is performed on the video frame.

10. A video processing device, characterized in that: The device comprises: A determining unit is configured to determine, based on a video frame of a video and a previous video frame of the video frame, a first image and a second image of the previous video frame, wherein the first image of the previous video frame includes an area whose motion condition meets the target condition, and the second image of the previous video frame includes an area whose motion condition does not meet the target condition; The determining unit is further configured to determine, based on a video frame of the video and a subsequent video frame of the video frame, a first image and a second image of the subsequent video frame, wherein the first image of the subsequent video frame includes an area whose motion condition meets the target condition, and the second image of the subsequent video frame includes an area whose motion condition does not meet the target condition; a fusion unit configured to acquire a fused feature map of the video frame based on a first feature map and a second feature map of the video frame, wherein the first feature map is determined based on the video frame, a first image of the previous video frame, and a first image of the next video frame, and the second feature map is determined based on the video frame, a second image of the previous video frame, and a second image of the next video frame; A generating unit is configured to generate a target video of the video based on the fused feature map of the multiple video frames of the video and the multiple video frames.

11. The video processing device according to claim 10, characterized in that: The determining unit comprises: A first acquisition subunit is configured to acquire a first residual image of the video frame, where the first residual image is a residual between the video frame and a previous video frame of the video frame; A determination subunit is configured to determine a pixel value threshold corresponding to the previous video frame based on the first residual map, wherein the pixel value threshold is a mean pixel value of pixel points in the first residual map; The generating subunit is configured to generate a first image and a second image of the previous video frame based on a pixel value threshold corresponding to the previous video frame and the first residual map.

12. The video processing device according to claim 11, characterized in that: The generating subunit comprises: A determination sub-subunit, configured to determine, based on the pixel value threshold, an area in the first residual image where the pixel value is greater than or equal to the pixel value threshold; A first generating sub-subunit is configured to generate a mask of the previous video frame based on an area in the previous video frame corresponding to the determined area; The second generating sub-subunit is configured to generate a first image and a second image of the previous video frame based on the mask of the previous video frame and the first image of the previous video frame.

13. The video processing device according to claim 12, characterized in that: The second generation sub-sub-unit is configured to execute: Multiplying the mask of the previous video frame by the previous video frame to obtain a first image of the previous video frame; The first image of the previous video frame is subtracted from the previous video frame to obtain the second image of the previous video frame.

14. The video processing device according to claim 10, characterized in that: The fusion unit comprises: A second acquisition subunit is configured to execute acquisition of a first feature map and a second feature map of the video frame; The fusion subunit is configured to perform fusion of the first feature map and the second feature map to obtain a fused feature map of the video frame.

15. The video processing device according to claim 14, characterized in that: The second acquisition subunit is configured to execute: splicing the video frame, the first image of the previous video frame, and the first image of the next video frame to obtain a first spliced ​​video frame of the video frame; The first spliced ​​video frame of the video frames is input into a first convolutional neural network, and features are extracted from the first spliced ​​video frame of the video frames by the first convolutional neural network to obtain a first feature map, wherein the first convolutional neural network is trained using a first sample video frame, and the first sample video frame includes an area whose motion meets the target condition.

16. The video processing device according to claim 14, characterized in that: The second acquisition subunit is further configured to execute: splicing the video frame, the second image of the previous video frame, and the second image of the next video frame to obtain a second spliced ​​video frame of the video frame; The second spliced ​​video frame of the video frames is input into a second convolutional neural network, and features are extracted from the second spliced ​​video frame of the video frames by the second convolutional neural network to obtain a second feature map, wherein the second convolutional neural network is trained using second sample video frames, and the second sample video frames include areas where the motion does not meet the target conditions.

17. The video processing device according to claim 10, characterized in that: The generating unit is configured to execute: For each video frame in the video, up-sampling the fused feature map of the video frame to obtain a target feature map; Upsampling the video frame to obtain an upsampled video frame corresponding to the video frame; Fusion of the target feature map and the upsampled video frame corresponding to the video frame to obtain a target video frame of the video frame; The target video frames corresponding to the multiple video frames are encoded to generate a target video.

18. The video processing device according to claim 10, characterized in that: When scene switching detection is performed based on the acquired video frame and it is detected that no scene switching occurs in the video frame, the determining unit, the fusion unit and the generating unit process the video frame.

19. A computer device, characterized in that: The computer device comprises: one or more processors; a memory for storing program code executable by the processor; The processor is configured to execute the program code to implement the video processing method according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that: When the program code in the computer-readable storage medium is executed by a processor of a server, the server is enabled to execute the video processing method according to any one of claims 1 to 9.

21. A computer program product, characterized in that The invention comprises a computer program, which implements the video processing method according to any one of claims 1 to 9 when being executed by a processor.

Images