Video processing method and device, electronic equipment and storage medium

Abstract

The invention provides a video processing method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining a to-be-processed video frame of a to-be-processed video and a reference frame corresponding to the to-be-processed video frame; the reference frame comprises a video frame in the video to be processed, wherein the number of interval frames between the video frame to be processed and the video frame to be processed does not exceed a preset threshold value; extracting a target single-channel feature of the reference frame, wherein the target single-channel feature can reflect the noise intensity degree of the reference frame; generating a de-noised video frame corresponding to the to-be-processed video frame based on the target single-channel features of the to-be-processed video frame and the reference frame; and generating a de-noised video based on the de-noised video frame. The to-be-processed video frame is subjected to noise reduction processing based on the target single-channel feature of the reference frame, so that rich time sequence information is provided to improve the accuracy of noise reduction processing and relieve inter-frame jitter, the processing data volume and the memory occupied space are reduced, and the computing resources required by noise reduction processing are reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video processing method, device, electronic device and storage medium. Background Art

[0002] Video noise reduction technology often requires the use of multi-frame information, which is then fused as the input of the noise reduction algorithm so that the noise reduction algorithm can more accurately determine the noise area based on the multi-frame information and help alleviate the inter-frame jitter problem in the video after noise reduction. Generally, the more frames used, the more significant the noise reduction effect and the smaller the inter-frame jitter, but it will consume more computing resources, especially when the technology is deployed locally on a personal computer or mobile terminal, which has a greater impact on device performance. Summary of the invention

[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, according to one or more embodiments of the present disclosure, a video processing method is provided, including:

[0005] Acquire a video frame to be processed of the video to be processed and a reference frame corresponding to the video frame to be processed; the reference frame includes a video frame in the video to be processed whose interval frame number with the video frame to be processed does not exceed a preset threshold;

[0006] Extracting a target single-channel feature of the reference frame, wherein the target single-channel feature can reflect the noise intensity of the reference frame;

[0007] Based on the target single-channel features of the video frame to be processed and the reference frame, generating a denoised video frame corresponding to the video frame to be processed;

[0008] A denoised video is generated based on the denoised video frames.

[0009] In a second aspect, according to one or more embodiments of the present disclosure, a video processing device is provided, including:

[0010] A video frame acquisition unit, used to acquire a to-be-processed video frame of the to-be-processed video and a reference frame corresponding to the to-be-processed video frame; the reference frame includes a video frame in the to-be-processed video whose interval frame number with the to-be-processed video frame does not exceed a preset threshold;

[0011] A feature extraction unit, used to extract a target single-channel feature of the reference frame, wherein the target single-channel feature can reflect the noise intensity of the reference frame;

[0012] An image generating unit, configured to generate a denoised video frame corresponding to the video frame to be processed based on the target single-channel features of the video frame to be processed and the reference frame;

[0013] A video generating unit is used to generate a denoised video based on the denoised video frame.

[0014] In a third aspect, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one memory and at least one processor; wherein the memory is used to store program code, and the processor is used to call the program code stored in the memory so that the electronic device executes the method provided according to one or more embodiments of the present disclosure.

[0015] In a fourth aspect, according to one or more embodiments of the present disclosure, a non-transitory computer storage medium is provided, wherein the non-transitory computer storage medium stores a program code, and when the program code is executed by a computer device, the computer device executes the method provided according to one or more embodiments of the present disclosure.

[0016] According to one or more embodiments of the present disclosure, by performing noise reduction processing on the video frame to be processed based on the target single-channel features of the video frame to be processed and the reference frame, not only rich timing information is provided to improve the accuracy of the noise reduction processing and reduce inter-frame jitter, but also the amount of processing data and memory usage are reduced, thereby reducing the computing resources required for the noise reduction processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0018] Figure 1 A flowchart of a video processing method provided by an embodiment of the present disclosure;

[0019] Figure 2 A flowchart of a video processing method provided by another embodiment of the present disclosure;

[0020] Figure 3 A schematic diagram of the structure of a video processing device provided by an embodiment of the present disclosure;

[0021] Figure 4 The figure is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] It should be understood that the steps described in the embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0024] As used herein, the term "including" and its variations are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The term "responsive to" and related terms refer to a signal or event being affected to a certain extent by another signal or event, but not necessarily completely or directly. If event x occurs "responsive to" event y, x may be directly or indirectly responsive to y. For example, the occurrence of y may ultimately lead to the occurrence of x, but there may be other intermediate events and / or conditions. In other cases, y may not necessarily lead to the occurrence of x, and x may occur even if y has not yet occurred. In addition, the term "responsive to" may also mean "at least partially responsive to".

[0025] The term "determine" broadly covers a variety of actions, which may include obtaining, calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, database, or other data structure), ascertaining, and similar actions, and may also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and similar actions, as well as parsing, selecting, choosing, establishing, and similar actions, etc. The relevant definitions of other terms will be given in the following description. The relevant definitions of other terms will be given in the following description.

[0026] It is understandable that the images generated by the methods provided in the embodiments of the present disclosure should be processed in accordance with the provisions of relevant laws and regulations. For example, technical measures should be taken in accordance with regulations to add a logo that does not affect user use, or a prominent logo should be placed in a reasonable location or area in accordance with regulations to inform the public of the depth synthesis situation.

[0027] It can be understood that the above-mentioned processing of the image is only illustrative and does not constitute a limitation on the implementation method of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation method of the present disclosure.

[0028] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0029] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0030] For the purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B).

[0031] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0032] refer to Figure 1 , Figure 1 A flowchart of a video processing method 100 provided in an embodiment of the present disclosure is shown. The method 100 includes steps S110 to S140.

[0033] Step S110: obtaining a to-be-processed video frame of the to-be-processed video and a reference frame corresponding to the to-be-processed video frame; the reference frame comprises a video frame in the to-be-processed video whose interval frame number with the to-be-processed video frame does not exceed a preset threshold.

[0034] In some embodiments, all video frames of the video to be processed may be used as the video frames to be processed, or some video frames of the video to be processed may be extracted as the video frames to be processed, which is not limited in the present disclosure.

[0035] In some embodiments, the reference frame includes a video frame in the video to be processed that is adjacent or close to the video frame to be processed. In a specific embodiment, the reference frame includes video frames adjacent to and adjacent to the video frame to be processed, or video frames with a spacing of no more than 2 frames before and after. For example, if the video frame to be processed is the i-th frame of the video, the reference frame can be the i-1th frame and the i+1th frame (a total of 2 frames) of the video, or the reference frame can be the i-2th frame, the i-1th frame, the i+1th frame and the i+2th frame (a total of 4 frames), but the present disclosure is not limited thereto, and those skilled in the art can use one or more video frames adjacent or close to the video frame to be processed as reference frames as needed.

[0036] Step S130: extracting target single-channel features of the reference frame, where the target single-channel features can reflect the noise intensity of the reference frame.

[0037] In some embodiments, the target single-channel feature includes a brightness channel feature, such as a Y channel feature of a YUV color space, or an L channel feature of an HSL color space, or a V channel feature of an HSV color space, or a grayscale image feature. Among them, YUV, HSL and HSV color spaces are different color encoding methods. In the YUV color space, Y represents brightness, while U and V represent chromaticity, which is used to describe image color and saturation. HSL and HSV (also known as HSB) are both a method of representing points in an RGB color model in a cylindrical coordinate system, wherein HSL refers to hue, saturation and brightness, and HSV refers to hue, saturation and brightness. A grayscale image is a single-channel image in which each pixel carries only information about brightness.

[0038] Compared with the three-channel color channel features (such as RGB channel features), the brightness channel features of an image not only have fewer channels and correspondingly require less computing resources, but also retain the visual structure and details of the original image (such as contours, edges, corners, textures, etc.) to a certain extent, and can reflect the noise intensity of the original image.

[0039] Step S140: generating a denoised video frame corresponding to the video frame to be processed based on the target single-channel features of the video frame to be processed and the reference frame.

[0040] In some embodiments, the color channel features of the video frame to be processed can be fused with the brightness channel features of the reference frame, and the fused image features can be subjected to noise reduction processing to obtain a denoised video frame corresponding to the video frame to be processed. Exemplarily, the noise reduction techniques that can be used include but are not limited to noise reduction techniques based on filtering methods (including spatial domain methods, transform domain methods, hybrid spatiotemporal transform methods), sparse expression, clustered low rank, Gaussian mixture models, or deep learning.

[0041] Step S150: generating a denoised video based on the denoised video frame.

[0042] In some embodiments, the obtained denoised video frames may be spliced ​​based on the timestamps of the video frames to be processed to finally obtain the denoised video.

[0043] According to one or more embodiments of the present disclosure, by performing denoising on the video frame to be processed based on the target single-channel features of the video frame to be processed and the reference frame, not only rich timing information is provided to improve the accuracy of the denoising and reduce inter-frame jitter, but also compared with the solution of directly using RGB images of multiple video frames to perform denoising on the video frame to be processed, the amount of processing data and memory usage are reduced, thereby significantly reducing the computing resources required for denoising.

[0044] In some embodiments, step S140 includes:

[0045] Step A1: generating a fusion feature corresponding to the video frame to be processed based on the color channel feature of the video frame to be processed and the target single channel feature of the reference frame;

[0046] Step A2: inputting the fusion features into an image denoising model;

[0047] Step A3: using the output result of the image denoising model as the denoised video frame.

[0048] The color channel features of the video frame to be processed may be RGB channel features, including R channel features, G channel features, and B channel features.

[0049] In a specific embodiment, the color channel features of the video frame to be processed and the target single channel features of the reference frame can be spliced ​​based on the time sequence between the video frame to be processed and the reference frame to generate the fused features. Exemplarily, the feature fusion operation can be implemented by a concatenation function (Concat), which can splice the target single channel features of the reference frame and the color channel features of the video frame to be processed in a time sequence in a specified dimension to obtain a fused feature containing more information.

[0050] refer to Figure 2 , which shows a schematic diagram of the processing flow of performing noise reduction processing on a sequence of video frames to be processed to obtain a denoised video frame sequence. When denoising the i-th frame of the video frame sequence to be processed, a total of 5 video frames from the i-2th frame to the i+2th frame can be collected through a sampling window, wherein the i-2th frame, the i-1th frame, the i+1th frame and the i+2th frame are used as reference frames for the i-th frame; the reference frame is decomposed into YUV channels to extract the Y, U and V channel features corresponding to each reference frame, and the RGB channel features of the i-th frame of the video frame to be processed are extracted, including the R channel features, the G channel features and the B channel features; according to the time sequence of each video frame in the sequence of video frames to be processed, the Y channel features of the i-2th frame, the Y channel features of the i-1th frame, the RGB channel features of the i-th frame, the Y channel features of the i+1th frame and the Y channel features of the i+2th frame are sequentially spliced ​​to obtain a fusion feature (such as Figure 2The fusion feature is input into the image denoising model, and the denoised video frame of the i-th frame is output. Similarly, the above method is repeatedly executed until all the video frames in the video frame sequence to be processed are processed.

[0051] In this embodiment, the image denoising model outputs the RGB image processing result of the image video frame to be processed, which can directly obtain the final denoising result of the video frame to be processed without performing a channel restoration operation.

[0052] In this embodiment, the target single-channel feature of the reference frame specifically adopts the brightness channel (i.e., Y channel) feature of the YUV color space. According to the experimental research of the inventor, the scheme of using the brightness channel feature of the reference frame in the YUV color space is consistent with the scheme of directly using the RGB image of the reference frame in terms of noise reduction effect, while the former consumes less computing resources and memory space. See Table 1 below, which shows the various parameters and evaluation results of Comparative Examples 1, 2 and Example 1 when the resolution of the video is 1920×1080.

[0053] Table 1

[0054]

[0055] Among them, in comparative example 1, the input of the denoising model is generated based on the RGB channel features of 2 reference frames and the RGB channel features of 1 frame to be processed (i.e., a total of 3 video frames are sampled and 9 channels are processed); in comparative example 2, the input of the denoising model is generated based on the RGB channel features of 4 reference frames and the RGB channel features of 1 frame to be processed (i.e., a total of 5 video frames are sampled and 15 channels are processed); in embodiment 1, the input of the denoising model is generated based on the Y channel features of 4 reference frames and the RGB channel features of 1 frame to be processed (i.e., a total of 5 video frames are sampled and 7 channels are processed, such as Figure 2 As shown in Table 1, the test results show that for the same scheme of sampling 5 video frames, the required computational complexity and memory space occupied by Example 1 are lower than those of Comparative Example 2, and even lower than those of Comparative Example 1 which samples 3 video frames. In terms of denoising results, the denoising effects of Example 1 and Comparative Example 2 are consistent as a whole, and are better than those of Comparative Example 1.

[0056] In some embodiments, the image denoising model may be a neural network model, such as a convolutional neural network, a recurrent neural network, etc., but the present disclosure is not limited thereto.

[0057] In some embodiments, the training method of the image denoising model includes: taking the color channel features of the first sample image and the target single channel features of the reference sample image corresponding to the first sample image as the input of the image denoising model, taking the first denoised image corresponding to the first sample image as the expected output of the image denoising model, and training to obtain the image denoising model.

[0058] Among them, the first sample image can be a video frame in a sample video, and the reference sample image can be a video frame in the sample video that is adjacent to or close to the first sample image, for example, including one or more frames before and after the first sample image; the color channel feature can be an RGB channel feature, including an R channel feature, a G channel feature and a B channel feature; the target single channel feature can be a brightness channel feature (for example, a Y channel feature), but the present disclosure is not limited to this.

[0059] In a specific embodiment, a fusion feature can be generated based on the color channel feature of the first sample image and the brightness channel feature of the reference sample image, and the fusion feature can be input into the image denoising model. The parameters of the image denoising model are adjusted based on the loss between the output of the image denoising model and the first denoised image so that the loss function converges.

[0060] According to one or more embodiments of the present disclosure, by performing noise reduction processing on the video frame to be processed based on the color channel features (such as RGB channel features) of the video frame to be processed and the brightness channel features of the reference frame, not only rich timing information is provided to improve the accuracy of noise reduction processing and reduce inter-frame jitter, but also compared with the solution of directly using the color channel features of the reference frame to perform noise reduction processing on the video frame to be processed, the amount of processing data and memory usage are reduced, and it is more suitable for devices with limited computing power such as personal computers or mobile terminals. In addition, the embodiments of the present disclosure more fully consider image information of different dimensions through the mixed input of color channel features and brightness channel features, thereby ensuring the video noise reduction effect.

[0061] Accordingly, refer to Figure 3 According to an embodiment of the present disclosure, a video processing device 300 is provided, including:

[0062] The video frame acquisition unit 301 is used to acquire a to-be-processed video frame of the to-be-processed video and a reference frame corresponding to the to-be-processed video frame; the reference frame includes a video frame in the to-be-processed video whose interval frame number with the to-be-processed video frame does not exceed a preset threshold;

[0063] A feature extraction unit 302 is used to extract a target single-channel feature of the reference frame, wherein the target single-channel feature can reflect the noise intensity of the reference frame;

[0064] An image generating unit 303 is configured to generate a denoised video frame corresponding to the video frame to be processed based on the target single-channel features of the video frame to be processed and the reference frame;

[0065] The video generating unit 304 is configured to generate a denoised video based on the denoised video frame.

[0066] In some embodiments, the target single-channel feature comprises a luminance channel feature.

[0067] In some embodiments, the brightness channel feature includes a brightness channel feature of a YUV color space, or a brightness channel feature of an HSL color space, or a lightness channel feature of an HSV color space, or a grayscale image feature.

[0068] In some embodiments, the image generation unit is used to generate a fusion feature corresponding to the video frame to be processed based on the color channel features of the video frame to be processed and the target single channel features of the reference frame, input the fusion feature into an image denoising model, and use the output result of the image denoising model as the denoised video frame.

[0069] In some embodiments, the generating of a fusion feature corresponding to the video frame to be processed based on the color channel features of the video frame to be processed and the target single channel features of the reference frame includes: based on the time sequence between the video frame to be processed and the reference frame, splicing the color channel features of the video frame to be processed and the target single channel features of the reference frame to generate the fusion feature.

[0070] In some embodiments, the training method of the image denoising model includes: taking the color channel features of a first sample image and the target single channel features of a reference sample image corresponding to the first sample image as inputs of the image denoising model, taking the first denoised image corresponding to the first sample image as the expected output of the image denoising model, and training to obtain the image denoising model.

[0071] For the embodiments of the device, since they basically correspond to the method embodiments, the relevant parts refer to the partial description of the method embodiments. The device embodiments described above are merely illustrative, and the modules described as separation modules may or may not be separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present embodiment. Those of ordinary skill in the art can understand and implement without paying creative work.

[0072] Accordingly, according to one or more embodiments of the present disclosure, there is provided an electronic device, including:

[0073] at least one memory and at least one processor;

[0074] The memory is used to store program codes, and the processor is used to call the program codes stored in the memory to enable the electronic device to execute the video processing method provided according to one or more embodiments of the present disclosure.

[0075] Accordingly, according to one or more embodiments of the present disclosure, a non-transitory computer storage medium is provided, which stores program code, and the program code can be executed by a computer device to enable the computer device to perform the video processing method provided according to one or more embodiments of the present disclosure.

[0076] Reference below Figure 4 , which shows a schematic diagram of the structure of an electronic device (such as a terminal device or a server) 800 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0077] like Figure 4 As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0078] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0079] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0080] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0081] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0082] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0083] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method of the present disclosure.

[0084] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0085] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0086] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0087] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0088] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0089] According to one or more embodiments of the present disclosure, a video processing method is provided, comprising: obtaining a to-be-processed video frame of a to-be-processed video and a reference frame corresponding to the to-be-processed video frame; the reference frame comprises a video frame in the to-be-processed video whose interval frame number with the to-be-processed video frame does not exceed a preset threshold; extracting a target single-channel feature of the reference frame, wherein the target single-channel feature can reflect the noise intensity of the reference frame; generating a denoised video frame corresponding to the to-be-processed video frame based on the target single-channel features of the to-be-processed video frame and the reference frame; and generating a denoised video based on the denoised video frame.

[0090] According to one or more embodiments of the present disclosure, the target single-channel feature includes a brightness channel feature.

[0091] According to one or more embodiments of the present disclosure, the brightness channel feature includes a brightness channel feature of a YUV color space, or a brightness channel feature of an HSL color space, or a lightness channel feature of an HSV color space, or a grayscale image feature.

[0092] According to one or more embodiments of the present disclosure, the generating of a denoised video frame corresponding to the video frame to be processed based on the target single-channel features of the video frame to be processed and the reference frame includes: generating a fusion feature corresponding to the video frame to be processed based on the color channel features of the video frame to be processed and the target single-channel features of the reference frame; inputting the fusion feature into an image denoising model; and using the output result of the image denoising model as the denoised video frame.

[0093] According to one or more embodiments of the present disclosure, the generating of fusion features corresponding to the video frame to be processed based on the color channel features of the video frame to be processed and the target single channel features of the reference frame includes: based on the time sequence between the video frame to be processed and the reference frame, splicing the color channel features of the video frame to be processed and the target single channel features of the reference frame to generate the fusion features.

[0094] According to one or more embodiments of the present disclosure, the training method of the image denoising model includes: taking the color channel features of a first sample image and the target single channel features of a reference sample image corresponding to the first sample image as inputs of the image denoising model, taking the first denoised image corresponding to the first sample image as the expected output of the image denoising model, and training to obtain the image denoising model.

[0095] According to one or more embodiments of the present disclosure, a video processing device is provided, comprising: a video frame acquisition unit, configured to acquire a to-be-processed video frame of a to-be-processed video and a reference frame corresponding to the to-be-processed video frame; the reference frame comprises a video frame in the to-be-processed video whose interval frame number with the to-be-processed video frame does not exceed a preset threshold; a feature extraction unit, configured to extract a target single-channel feature of the reference frame, wherein the target single-channel feature can reflect the noise intensity of the reference frame; an image generation unit, configured to generate a denoised video frame corresponding to the to-be-processed video frame based on the target single-channel features of the to-be-processed video frame and the reference frame; and a video generation unit, configured to generate a denoised video based on the denoised video frame.

[0096] According to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one memory and at least one processor; wherein the memory is used to store program code, and the processor is used to call the program code stored in the memory so that the electronic device executes the video processing method provided according to one or more embodiments of the present disclosure.

[0097] According to one or more embodiments of the present disclosure, a non-transitory computer storage medium is provided, wherein the non-transitory computer storage medium stores program code, and when the program code is executed by a computer device, the computer device executes a video processing method provided according to one or more embodiments of the present disclosure.

[0098] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0099] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0100] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A video processing method, characterized in that: include: Acquire a video frame to be processed of the video to be processed and a reference frame corresponding to the video frame to be processed; the reference frame includes a video frame in the video to be processed whose interval frame number with the video frame to be processed does not exceed a preset threshold; Extracting a target single-channel feature of the reference frame, wherein the target single-channel feature can reflect the noise intensity of the reference frame; Based on the target single-channel features of the video frame to be processed and the reference frame, generating a denoised video frame corresponding to the video frame to be processed; A denoised video is generated based on the denoised video frames.

2. The method according to claim 1, characterized in that The target single-channel feature includes a brightness channel feature.

3. The method according to claim 2, characterized in that The brightness channel feature includes a brightness channel feature of a YUV color space, a brightness channel feature of an HSL color space, a lightness channel feature of an HSV color space, or a grayscale image feature.

4. The method according to claim 1, characterized in that: The generating a denoised video frame corresponding to the video frame to be processed based on the target single-channel features of the video frame to be processed and the reference frame comprises: Generate a fusion feature corresponding to the video frame to be processed based on the color channel feature of the video frame to be processed and the target single channel feature of the reference frame; Inputting the fused features into an image denoising model; The output result of the image denoising model is used as the denoised video frame.

5. The method according to claim 4, characterized in that The generating, based on the color channel feature of the video frame to be processed and the target single channel feature of the reference frame, a fusion feature corresponding to the video frame to be processed comprises: Based on the time sequence between the video frame to be processed and the reference frame, the color channel features of the video frame to be processed and the target single channel features of the reference frame are spliced ​​to generate the fusion features.

6. The method according to claim 4, characterized in that The training method of the image denoising model includes: The image denoising model is obtained by training by using the color channel features of the first sample image and the target single channel features of the reference sample image corresponding to the first sample image as inputs of the image denoising model, and using the first denoised image corresponding to the first sample image as the expected output of the image denoising model.

7. A video processing device, characterized in that: include: A video frame acquisition unit, used to acquire a to-be-processed video frame of the to-be-processed video and a reference frame corresponding to the to-be-processed video frame; the reference frame includes a video frame in the to-be-processed video whose interval frame number with the to-be-processed video frame does not exceed a preset threshold; A feature extraction unit, used to extract a target single-channel feature of the reference frame, wherein the target single-channel feature can reflect the noise intensity of the reference frame; An image generating unit, configured to generate a denoised video frame corresponding to the video frame to be processed based on the target single-channel features of the video frame to be processed and the reference frame; A video generating unit is used to generate a denoised video based on the denoised video frame.

8. An electronic device, characterized in that: include: at least one memory and at least one processor; The memory is used to store program codes, and the processor is used to call the program codes stored in the memory to enable the electronic device to execute the method according to any one of claims 1 to 6.

9. A non-transitory computer storage medium, characterized in that The non-transitory computer storage medium stores a program code, and when the program code is executed by a computer device, the computer device executes the method according to any one of claims 1 to 6.

Images