Video processing methods and devices

Abstract

This disclosure relates to a video processing method and apparatus. The video processing method includes: acquiring features of a current frame image and features of a previous frame image of a video to be processed; calculating the correlation between the features of the current frame image and the features of the previous frame image; determining alignment features between the current frame image and the previous frame image based on the correlation and hidden layer features of the previous frame image; and processing the current frame image based on the alignment features. According to the video processing method and apparatus of this disclosure, video processing efficiency and effectiveness can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of video technology. More specifically, this disclosure relates to a video processing method and apparatus. Background Technology

[0002] Video, as a medium for recording human life, is an indispensable part of modern life. With technological advancements, people have increasingly higher demands for video quality, maintaining a perpetual pursuit of superior image quality. This primarily includes deblurring, noise reduction, and super-resolution. Limited by shooting equipment, transmission, and encoding / decoding capabilities, many users cannot obtain high-quality videos from downloaded feeds. Therefore, external algorithms are needed to help users obtain higher-definition videos. Taking video super-resolution as an example, previous algorithms using recurrent convolutional networks calculated the optical flow between the previous and current frames, using this as a guide to compensate for hidden layer features from the previous moment in the current moment. However, optical flow estimation is inaccurate, especially in scenes with significant occlusion or motion. Furthermore, optical flow estimation is extremely time-consuming; pixel-by-pixel optical flow estimation consumes enormous computational resources. Therefore, finding an alternative solution for optical flow is essential. Summary of the Invention

[0003] Exemplary embodiments of this disclosure provide a video processing method and apparatus to at least solve the video processing problems in the related art, or may not solve any of the aforementioned problems.

[0004] According to an exemplary embodiment of this disclosure, a video processing method is provided, comprising: acquiring features of a current frame image of a video to be processed and features of a previous frame image of the current frame; calculating the correlation between the features of the current frame image and the features of the previous frame image; determining alignment features between the current frame image and the previous frame image based on the correlation and the hidden layer features of the previous frame image, wherein the hidden layer features of the previous frame image include features of a hidden layer between an input layer and an output layer for processing the previous frame image; and processing the current frame image based on the alignment features.

[0005] Optionally, the step of determining the alignment features of the current frame image and the previous frame image may include: determining the position with the greatest correlation between the current frame image and the previous frame image based on the correlation between the features of the current frame image and the features of the previous frame image; and performing an alignment operation on the hidden layer features of the previous frame image based on the position with the greatest correlation to obtain the alignment features of the current frame image and the previous frame image.

[0006] Optionally, the step of aligning the hidden features of the previous frame image based on the position with the highest correlation may include: using the position with the highest correlation as an index to transfer the hidden features of the previous frame image.

[0007] Optionally, the step of obtaining the features of the current frame image and the features of the previous frame image of the video to be processed may include: performing feature extraction on the current frame image and the previous frame image respectively through the same feature extraction network to obtain the features of the current frame image and the features of the previous frame image.

[0008] Optionally, the step of calculating the correlation between the features of the current frame image and the features of the previous frame image may include: calculating the norm of the features of the current frame image, dividing the features of the current frame image by the norm of the features of the current frame image to obtain a first feature; calculating the norm of the features of the previous frame image, dividing the features of the previous frame image by the norm of the features of the previous frame image to obtain a second feature; and using the inner product of the first feature and the second feature as the correlation between the features of the current frame image and the features of the previous frame image.

[0009] Optionally, the step of processing the current frame image based on the alignment feature may include: concatenating the current frame image and the alignment feature to obtain a concatenated feature; and inputting the concatenated feature into a preset convolutional neural network to obtain the processed image of the current frame image and the hidden layer features of the current frame image.

[0010] Optionally, the processed image of the current frame image may include a super-resolution processed image of the current frame image or an image enhancement processed image of the current frame image.

[0011] According to an exemplary embodiment of the present disclosure, a video processing apparatus is provided, comprising: a feature acquisition unit configured to acquire features of a current frame image of a video to be processed and features of a previous frame image of the current frame; a correlation determination unit configured to determine the correlation between the features of the current frame image and the features of the previous frame image; a feature alignment unit configured to determine alignment features between the current frame image and the previous frame image based on the correlation and hidden layer features of the previous frame image, wherein the hidden layer features of the previous frame image include features of a hidden layer between an input layer and an output layer for processing the previous frame image; and an image processing unit configured to process the current frame image based on the alignment features.

[0012] Optionally, the feature alignment unit can be configured to: determine the position with the highest correlation between the features of the current frame image and the features of the previous frame image based on the correlation between the features of the current frame image and the features of the previous frame image; and perform an alignment operation on the hidden layer features of the previous frame image based on the position with the highest correlation to obtain the alignment features of the current frame image and the previous frame image.

[0013] Optionally, the feature alignment unit can be configured to use the position with the highest correlation as an index to transfer the hidden features of the previous frame image.

[0014] Optionally, the feature acquisition unit can be configured to: extract features from the current frame image and the previous frame image respectively through the same feature extraction network to obtain the features of the current frame image and the features of the previous frame image.

[0015] Optionally, the correlation determination unit may be configured to: calculate the norm of the features of the current frame image, divide the features of the current frame image by the norm of the features of the current frame image to obtain a first feature; calculate the norm of the features of the previous frame image, divide the features of the previous frame image by the norm of the features of the previous frame image to obtain a second feature; and use the inner product of the first feature and the second feature as the correlation between the features of the current frame image and the features of the previous frame image.

[0016] Optionally, the image processing unit may be configured to: concatenate the current frame image and the alignment features to obtain concatenated features; and input the concatenated features into a preset convolutional neural network to obtain the processed image of the current frame image and the hidden layer features of the current frame image.

[0017] Optionally, the processed image of the current frame image includes a super-resolution processed image of the current frame image or an image enhancement processed image of the current frame image.

[0018] According to an exemplary embodiment of the present disclosure, an electronic device is provided, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a video processing method according to an exemplary embodiment of the present disclosure.

[0019] According to exemplary embodiments of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor of an electronic device, causes the electronic device to perform a video processing method according to exemplary embodiments of the present disclosure.

[0020] According to exemplary embodiments of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement a video processing method according to exemplary embodiments of the present disclosure.

[0021] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0022] By aligning the hidden layer features of the previous time step, the problem of motion differences between time steps is solved, thus improving the video processing effect;

[0023] Alignment can be achieved with fewer parameters and is less prone to error accumulation, thus improving video processing performance.

[0024] Without the need to introduce additional reference images, the variables required for the deformer can be constructed using only low-resolution video sequences, improving video processing performance;

[0025] Performing calculations at low resolution improves video processing efficiency.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0028] Figure 1 A schematic diagram of image super-resolution using TTSR is shown.

[0029] Figure 2 Exemplary system architectures in which exemplary embodiments of this disclosure can be applied are shown.

[0030] Figure 3 A flowchart illustrating a video processing method according to an exemplary embodiment of the present disclosure is shown.

[0031] Figure 4 This diagram illustrates the use of a super-resolution network to perform super-resolution processing on low-resolution video.

[0032] Figure 5 This diagram illustrates the use of a super-resolution network including an alignment module to perform super-resolution processing on low-resolution video.

[0033] Figure 6 This diagram illustrates how the alignment module is used to determine the alignment features between the current frame image and the previous frame image.

[0034] Figure 7 A block diagram of a video processing apparatus according to an exemplary embodiment of the present disclosure is shown.

[0035] Figure 8 This is a block diagram of an electronic device 800 according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0036] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0037] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0038] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. As another example, "performing at least one of step one and step two" indicates the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0039] With the development of deformers in computer vision in recent years, they have shown considerable research and application prospects in segmentation, detection, and medical imaging. Thanks to the design structure of deformers, they can be used as a solution for capturing temporal information in videos. Specifically, deformers have three important properties: (1) self-attention mechanism, which is very useful in temporal modeling tasks. (2) computational degrees of freedom, by reasonably setting the three variables k, v, and q, the content of v can be mapped to a new space. (3) fast learning of global dependencies, the correlation calculation between two positions in the self-attention mechanism is independent of the number of layers in the neural network, so it can quickly learn the global dependencies of information. Here, global can be information within an image, information between images in a video, or the relationship between features and images.

[0040] Deformers were initially used in Natural Language Processing (NLP) to establish global relationships between words. This computational approach has been widely adopted in computer vision, breaking the limitation of convolutional neural networks that rely on stacking layers to expand the receptive field. It allows the establishment of global connections between features using only a few multi-head layers. Initially, these global connections were used as a soft attention mechanism in high-level semantic tasks such as recognition, detection, and retrieval, with limited application in image enhancement. The Texture Transformer Network for Image Super-Resolution (TTSR) applies deformers to the field of reference image-based super-resolution, expanding the uses of deformers. TTSR uses the deformer's attention mechanism for texture transfer between reference and high-resolution images, improving the performance of reference image-based super-resolution algorithms.

[0041] Figure 1 This diagram illustrates image super-resolution using TTSR. Figure 1 In the TTSR, there are four input variables, from left to right: low-resolution image (LR), reference image (Ref), upsampling and downsampling of the reference image, and upsampling of the low-resolution image. The deformer's V is the reference image, K is the upsampling and downsampling of the reference image, and Q is the upsampling of the low-resolution image, representing the three elements used for texture transfer.

[0042] Q = LTE (LR↑),

[0043] K = LTE (Ref↓↑),

[0044] V = LTE(Ref).

[0045] Subsequently, TTSR applied the properties of the deformer to find similar textures between the reference image and the low-resolution image.

[0046] According to the deformer's calculation rules, K and Q will calculate a similarity to represent the correlation between the low-resolution image and the reference image, represented by the following inner product:

[0047]

[0048] Here, r represents the "correlation between each point in the low-resolution image and the reference image", but it does not indicate "which information should be transferred from the reference image to the low-resolution image". The next step is to find the most relevant information from r.

[0049] For example, through formula To determine the most relevant information.

[0050] h iThe most relevant position in the reference image is represented by the i-th position in the LR image. This describes the process by which TTSR uses the properties of deformers to find similar textures between the reference image and the low-resolution image.

[0051] TTSR requires the introduction of reference images when designing K, V, and Q parameters. However, selecting suitable and paired reference images in real-world scenarios is extremely difficult. On one hand, choosing reference images unrelated to the low-resolution image will not achieve texture transfer; on the other hand, high-resolution and lossless video sources are rare in real-world scenarios, making suitable sources for low-resolution images even rarer. TTSR's K, V, and Q calculations are performed at upsampled resolution, resulting in significant computational overhead. Furthermore, TTSR has not been further validated in other tasks, such as feature alignment in recurrent convolutional networks, therefore, the K, V, and Q parameters need to be redesigned for different scenarios.

[0052] Below, we will refer to Figures 2 to 8 This invention provides a detailed description of a video processing method and apparatus according to exemplary embodiments of the present disclosure.

[0053] Figure 2 An exemplary system architecture 100 is shown that can be applied to an exemplary embodiment of this disclosure.

[0054] like Figure 2 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables. Users can use terminal devices 101, 102, and 103 to interact with server 105 through network 104 to receive or send messages (e.g., video processing requests). Various video applications may be installed on terminal devices 101, 102, and 103. Terminal devices 101, 102, and 103 can be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays capable of audio and video playback, recording, and editing, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module. No specific limitations are made here.

[0055] Terminal devices 101, 102, and 103 may be equipped with image acquisition devices (e.g., cameras) to capture video data. In practice, the smallest visual unit that makes up video is a frame. Each frame is a static image. Combining temporally consecutive frame sequences creates a dynamic video. Furthermore, terminal devices 101, 102, and 103 may also be equipped with components for converting electrical signals into sound (e.g., speakers) to play sound, and may also be equipped with devices for converting analog audio signals into digital audio signals (e.g., microphones) to capture sound.

[0056] Server 105 can be a server that provides various services, such as a backend server that supports multimedia applications installed on terminal devices 101, 102, and 103. The backend server can parse, store, and process data such as received video processing requests.

[0057] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0058] It should be noted that the video processing method provided in this disclosure is typically executed by a terminal device, but it can also be executed by a server, or it can be executed collaboratively by a terminal device and a server. Accordingly, the video processing apparatus can be located in the terminal device, the server, or both.

[0059] It should be understood that Figure 2 The number of terminal devices, networks, and servers shown is merely illustrative. Any number of terminal devices, networks, and servers can be used as needed, and this disclosure places no limitation on this.

[0060] Due to its efficiency and simple structure, Recurrent Convolutional Neural Networks (RCNs) are widely used in video enhancement. A key characteristic of RCNs is that they use the hidden features from the previous time step as input for the next. However, due to motion differences, the hidden features from the previous time step can differ significantly from those of the current time step. Limited by the receptive field of the convolutional network, a large number of convolutional layers are often required to capture relevant information from previous and subsequent time steps. As mentioned earlier, the advantage of RCNs lies in their simplicity, therefore, information alignment between time steps can only be achieved through external algorithms. Optical flow-based compensation is the most common and widely used approach, but it suffers from significant computational overhead and error accumulation. To address this issue, this disclosure introduces the global properties of deformers, enabling the capture of global information with fewer layers, thereby achieving information alignment between time steps.

[0061] Figure 3 A flowchart illustrating a video processing method according to an exemplary embodiment of the present disclosure is shown.

[0062] Reference Figure 3 In step S301, the features of the current frame image of the video to be processed and the features of the previous frame image of the current frame are obtained.

[0063] In an exemplary embodiment of this disclosure, when acquiring the features of the current frame image and the features of the previous frame image of the video to be processed, features can be extracted from the current frame image and the previous frame image respectively using the same feature extraction network to obtain the features of the current frame image and the features of the previous frame image. Here, the feature extraction network may include 2D convolutional layers, activation functions, and residual modules. The feature extraction network is a pre-trained network.

[0064] In step S302, the correlation between the features of the current frame image and the features of the previous frame image is determined.

[0065] In an exemplary embodiment of this disclosure, when determining the correlation between the features of the current frame image and the features of the previous frame image, the norm of the features of the current frame image can be calculated first, the features of the current frame image can be divided by the norm of the features of the current frame image to obtain a first feature, then the norm of the features of the previous frame image can be calculated, the features of the previous frame image can be divided by the norm of the features of the previous frame image to obtain a second feature, and the inner product of the first feature and the second feature can be used as the correlation between the features of the current frame image and the features of the previous frame image.

[0066] In step S303, based on the correlation and the hidden layer features of the previous frame image, the alignment features between the current frame image and the previous frame image are determined. Here, the hidden layer features of the previous frame image include features of the hidden layer between the input layer and the output layer that processes the previous frame image. The hidden layer features can be, for example, global abstract features.

[0067] In an exemplary embodiment of this disclosure, when determining the alignment features of the current frame image and the previous frame image, the position with the greatest correlation between the current frame image and the previous frame image can be determined first based on the correlation between the features of the current frame image and the features of the previous frame image. Then, the hidden layer features of the previous frame image are aligned based on the position with the greatest correlation to obtain the alignment features of the current frame image and the previous frame image.

[0068] In an exemplary embodiment of this disclosure, when aligning the hidden features of the previous frame image based on the position with the highest correlation, the position with the highest correlation can be used as an index to migrate the hidden features of the previous frame image.

[0069] In step S304, the current frame image is processed based on the alignment features.

[0070] In an exemplary embodiment of this disclosure, when processing the current frame image based on the alignment features, the current frame image and the alignment features are first concatenated to obtain concatenated features. Then, the concatenated features are input into a preset convolutional neural network to obtain the processed image of the current frame image and the hidden layer features of the current frame image. Here, the preset convolutional neural network may include 2D convolutional layers, activation functions, and residual modules. The preset convolutional neural network is a pre-trained network. The preset convolutional neural network may include a convolutional neural network for super-resolution processing or a convolutional neural network for image enhancement processing. The hidden layer features may be features of the hidden layers in the process of the preset convolutional neural network processing the current frame image based on the alignment features, that is, features of the intermediate layer between the input and output layers of the preset convolutional neural network.

[0071] The video processing method according to exemplary embodiments of this disclosure is applicable not only to video super-resolution but also to video enhancement. In exemplary embodiments of this disclosure, the processed image of the current frame image may include a super-resolution processed image of the current frame image or an enhanced image of the current frame image.

[0072] According to the video processing method disclosed herein, by first obtaining the features of the current frame image of the video to be processed and the features of the previous frame image, calculating the correlation between the features of the current frame image and the features of the previous frame image, and determining the alignment features of the current frame image and the previous frame image based on the correlation and the hidden layer features of the previous frame image, and then processing the current frame image based on the alignment features, the efficiency and effect of video processing are improved.

[0073] Figure 4 This diagram illustrates the use of a super-resolution network to perform super-resolution processing on low-resolution video. Figure 5 This diagram illustrates the use of a super-resolution network including an alignment module to perform super-resolution processing on low-resolution video. Figure 6 This diagram illustrates how the alignment module is used to determine the alignment features between the current frame image and the previous frame image.

[0074] like Figure 4 As shown, when performing super-resolution processing on low-resolution video, the input is the video sequence to be processed (i.e., the low-resolution video sequence). Two frames are input each time, and one frame is super-resolution processed as the output. The features of the hidden layer are used as the input for the next time step. Figure 5 As shown, the three inputs of the super-resolution network are the previous and next frame images, and the hidden features from the previous time step (i.e., the hidden features of the previous frame image). After passing through the alignment module, the hidden features h are transformed into aligned features, which are called... It is then concatenated with the current frame Xt (C represents the concatenation of feature dimensions; for example, concatenating HxWxC and HxWxC images results in HxWx2C). The concatenated features are processed through 2D convolutional layers and activation functions, and then through a preset number of residual modules to obtain the super-resolution result O and the hidden features at the current time step (i.e., the hidden features of the current frame image) h.

[0075] exist Figure 6 In the alignment module shown, K, V, and Q are defined as:

[0076] K=Res(relu(conv(X_{t-1}))),

[0077] V = relu(conv(h_{t-1})),

[0078] Q = Res(relu(conv(X_t))).

[0079] like Figure 6As shown, the alignment module has three inputs: X_{t-1}, X_t, and h. X_{t-1} and X_t are processed by the same Conv2D, reLU, and residual module (e.g., a two-layer residual module) to obtain the corresponding features F_{t-1} and F_t, respectively. Then, in the deformer, they are processed... Calculate the relationship between k and q, and based on r i,j This guides the alignment of hidden features. For example, based on Determine the most relevant information h i Then this h i As an index, hidden layer features are transferred. The alignment module in this disclosure (e.g., Figure 6 In addition to its application in video super-resolution, the alignment module in the video enhancement network can also be applied to other video enhancement networks, such as, but not limited to, denoising, deblurring, and deblocking.

[0080] The above has been combined Figures 1 to 6 A video processing method according to exemplary embodiments of the present disclosure has been described. Hereinafter, reference will be made to... Figure 7 A video processing apparatus and its units according to exemplary embodiments of the present disclosure will be described.

[0081] Figure 7 A block diagram of a video processing apparatus according to an exemplary embodiment of the present disclosure is shown.

[0082] Reference Figure 7 The video processing device includes a feature acquisition unit 71, a correlation determination unit 72, a feature alignment unit 73, and an image processing unit 74.

[0083] The feature acquisition unit 71 is configured to acquire the features of the current frame image of the video to be processed and the features of the previous frame image of the current frame.

[0084] In an exemplary embodiment of this disclosure, the feature acquisition unit 71 may be configured to: extract features from the current frame image and the previous frame image respectively through the same feature extraction network to obtain the features of the current frame image and the features of the previous frame image.

[0085] The correlation determination unit 72 is configured to determine the correlation between the features of the current frame image and the features of the previous frame image.

[0086] In an exemplary embodiment of this disclosure, the correlation determination unit 72 may be configured to: calculate the norm of the features of the current frame image, divide the features of the current frame image by the norm of the features of the current frame image to obtain a first feature; calculate the norm of the features of the previous frame image, divide the features of the previous frame image by the norm of the features of the previous frame image to obtain a second feature; and use the inner product of the first feature and the second feature as the correlation between the features of the current frame image and the features of the previous frame image.

[0087] Feature alignment unit 73 is configured to determine alignment features between the current frame image and the previous frame image based on the correlation and the hidden layer features of the previous frame image. Here, the hidden layer features of the previous frame image include features of the hidden layer between the input layer and the output layer that processes the previous frame image. The hidden layer features can be, for example, global abstract features.

[0088] In an exemplary embodiment of this disclosure, the feature alignment unit 73 may be configured to: determine the position with the greatest correlation between the features of the current frame image and the features of the previous frame image based on the correlation between the features of the current frame image and the features of the previous frame image; and perform an alignment operation on the hidden layer features of the previous frame image based on the position with the greatest correlation to obtain the alignment features of the current frame image and the previous frame image.

[0089] In an exemplary embodiment of this disclosure, the feature alignment unit 73 may be configured to: use the position with the highest correlation as an index to transfer the hidden layer features of the previous frame image.

[0090] The image processing unit 74 is configured to process the current frame image based on the alignment features.

[0091] In an exemplary embodiment of this disclosure, the image processing unit 74 may be configured to: concatenate the current frame image and the alignment feature to obtain a concatenated feature; and input the concatenated feature into a preset convolutional neural network to obtain a processed image of the current frame image and hidden layer features of the current frame image.

[0092] In an exemplary embodiment of this disclosure, the processed image of the current frame image may include a super-resolution processed image of the current frame image or an image enhancement processed image of the current frame image.

[0093] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0094] The above has been combined Figure 7A video processing apparatus according to exemplary embodiments of the present disclosure has been described. Next, in conjunction with... Figure 8 An electronic device according to exemplary embodiments of the present disclosure will be described.

[0095] Figure 8 This is a block diagram of an electronic device 800 according to an exemplary embodiment of the present disclosure.

[0096] Reference Figure 8 The electronic device 800 includes at least one memory 801 and at least one processor 802. The at least one memory 801 stores a set of computer-executable instructions that, when executed by the at least one processor 802, perform a method of video processing according to an exemplary embodiment of the present disclosure.

[0097] In exemplary embodiments of this disclosure, electronic device 800 may be a PC computer, tablet device, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, electronic device 800 is not necessarily a single electronic device, but may be a collection of any devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 800 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0098] In electronic device 800, processor 802 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0099] The processor 802 can execute instructions or code stored in the memory 801, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transmission protocol.

[0100] The memory 801 may be integrated with the processor 802, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 801 may include a separate device, such as an external disk drive, a storage array, or other storage device that can be used by any database system. The memory 801 and the processor 802 may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 802 to read files stored in the memory.

[0101] In addition, the electronic device 800 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 800 can be interconnected via a bus and / or network.

[0102] According to exemplary embodiments of this disclosure, a computer-readable storage medium including instructions is also provided, such as a memory 801 including instructions, which can be executed by a processor 802 of the apparatus 800 to perform the above-described method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0103] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, the computer program product including a computer program / instructions that, when executed by a processor, implement a method for video processing according to exemplary embodiments of the present disclosure.

[0104] The above has been referred to Figures 2 to 8 A video processing method and apparatus according to exemplary embodiments of the present disclosure have been described. However, it should be understood that: Figure 7 The video processing apparatus and its units shown can be configured as software, hardware, firmware, or any combination thereof to perform specific functions. Figure 8 The electronic device shown is not limited to the components shown above, but some components may be added or removed as needed, and the above components may also be combined.

[0105] According to the video processing method and apparatus disclosed herein, by first acquiring the features of the current frame image of the video to be processed and the features of the previous frame image, calculating the correlation between the features of the current frame image and the features of the previous frame image, and determining the alignment features of the current frame image and the previous frame image based on the correlation and the hidden layer features of the previous frame image, and then processing the current frame image based on the alignment features, the efficiency and effect of video processing are improved.

[0106] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0107] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video processing method, characterized in that, include: Obtain the features of the current frame image and the features of the previous frame image of the video to be processed; Determine the correlation between the features of the current frame image and the features of the previous frame image; Based on the correlation and the hidden layer features of the previous frame image, the alignment features of the current frame image and the previous frame image are determined, wherein the hidden layer features of the previous frame image include the features of the hidden layer between the input layer and the output layer that processes the previous frame image. The current frame image is processed based on the alignment features. The step of determining the correlation between the features of the current frame image and the features of the previous frame image includes: Calculate the norm of the features of the current frame image, and divide the features of the current frame image by the norm of the features of the current frame image to obtain the first feature; Calculate the norm of the features of the previous frame image, and divide the features of the previous frame image by the norm of the features of the previous frame image to obtain the second feature; The inner product of the first feature and the second feature is used as the correlation between the features of the current frame image and the features of the previous frame image.

2. The video processing method according to claim 1, characterized in that, The steps for determining the alignment features between the current frame image and the previous frame image include: The position with the highest correlation between the features of the current frame image and the features of the previous frame image is determined based on the correlation between the features of the current frame image and the features of the previous frame image. Alignment operations are performed on the hidden layer features of the previous frame image based on the position with the highest correlation to obtain the alignment features of the current frame image and the previous frame image.

3. The video processing method according to claim 2, characterized in that, The steps for aligning the hidden features of the previous frame image based on the position with the highest correlation include: Using the position with the highest correlation as an index, the hidden features of the previous frame image are transferred.

4. The video processing method according to claim 1, characterized in that, The steps for obtaining the features of the current frame image and the features of the previous frame image of the video to be processed include: The same feature extraction network is used to extract features from the current frame image and the previous frame image respectively, so as to obtain the features of the current frame image and the features of the previous frame image.

5. The video processing method according to claim 1, characterized in that, The steps for processing the current frame image based on the alignment features include: The current frame image and the alignment feature are concatenated to obtain the concatenated feature; The stitched features are input into a preset convolutional neural network to obtain the processed image of the current frame and the hidden layer features of the current frame.

6. The video processing method according to claim 5, characterized in that, The processed image of the current frame image includes either a super-resolution processed image of the current frame image or an image enhancement processed image of the current frame image.

7. A video processing apparatus, characterized in that, include: The feature acquisition unit is configured to acquire the features of the current frame image of the video to be processed and the features of the previous frame image of the current frame. The correlation determination unit is configured to determine the correlation between the features of the current frame image and the features of the previous frame image; A feature alignment unit is configured to determine alignment features between the current frame image and the previous frame image based on the correlation and the hidden layer features of the previous frame image, wherein the hidden layer features of the previous frame image include features of the hidden layer between the input layer and the output layer that processes the previous frame image; and An image processing unit is configured to process the current frame image based on the alignment features. The correlation determination unit is configured as follows: Calculate the norm of the features of the current frame image, and divide the features of the current frame image by the norm of the features of the current frame image to obtain the first feature; Calculate the norm of the features of the previous frame image, and divide the features of the previous frame image by the norm of the features of the previous frame image to obtain the second feature; The inner product of the first feature and the second feature is used as the correlation between the features of the current frame image and the features of the previous frame image.

8. The video processing apparatus according to claim 7, characterized in that, The feature alignment unit is configured as follows: The position with the highest correlation between the features of the current frame image and the features of the previous frame image is determined based on the correlation between the features of the current frame image and the features of the previous frame image. Alignment operations are performed on the hidden layer features of the previous frame image based on the position with the highest correlation to obtain the alignment features of the current frame image and the previous frame image.

9. The video processing apparatus according to claim 7, characterized in that, The feature alignment unit is configured as follows: Using the position with the highest correlation as an index, the hidden features of the previous frame image are transferred.

10. The video processing apparatus according to claim 7, characterized in that, The feature acquisition unit is configured as follows: The same feature extraction network is used to extract features from the current frame image and the previous frame image respectively, so as to obtain the features of the current frame image and the features of the previous frame image.

11. The video processing apparatus according to claim 7, characterized in that, The image processing unit is configured as follows: The current frame image and the alignment feature are concatenated to obtain the concatenated feature; The stitched features are input into a preset convolutional neural network to obtain the processed image of the current frame and the hidden layer features of the current frame.

12. The video processing apparatus according to claim 11, characterized in that, The processed image of the current frame image includes either a super-resolution processed image of the current frame image or an image enhancement processed image of the current frame image.

13. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the video processing method as described in any one of claims 1 to 6.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor of the electronic device, the electronic device performs the video processing method as described in any one of claims 1 to 6.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the video processing method according to any one of claims 1 to 6.

Images