Method for video quality enhancement and related device

Abstract

The embodiment of the specification discloses a video quality enhancement method and related equipment, and relates to the field of image processing. For quality enhancement of a video frame, first, a first frame and a second frame are subjected to feature modulation processing to extract position offset information of a moving object on the two video frames; further, a look-up table is subjected to deformation operation according to the two position offset information, motion feature information corresponding to the two video frames is determined, and the two motion feature information is fused to obtain fused feature information; further, a plurality of fused feature information under different resolutions is obtained to perform quality enhancement processing on the second frame to obtain a target second frame, that is, feature information of different receptive fields is integrated during video quality enhancement, and each frame in a video sequence is subjected to quality enhancement to obtain an entire target video.

Description

Technical Field

[0001] This specification relates to the field of image processing, and in particular to a video quality enhancement method and related equipment. Background Technology

[0002] With the rapid development of internet technology, the amount of video data on the network has exploded. To cope with the transmission demands of massive amounts of video data on the network, video compression technology has been widely used to reduce bandwidth and storage space consumption. For example, H.264 / AVC and H.265 / HEVC are currently the mainstream video coding standards. However, in low bitrate or low latency scenarios, video compression often introduces severe compression artifacts (such as blurring, ringing, and blocking), thus significantly reducing the user's viewing experience (Quality of Experience, QoE).

[0003] While traditional methods have improved the quality of compressed videos to some extent in response to the various problems caused by video compression, they are usually inflexible and have limited performance, making them difficult to adapt to the task of compressed video quality enhancement (CVQE) in low-latency scenarios. Summary of the Invention

[0004] This specification provides a video quality enhancement method and related equipment, which can solve the above-mentioned problems. The technical solution is as follows:

[0005] In a first aspect, embodiments of this specification provide a video quality enhancement method, the method comprising:

[0006] Obtain multiple video frames ordered chronologically; wherein the multiple video frames include a first frame and a second frame that follows the first frame;

[0007] Feature modulation processing is performed on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame; wherein, the number of the moving object is at least one.

[0008] The first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information are determined by transforming the lookup table, and the first motion feature information and the second motion feature information are fused to obtain fused feature information.

[0009] Multiple video frame combinations, including a first frame and a second frame, are acquired. The fusion feature information corresponding to the moving object in each of the multiple video frame combinations is obtained through the above steps. The first frame and the second frame included in the video frame combination have the same resolution, while the resolutions of the multiple video frame combinations are different.

[0010] The second frame is enhanced based on the multiple fusion feature information corresponding to the moving object to obtain the target second frame.

[0011] Secondly, embodiments of this specification provide a video quality enhancement device, the device comprising:

[0012] A video frame acquisition module is used to acquire multiple video frames ordered in chronological order; wherein the multiple video frames include a first frame and a second frame that follows the first frame.

[0013] A convolutional processing module is used to perform feature modulation processing on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame; wherein, the number of the moving object is at least one.

[0014] The lookup table processing module is used to determine the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information through the transformation operation of the lookup table, and to fuse the first motion feature information and the second motion feature information to obtain fused feature information.

[0015] A multi-scale fusion module is used to acquire multiple video frame combinations, including a first frame and a second frame, and obtain the fusion feature information of the moving object in the multiple video frame combinations through the above steps; wherein, the first frame and the second frame included in the video frame combination have the same resolution, and the resolutions of the multiple video frame combinations are different.

[0016] The feature fusion module is used to perform quality enhancement processing on the second frame based on multiple fused feature information corresponding to the moving object, so as to obtain the target second frame.

[0017] Thirdly, embodiments of this specification provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the above-described method steps.

[0018] Fourthly, embodiments of this specification provide a computer program product that stores multiple instructions adapted for loading by a processor and executing the above-described method steps.

[0019] Fifthly, embodiments of this specification provide an electronic device that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0020] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0021] To enhance the quality of video frames, feature modulation processing is first performed on the first and second frames to extract the position offset information of the moving object in the two video frames. This allows the temporal information between multiple video frames to be fully captured during quality enhancement, thereby improving the effect of video quality enhancement.

[0022] Furthermore, based on the two position offset information, a lookup table transformation operation is performed to determine the motion feature information corresponding to the two video frames respectively, and the two motion feature information are fused to obtain fused feature information. By replacing the neural network algorithm with a low-complexity lookup table algorithm, the amount of computation and the performance requirements of computing devices are greatly reduced. Moreover, the computation task can be completed efficiently and quickly in low-latency scenarios, with high real-time performance.

[0023] Furthermore, multiple fused feature information at different resolutions are obtained to perform quality enhancement processing on the second frame to obtain the target second frame. That is, feature information from different receptive fields is incorporated into the video quality enhancement, which effectively solves the problem of insufficient performance when extending the single-frame quality enhancement method to multi-frame quality enhancement processing scenarios, and greatly improves the quality enhancement effect of the second frame and even the quality enhancement effect of the entire video. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the architecture of a video quality enhancement method provided in the embodiments of this specification;

[0026] Figure 2 This is a schematic flowchart of a video quality enhancement method provided in the embodiments of this specification;

[0027] Figure 3 This is a schematic diagram of a scene for obtaining the first frame and the second frame, provided in an embodiment of this specification.

[0028] Figure 4This is a schematic flowchart of a video quality enhancement method provided in the embodiments of this specification;

[0029] Figure 5 This is a schematic diagram of a process for obtaining first position offset information and second position offset information provided in an embodiment of this specification;

[0030] Figure 6 This is a schematic flowchart of a video quality enhancement method provided in the embodiments of this specification;

[0031] Figure 7 This is a schematic diagram of a process for obtaining first motion feature information provided in an embodiment of this specification;

[0032] Figure 8 This is a schematic diagram of the structure of a video quality enhancement device provided in the embodiments of this specification;

[0033] Figure 9 This is a schematic diagram of the structure of an electronic device provided in the embodiments of this specification. Detailed Implementation

[0034] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0035] In the description of this specification, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this specification, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this specification based on the specific circumstances. Furthermore, in the description of this specification, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0036] The present specification will now be described in detail with reference to specific embodiments.

[0037] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this specification are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the features, information, and data involved in this specification were all obtained under full authorization.

[0038] With the rapid development of internet technology, the amount of video data on the network has exploded. To cope with the transmission demands of massive amounts of video data on the network, video compression technology has been widely used to reduce bandwidth and storage space consumption. For example, H.264 / AVC and H.265 / HEVC are currently the mainstream video coding standards. However, in low bitrate or low latency scenarios, video compression often introduces severe compression artifacts (such as blurring, ringing, and blocking), thus significantly reducing the user's viewing experience (Quality of Experience, QoE).

[0039] While traditional methods have improved the quality of compressed videos to some extent in response to the various problems caused by video compression, they are usually inflexible and have limited performance, making them difficult to adapt to the task of compressed video quality enhancement (CVQE) in low-latency scenarios.

[0040] In recent years, the rapid development of deep learning has driven research on CVQE based on deep neural networks. Early methods mostly focused on quality enhancement of single-frame images. To better utilize the temporal information of videos, subsequent researchers have proposed various multi-frame-based compressed video quality enhancement methods. However, these methods are generally computationally complex and heavily dependent on hardware, making it difficult to meet the real-time requirements of low-latency scenarios.

[0041] Look-up table (LUT) based algorithms have gained increasing attention due to their efficiency and have shown great potential in the field of image / video super-resolution. However, most current LUT-based methods focus on single-frame image processing, while existing multi-frame methods only utilize simple temporal information through LUT weighting. Therefore, there is still a technological gap in research on low-latency CVQE tasks, and there is an urgent need to propose efficient and real-time solutions to meet the needs of practical engineering applications.

[0042] Therefore, to address the above problems, this specification provides a video quality enhancement method and related equipment in its embodiments. For example... Figure 1 As shown, Figure 1 This is a schematic diagram of the architecture of a video quality enhancement method provided in the embodiments of this specification. Figure 1 It includes at least a server 101 for performing a video quality enhancement method, and also includes multiple electronic devices for uploading multiple video frames and execution instructions for instructing the server to perform the video quality enhancement method. The multiple electronic devices include at least electronic device 1021, electronic device 1022, and electronic device 1023. It is understood that... Figure 1 The number of servers and electronic devices shown is for illustrative purposes only, and the embodiments in this specification do not impose any limitations on them.

[0043] The aforementioned server 101 includes, but is not limited to, physical or virtual processors, mobile stations (MS), mobile terminals, mobile phones, handsets and portable equipment, Bluetooth headsets, smartwatches, etc. The server 101 can communicate with one or more core networks via a radio access network (RAN).

[0044] For example, server 101 consists of multiple physical servers, which are hardware-independent. Alternatively, server 101 can consist of multiple virtual servers deployed in the same hardware resource pool. The deployment methods for virtual servers include, but are not limited to, VMware, VirtualBox, and Virtual PC.

[0045] It is understood that server 10 also has other service capabilities and functions to complete the tasks described in the following embodiments. For example, server 1011 also provides portal services, resource management services, and CI / CD services, etc.

[0046] In the embodiments of this specification, electronic devices such as electronic devices 1021, 1022, and 1023 may also be equipped with display devices. These display devices can be various devices capable of display functions, such as cathode ray tube displays (CR), light-emitting diode displays (LED), electronic ink screens, liquid crystal displays (LCD), and plasma display panels (PDP). For example, a user can use the display device on electronic device 1021 to send multiple video frames to server 101, or view a quality-enhanced target video sent by server 101.

[0047] Multiple electronic devices and servers can communicate through communication links established by communication protocols, such as gRPC. gRPC is a high-performance, general-purpose, open-source Remote Procedure Call (RPC) framework primarily designed for mobile application development and based on the HTTP / 2 protocol standard. It is developed using Protocol Buffers (PB) serialization and supports numerous programming languages. Furthermore, communication links can be wireless or wired. For example, wired communication links include fiber optic cables, twisted-pair cables, or coaxial cables, while wireless communication links include Bluetooth, Wi-Fi, or microwave communication links.

[0048] In one embodiment, such as Figure 2 The diagram shown is a flowchart illustrating a video quality enhancement method provided in an embodiment of this specification. This method can be implemented using a computer program and can run on a video quality enhancement device based on the von Neumann architecture. The computer program can be integrated into an application or run as a standalone utility application.

[0049] Specifically, the video quality enhancement method includes:

[0050] S102. Obtain multiple video frames sorted in chronological order, including a first frame and a second frame following the first frame.

[0051] Video can be acquired through various methods, including local files, network streams, webcams, and wireless devices. For example, server 101 can receive video streams from other electronic devices via network protocols such as RTSP, RTMP, and HLS. Alternatively, server 101 can be equipped with an image acquisition device to capture video.

[0052] A video frame can be understood as a segmentation of a video based on a preset frame rate, resulting in multiple video frames. Each video frame contains various patterns. For example, if the preset frame rate is 10 frames per minute, a 2-minute video can yield 20 video frames. The specific frame rate value can be set based on the desired video quality enhancement or can be customized by relevant personnel as needed.

[0053] In another embodiment, multiple video frames can be directly acquired and then fused to obtain a video. For example, server 101 can directly receive multiple video frames sent from electronic device 1021.

[0054] Multiple video frames are sorted chronologically, specifically from earliest to latest based on their occurrence time. Alternatively, this can be understood as sorting them from front to back based on the generation time of each individual video frame. Another interpretation is that the order of the video frames reflects the order in which the video was segmented during chronological playback.

[0055] like Figure 3 As shown, Figure 3 This is a schematic diagram illustrating a scenario for obtaining a first frame and a second frame, provided in an embodiment of this specification. Multiple video frames are sorted chronologically, with the generation time of the first frame 201 preceding the generation time of the second frame 202. It is understood that any two video frames can be extracted from the multiple video frames as the first and second frames, or two video frames that are immediately adjacent in the order can be extracted as the first and second frames (e.g.,...). Figure 3 As shown in the figure, the latter extraction method can better capture the temporal information between two video frames and improve the video quality enhancement effect.

[0056] S104. Perform feature modulation processing on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame; wherein, the number of moving objects is at least one.

[0057] Feature modulation (EM) can be understood as a type of convolution processing and is a common method used in computer vision to extract image features. EEM can extract features such as edges and textures from the first and second frames, especially the motion changes of at least one moving object from the first frame to the second frame. Typically, convolution kernels (such as edge detection kernels and blur kernels) are used to perform EEM on the first and second frames, resulting in first and second convolution information, which are also known as first and second position offset information.

[0058] There is a moving object in both the first frame and the second frame, and there is at least one moving object. The processing steps for obtaining the first position offset information of each moving object in the first frame and the second position offset information in the second frame are the same.

[0059] For example, the first and second frames are preprocessed, such as converting them to grayscale images to reduce the complexity of convolution processing. Next, a convolution kernel is selected; a convolution kernel is a filter used to process the image. Commonly used convolution kernels include edge detection kernels (such as Sobel or Prewitt kernels), blurring kernels, and sharpening kernels. The selected kernel is applied to the first and second frames, and the convolution results are extracted, including convolution result conv1 obtained by convolving the first frame and convolution result conv2 obtained by convolving the second frame. The gradient magnitudes of the first and second frames are calculated using the two convolution results. If the gradients of certain regions change significantly between the two frames, those regions may contain moving objects. Image processing methods (such as contour detection and region filling) can then be used to analyze the shape and position of the moving objects, obtaining the first positional offset information of the moving objects in the first frame and the second positional offset information in the second frame.

[0060] In one embodiment, the temporal data of the first frame and the second frame are aligned, and feature modulation processing is performed on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame.

[0061] Aligning the temporal data of the first and second frames ensures that pixels in video frames at different times correspond to the same physical region (i.e., different moments in the same scene). If the first and second frames are not aligned, the image content at the two moments may shift due to object movement, camera shake, or other factors, leading to inaccurate detection of moving objects and affecting subsequent analysis. In other words, aligning the temporal data avoids false or missed detections of moving objects due to misaligned video frame content, and allows for the extraction of the correct first and second position offset information of the moving object.

[0062] In this embodiment, the temporal data of the first and second frames can be aligned by comparing the positional offset information of the same region in the first and second frames, respectively, thus more accurately detecting the movement of moving objects. In another embodiment, the temporal data of the first and second frames can also be aligned using image alignment methods based on feature points or optical flow, as needed by relevant personnel.

[0063] like Figure 4 As shown, Figure 4 This is a flowchart illustrating a video quality enhancement method provided in an embodiment of this specification. The first frame 301 and the second frame 302 are used as input for encoding. The encoded data corresponding to the first frame and the second frame are then input into a Motion Feature Modulation Network for convolution processing to obtain the first position offset information of the moving object in the first frame and the second position offset information in the second frame. For example... Figure 4 As shown, the moving object is a volleyball, and its pixel position is different in the first frame Sample and the second frame Offsets.

[0064] S106. Determine the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information through the transformation operation of the lookup table, and fuse the first motion feature information and the second motion feature information to obtain fused feature information.

[0065] The table used in a lookup table (LUT) is a mapping table that returns predefined information based on certain input values ​​(such as convolution outputs, specific features, etc.). A LUT can be a simple array or matrix, where the indices are the feature values ​​or activation values ​​of the positional offset information. The lookup table can be optimized through training to better predict and select appropriate motion feature information based on the features in the positional offset information.

[0066] Based on the first position offset information, a lookup table (LUT) is used to determine the first motion feature information corresponding to the first frame, and based on the second position offset information, a lookup table is used to determine the second motion feature information corresponding to the second frame. For example, the first position offset information is a convolutional feature map containing a large number of activation values. These activation values ​​are mapped to the index of the lookup table to obtain the motion features associated with each convolutional value in the convolutional feature map until the first motion feature information is obtained.

[0067] The first motion feature information represents the motion pattern, direction, or speed of a moving object in the first frame, while the second motion feature information represents the same characteristics in the second frame. After obtaining the first and second motion feature information, they are fused to form a more comprehensive set of motion feature information. Fusion can be performed in various ways, with common methods including weighted fusion, concatenation, and element-wise summation.

[0068] like Figure 4 As shown, the first position offset information and the second position offset information are further matched by a lookup table in the Temporal Feature Extraction Module to obtain the first motion feature information corresponding to the first position offset information and the second motion feature information associated with the second position offset information, and then the two motion feature information are fused to obtain the fused feature information.

[0069] In this embodiment, the number of moving objects is taken as an example. In other embodiments, the number of moving objects can be multiple. That is, the fused motion feature information corresponding to multiple moving objects is obtained by the temporal feature extraction module TFEM. Each motion feature information represents the motion feature of the moving object based on the temporal information of the first frame and the second frame.

[0070] S108. Obtain multiple video frame combinations including a first frame and a second frame, and obtain the fusion feature information corresponding to the moving object in the multiple video frame combinations through the above steps; wherein, the first frame and the second frame included in the video frame combination have the same resolution, and the resolutions of the multiple video frame combinations are different.

[0071] Simultaneously changing the resolution of the first and second frames yields another set of video frames that include both the first and second frames. For example, as... Figure 3 The first and second frames shown have a resolution of 1080P. Based on the first frame 201 and the second frame 202 with a resolution of 1080P, the fused motion feature information corresponding to the moving object is obtained. The resolution of the first frame 201 and the second frame 202 is changed to 720P, and the fused motion feature information corresponding to the moving object is obtained based on the above steps. The resolution of the first frame 201 and the second frame 202 is changed again to 480P, and the fused motion feature information corresponding to the moving object is obtained based on the above steps.

[0072] like Figure 4 As shown, video frames corresponding to different resolutions are combined and fused feature information is obtained by using the Motion Feature Modulation Network (MFMN) and the Temporal Feature Extraction Module (TFEM).

[0073] S110. Perform quality enhancement processing on the second frame based on multiple fusion feature information corresponding to the moving object to obtain the target second frame.

[0074] Multiple fusion feature information corresponding to the moving object are fused, and the second frame is then subjected to quality enhancement processing to obtain a target second frame with enhanced quality. For example... Figure 4 As shown, multiple resolutions are obtained as fusion feature information. The multiple fusion feature information are then fused by the Multi-Scale Fusion Module, which combines the feature information of different receptive fields to enhance the quality of the second frame, resulting in the target second frame 303.

[0075] Multi-Scale Fusion Module (MSFM) is a structure in convolutional neural networks, particularly effective in tasks such as video processing, dynamic scene rendering, and image segmentation. It combines information from different receptive fields (resolutions). This MSFM module considers information from local details to global features simultaneously, thus improving the quality of the second frame. This is because in video enhancement, the details of each frame may differ at different resolutions. For example, high-resolution features capture detailed information, while low-resolution features capture global information. MSFM combines features from these different receptive fields, enhancing the model's ability to understand and process diverse information.

[0076] Understandable, Figure 4 Taking the example of one moving object in the first frame and one moving object in the second frame, when there are multiple moving objects, multiple fused motion feature information corresponding to multiple moving objects are obtained, as well as multiple fused motion feature information at different resolutions corresponding to each moving object. The multiple fused motion feature information at different resolutions corresponding to each moving object are fused in turn until the multiple fused motion feature information corresponding to all moving objects is fused. Then, the second frame is subjected to quality enhancement processing to obtain the target second frame.

[0077] To enhance video frame quality, feature modulation processing is first applied to the first and second frames to extract the positional offset information of moving objects in the two frames. This allows for the utilization of temporal information between multiple video frames during quality enhancement, fully capturing the correlation between them and improving the overall video quality enhancement effect. Furthermore, a lookup table transformation operation is performed based on the two positional offset information to determine the motion feature information corresponding to each of the two video frames, and the two motion feature information are fused to obtain fused feature information. Replacing the neural network algorithm with a low-complexity lookup table algorithm significantly reduces the computational load and the performance requirements of computing devices, and can efficiently and quickly complete computational tasks even in low-latency scenarios, exhibiting high real-time performance. Further, multiple fused feature information at different resolutions is obtained to enhance the second frame, resulting in the target second frame. This means that feature information from different receptive fields is incorporated into the video quality enhancement process, effectively solving the performance limitations when extending single-frame quality enhancement methods to multi-frame quality enhancement scenarios, greatly improving the quality enhancement effect on the second frame and even the overall video quality enhancement effect.

[0078] based on Figure 2 - Figure 4 The illustrated embodiment is shown in Example 1 and can be found in [other examples]. Figure 5 , Figure 5 This is a schematic flowchart illustrating an embodiment of the present specification for obtaining first position offset information and second position offset information. Step S104 includes:

[0079] S202. The first and second frames are processed by feature modulation using the motion vector of the moving object to obtain the multi-scale convolution kernel offset.

[0080] Motion vectors (MVs) reflect the displacement or direction of motion of a moving object and can be used to guide convolution processing, enabling multi-scale convolution kernels to adapt to motion across multiple video frames. Feature modulation aims to dynamically adjust the convolution process based on motion information, thereby better capturing the motion changes of a moving object between consecutive first and second frames. The offset of the multi-scale convolution kernel is generated based on the horizontal and vertical components of the motion vectors.

[0081] like Figure 4As shown, the encoded data corresponding to the first and second frames are input into the Motion Feature Modulation Network (MFMN). The MFMN processes the first and second frames through motion modulation. This feature modulation process utilizes a spatiotemporal convolutional network to extract motion features of moving objects from multiple videos, paying attention to changes between frames. Motion modulation is a crucial step in the convolutional processing. In this stage, the MFMN enhances the perception of motion features based on the extracted motion features using specific modulation mechanisms, highlighting dynamic information within the video frames.

[0082] S204. Perform feature modulation processing on the first and second frames based on the multi-scale convolution kernel offset to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame.

[0083] Specifically, the first and second frames are sampled according to the multi-scale convolutional kernel offset to achieve intra-frame feature refinement and inter-frame information alignment, extracting the first position offset information of the moving object in the first frame and the second position offset information in the second frame. The design of the multi-scale convolutional kernel allows the Motion Feature Modulation Network (MFMN) to perform feature modulation processing on the first and second frames at different resolutions. The size or offset of the convolutional kernel at each scale can be adjusted according to the motion vectors of the first and second frames.

[0084] In this embodiment, the first and second frames are subjected to feature modulation processing using the motion vector of the moving object to obtain the multi-scale convolution kernel offset. The first and second frames are then sampled using the multi-scale convolution kernel offset to obtain the first position offset information of the moving object in the first frame and the second position offset information in the second frame. This efficiently aligns the temporal data of the first and second frames, allowing for better focus on the motion features of the moving object obtained based on the temporal information of consecutive frames.

[0085] based on Figure 2 - Figure 4 The embodiments shown in the specification also include the following embodiments:

[0086] S104. Before performing feature modulation processing on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame, the process further includes:

[0087] Background separation processing is performed on the background region and region of interest in the first frame to obtain a separated frame.

[0088] Background subtraction is used in tasks such as video analysis, object tracking, and motion detection. In this embodiment, by performing background subtraction on the first frame, the background region and the region of interest (foreground object) including the moving object are extracted, resulting in a subtracted frame after background subtraction processing. Figure 4 As shown, the input video frames include a first frame 301, a separated frame that performs background separation processing on the first frame 301, and a second frame 302.

[0089] S104. Perform feature modulation processing on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame, specifically including:

[0090] Based on the separation of frames, feature modulation processing is performed on the first and second frames to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame.

[0091] Specifically, because the separated frames initially distinguish between the background region and the region of interest, which includes moving objects, the accuracy of capturing moving objects present in both the first and second frames can be improved. For example... Figure 4 As shown, the first frame 301, the second frame 302, and the separated frames corresponding to the first frame 301 are encoded and input into the motion feature modulation network MFMN to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame.

[0092] In this embodiment, a separated frame is obtained by performing background separation processing on the first frame. The first position offset information and the second position offset information of the moving object are obtained through the separated frame, which makes the extraction of the moving object more accurate and reduces the possibility of identifying the background area as a moving object and performing convolution processing.

[0093] based on Figure 2 The embodiments shown in the specification also include the following embodiments:

[0094] S110. After performing quality enhancement processing on the second frame based on multiple fused feature information corresponding to the moving object to obtain the target second frame, the process also includes:

[0095] The first and second frames are obtained sequentially from multiple video frames. The quality enhancement target second frame corresponding to the second frame is obtained based on the above steps, until the target second frame corresponding to the reversed first frame is obtained. The multiple target second frames are then subjected to video fusion processing to obtain the target video.

[0096] For example, multiple video frames include video frame 01, video frame 02, video frame 03, and video frame 04, and these video frames 01, 02, 03, and 04 are ordered from front to back in chronological order. In other words, video frame 01 is the first frame in forward order, and video frame 04 is the first frame in reverse order.

[0097] Extract video frame 01 and video frame 02 as the first and second frames, and then... Figure 2 As shown in S102-S110, the target second frame 02 for quality enhancement of video frame 02 is obtained. Video frames 02 and 03 are extracted as the first and second frames, respectively, and then processed... Figure 2 As shown in S102-S110, the target second frame 03 for quality enhancement of video frame 03 is obtained. Video frame 03 and video frame 04 are extracted as the first frame and the second frame, respectively, and then processed... Figure 2 As shown in S102-S110, the target second frame 04 is obtained for quality enhancement of video frame 04.

[0098] The aforementioned video frame 01, target second frame 02, target second frame 03, and target second frame 04 are subjected to video fusion processing to obtain the target video. This target video has been enhanced in quality compared to the original video obtained from video frames 01, 02, 03, and 04, reducing problems such as blurring, ringing, and blockiness present in the original video.

[0099] In one embodiment, such as Figure 6 The diagram shown is a flowchart illustrating a video quality enhancement method provided in an embodiment of this specification. This method can be implemented using a computer program and can run on a video quality enhancement device based on the von Neumann architecture. The computer program can be integrated into an application or run as a standalone utility application.

[0100] Specifically, the video quality enhancement method includes:

[0101] S302. Obtain multiple video frames sorted in chronological order.

[0102] The above S102 will not be repeated here.

[0103] S304. Perform feature modulation processing on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame.

[0104] See S104 above, which will not be repeated here.

[0105] S306. Obtain the feature information table and the condition table; wherein, the condition table includes multiple condition parameters used in the table lookup process.

[0106] The table used in a lookup table (LUT) is a feature information table that returns corresponding predefined information based on certain input values ​​(such as convolution output, specific features, etc.). A LUT can be a simple array or matrix, where the index is the feature value or activation value of the positional offset information, and it stores motion feature information that matches that positional offset information.

[0107] In this embodiment, a conditional LUT is introduced to assist in finding the motion feature information corresponding to the position offset information in the feature information table. The condition table contains multiple conditions used in the lookup table and multiple parameters involved in those conditions, which take effect when performing queries, filtering, or calculations.

[0108] S308. Based on the condition table, determine the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information by performing a lookup table transformation operation in the feature information table.

[0109] Based on the conditions and parameters provided in the condition table, during the lookup process, the target conditions that the first position offset information meets in the condition table are obtained, and at least one parameter corresponding to the target condition is acquired. Then, the target condition and at least one parameter are matched in the feature information table to obtain the first motion feature information associated with the first position offset information. The steps for obtaining the second motion feature information based on the second position offset information are as described above.

[0110] In one embodiment, during the lookup table process, the first position offset information and the second position offset information are matched in the feature information table using a tetrahedral interpolation algorithm and a condition table, respectively, to obtain the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information.

[0111] Tetrahedral interpolation is an algorithm for interpolating data in three-dimensional space. It is commonly used in numerical simulation, computer graphics, geographic information systems (GIS), physical simulation, and other fields, and is particularly effective when processing unstructured meshes or three-dimensional data.

[0112] The 4D-LUT algorithm combines the speed of table lookup with the accuracy of interpolation, and can significantly improve computational efficiency, especially in video quality enhancement that requires fast calculations.

[0113] In one embodiment, during the lookup table process, the first position offset information is quantized using a tetrahedral interpolation algorithm to obtain first low-order data and first high-order data; the first interpolation parameter is obtained in the condition table based on the first low-order data, and interpolation calculation is performed in the feature information table based on the first interpolation parameter being greater than the first high-order data to obtain the first motion feature information associated with the first position offset information.

[0114] like Figure 7 As shown, Figure 7 This is a flowchart illustrating a method for obtaining first motion feature information provided in an embodiment of this specification. Taking the first position offset information 401 as an example, the four-dimensional first position offset information 401 is quantized to a size of 17*17*17*17, that is, the four-dimensional data (usually floating-point numbers) is quantized into integer values. By reducing precision, the computational burden in the search process is reduced. The value of each dimension is quantized into 17 discrete levels according to its range.

[0115] During the table lookup process, the first low-order data (LSB) obtained through quantization is used to retrieve the conditional parameters Query, including interpolation weights and interpolation points, from the lookup table. The lower four bits (low precision) are used as an index to find the interpolation weights and interpolation points. These interpolation points represent certain positions within the feature information table (LUT), while the interpolation weights indicate how these points are weighted. Further, the interpolation points are weighted using the higher four bits (MSB) to obtain the final lookup result; the calculation formula is described in [link to formula]. Figure 7 The expression for Output is used to obtain the first motion feature information corresponding to the first position offset information 401.

[0116] Furthermore, the second position offset information is quantized using the tetrahedral interpolation algorithm to obtain the second low-order data and the second high-order data; the second interpolation parameter is obtained in the condition table based on the second low-order data, and the second high-order data is interpolated in the feature information table based on the second interpolation parameter to obtain the second motion feature information associated with the second position offset information.

[0117] The second motion feature information is obtained based on the second position offset information using the tetrahedral interpolation algorithm. (See [link]) Figure 7 The first motion feature information is obtained through the first position offset information 401, as shown in the figure, which will not be elaborated here.

[0118] In this embodiment, continuous input data is compressed into discrete indices through quantization, which facilitates rapid lookup. The lookup of the lower four bits can be regarded as a rapid positioning process to obtain the position of the interpolation point and the initial interpolation weights, while the weighted adjustment of the higher four bits is a fine adjustment of the interpolation point, ensuring higher precision interpolation results and improving the accuracy of the obtained motion feature information.

[0119] S310. The first motion feature information and the second motion feature information are fused to obtain fused feature information.

[0120] The first motion feature information represents the motion pattern, direction, or speed of a moving object in the first frame, while the second motion feature information represents the same characteristics in the second frame. After obtaining the first and second motion feature information, they are fused to form a more comprehensive set of motion feature information. Fusion can be performed in various ways, with common methods including weighted fusion, concatenation, and element-wise summation.

[0121] S312. Obtain multiple video frame combinations including a first frame and a second frame, and obtain the fusion feature information corresponding to the moving object in the multiple video frame combinations through the above steps; wherein, the first frame and the second frame included in the video frame combination have the same resolution, and the resolutions of the multiple video frame combinations are different.

[0122] Participation in S108 mentioned above will not be repeated here.

[0123] S316. Perform quality enhancement processing on the second frame based on multiple fusion feature information corresponding to the moving object to obtain the target second frame.

[0124] The participation in S110 mentioned above will not be repeated here.

[0125] To enhance video frame quality, feature modulation processing is first applied to the first and second frames to extract the positional offset information of moving objects in the two frames. This allows for the utilization of temporal information between multiple video frames during quality enhancement, fully capturing the correlation between them and improving the overall video quality enhancement effect. Furthermore, a lookup table transformation operation is performed based on the two positional offset information to determine the motion feature information corresponding to each of the two video frames, and the two motion feature information are fused to obtain fused feature information. Replacing the neural network algorithm with a low-complexity lookup table algorithm significantly reduces the computational load and the performance requirements of computing devices, and can efficiently and quickly complete computational tasks even in low-latency scenarios, exhibiting high real-time performance. Further, multiple fused feature information at different resolutions is obtained to enhance the second frame, resulting in the target second frame. This means that feature information from different receptive fields is incorporated into the video quality enhancement process, effectively solving the performance limitations when extending single-frame quality enhancement methods to multi-frame quality enhancement scenarios, greatly improving the quality enhancement effect on the second frame and even the overall video quality enhancement effect.

[0126] The following are embodiments of the apparatus described in this specification, which can be used to execute the embodiments of the methods described in this specification. For details not disclosed in the apparatus embodiments of this specification, please refer to the embodiments of the methods described in this specification.

[0127] Please see Figure 8 This diagram illustrates a structural schematic of a video quality enhancement apparatus provided in an exemplary embodiment of this specification. The video quality enhancement apparatus can be implemented as all or part of a device through software, hardware, or a combination of both. The device includes a video frame acquisition module 501, a convolution processing module 502, a lookup table processing module 503, a multi-scale fusion module 504, and a feature fusion module 505.

[0128] The video frame acquisition module 501 is used to acquire multiple video frames ordered in chronological order; wherein the multiple video frames include a first frame and a second frame that follows the first frame.

[0129] The convolution processing module 502 is used to perform feature modulation processing on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame; wherein, the number of the moving object is at least one.

[0130] The lookup table processing module 503 is used to determine the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information through the transformation operation of the lookup table, and to fuse the first motion feature information and the second motion feature information to obtain fused feature information.

[0131] The multi-scale fusion module 504 is used to acquire multiple video frame combinations including a first frame and a second frame, and obtain the fusion feature information corresponding to the moving object in the multiple video frame combinations through the above steps; wherein, the first frame and the second frame included in the video frame combination have the same resolution, and the resolutions of the multiple video frame combinations are different.

[0132] The feature fusion module 505 is used to perform quality enhancement processing on the second frame based on multiple fusion feature information corresponding to the moving object to obtain the target second frame.

[0133] In one embodiment, the video quality enhancement device further includes:

[0134] The table acquisition module is used to acquire a feature information table and a condition table; wherein, the condition table includes multiple condition parameters used in the table lookup process;

[0135] The lookup table processing module 503 includes:

[0136] The table processing unit is configured to determine, based on the condition table, the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information by performing a lookup table transformation operation in the feature information table;

[0137] The information fusion unit is used to fuse the first motion feature information and the second motion feature information to obtain fused feature information.

[0138] In one embodiment, the table processing unit includes:

[0139] The lookup table subunit is used to match the first position offset information and the second position offset information in the feature information table respectively through the tetrahedral interpolation algorithm and the condition table during the lookup table process, so as to obtain the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information.

[0140] In one embodiment, the lookup table subunit is specifically used for:

[0141] During the lookup table process, the first position offset information is quantized using a tetrahedral interpolation algorithm to obtain the first low-order data and the first high-order data;

[0142] The first interpolation parameter is obtained from the condition table based on the first low-order data, and the first high-order data is interpolated in the feature information table based on the first interpolation parameter to obtain the first motion feature information associated with the first position offset information.

[0143] The second position offset information is quantized using the tetrahedral interpolation algorithm to obtain the second low-order data and the second high-order data;

[0144] The second interpolation parameter is obtained from the condition table based on the second low-order data. The second high-order data is then interpolated in the feature information table based on the second interpolation parameter to obtain the second motion feature information associated with the second position offset information.

[0145] In one embodiment, the convolution processing module 502 includes:

[0146] The alignment processing unit is used to align the temporal data of the first frame and the second frame, and to perform feature modulation processing on the first frame and the second frame to extract the first position offset information of the moving object on the first frame and the second position offset information on the second frame.

[0147] In one embodiment, the convolution processing module 502 includes:

[0148] The feature modulation unit is used to perform feature modulation processing on the first frame and the second frame using the motion vector of the moving object to obtain the multi-scale convolution kernel offset.

[0149] The convolution processing unit is used to perform feature modulation processing on the first frame and the second frame according to the multi-scale convolution kernel offset, and extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame.

[0150] In one embodiment, the video quality enhancement device includes:

[0151] A separation unit is used to perform background separation processing on the background region and the region of interest in the first frame to obtain a separated frame;

[0152] Convolution processing module 502 includes:

[0153] The split convolutional unit is used to perform feature modulation processing on the first frame and the second frame based on the split frame, and extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame.

[0154] In one embodiment, the video quality enhancement device includes:

[0155] The frame enhancement unit is used to sequentially obtain two adjacent video frames from multiple video frames as the first frame and the second frame, and to obtain a target second frame with quality enhancement corresponding to the second frame based on the above steps, until the target second frame corresponding to the reversed first frame is obtained.

[0156] The video enhancement unit is used to perform video fusion processing on the multiple target second frames to obtain the target video.

[0157] To enhance video frame quality, feature modulation processing is first applied to the first and second frames to extract the positional offset information of moving objects in the two frames. This allows for the utilization of temporal information between multiple video frames during quality enhancement, fully capturing the correlation between them and improving the overall video quality enhancement effect. Furthermore, a lookup table transformation operation is performed based on the two positional offset information to determine the motion feature information corresponding to each of the two video frames, and the two motion feature information are fused to obtain fused feature information. Replacing the neural network algorithm with a low-complexity lookup table algorithm significantly reduces the computational load and the performance requirements of computing devices, and can efficiently and quickly complete computational tasks even in low-latency scenarios, exhibiting high real-time performance. Further, multiple fused feature information at different resolutions is obtained to enhance the second frame, resulting in the target second frame. This means that feature information from different receptive fields is incorporated into the video quality enhancement process, effectively solving the performance limitations when extending single-frame quality enhancement methods to multi-frame quality enhancement scenarios, greatly improving the quality enhancement effect on the second frame and even the overall video quality enhancement effect.

[0158] It should be noted that the video quality enhancement device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the video quality enhancement method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the video quality enhancement device and the video quality enhancement method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0159] The example numbers in this specification are for descriptive purposes only and do not represent the superiority or inferiority of the examples.

[0160] This specification also provides a computer storage medium that can store multiple instructions adapted to be loaded and executed by a processor as described above. Figure 1 - Figure 7 The video quality enhancement method described in the illustrated embodiment can be found in the following documentation for its specific execution process. Figure 1 - Figure 7 The specific details of the illustrated embodiments will not be elaborated here.

[0161] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figure 1 - Figure 7 The video quality enhancement method described in the illustrated embodiment can be found in the following documentation for its specific execution process. Figure 1 - Figure 7The specific details of the illustrated embodiments will not be elaborated here.

[0162] Please see Figure 9 This document provides a schematic diagram of the structure of an electronic device as an embodiment of the present specification. Figure 9 As shown, the electronic device 600 may include: at least one processor 601, at least one network interface 604, a user interface 603, a memory 605, and at least one communication bus 602.

[0163] The communication bus 602 is used to enable communication between these components.

[0164] The user interface 603 may include a display screen and a camera. Optionally, the user interface 603 may also include a standard wired interface and a wireless interface.

[0165] The network interface 604 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0166] The processor 601 may include one or more processing cores. The processor 601 connects to various parts of the server 600 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 605, and by calling data stored in the memory 605. Optionally, the processor 601 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 601 may integrate one or a combination of several of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 601 and may be implemented as a separate chip.

[0167] The memory 605 may include random access memory (RAM) or read-only memory. Optionally, the memory 605 may include a non-transitory computer-readable storage medium. The memory 605 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 605 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 605 may also be at least one storage device located remotely from the aforementioned processor 601. Figure 9 As shown, the memory 605, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a video quality enhancement application.

[0168] exist Figure 9 In the illustrated electronic device 600, the user interface 603 is mainly used to provide an input interface for the user and to acquire user input data; while the processor 601 can be used to call the video quality enhancement application stored in the memory 605 and specifically perform the following operations:

[0169] Obtain multiple video frames ordered chronologically; wherein the multiple video frames include a first frame and a second frame that follows the first frame;

[0170] Feature modulation processing is performed on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame; wherein, the number of the moving object is at least one.

[0171] The first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information are determined by transforming the lookup table, and the first motion feature information and the second motion feature information are fused to obtain fused feature information.

[0172] Multiple video frame combinations, including a first frame and a second frame, are acquired. The fusion feature information corresponding to the moving object in each of the multiple video frame combinations is obtained through the above steps. The first frame and the second frame included in the video frame combination have the same resolution, while the resolutions of the multiple video frame combinations are different.

[0173] The second frame is enhanced based on the multiple fusion feature information corresponding to the moving object to obtain the target second frame.

[0174] In one embodiment, before the processor 601 performs the transformation operation of the lookup table to determine the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information, and before fusing the first motion feature information and the second motion feature information to obtain fused feature information, it further performs the following:

[0175] Obtain a feature information table and a condition table; wherein, the condition table includes multiple condition parameters used during the table lookup process;

[0176] The processor 601 executes the transformation operation through the lookup table to determine the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information, and fuses the first motion feature information and the second motion feature information to obtain fused feature information. Specifically, the following steps are performed:

[0177] According to the condition table, the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information are determined by a lookup table transformation operation in the feature information table.

[0178] The first motion feature information and the second motion feature information are fused to obtain fused feature information.

[0179] In one embodiment, the processor 601 executes the step of determining, based on the condition table, the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information through a lookup table transformation operation in the feature information table, specifically:

[0180] During the lookup process, the first position offset information and the second position offset information are matched in the feature information table by the tetrahedral interpolation algorithm and the condition table respectively, to obtain the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information.

[0181] In one embodiment, during the lookup table process, the processor 601 matches the first position offset information and the second position offset information in the feature information table using a tetrahedral interpolation algorithm and the condition table, respectively, to obtain the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information. Specifically, the process is as follows:

[0182] During the lookup table process, the first position offset information is quantized using a tetrahedral interpolation algorithm to obtain the first low-order data and the first high-order data;

[0183] The first interpolation parameter is obtained from the condition table based on the first low-order data, and the first high-order data is interpolated in the feature information table based on the first interpolation parameter to obtain the first motion feature information associated with the first position offset information.

[0184] The second position offset information is quantized using the tetrahedral interpolation algorithm to obtain the second low-order data and the second high-order data;

[0185] The second interpolation parameter is obtained from the condition table based on the second low-order data. The second high-order data is then interpolated in the feature information table based on the second interpolation parameter to obtain the second motion feature information associated with the second position offset information.

[0186] In one embodiment, the processor 601 performs feature modulation processing on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame, specifically:

[0187] Align the temporal data of the first frame and the second frame, and perform feature modulation processing on the first frame and the second frame to extract the first position offset information of the moving object on the first frame and the second position offset information on the second frame.

[0188] In one embodiment, the processor 601 performs feature modulation processing on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame, specifically:

[0189] The first and second frames are subjected to feature modulation processing using the motion vectors of moving objects to obtain multi-scale convolution kernel offsets.

[0190] Based on the multi-scale convolution kernel offset, feature modulation processing is performed on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame.

[0191] In one embodiment, before the processor 601 performs the feature modulation processing on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame, it further performs the following:

[0192] The background region and region of interest in the first frame are separated to obtain a separated frame.

[0193] Processor 601 performs feature modulation processing on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame. Specifically, it performs the following:

[0194] Based on the separated frames, feature modulation processing is performed on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame.

[0195] In one embodiment, after the processor 601 performs quality enhancement processing on the second frame based on the multiple fused feature information corresponding to the moving object to obtain the target second frame, it specifically performs the following:

[0196] The first frame and the second frame are obtained sequentially from multiple video frames, and the target second frame corresponding to the second frame with quality enhancement is obtained based on the above steps, until the target second frame corresponding to the reverse order of the first frame is obtained.

[0197] The second frames of the multiple targets are fused together to obtain the target video.

[0198] To enhance video frame quality, feature modulation processing is first applied to the first and second frames to extract the positional offset information of moving objects in the two frames. This allows for the utilization of temporal information between multiple video frames during quality enhancement, fully capturing the correlation between them and improving the overall video quality enhancement effect. Furthermore, a lookup table transformation operation is performed based on the two positional offset information to determine the motion feature information corresponding to each of the two video frames, and the two motion feature information are fused to obtain fused feature information. Replacing the neural network algorithm with a low-complexity lookup table algorithm significantly reduces the computational load and the performance requirements of computing devices, and can efficiently and quickly complete computational tasks even in low-latency scenarios, exhibiting high real-time performance. Further, multiple fused feature information at different resolutions is obtained to enhance the second frame, resulting in the target second frame. This means that feature information from different receptive fields is incorporated into the video quality enhancement process, effectively solving the performance limitations when extending single-frame quality enhancement methods to multi-frame quality enhancement scenarios, greatly improving the quality enhancement effect on the second frame and even the overall video quality enhancement effect.

[0199] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0200] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0201] The above-disclosed embodiments are merely preferred embodiments of this specification and should not be construed as limiting the scope of this specification. Therefore, any equivalent variations made in accordance with the claims of this specification shall still fall within the scope of this specification.

Claims

1. A video quality enhancement method, the method comprising: Retrieve multiple video frames sorted chronologically; The plurality of video frames include a first frame and a second frame that follows the first frame; Feature modulation processing is performed on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame; wherein, the number of the moving object is at least one. The first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information are determined by transforming the lookup table, and the first motion feature information and the second motion feature information are fused to obtain fused feature information. Multiple video frame combinations, including a first frame and a second frame, are acquired. The fusion feature information corresponding to the moving object in each of the multiple video frame combinations is obtained through the above steps. The first frame and the second frame included in the video frame combination have the same resolution, while the resolutions of the multiple video frame combinations are different. The second frame is subjected to quality enhancement processing based on the multiple fused feature information corresponding to the moving object to obtain the target second frame; The method further includes, before determining the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information through the transformation operation of the lookup table, and before fusing the first motion feature information and the second motion feature information to obtain fused feature information: Obtain a feature information table and a condition table; wherein, the condition table includes multiple condition parameters used during the table lookup process; The step of determining the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information through a lookup table transformation operation, and fusing the first motion feature information and the second motion feature information to obtain fused feature information, includes: According to the condition table, the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information are determined by a lookup table transformation operation in the feature information table. The first motion feature information and the second motion feature information are fused to obtain fused feature information; The step of determining the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information in the feature information table by means of a lookup table transformation operation, based on the condition table, includes: During the lookup process, the first position offset information and the second position offset information are matched in the feature information table by the tetrahedral interpolation algorithm and the condition table respectively, to obtain the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information.

2. The video quality enhancement method according to claim 1, wherein during the lookup table process, the first position offset information and the second position offset information are matched in the feature information table respectively using a tetrahedral interpolation algorithm and the condition table to obtain first motion feature information associated with the first position offset information and second motion feature information associated with the second position offset information, comprising: During the lookup table process, the first position offset information is quantized using a tetrahedral interpolation algorithm to obtain the first low-order data and the first high-order data; The first interpolation parameter is obtained from the condition table based on the first low-order data, and the first high-order data is interpolated in the feature information table based on the first interpolation parameter to obtain the first motion feature information associated with the first position offset information. The second position offset information is quantized using a tetrahedral interpolation algorithm to obtain the second low-order data and the second high-order data; The second interpolation parameter is obtained from the condition table based on the second low-order data. The second high-order data is then interpolated in the feature information table based on the second interpolation parameter to obtain the second motion feature information associated with the second position offset information.

3. The video quality enhancement method according to claim 1, wherein performing feature modulation processing on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame includes: Align the temporal data of the first frame and the second frame, and perform feature modulation processing on the first frame and the second frame to extract the first position offset information of the moving object on the first frame and the second position offset information on the second frame.

4. The video quality enhancement method according to claim 1, wherein performing feature modulation processing on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame includes: The first and second frames are subjected to feature modulation processing using the motion vectors of moving objects to obtain multi-scale convolution kernel offsets. Based on the multi-scale convolution kernel offset, feature modulation processing is performed on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame.

5. The video quality enhancement method according to claim 1, further comprising, before performing feature modulation processing on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame: The background region and region of interest in the first frame are separated to obtain a separated frame. The step of performing feature modulation processing on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame includes: Based on the separated frames, feature modulation processing is performed on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame.

6. The video quality enhancement method according to claim 1, wherein after performing quality enhancement processing on the second frame based on the multiple fusion feature information corresponding to the moving object to obtain the target second frame, the method further includes: The first frame and the second frame are obtained sequentially from multiple video frames, and the target second frame corresponding to the second frame with quality enhancement is obtained based on the above steps, until the target second frame corresponding to the reverse order of the first frame is obtained. The second frames of the multiple targets are fused together to obtain the target video.

7. A video quality enhancement device, the device comprising: The video frame acquisition module is used to acquire multiple video frames sorted in chronological order. The plurality of video frames include a first frame and a second frame that follows the first frame; A convolutional processing module is used to perform feature modulation processing on the first frame and the second frame to extract the first position offset information of the moving object in the first frame and the second position offset information in the second frame; wherein, the number of the moving object is at least one. The lookup table processing module is used to determine the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information through the transformation operation of the lookup table, and to fuse the first motion feature information and the second motion feature information to obtain fused feature information. A multi-scale fusion module is used to acquire multiple video frame combinations, including a first frame and a second frame, and obtain the fusion feature information of the moving object in the multiple video frame combinations through the above steps; wherein, the first frame and the second frame included in the video frame combination have the same resolution, and the resolutions of the multiple video frame combinations are different. The feature fusion module is used to perform quality enhancement processing on the second frame based on multiple fused feature information corresponding to the moving object, so as to obtain the target second frame; The device also includes The table acquisition module is used to acquire a feature information table and a condition table; wherein, the condition table includes multiple condition parameters used in the table lookup process; The lookup table processing module includes: The table processing unit is configured to determine, based on the condition table, the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information by performing a lookup table transformation operation in the feature information table; An information fusion unit is used to fuse the first motion feature information and the second motion feature information to obtain fused feature information; The table processing unit includes: The lookup table subunit is used to match the first position offset information and the second position offset information in the feature information table respectively through the tetrahedral interpolation algorithm and the condition table during the lookup table process, so as to obtain the first motion feature information associated with the first position offset information and the second motion feature information associated with the second position offset information.

8. A computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method steps of any one of claims 1 to 6.

9. A computer program product storing a plurality of instructions adapted for loading by a processor and executing the method steps of any one of claims 1 to 6.

10. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed the method steps as claimed in any one of claims 1 to 6.

Images