Video super-resolution reconstruction method and device, equipment and storage medium

Abstract

The invention relates to the field of video processing, and discloses a video super-resolution reconstruction method and device, equipment and a storage medium. The method comprises the following steps: acquiring an original video sequence, and preprocessing the original video sequence to obtain a video frame sequence; calculating a relative difference between adjacent frames in the video frame sequence based on the sliding window, and dividing the video frame sequence into a plurality of sub-video sequences according to the relative difference; the absolute difference of adjacent frames in each sub-video sequence is calculated, the motion state of each sub-video sequence is judged according to the absolute difference, and the motion state comprises a high dynamic state and a static state; and processing each sub-video sequence according to the motion state of each sub-video sequence to obtain a plurality of processed sub-video sequences, and reconstructing all the processed sub-video sequences to obtain a high-resolution video sequence. According to the method, computing resources are effectively saved, and time sequence information is better captured to achieve fine video super-resolution reconstruction quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video processing, and in particular to a video super-resolution reconstruction method, device, equipment and storage medium. BACKGROUND

[0002] With the popularity of high-resolution display devices (such as 4K / 8K televisions, AR / VR headsets), the market demand for high-quality video content has increased dramatically. However, due to the cost of high-definition camera equipment, network transmission bandwidth and storage pressure, a large number of existing videos still exist in the form of low resolution (such as 720p, 1080p), resulting in problems such as blurring and jaggedness when played on high-end devices.

[0003] The current mainstream video super-resolution technology mainly relies on deep learning models, but the existing technology has not effectively solved the balance problem of dynamic scene adaptive processing and hardware resource optimization, resulting in resource waste, large calculation delay and large optical flow calculation error.

[0004] Therefore, it is urgent to propose a video super-resolution reconstruction technology with small calculation delay, small error and high quality. SUMMARY

[0005] Therefore, the purpose of the present application is to overcome the deficiencies in the prior art and provide a video super-resolution reconstruction method, device, equipment and storage medium.

[0006] The present application provides the following technical solutions: In a first aspect, the present application provides a video super-resolution reconstruction method, which comprises: obtaining an original video sequence, pre-processing the original video sequence to obtain a video frame sequence; calculating the relative difference between adjacent frames in the video frame sequence based on a sliding window, and dividing the video frame sequence into a plurality of sub-video sequences according to the relative difference; calculating the absolute difference between adjacent frames in each sub-video sequence, judging the motion state of each sub-video sequence according to the absolute difference, and the motion state includes a high dynamic state and a static state; processing each sub-video sequence according to the motion state of each sub-video sequence to obtain a plurality of processed sub-video sequences, and reconstructing all the processed sub-video sequences to obtain a high-resolution video sequence.

[0007] In an optional embodiment, the pre-processing of the original video sequence to obtain a video frame sequence comprises: normalizing the original video sequence to obtain a normalized video sequence; determining a level of the normalized video sequence, the level comprising a pixel level and a feature level; if the level of the normalized video sequence is the pixel level, determining the normalized video sequence as the video frame sequence; if the level of the normalized video sequence is the feature level, inputting the normalized video sequence into a preset shallow neural network, performing feature extraction on the normalized video sequence by the preset shallow neural network, and obtaining the video frame sequence.

[0008] In an optional implementation, the calculating the relative difference between adjacent frames in the video frame sequence based on a sliding window comprises: The relative difference between adjacent frames in the video frame sequence is calculated according to the following formula:

[0009] In the formula, the relative difference is the relative difference, the i-th frame, the i-th frame, the i-th frame, the i-th frame, is a distance function.

[0010] In an optional implementation, the dividing the video frame sequence into a plurality of sub-video sequences according to the relative difference comprises: determining whether the relative difference is greater than a preset relative difference threshold; if the relative difference is greater than the preset relative difference threshold, dividing a new sub-video sequence starting from the i-th frame; if the relative difference is not greater than the preset relative difference threshold, dividing a new sub-video sequence starting from the i-th frame; if no new sub-video sequence is divided for N consecutive frames, dividing a new sub-video sequence starting from the N+1-th frame, where N is set according to the size of a hardware display memory.

[0011] In an optional implementation, the calculating the absolute difference between adjacent frames in each of the sub-video sequences comprises: The absolute difference between adjacent frames in the video frame sequence is calculated according to the following formula:

[0012] In the formula, the absolute difference is the absolute difference.

[0013] In an optional implementation, the determining the motion state of each of the sub-video sequences according to the absolute difference comprises: determining whether the absolute difference is greater than a preset absolute difference threshold; if the absolute difference is greater than or equal to the preset absolute difference threshold, the motion state of the corresponding sub-video sequence is determined as the high dynamic state; if the absolute difference is less than the preset absolute difference threshold, the motion state of the corresponding sub-video sequence is determined as the static state.

[0014] In an optional implementation, the processing of each of the sub-video sequences according to the motion state of each of the sub-video sequences to obtain a plurality of processed sub-video sequences comprises: if the motion state of the sub-video sequence is the high dynamic state, a large-scale feature capture manner is used to extract feature information of the sub-video sequence, a large-scale optical flow network is used to model fine-grained motion information and feature interaction, and a super-resolution network is used to perform up-sampling processing on the sub-video sequence to obtain a plurality of processed high dynamic sub-video sequences; if the motion state of the sub-video sequence is the static state, a small-scale feature capture manner is used to extract feature information of the sub-video sequence, a small-scale optical flow network is used to model coarse motion information and feature interaction, and the super-resolution network is used to perform up-sampling processing on the sub-video sequence to obtain a plurality of processed static sub-video sequences; each of the processed high dynamic sub-video sequences and each of the processed static sub-video sequences are merged into a plurality of processed sub-video sequences.

[0015] In a second aspect, the embodiments of the present disclosure provide a video super-resolution reconstruction device, and the device comprises: a preprocessing module configured to obtain an original video sequence, and perform preprocessing on the original video sequence to obtain a video frame sequence; a first calculation module configured to calculate a relative difference of adjacent frames in the video frame sequence based on a sliding window, and divide the video frame sequence into a plurality of sub-video sequences according to the relative difference; a second calculation module configured to calculate an absolute difference of adjacent frames in each of the sub-video sequences, and determine a motion state of each of the sub-video sequences according to the absolute difference, wherein the motion state comprises a high dynamic state and a static state; a processing module configured to process each of the sub-video sequences according to the motion state of each of the sub-video sequences to obtain a plurality of processed sub-video sequences, and reconstruct all the processed sub-video sequences to obtain a high-resolution video sequence.

[0016] In a third aspect, the embodiments of the present disclosure provide a computer device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the video super-resolution reconstruction method in the first aspect when executing the computer program.

[0017] In a fourth aspect, the present disclosure provides a computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps of the video super-resolution reconstruction method in the first aspect.

[0018] Advantages of the present application: The video super-resolution reconstruction method provided by the embodiments of the present application can calculate the absolute difference and the relative difference of a video sequence, judge the motion state of a current video frame, divide a video sub-sequence through an adaptive algorithm, and then call a neural network of different scales to save computing resources and better capture time sequence information to achieve fine video super-resolution reconstruction quality.

[0019] In order to make the above objectives, features and advantages of the present application more apparent and easy to understand, the following will describe preferred embodiments in detail, and the accompanying drawings will be described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, and it should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor. In each drawing, similar components are marked with similar reference numerals.

[0021] Figure 1 A flow chart of a video super-resolution reconstruction method provided by the embodiments of the present application is shown; Figure 2 A structural schematic diagram of a video super-resolution reconstruction device provided by the embodiments of the present application is shown; Figure 3 A structural schematic diagram of a computer device provided by the embodiments of the present application is shown. DETAILED DESCRIPTION

[0022] The embodiments of the present application will be described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation on the present application.

[0023] It should be noted that the terms "first", "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or a specific number of the technical features indicated. Therefore, the features defined as "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the template herein is only for the purpose of describing specific embodiments and is not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0025] Embodiment 1 The current mainstream video super-resolution technology mainly relies on deep learning models, but the existing technology has not effectively solved the balance problem of dynamic scene adaptive processing and hardware resource optimization: 1) high motion area needs strong temporal modeling, while static background can simplify the calculation, but the existing scheme (such as BASICVSR++, VRT, etc.) uses a unified processing flow, causing resource waste; 2) most existing methods rely on optical flow alignment / temporal attention / time-related correlation, and use a fixed grouping strategy. But in real business scenarios, when there is a scene switch or a scene with intense motion, this fixed grouping strategy is easily ineffective, and the optical flow calculation error causes the reconstruction quality to be damaged.

[0026] Based on the problems existing in the prior art, such as Figure 1 As shown in FIG. 1, a flowchart of a video super-resolution reconstruction method according to an embodiment of the present application is shown, and the video super-resolution reconstruction method provided by the present application includes the following steps: Step S110, obtaining an original video sequence, pre-processing the original video sequence to obtain a video frame sequence.

[0027] In this embodiment, first, an original video sequence is obtained , then the original video sequence is normalized and pre-processed, for example, the normalized method can be one of the following methods: (1) directly divide the pixel value (range [0, 255]) of the video frame in the original video sequence by 255, and normalize it to [0, 1]; (2) After dividing the pixel values ​​of the video frames in the original video sequence by 255 to [0, 1], each channel (RGB three channels: red channel, green channel, blue channel) is subtracted from the preset mean (such as [0.485, 0.456, 0.406]), and then divided by the preset standard deviation (such as [0.229, 0.224, 0.225]) to standardize the pixel values ​​of different channels to a more stable numerical range.

[0028] Normalization processing unifies the pixel value distribution of the original video frame, eliminates the interference of brightness and color differences in different scenes on the model, makes subsequent feature extraction and difference calculation more stable, and improves the efficiency and robustness of model learning.

[0029] Furthermore, the level of the normalized video sequence is determined, including the pixel level and the feature level. If the level of the normalized video sequence is the pixel level, the normalized video sequence is directly determined as a video frame sequence. If the level of the normalized video sequence is the feature level, the normalized video sequence is input into a preset shallow neural network, and features are extracted from the normalized video sequence by the preset shallow neural network to obtain a video frame sequence.

[0030] The principle behind differentiated processing at the pixel and feature levels is as follows: Pixel-level sequences bypass feature extraction, retaining spatial correlations between pixels. This saves hardware resources and reduces computational overhead, making them suitable for static scenes. Feature-level sequences are processed through a pre-set shallow neural network to extract visual features. These features, compared to raw pixels, better reflect inter-frame motion differences and scene changes, enhancing feature representation and making them suitable for scenes with high motion. The processing logic at both levels essentially pre-selects between "lightweight pixel representation" and "enhanced feature representation" based on the motion state of the video scene (determined through subsequent difference calculations). This balances computational efficiency with feature capture, achieving the dual goals of "resource conservation" and "fine reconstruction."

[0031] It should be noted that, in this embodiment, the preset shallow neural network can be a simple 3x3 convolutional layer with 3 input channels, 64 output channels, and a step size of 1 or 2. The specific neural network can be determined according to actual conditions, and this embodiment does not limit this.

[0032] Step S120 : calculating relative differences between adjacent frames in the video frame sequence based on a sliding window, and dividing the video frame sequence into a plurality of sub-video sequences according to the relative differences.

[0033] Understandably, the present application introduces the role of relative difference is to make segmentation at the whole video level, because different video sequences, the overall motion situation / scene change degree is different, so when performing adaptive grouping strategy, relative difference needs to be calculated, and absolute difference cannot be calculated. The relative difference can be calculated at the pixel level or the feature level, and the relative difference of adjacent frames in the video frame sequence is calculated in a sliding window manner. The calculation formula of the relative difference is:

[0034] In the formula, is the relative difference, is the first frame, is the first frame, is the distance function, which can be L1 distance, L2 distance, cosine similarity, KL divergence after normalization, etc., without limitation.

[0035] After calculating the relative difference, it is judged whether the relative difference is greater than a preset relative difference threshold RT1 (for example, 200%). If the relative difference is greater than the preset relative difference threshold, a new sub-video sequence is divided with as the starting point. Understandably, for a video frame sequence, relative difference = 100% represents that the difference between the current adjacent two frames is equal to the average difference of the overall video frames; relative difference < 100% represents that the difference between the current adjacent two frames is less than the overall video, and the motion information is relatively less, which is static; and relative difference > 100% represents that the difference between the current adjacent two frames is greater than the overall video, and the motion information is relatively large, which has violent motion or scene switching.

[0036] The adaptive grouping based on the relative difference solves the problem that the “fixed grouping strategy fails in scene switching or violent motion” in the prior art: by calculating the ratio of the difference between adjacent frames and the historical average difference through a sliding window, scene switching or violent motion can be dynamically identified, ensuring that the time correlation within the sub-video sequence is strong, reducing the optical flow calculation error, and improving the reconstruction quality.

[0037] In actual scenarios, there may be video clips with no obvious scene switching for a long time and smooth motion (such as static monitoring pictures and fixed-lens landscape videos). At this time, if only the relative difference is relied on for judgment, the number of frames in the sub-video sequence may continue to accumulate, and the length may increase indefinitely. However, because of limited resources, it is impossible to process a sub-video sequence that is too long, and the length of each sub-video sequence needs to have an upper limit. Therefore, if N consecutive frames are not divided into new sub-video sequences, a new sub-video sequence is divided with the N+1th frame as the starting point, which is a supplement to the adaptive grouping strategy, used to hard constrain the maximum length of the sub-video sequence, avoid the hardware resource overrun problem caused by too many frames, and ensure the stability of the processing process.

[0038] It should be noted that N is set according to the size of the hardware GPU video memory. The core logic is: N must be less than or equal to the maximum number of consecutive frames that a single GPU can stably carry in super-resolution processing. The larger the video memory (such as 24GB and above), the more consecutive frames can be loaded and processed at a time, and N can be set to a larger value (for example, 30 frames), allowing longer sub-video sequences to exist, reducing unnecessary group truncation, and thus reducing the computational overhead caused by group switching; the smaller the video memory (such as 8GB and below), the more consecutive frames that can be carried at a time are limited, and N needs to be set to a smaller value (for example, 10 frames). By controlling the length of the sub-video sequence through more frequent forced truncation, it is ensured that the frame data processed in each batch does not exceed the upper limit of the video memory, further optimizing the efficiency of hardware resource utilization.

[0039] Step S130 , calculating the absolute difference between adjacent frames in each sub-video sequence, and determining the motion state of each sub-video sequence according to the absolute difference, where the motion state includes a high-dynamic state and a static state.

[0040] It is understandable that the purpose of introducing absolute difference in this application is to judge the motion of video frames or scene switching in each sub-video sequence. The absolute difference between adjacent frames in the video frame sequence is calculated. The calculation formula of absolute difference is:

[0041] Where, is the absolute difference, For the frames, No. frames, The distance function may be L1 distance, L2 distance, cosine similarity, normalized KL divergence, etc., without limitation.

[0042] After calculating the absolute difference, a determination is made as to whether the absolute difference is greater than a preset absolute difference threshold RT2 (e.g., 0.2). If the absolute difference is greater than or equal to the preset absolute difference threshold, then significant motion changes are present within the sub-video sequence, and the motion state of the sub-video sequence is determined to be highly dynamic. If the absolute difference is less than the preset absolute difference threshold, then minimal motion changes are present within the sub-video sequence, and the motion state of the corresponding sub-video sequence is determined to be static.

[0043] The motion state judgment based on absolute difference realizes the quantitative distinction of the degree of motion within the sub-video sequence. By directly calculating the difference between adjacent frames and comparing it with the threshold, it can accurately identify high-dynamic states (violent motion) and static states (smooth changes), providing a clear basis for subsequent differentiated processing, and avoiding the problems of "resource waste caused by unified processing flow (redundant calculation of static scenes) or insufficient quality (insufficient modeling of dynamic scenes)" in existing technologies.

[0044] Step S140, according to the motion state of each sub-video sequence, processing each sub-video sequence to obtain a plurality of processed sub-video sequences, and reconstructing all the processed sub-video sequences to obtain a high-resolution video sequence.

[0045] Understandably, this application matches different precision feature capture and timing modeling strategies according to the motion state (high dynamic / static) of the sub-video sequence, optimizes the consumption of computing resources while ensuring the quality of reconstruction. Specifically, high dynamic scenes require more detailed feature and motion modeling to avoid reconstruction distortion, and static scenes can be simplified to save resources. Both share the core structure of the super-resolution network, only differ in feature scale and timing modeling accuracy: (1) When the sub-video sequence is determined to be in a high dynamic state, it means that there is a dramatic motion (such as fast-moving objects, fast lens push-pull and pan) or a significant local dynamic change in the sequence, which requires detailed processing through "large-scale feature capture + large-scale optical flow network": a. Large-scale feature capture is used to extract higher resolution feature information, which specifically means that the original features (pixel-level or feature-level frames) are down-sampled by 2 times. This processing retains more detailed information (such as object edges, texture changes), providing richer input for subsequent motion modeling; b. Large-scale optical flow network is based on large-scale features to model fine-grained motion information and feature interaction. Fine-grained motion information: calculate the inter-frame pixel-level displacement (such as optical flow vector) through high-resolution features, accurately align the dynamic areas of adjacent frames (such as the position changes of moving objects), avoid ghosting, blurring and other problems caused by alignment errors; Feature interaction: strengthen the inter-frame feature fusion of dynamic areas (such as the feature association of the same object in the previous and subsequent frames), ensure the continuity of the motion process (such as the smoothness of the object motion trajectory); c. The super-resolution network performs up-sampling on the features that have undergone fine timing modeling, and the process is consistent with the general super-resolution network (including feature extraction, temporal consistency modeling, sub-pixel convolution / Pixel shuffle up-sampling, fusion reconstruction, etc.). Finally, it outputs a plurality of processed high dynamic sub-video sequences that meet the 4K resolution. Since the input features retain more dynamic details, the clarity of the high dynamic scene can be restored after up-sampling.

[0046] (2) When the sub-video sequence is determined to be in a static state, it means that the sequence is mainly composed of static scenes (such as stationary backgrounds under a fixed camera, slight changes in lighting), which can be simplified through "small-scale feature capture + small-scale optical flow network": a. Low-resolution feature information is extracted using a small-scale feature capture method, which specifically means that the original features are down-sampled by 4 or 8 times. This processing compresses the spatial resolution of the features (reduces the number of pixels / feature points), but in a static scene, the dynamic changes are weak (such as the background hardly changes), and the low-resolution features are sufficient to represent the inter-frame differences, and can significantly reduce the memory usage and computational complexity of subsequent calculations; b. The small-scale optical flow network is based on small-scale features to model rough motion information and feature interaction. Rough motion information: for slight motion in a static scene (such as slight camera shake, slow light changes), the approximate inter-frame displacement is calculated based on low-resolution features, without the need for pixel-level accuracy to meet the alignment requirements; feature interaction: weaken the feature fusion strength of dynamic regions, only keep the inter-frame consistency of static backgrounds (such as the features of static regions such as walls and floors), and reduce unnecessary computational overhead; c. The super-resolution network is based on simplified features for up-sampling. Although the input feature resolution is low, the details of the static scene change little, and the low-resolution features can support high-resolution reconstruction (such as static buildings, landscapes, etc.). The up-sampling process shares the super-resolution network structure with high-dynamic scenes, and finally outputs multiple processed static sub-video sequences with 4K resolution, with significantly lower computational time than high-dynamic scenes.

[0047] Preferably, after the high-dynamic and static sub-video sequences are processed differently, they need to be merged in the original video time sequence. When merging, the frame sequence continuity of each sub-video sequence is maintained (such as the last frame of sub-video sequence A and the first frame of sub-video sequence B are adjacent in time), to avoid time sequence disorder. Since the output of the super-resolution network of both types of sub-video sequences is 4K resolution, and both need to go through a unified post-processing process (such as denoising and sharpening) to further improve the visual quality, the merged video sequence can form a complete high-resolution video sequence, taking into account the clarity of dynamic scenes and the efficient processing of static scenes.

[0048] The video super-resolution reconstruction method provided by the embodiments of the present application can judge the motion state of the current video frame by calculating the absolute difference and the relative difference of the video sequence, divide the video sub-sequence by an adaptive algorithm, and then call different scale neural networks to save computing resources and better capture time sequence information to achieve fine video super-resolution reconstruction quality.

[0049] Embodiment 2 As shown in Figure 2 Fig. 1 is a structural schematic diagram of a video super-resolution reconstruction device 200 in an embodiment of the present application. The device includes: A preprocessing module 210 is configured to obtain an original video sequence, and pre-process the original video sequence to obtain a video frame sequence. The first calculation module 220 is configured to calculate relative differences between adjacent frames in the video frame sequence based on a sliding window, and divide the video frame sequence into a plurality of sub-video sequences according to the relative differences. The second calculation module 230 is configured to calculate absolute differences between adjacent frames in each of the sub-video sequences, and determine a motion state of each of the sub-video sequences according to the absolute differences, wherein the motion state includes a high dynamic state and a static state. The processing module 240 is configured to process each of the sub-video sequences according to the motion state of each of the sub-video sequences, obtain a plurality of processed sub-video sequences, and reconstruct all the processed sub-video sequences to obtain a high-resolution video sequence.

[0050] The video super-resolution reconstruction device provided by the embodiment can implement each process of the video super-resolution reconstruction method corresponding to the embodiment 1, and achieve the same technical effects. To avoid repetition, details are not described herein.

[0051] The video super-resolution reconstruction device provided by the embodiment can calculate absolute differences and relative differences of a video sequence, determine a motion state of a current video frame, divide video sub-sequences through an adaptive algorithm, and then call a neural network of different scales, so as to save computing resources, better capture time sequence information, and achieve fine video super-resolution reconstruction quality.

[0052] Embodiment 3 The embodiment of the application further provides a computer device. For details, please refer to Figure 3 , Figure 3 The computer device is a basic structure block diagram of the embodiment.

[0053] The computer device 3 includes a memory 31, a processor 32, and a network interface 33 which are connected to each other through a system bus. It should be noted that only the computer device 3 with the memory 31, the processor 32, and the network interface 33 is shown in the figure, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented. Among them, those skilled in the art can understand that the computer device herein is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0054] The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server, or the like. The computer device can interact with a user through a keyboard, a mouse, a remote controller, a touchpad, a voice control device, or the like.

[0055] The memory 31 can include at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or D slot compatible memory, or the like), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, or the like. In some embodiments, the memory 31 can be an internal storage unit of the computer device 3, such as a hard disk or a memory of the computer device 3. In other embodiments, the memory 31 can also be an external storage device of the computer device 3, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or the like. Of course, the memory 31 can include both an internal storage unit and an external storage device of the computer device 3. In this embodiment, the memory 31 is generally used to store an operating system and various application software installed in the computer device 3, such as computer readable instructions of the slot compatibility test method, or the like. In addition, the memory 31 can also be used to temporarily store various data that has been output or will be output.

[0056] The processor 32 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other video super-resolution reconstruction chips in some embodiments. The processor 32 is generally used to control the overall operation of the computer device 3. In this embodiment, the processor 32 is used to run computer readable instructions or process data stored in the memory 31, such as computer readable instructions of the slot compatibility test method.

[0057] The network interface 33 can include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the computer device 3 and other electronic devices.

[0058] The computer device provided in this embodiment can execute the video super-resolution reconstruction method described above. The video super-resolution reconstruction method can be the video super-resolution reconstruction method of each of the above-described embodiments.

[0059] Embodiment 4 The embodiment also provides a computer readable storage medium, the computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement steps of the video super-resolution reconstruction method in the embodiment.

[0060] In the embodiment, the computer readable storage medium includes a flash memory, a hard disk, a multimedia card, a card memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a programmable read only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the computer readable storage medium can be an internal storage unit of a computer device, for example, a hard disk or a memory of the computer device. In other embodiments, the computer readable storage medium can also be an external storage device of the computer device, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Of course, the computer readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In the embodiment, the computer readable storage medium is usually used to store an operating system and various application software installed on the computer device. In addition, the computer readable storage medium can also be used to temporarily store various data that have been output or will be output.

[0061] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can also be implemented by other means. The apparatus embodiments described above are only illustrative. For example, the flowchart and block diagram in the drawings show the possible implementation architecture, function and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code, which includes one or more executable instructions for implementing the specified logic function. It should also be noted that in alternative implementation, the functions noted in the block can occur in different order from that shown in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for executing the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0062] In addition, each functional module or unit in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0063] If the functions are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the prior art that essentially contributes or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium can be a non-volatile storage medium or a volatile storage medium, for example, the storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various storage medium capable of storing program codes.

[0064] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A video super-resolution reconstruction method, characterized in that: The method comprises: Obtaining an original video sequence, and preprocessing the original video sequence to obtain a video frame sequence; Calculating relative differences between adjacent frames in the video frame sequence based on a sliding window, and dividing the video frame sequence into a plurality of sub-video sequences according to the relative differences; calculating an absolute difference between adjacent frames in each of the sub-video sequences, and determining a motion state of each of the sub-video sequences based on the absolute difference, wherein the motion state includes a highly dynamic state and a static state; Each of the sub-video sequences is processed according to its motion state to obtain a plurality of processed sub-video sequences, and all of the processed sub-video sequences are reconstructed to obtain a high-resolution video sequence.

2. The video super-resolution reconstruction method according to claim 1, characterized in that The preprocessing of the original video sequence to obtain a video frame sequence includes: Normalizing the original video sequence to obtain a normalized video sequence; Determining a level of the normalized video sequence, the level including a pixel level and a feature level; If the level of the normalized video sequence is the pixel level, determining the normalized video sequence as the video frame sequence; If the level of the normalized video sequence is the feature level, the normalized video sequence is input into a preset shallow neural network, and features of the normalized video sequence are extracted by the preset shallow neural network to obtain the video frame sequence.

3. The video super-resolution reconstruction method according to claim 1, characterized in that The calculating the relative difference between adjacent frames in the video frame sequence based on the sliding window includes: The relative difference between adjacent frames in the video frame sequence is calculated using the following formula: Where, For the relative difference, For the frames, For the frames, is the distance function.

4. The video super-resolution reconstruction method according to claim 3, characterized in that The dividing the video frame sequence into a plurality of sub-video sequences according to the relative differences comprises: Determining whether the relative difference is greater than a preset relative difference threshold; If the relative difference is greater than the preset relative difference threshold, Divide a new sub-video sequence for the starting point; If N consecutive frames are not divided into a new sub-video sequence, a new sub-video sequence is divided starting from the N+1th frame, where N is set according to the hardware video memory size.

5. The video super-resolution reconstruction method according to claim 3, characterized in that: The calculating the absolute difference between adjacent frames in each sub-video sequence includes: The absolute difference between adjacent frames in the video frame sequence is calculated using the following formula: Where, is the absolute difference.

6. The video super-resolution reconstruction method according to claim 1, characterized in that The determining the motion state of each of the sub-video sequences according to the absolute difference includes: Determining whether the absolute difference is greater than a preset absolute difference threshold; If the absolute difference is greater than or equal to the preset absolute difference threshold, determining the motion state of the corresponding sub-video sequence as the high dynamic state; If the absolute difference is less than the preset absolute difference threshold, the motion state of the corresponding sub-video sequence is determined to be the static state.

7. The video super-resolution reconstruction method according to claim 1, characterized in that: The processing of each sub-video sequence according to the motion state of each sub-video sequence to obtain a plurality of processed sub-video sequences includes: If the motion state of the sub-video sequence is a high-dynamic state, a large-scale feature capture method is used to extract the feature information of the sub-video sequence, fine-grained motion information and feature interaction are modeled through a large-scale optical flow network, and the sub-video sequence is upsampled through a super-resolution network to obtain multiple processed high-dynamic sub-video sequences; If the motion state of the sub-video sequence is a static state, a small-scale feature capture method is used to extract feature information of the sub-video sequence, rough motion information and feature interaction are modeled through a small-scale optical flow network, and the sub-video sequence is upsampled through the super-resolution network to obtain multiple processed static sub-video sequences; The processed high-dynamic sub-video sequences and the processed static sub-video sequences are merged into a plurality of processed sub-video sequences.

8. A video super-resolution reconstruction device, characterized in that: The device comprises: A preprocessing module is used to obtain an original video sequence and preprocess the original video sequence to obtain a video frame sequence; a first calculation module, configured to calculate relative differences between adjacent frames in the video frame sequence based on a sliding window, and divide the video frame sequence into a plurality of sub-video sequences according to the relative differences; a second calculation module, configured to calculate an absolute difference between adjacent frames in each sub-video sequence, and determine a motion state of each sub-video sequence according to the absolute difference, wherein the motion state includes a high-dynamic state and a static state; The processing module is used to process each sub-video sequence according to the motion state of each sub-video sequence to obtain multiple processed sub-video sequences, and reconstruct all the processed sub-video sequences to obtain a high-resolution video sequence.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the video super-resolution reconstruction method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video super-resolution reconstruction method according to any one of claims 1 to 7 are implemented.

Images