Video resolution improving method based on AI visual engine and related equipment

Abstract

The invention relates to the technical field of computer vision and deep learning, in particular to a video resolution improving method based on an AI vision engine and related equipment. The method comprises the following steps: receiving an input low-resolution video, and carrying out continuous frame group division on the low-resolution video; multi-scale feature extraction is carried out on each frame group to obtain an initial feature image corresponding to each frame group, and multi-scale features comprise shallow edge textures and deep semantic contents; performing feature alignment and fusion on the preset frame number of initial feature images to obtain a target spatial-temporal feature image frame; performing up-sampling processing on the target spatial-temporal feature image frame to obtain a target high-resolution image frame; and integrating all the target high-resolution image frames, and outputting a target high-resolution video stream. According to the method, video time sequence information can be efficiently and accurately utilized, vivid details can be generated while the resolution is improved, and the calculation efficiency is considered.

Description

Technical Field

[0001] This application relates to the fields of computer vision and deep learning technology, and in particular to a method and related equipment for improving video resolution based on an AI vision engine. Background Technology

[0002] Video super-resolution (VSR), which involves reconstructing high-resolution (HR) video from low-resolution (LR) video sequences, is a core topic in computer vision. Traditional methods include interpolation-based (such as bicubic interpolation) and reconstruction-based methods, but these methods often lead to image blurring, loss of detail, and failure to generate satisfactory high-frequency details. Currently, with the development of deep learning, video super-resolution techniques based on convolutional neural networks (CNNs) have made significant progress. These methods typically utilize neural networks to learn an end-to-end mapping function from LR to HR, and their performance far surpasses that of traditional methods.

[0003] However, despite the achievements of existing deep learning-based VSR methods, they still suffer from one or more of the following inherent drawbacks: 1) Insufficient utilization of temporal information: Many methods simply stack adjacent frames or use optical flow for alignment. Optical flow estimation itself is computationally complex and error-prone. When dealing with fast-moving or occluded areas, it can lead to the accumulation of errors in subsequent reconstruction, resulting in artifacts (such as ghosting and trailing).

[0004] 2) The contradiction between detail restoration and smoothing: When generating details, the network is prone to introducing unnatural noise or texture; while excessive pursuit of smoothing will make the image lose texture, appear too plasticky, and fail to restore "realistic" details.

[0005] 3) High computational complexity: Existing high-performance models typically have a huge number of parameters and high computational overhead, making it difficult to achieve real-time processing on ordinary consumer-grade GPUs or mobile devices, thus limiting their practical applications.

[0006] 4) Weak generalization ability: Models trained on specific datasets show significant performance degradation and insufficient stability when dealing with complex and varied video content in the real world (such as movies, animations, game recordings, and surveillance videos).

[0007] Therefore, there is an urgent need in this field for a new solution that can efficiently and accurately utilize video temporal information, generate realistic details while improving resolution, and take into account computational efficiency. Summary of the Invention

[0008] The purpose of this application is to overcome the above-mentioned technical problems and provide a method and related device for improving video resolution based on an AI vision engine. This method can efficiently and accurately utilize video temporal information, generate realistic details while improving resolution, and take into account computational efficiency.

[0009] Firstly, one embodiment of this application discloses a method for improving video resolution based on an AI visual engine, which employs the following scheme: A method for video resolution enhancement based on an AI vision engine includes: receiving an input low-resolution video and dividing the low-resolution video into consecutive frame groups; extracting multi-scale features for each frame group to obtain an initial feature image corresponding to each frame group, wherein the multi-scale features include shallow edge textures and deep semantic content; aligning and fusing the initial feature images of a preset number of frames to obtain target spatiotemporal feature image frames; upsampling the target spatiotemporal feature image frames to obtain target high-resolution image frames; and integrating all the target high-resolution image frames to output a target high-resolution video stream, wherein the resolution of the high-resolution video stream is higher than that of the low-resolution video.

[0010] By adopting the above technical solution, the input low-resolution video is received and divided into continuous frame groups, which facilitates subsequent targeted processing of the video. Multi-scale feature extraction is performed on each frame group, extracting shallow edge texture and deep semantic content as initial feature images. This avoids the problems of image blurring and detail loss caused by traditional methods and provides richer and more accurate features for subsequent reconstruction. The initial feature images of a preset number of frames are feature-aligned and fused to obtain target spatiotemporal feature image frames. This efficiently and accurately utilizes the temporal information of the video, overcomes the shortcomings of existing deep learning-based VSR methods in utilizing temporal information, and reduces the generation of artifacts. The target spatiotemporal feature image frames are upsampled to obtain target high-resolution image frames. While improving the resolution, realistic details are generated, which solves the contradiction between detail restoration and smoothing. All target high-resolution image frames are integrated to output a target high-resolution video stream. The resolution of the high-resolution video stream is higher than that of the low-resolution video, which effectively improves the video resolution. At the same time, this method can take into account computational efficiency and can be processed in real time on ordinary consumer-grade GPUs or mobile devices. It also has strong generalization ability and can handle complex and varied video content in the real world.

[0011] Optionally, the step of performing feature alignment and fusion on the initial feature images of a preset number of frames to obtain a target spatiotemporal feature image frame includes: obtaining the predicted offset between adjacent image frames and the current reference image frame in the preset number of frames; performing feature alignment on the initial feature images corresponding to the preset number of frames according to the predicted offset; matching different regions in the aligned initial feature images with different preset weights to perform feature fusion to obtain the target spatiotemporal feature image frame.

[0012] By adopting the above technical solution, the predicted offset between adjacent image frames and the current reference image frame within a preset number of frames can be obtained, which can accurately determine the positional relationship between adjacent image frames and the reference image frame, providing an accurate basis for subsequent feature alignment and avoiding feature matching errors caused by inaccurate positions. By aligning the initial feature images corresponding to the preset number of frames according to the predicted offset, the features of different image frames can be accurately matched in space, effectively utilizing the temporal information of the video and reducing artifacts caused by motion or occlusion. By matching different regions in the aligned initial feature images with different preset weights for feature fusion, differentiated processing can be performed on the characteristics of different regions of the image, improving resolution while better restoring image details, so that the generated target spatiotemporal feature image frames contain richer and more accurate information.

[0013] Optionally, matching different regions in the aligned initial feature image with different preset weights includes: identifying regions in each initial feature image and determining whether there is a first region that is occluded or motion-blurred, and a second region that is rich in information; when the first region exists, matching it with a first preset weight; when the second region exists, matching it with a second preset weight, wherein the second preset weight is greater than the first preset weight.

[0014] By adopting the above technical solution, the regions in each initial feature image are identified and it is determined whether there are occluded or motion-blurred regions as well as information-rich regions, which can accurately distinguish regions with different features. When there are occluded or motion-blurred regions, a first preset weight is matched to avoid the erroneous features of the region from having an adverse effect on subsequent reconstruction and to reduce the generation of artifacts. When there are information-rich regions, a larger second preset weight is matched to highlight the features of the region, which helps to more accurately restore realistic details and improve the quality of the video image.

[0015] Optionally, the upsampling process of the target spatiotemporal feature image frame to obtain the target high-resolution image frame includes: upsampling the target spatiotemporal feature image frame through a subpixel convolutional layer to reconstruct the high-resolution image frame; determining whether the resolution difference between the high-resolution image frame and the preset high-resolution frame meets the preset difference; if so, the high-resolution image frame is used as the target high-resolution image frame.

[0016] By adopting the above technical solution, the target spatiotemporal feature image frame is upsampled through a subpixel convolutional layer and reconstructed into a high-resolution image frame, which can effectively improve the resolution of the image. Determining whether the resolution difference between the high-resolution image frame and the preset high-resolution frame meets the preset difference ensures that the reconstructed high-resolution image frame meets the expected resolution standard. Using the high-resolution image frame that meets the preset resolution difference requirement as the target high-resolution image frame ensures that the final output target high-resolution image frame meets the actual application requirements.

[0017] Optionally, the step of integrating all the target high-resolution image frames and outputting a target high-resolution video stream includes: performing effect processing on each of the target high-resolution image frames, the effect processing including color correction and sharpening; and arranging and integrating all the processed target high-resolution image frames in chronological order to output a target high-resolution video stream.

[0018] By employing the above technical solutions, color correction and sharpening effects are applied to each target high-resolution image frame, which optimizes the color performance of the image, making the colors more accurate, vibrant, and natural. At the same time, it enhances the edges and details of the image, making the image clearer and sharper, thus improving the visual effect. Arranging and integrating all the processed target high-resolution image frames in chronological order to output a target high-resolution video stream ensures the continuity and smoothness of the video, allowing viewers to watch the video content normally. Furthermore, the output video stream has a higher resolution than the input low-resolution video, achieving an effective improvement in video resolution.

[0019] Optionally, before performing multi-scale feature extraction on each frame group, the method further includes: normalizing the pixel values ​​of each frame group; obtaining the predicted offset between adjacent image frames and the current reference image frame in the preset number of frames includes: obtaining the predicted offset between the current frame and each of the two frames before and after it based on the offset prediction network.

[0020] By adopting the above technical solution, the pixel values ​​of each frame group are normalized, which unifies the pixel values ​​to a specific range, reduces the impact of pixel value differences between different frame groups, and makes subsequent multi-scale feature extraction more stable and accurate. According to the offset prediction network, the predicted offset between the current frame and each of the two frames before and after can make fuller use of the temporal information of adjacent frames, improve the accuracy of feature alignment, reduce artifacts caused by fast motion or improper handling of occluded areas, and thus perform feature fusion more accurately, thereby improving the effect of video resolution enhancement.

[0021] Secondly, one embodiment of this application discloses a system for video resolution enhancement based on an AI vision engine, which adopts the following solution: A system for video resolution enhancement based on an AI vision engine, used to execute the method described above, includes: a frame group segmentation module for receiving input low-resolution video and segmenting the low-resolution video into consecutive frame groups; a multi-scale extraction module for performing multi-scale feature extraction on each frame group to obtain an initial feature image corresponding to each frame group, wherein the multi-scale features include shallow edge texture and deep semantic content; an alignment and fusion module for aligning and fusing the initial feature images of a preset number of frames to obtain target spatiotemporal feature image frames; a processing module for upsampling the target spatiotemporal feature image frames to obtain target high-resolution image frames; and an integration module for integrating all the target high-resolution image frames to output a target high-resolution video stream, wherein the resolution of the high-resolution video stream is higher than that of the low-resolution video.

[0022] By adopting the above technical solution, the input low-resolution video is received and divided into continuous frame groups, which can segment the video into units that are easy to process, laying the foundation for subsequent accurate analysis and processing. Multi-scale feature extraction is performed on each frame group to obtain shallow edge texture and deep semantic content, which can avoid the image blurring and detail loss problems caused by traditional methods and capture video features more comprehensively. The initial feature images of a preset number of frames are feature aligned and fused to obtain target spatiotemporal feature image frames, which can efficiently and accurately utilize video temporal information, solve the problem of insufficient utilization of temporal information in existing methods, and reduce the generation of artifacts. The target spatiotemporal feature image frames are upsampled to obtain target high-resolution image frames, which can generate realistic details while improving resolution and alleviate the contradiction between detail recovery and smoothing. The target high-resolution image frames are integrated and output as a target high-resolution video stream, which can balance computational efficiency, solve the problem of high computational complexity of existing high-performance models, and improve the generalization ability of the model, enabling it to stably process complex and variable video content in the real world.

[0023] Optionally, the alignment and fusion module includes: a prediction offset acquisition unit, used to acquire the prediction offset between adjacent image frames and the current reference image frame in the preset number of frames; a feature alignment unit, used to perform feature alignment on the initial feature image corresponding to the preset number of frames according to the prediction offset; and a feature fusion unit, used to match different regions in the aligned initial feature image with different preset weights to perform feature fusion and obtain the target spatiotemporal feature image frame.

[0024] By adopting the above technical solution, the prediction offset acquisition unit obtains the predicted offset between adjacent image frames and the current reference image frame within a preset number of frames, which can provide an accurate basis for subsequent feature alignment and help to utilize the temporal information of the video more accurately. The feature alignment unit performs feature alignment on the initial feature images corresponding to the preset number of frames based on the predicted offset, which can reduce the accumulation of errors caused by the complexity and error-proneness of optical flow estimation in traditional methods and avoid problems such as artifacts. The feature fusion unit matches different regions in the aligned initial feature images with different preset weights to perform feature fusion to obtain the target spatiotemporal feature image frame, which can optimize the processing of the characteristics of different regions, balance the relationship between detail restoration and smoothing, and better restore the realistic details in the video.

[0025] Thirdly, another embodiment of this application discloses an electronic device that adopts the following solution: An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the video resolution enhancement method based on the AI ​​vision engine described above.

[0026] Fourthly, another embodiment of this application discloses a computer-readable storage medium, which adopts the following scheme: A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the video resolution enhancement method based on the AI ​​vision engine described above.

[0027] In summary, this application includes at least one of the following beneficial technical effects: 1. The method receives low-resolution input video and divides it into consecutive frame groups for targeted processing. Multi-scale feature extraction is performed on each frame group, extracting shallow edge textures and deep semantic content as initial feature images. This avoids the problems of image blurring and detail loss caused by traditional methods, providing richer and more accurate features for subsequent reconstruction. The initial feature images of a predetermined number of frames are aligned and fused to obtain target spatiotemporal feature image frames. This efficiently and accurately utilizes video temporal information, overcoming the shortcomings of existing deep learning-based VSR methods in utilizing insufficient temporal information and reducing artifact generation. The target spatiotemporal feature image frames are upsampled to obtain target high-resolution image frames, improving resolution while generating realistic details, resolving the contradiction between detail restoration and smoothing. All target high-resolution image frames are integrated to output a target high-resolution video stream. The resolution of the high-resolution video stream is higher than that of the low-resolution video, effectively improving video resolution. This method also considers computational efficiency, enabling real-time processing on ordinary consumer-grade GPUs or mobile devices, and has strong generalization ability, capable of handling complex and varied video content in the real world. 2. Obtaining the predicted offset between adjacent image frames and the current reference image frame within a preset number of frames allows for precise determination of the positional relationship between adjacent image frames and the reference image frame, providing an accurate basis for subsequent feature alignment and avoiding feature matching errors caused by inaccurate positioning. Aligning the initial feature images corresponding to the preset number of frames based on the predicted offset ensures accurate spatial correspondence of features between different image frames, effectively utilizing the temporal information of the video and reducing artifacts caused by motion or occlusion. Matching different regions in the aligned initial feature images with different preset weights for feature fusion allows for differentiated processing based on the characteristics of different regions of the image, improving resolution while better restoring image details, resulting in generated target spatiotemporal feature image frames containing richer and more accurate information. 3. Upsampling the target spatiotemporal feature image frame through subpixel convolutional layers and reconstructing a high-resolution image frame can effectively improve the image resolution; judging whether the resolution difference between the high-resolution image frame and the preset high-resolution frame meets the preset difference can ensure that the reconstructed high-resolution image frame meets the expected resolution standard; using the high-resolution image frame that meets the preset resolution difference requirement as the target high-resolution image frame ensures that the final output target high-resolution image frame meets the actual application requirements. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating a method for improving video resolution based on an AI vision engine, as disclosed in an embodiment of this application. Figure 2 for Figure 1A detailed flowchart of step S30 in the video resolution enhancement method based on an AI vision engine disclosed in the paper; Figure 3 for Figure 1 A schematic diagram of the specific process of feature alignment and fusion in step S30 of the video resolution enhancement method based on AI vision engine disclosed in the paper; Figure 4 for Figure 1 A flowchart illustrating an example of a video resolution enhancement method based on an AI vision engine, as disclosed in the document. Figure 5 This is a schematic diagram of the system structure of a video resolution enhancement system based on an AI vision engine disclosed in another embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device disclosed in another embodiment of this application. Detailed Implementation

[0029] The present application will be further described in detail below with reference to the accompanying drawings.

[0030] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0031] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a” and “the” as used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0032] It should be understood that although the terms "first," "second," etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0033] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0034] [First Embodiment] See Figure 1 The first embodiment of this application discloses a method for improving video resolution based on an AI vision engine. The method includes the steps of receiving low-resolution video, dividing frame groups, extracting multi-scale features, aligning and fusing features, upsampling processing, and integrating and outputting image frames.

[0035] This method achieves efficient and accurate utilization of video temporal information by sequentially performing frame grouping, multi-scale feature extraction, feature alignment and fusion, upsampling, and image frame integration output on low-resolution videos. It generates realistic details while improving resolution, while also maintaining computational efficiency. This is because continuous frame grouping facilitates subsequent processing, multi-scale feature extraction captures shallow edge textures and deep semantic content, feature alignment and fusion effectively integrates temporal information, upsampling improves resolution, and image frame integration outputs the final high-resolution video stream.

[0036] Specifically, the steps are as follows: S10. Receive the input low-resolution video and divide the low-resolution video into consecutive frame groups.

[0037] Receiving low-resolution video can be achieved through video interfaces such as HDMI and USB, which can stably transmit video data. Dividing the video into consecutive frame groups can be done based on the video's frame rate. The number of frames in each frame group is determined based on processing requirements; for example, several consecutive frames can be divided into a frame group.

[0038] S20. Perform multi-scale feature extraction on each frame group to obtain the initial feature image corresponding to each frame group.

[0039] Multi-scale feature extraction is achieved through convolutional neural networks. For example, convolutional layers with different kernel sizes are used to extract multi-level features within each frame group, such as shallow edge texture and deep semantic content. Shallow edge texture is extracted using shallow convolutional layers to capture edge texture information, such as object outlines and lines. Deep semantic content is extracted using deep convolutional layers to capture semantic content, such as identifying object categories. Here, different kernel sizes can capture features at different scales, thus comprehensively acquiring the image's feature information.

[0040] Additionally, before performing step S20, the following steps are also included: S11. Normalize the pixel values ​​of each frame group.

[0041] The normalization process uses a common linear normalization method to map pixel values ​​to a specific range, such as the [0, 1] interval, which makes subsequent feature extraction more stable and accurate.

[0042] S30. Perform feature alignment and fusion on the initial feature images of a preset number of frames to obtain the target spatiotemporal feature image frames.

[0043] The preset number of frames is, for example, 5 frames, corresponding to the current frame and two frames before and after it. Of course, this configuration can be modified according to needs. See also... Figure 2 and Figure 3 Step S30 specifically includes: S31. Obtain the prediction offset between adjacent image frames and the current reference image frame within a preset number of frames.

[0044] Step S31 involves using an offset prediction network to predict the offset between adjacent image frames (F_t-1, F_t+1, ...) and the current reference image frame (center frame feature F_t) within a preset number of frames.

[0045] The offset prediction network is a current technology used to predict the positional offset of a target object relative to a reference point (such as an anchor box or initial prediction box). These offsets are then used to correct the initial position, thus more accurately locating the target object. This network can learn the feature relationships between image frames and directly predict the offset. This method may be more accurate and efficient than traditional feature matching algorithms.

[0046] S32. Align the initial feature images corresponding to the preset number of frames according to the predicted offset.

[0047] In particular, step S32 involves using a deformable convolutional layer to adaptively learn the offset between the adjacent frames and the current reference frame obtained in step S31, thereby achieving fine-grained and non-rigid alignment of multiple initial feature images, thus overcoming the shortcomings of traditional optical flow methods under complex motion.

[0048] S33. Match different regions in the aligned initial feature image with different preset weights to perform feature fusion and obtain the target spatiotemporal feature image frame.

[0049] Step S33 involves employing an attention mechanism to automatically learn and weight the importance of features at different locations and in different frames. Occluded or motion-blurred regions are given lower weights, while information-rich key regions are given higher weights, thereby achieving adaptive feature fusion.

[0050] Specifically, step S33 includes: For each initial feature image, the regions are identified, and it is determined whether there is a first region that is occluded or motion-blurred, and a second region that is rich in information. If the first region exists, it is matched with a first preset weight; if the second region exists, it is matched with a second preset weight, wherein the second preset weight is greater than the first preset weight.

[0051] Region recognition, for example, uses image segmentation algorithms, such as threshold-based segmentation and edge-based segmentation, to divide the image into different regions. Then, based on the characteristics of each region, it is determined whether it belongs to an occluded or motion-blurred region or an information-rich region. Occluded or motion-blurred regions are given smaller weights because the information in these regions may be inaccurate; information-rich regions are given larger weights to highlight their features, thereby achieving adaptive feature fusion.

[0052] It is worth mentioning that in step S30 above, an alignment and fusion module combining deformable convolution and attention mechanism is used, which makes more accurate use of temporal information, effectively reduces artifacts, and maintains excellent temporal consistency.

[0053] S40. Upsample the spatiotemporal feature image frame of the target to obtain a high-resolution image frame of the target.

[0054] In particular, step S40 is an operation based on inputting the fused spatiotemporal features (target spatiotemporal feature image frame) from step S30 into a detail enhancement module based on a generative adversarial network (GAN) to generate a higher resolution image frame through upsampling processing.

[0055] Specifically, step S40 includes: S41. The target spatiotemporal feature image frame is upsampled through a subpixel convolutional layer to reconstruct a high-resolution image frame.

[0056] In step S41, the generator performs upsampling through subpixel convolutional layers, with upsampling factors such as 4x (configurable 2x, 4x, and 8x are supported), gradually reconstructing high-resolution image frames. In this embodiment, the generator's loss function combines content loss (such as L1 / L2 loss) and perceptual loss (feature loss based on the VGG network); the generator's backbone network is an RRDB (Residual-in-Residual Dense Block) structure.

[0057] S42. Determine whether the resolution difference between the high-resolution image frame and the preset high-resolution frame meets the preset difference; if so, use the high-resolution image frame as the target high-resolution image frame.

[0058] In this step S42, executed by the discriminator, the resolution difference between the generated high-resolution frame and the real high-definition frame is determined. Specifically, the resolution difference is determined by calculating the image's resolution parameters, such as the number of pixels and image size, and comparing them with the corresponding parameters of the preset high-definition frame. If the difference is within a preset range, the requirement is considered met. In this embodiment, the adversarial loss forces the generator to produce more realistic and detailed textures, thereby avoiding an overly smooth result.

[0059] S50: Integrate all target high-resolution image frames and output the target high-resolution video stream; In this step S50, the final high-resolution video stream has a higher resolution than the low-resolution video, thus effectively improving the video resolution. At the same time, the method can take into account the computational efficiency, can be processed in real time on ordinary consumer-grade GPUs or mobile devices, and has strong generalization ability, capable of handling complex and varied video content in the real world.

[0060] Specifically, step S50 includes: S51. Perform effects processing on each target high-resolution image frame.

[0061] The effects processing includes color correction and sharpening. Color correction adjusts the color channel values ​​of an image to make the colors more realistic and natural. Sharpening enhances the edges and details of an image, making it clearer. This optimizes the color performance of the image, making the colors more accurate, vibrant, and natural, while enhancing the edges and details to make the image clearer and sharper, thus improving the visual effect.

[0062] S52. Arrange and integrate all the processed target high-resolution image frames in chronological order to output the target high-resolution video stream.

[0063] In step S52, image frames are combined into a video stream using video encoding formats such as H.264 and H.265. This ensures the continuity and smoothness of the video, allowing viewers to watch the content normally. Furthermore, the output video stream has a higher resolution than the input low-resolution video, effectively improving the video resolution.

[0064] Additionally, in this embodiment, the network training phase includes: Datasets: Large video super-resolution datasets such as Vimeo-90K and REDS are used.

[0065] Training settings: The Adam optimizer is used, with an initial learning rate of 1e-4 and a gradual decay strategy. The batch size is set to 16. The generator loss is: L_total = L_content + λ1 * L_perceptual + λ2 * L_gan, where λ1 and λ2 are balancing hyperparameters.

[0066] The training process involves first training the generator separately for several rounds, then fixing the generator and training the discriminator; subsequently, adversarial training of the generator and discriminator is performed alternately until the model converges.

[0067] See Figure 4 To more clearly describe the video resolution enhancement method based on the AI ​​vision engine disclosed in this embodiment, the specific implementation steps are as follows: S61, Low-resolution video input.

[0068] For example, a user submits a low-resolution 480p video as input. This corresponds to step S10 above, and a detailed description can be found above; it will not be repeated here.

[0069] S62, The preprocessing module performs frame grouping / normalization.

[0070] The system reads the video frame by frame and normalizes the pixel values. For a detailed description of step S11, please refer to the above; it will not be repeated here.

[0071] The S63 AI vision engine core incorporates a multi-scale feature extraction network.

[0072] The preprocessed image data is fed into a pre-trained AI vision engine. The multi-scale feature extraction network in the engine first extracts features from each frame. This corresponds to step S20, which is described above and will not be repeated here.

[0073] S64, the Spatiotemporal Feature Alignment and Fusion Module (DAF Module) in the core of the AI ​​Vision Engine.

[0074] Subsequently, the DAF module performs feature alignment and fusion on five consecutive frames (the current frame and the two frames before and after it). For a detailed description of step S30, please refer to the above; it will not be repeated here.

[0075] S65, the core high-frequency detail reconstruction module of the AI ​​vision engine (based on GAN).

[0076] The fused feature image frames are then fed into the generator, upsampled by a factor of 4, to generate a high-resolution 1080p frame. This corresponds to step S40, which is described above and will not be repeated here.

[0077] S66, Post-processing module: Color correction / sharpening.

[0078] The post-processing module performs a slight sharpening enhancement. For a detailed description of step S51, please refer to the above; it will not be repeated here.

[0079] S67, high-resolution video output.

[0080] Finally, output a 1080p high-definition video. For a detailed description of step S52, please refer to the above; it will not be repeated here.

[0081] The implementation principle of this embodiment is as follows: By performing a series of processing steps on low-resolution video, including frame grouping, multi-scale feature extraction, feature alignment and fusion, upsampling, and image frame integration output, the temporal information of the video can be utilized efficiently and accurately. Multi-scale feature extraction obtains the shallow edge texture and deep semantic content of the image, providing rich feature information for subsequent processing. Feature alignment and fusion effectively integrates temporal information by reasonably matching the weights of different regions, avoiding the problems caused by optical flow estimation in traditional methods. Upsampling and effect processing improve the resolution and quality of the image, ultimately outputting a high-resolution video stream. Compared with existing technologies, this embodiment solves problems such as insufficient utilization of temporal information, the contradiction between detail restoration and smoothing, high computational complexity, and weak generalization ability, improving the effect and efficiency of video resolution enhancement, and has stronger practicality and stability.

[0082] [Second Embodiment] See Figure 5 The second embodiment of this application discloses a video resolution enhancement system based on an AI vision engine. The system includes a frame group division module 210, a multi-scale extraction module 220, an alignment and fusion module 230, a processing module 240, and an integration module 250.

[0083] In this process, the modules work in sequence: frame grouping module 210 divides the input low-resolution video into frames; multi-scale extraction module 220 extracts multi-scale features; alignment and fusion module 230 aligns and fuses features; processing module 240 performs upsampling; and integration module 250 integrates and outputs the image frames. This achieves efficient and accurate utilization of video temporal information, generating realistic details while improving resolution and maintaining computational efficiency. This is because each module has a clear division of labor, works collaboratively, and processes the video according to a reasonable process, fully utilizing its respective functions.

[0084] Specifically, the frame group segmentation module 210 receives the input low-resolution video and divides the low-resolution video into continuous frame groups; the multi-scale extraction module 220 performs multi-scale feature extraction on each frame group to obtain an initial feature image corresponding to each frame group, wherein the multi-scale features include shallow edge texture and deep semantic content; the alignment and fusion module 230 performs feature alignment and fusion on the initial feature images of a preset number of frames to obtain target spatiotemporal feature image frames; the processing module 240 performs upsampling processing on the target spatiotemporal feature image frames to obtain target high-resolution image frames; and the integration module 250 integrates all the target high-resolution image frames to output a target high-resolution video stream, wherein the resolution of the high-resolution video stream is higher than the resolution of the low-resolution video.

[0085] Furthermore, the alignment and fusion module 230 includes a prediction offset acquisition unit 231, a feature fusion unit 232, and a feature alignment unit 233.

[0086] The prediction offset acquisition unit 231 is used to acquire the prediction offset between adjacent image frames and the current reference image frame in the preset number of frames; the feature alignment unit 232 is used to perform feature alignment on the initial feature image corresponding to the preset number of frames according to the prediction offset; and the feature fusion unit 233 is used to match different regions in the aligned initial feature image with different preset weights to perform feature fusion and obtain the target spatiotemporal feature image frame.

[0087] It should be noted that the video resolution enhancement system based on an AI vision engine disclosed in this embodiment implements the same method as described in the above embodiment, and therefore will not be described in detail here. Optionally, the various modules and other operations or functions in this embodiment are respectively for implementing the methods in the aforementioned embodiments.

[0088] [Third Embodiment] See Figure 6 In the third embodiment of this application, an electronic device is disclosed, which includes a memory 310 and a processor 320. The memory 310 is used to store a computer program; the processor 320 is used to execute the computer program to implement the steps of the video resolution improvement method based on an AI vision engine described in the first embodiment above. For details, please refer to the above description and will not be described in detail here.

[0089] The technical effect of the electronic device provided in this embodiment in practical application is the same as the technical effect of the video resolution improvement method based on AI vision engine in the first embodiment.

[0090] [Fourth Embodiment] A computer-readable storage medium is disclosed in the fourth embodiment of this application. The computer-readable storage medium is, for example, a non-volatile memory, such as magnetic media (e.g., hard disks, floppy disks, and magnetic tapes), optical media (e.g., CD-ROMs and DVDs), magneto-optical media (e.g., optical discs), and hardware devices specifically configured to store and execute computer-executable instructions (e.g., read-only memory (ROM), random access memory (RAM), flash memory, etc.). A computer program is stored on the computer-readable storage medium. The computer-readable storage medium can be executed by one or more processors or processing devices to implement the video resolution enhancement method based on an AI vision engine as described in the foregoing embodiments.

[0091] Furthermore, it is understood that the foregoing embodiments are merely illustrative examples of the present invention. Provided that the technical features do not conflict, the structure is not contradictory, and the purpose of the invention is not violated, the technical solutions of the various embodiments can be arbitrarily combined and used.

[0092] In the embodiments provided by this invention, it should be understood that the disclosed methods, systems, and measuring devices can be implemented in other ways. For example, the modules included in the systems described above are merely illustrative, and the division of modules is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0093] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] Furthermore, in the various embodiments of the present invention, the functional units / modules can be integrated into one processing unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated into one unit / module. The integrated unit / module described above can be implemented in hardware or in the form of hardware plus software functional units / modules.

[0095] The integrated units / modules implemented as software functional units / modules described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause one or more processors of a computer measurement device (which may be a personal computer, server, or network measurement device, etc.) to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for improving video resolution based on an AI vision engine, characterized in that, include: Receive input low-resolution video and divide the low-resolution video into consecutive frame groups; Multi-scale feature extraction is performed on each frame group to obtain the initial feature image corresponding to each frame group. The multi-scale features include shallow edge texture and deep semantic content. The initial feature images of a preset number of frames are aligned and fused to obtain target spatiotemporal feature image frames; Upsampling is performed on the spatiotemporal feature image frames of the target to obtain high-resolution image frames of the target; All the target high-resolution image frames are integrated to output a target high-resolution video stream, wherein the resolution of the high-resolution video stream is higher than that of the low-resolution video.

2. The method according to claim 1, characterized in that, The step of aligning and fusing the initial feature images of a preset number of frames to obtain target spatiotemporal feature image frames includes: Obtain the predicted offset between adjacent image frames and the current reference image frame within the preset number of frames; Based on the predicted offset, the initial feature image corresponding to the preset number of frames is aligned. Different regions in the aligned initial feature image are matched with different preset weights to perform feature fusion and obtain the target spatiotemporal feature image frame.

3. The method according to claim 2, characterized in that, The step of matching different regions in the aligned initial feature image with different preset weights includes: The regions in each of the initial feature images are identified, and it is determined whether there is a first region that is occluded or motion-blurred, and a second region that is rich in information; If the first region exists, then the first preset weight is matched; When the second region exists, a second preset weight is matched, wherein the second preset weight is greater than the first preset weight.

4. The method according to claim 1, characterized in that, The upsampling process of the target spatiotemporal feature image frame to obtain the target high-resolution image frame includes: The target spatiotemporal feature image frame is upsampled through a subpixel convolutional layer to reconstruct a high-resolution image frame; Determine whether the resolution difference between the high-resolution image frame and the preset high-definition frame meets the preset difference. If so, the high-resolution image frame is taken as the target high-resolution image frame.

5. The method according to claim 1, characterized in that, The step of integrating all the target high-resolution image frames and outputting the target high-resolution video stream includes: Each of the target high-resolution image frames is subjected to effect processing, including color correction and sharpening. All the processed target high-resolution image frames are arranged and integrated in chronological order to output a target high-resolution video stream.

6. The method according to claim 2, characterized in that, Before performing multi-scale feature extraction on each frame group, the method further includes: The pixel values ​​of each frame group are normalized. The step of obtaining the predicted offset between adjacent image frames and the current reference image frame within the preset number of frames includes: Based on the offset prediction network, obtain the predicted offset of the current frame and the two frames before and after it.

7. A system for video resolution enhancement based on an AI vision engine, characterized in that, For performing the method according to any one of claims 1 to 6, comprising: A frame group partitioning module is used to receive the input low-resolution video and partition the low-resolution video into consecutive frame groups. The multi-scale extraction module is used to extract multi-scale features for each frame group to obtain an initial feature image corresponding to each frame group. The multi-scale features include shallow edge texture and deep semantic content. The alignment and fusion module is used to perform feature alignment and fusion on the initial feature images of a preset number of frames to obtain target spatiotemporal feature image frames. The processing module is used to upsample the spatiotemporal feature image frame of the target to obtain a high-resolution image frame of the target; An integration module is used to integrate all the target high-resolution image frames and output a target high-resolution video stream, wherein the resolution of the high-resolution video stream is higher than the resolution of the low-resolution video.

8. The system according to claim 7, characterized in that, The alignment and fusion module includes: The prediction offset acquisition unit is used to acquire the prediction offset between adjacent image frames and the current reference image frame in the preset number of frames; A feature alignment unit is used to perform feature alignment on the initial feature image corresponding to the preset number of frames according to the predicted offset. The feature fusion unit is used to match different regions in the aligned initial feature image with different preset weights to perform feature fusion and obtain the target spatiotemporal feature image frame.

9. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method for improving video resolution based on an AI vision engine according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the video resolution enhancement method based on an AI vision engine according to any one of claims 1 to 6.

Images