A method, system, and media for real-time enhancement of ultra-high definition images and videos

Abstract

The application provides a kind of ultra-high definition image and video real-time enhancement method, system and medium, comprising the following steps: video frame extraction operation is carried out to source video to obtain corresponding image sequence;Image sequence is input to visual language model, and the text features corresponding to each image are obtained by visual language model to image text description to image sequence;Color retrieval enhanced gallery is constructed, and image sequence is input to color retrieval enhanced gallery for retrieval to obtain the color features that need to be supplemented;Image sequence, text features, color features are input to DA 2 Net model, and the output result of DA 2 Net model is obtained;Convolution operation is carried out to the output result of DA 2 Net model and image sequence fusion to obtain recovery image, and enhanced video is generated based on recovery image.The application can efficiently carry out image enhancement processing of ultra-high definition video under the premise of preserving color features and image details.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of super high-definition video enhancement, and particularly relates to a super high-definition image and video real-time enhancement method, system and medium. BACKGROUND

[0002] Creating images with high contrast, vivid colors, and rich details is the core goal of photography. However, uneven lighting conditions in scenes often lead to overexposure or underexposure, resulting in loss of details and color distortion. This decline in image quality can also adversely affect downstream tasks such as object detection and image segmentation. To address this challenge, current methods typically reconstruct abnormal exposure regions by inferring details from surrounding pixels. However, the reconstructed details and color information often lack realism and are prone to hallucinations, distorting the representation of the real scene.

[0003] Classic color correction methods in image enhancement include histogram equalization, gamma correction, and Retinex algorithms. Traditional methods extend histogram equalization from grayscale images to color images to achieve better saturation, while gamma correction methods use nonlinear transformations to enhance dark details and suppress bright regions. Retinex-based methods enhance details and colors by estimating reflectance and illumination. However, these methods often introduce noise or cause color distortion under complex lighting or uneven brightness.

[0004] Currently, CSEC uses a UNet-based network structure, combined with a pseudo-normal feature generator to extract color features, and through color offset estimation (COSE) and color modulation (COMO) modules to correct the color offset of overexposed and underexposed regions. By utilizing a customized cross-attention mechanism and an optimized loss function, CSEC can enhance the brightness and color accuracy of images under suboptimal lighting conditions. However, its performance is largely dependent on the quality of the pseudo-normal feature map, and the training process is relatively complex.

[0005] Deep learning-based methods, such as CSEC, have been widely applied to various image enhancement tasks, including high dynamic range (HDR) reconstruction, low-light enhancement, and underwater image enhancement. These methods perform well under challenging conditions such as extreme lighting or poor visibility. However, due to the need for targeted design for different tasks, there is still a lack of a unified framework to address various image enhancement tasks.

[0006] With the increasing popularity of 4K video (resolution 3840×2160), the enhancement of high-resolution video faces higher technical requirements. Existing 4K video enhancement technologies mainly rely on feature extraction in the spatial and temporal domains, but they often have the following shortcomings in the high-resolution video processing process: First, traditional enhancement methods fail to effectively cope with dynamic scene changes in high-resolution videos, which can easily lead to timing inconsistencies, flickering, or color drift. Second, 4K videos typically contain a large amount of detailed information, which places extremely high demands on computing power. Traditional methods are prone to computational bottlenecks when processing large-scale data. Finally, existing enhancement methods often ignore the semantic information in the video content, especially in complex lighting changes or scene transitions, which can easily lead to unnatural or incoherent enhancement effects. Summary of the Invention

[0007] The purpose of the present invention is to overcome the above-mentioned technical deficiencies and provide a method, system and medium for real-time enhancement of ultra-high-definition images and videos. When processing ultra-high-definition videos, the system can ensure that each frame can be accurately enhanced through the effective fusion of frame information and the enhancement of multimodal information, while ensuring the overall visual smoothness and color consistency of the video.

[0008] To achieve the above technical objectives, in a first aspect, the technical solution of the present invention provides a method for real-time enhancement of ultra-high-definition images and videos, the method comprising the following steps:

[0009] Perform a video frame extraction operation on the source video to obtain the corresponding image sequence;

[0010] Inputting the image sequence into a visual language model, and performing image text description on the image sequence using the visual language model to obtain text features corresponding to each image;

[0011] Constructing a color retrieval enhancement gallery, inputting the image sequence into the color retrieval enhancement gallery to retrieve and obtain the color features that need to be supplemented;

[0012] The image sequence, the text features, and the color features are input into the D 2 Net model, and obtain the DA 2 Output of Net model;

[0013] Regarding the DA 2 The output result of the Net model is convolved and fused with the image sequence to obtain a restored image, and an enhanced video is generated based on the restored image.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] First, the 4K video is frame-extracted to obtain a certain number of video frames; then, given a color-distorted video frame, the video frame is simultaneously input into the Color Retrieval Enhancement Gallery (CRAG) and the open source visual language model CogVLM. The Color Retrieval Enhancement Gallery (CRAG) retrieves the most needed k word vectors to supplement the semantic information of the input image in the color semantic space; the CogVLM visual language model extracts text prior information from the image, and then the original image, the retrieved color information and the text prior information are jointly input into the DARPA. 2 In the Net model, DA 2 The output of the .Net model is added to the original input image to generate a restored and enhanced image. Finally, the enhanced video frames are reassembled to restore the video. Because video content is often highly dynamic and contains complex spatiotemporal information, this method, when processing 4K video, effectively fuses frame information and enhances multimodal information, ensuring accurate enhancement of each frame while maintaining overall visual smoothness and color consistency.

[0016] According to some embodiments of the present invention, the image sequence is input into a visual language model, and the image sequence is subjected to an image text description by the visual language model to obtain a text feature corresponding to each image, including the steps of:

[0017] inputting the image sequence into a visual language model;

[0018] Input a prompt, and the visual language model performs a text description on the image sequence based on the prompt;

[0019] The text description of the image sequence is compiled by a text compiler to obtain text features.

[0020] According to some embodiments of the present invention, constructing a color retrieval enhanced gallery includes the steps of:

[0021] Applying a Gaussian blur kernel to the image sequence to blur texture details and preserve overall color distribution;

[0022] Extracting image feature vectors from the image sequence using an embedding model ResNet;

[0023] The image feature vector is stored in a vector database to obtain the color retrieval enhancement gallery.

[0024] According to some embodiments of the present invention, inputting the image sequence into the color retrieval enhancement gallery to retrieve and obtain the color features that need to be supplemented includes the following steps:

[0025] Given a feature vector of the image for query, construct a classifier to query the index of the most desired color vector;

[0026] The most needed k word vectors are selected, and based on the image feature vector and the classifier, the color retrieval enhancement gallery is searched to obtain the color features that need to be supplemented.

[0027] According to some embodiments of the present invention, the DA 2 Net models include:

[0028] Multiple residual Transformer attention blocks and convolutional layers. The residual Transformer attention blocks are used to extract and fuse features.

[0029] The residual Transformer attention block includes multiple local Transformer attention layers and convolutional layers;

[0030] The local Transformer attention layer includes: differential proxy attention mechanism, layer normalization and multi-layer perceptron, and the differential proxy attention mechanism is used to calculate attention features.

[0031] According to some embodiments of the present invention, the differential proxy attention mechanism introduces a set of additional proxy tokens A into the traditional attention module, represented as a four-tuple (Q, A, K, V). The proxy token A first acts as a proxy for the query token Q, responsible for aggregating information from K and V, and then broadcasting it back to Q, which can be expressed as:

[0032] O A =Attn S (Q,A,Attn S (A,K,V))=σ(QA T )σ(AK T )V

[0033] in, are the query, key, and value matrices, is the proxy token and σ(·) represents the Softmax function.

[0034] According to some embodiments of the present invention, extracting and fusing features in a residual Transformer attention block includes the steps of:

[0035] Given the input feature F of the i-th residual Transformer attention block i,0 , first use L local Transformer attention layers (LTAL) to extract the intermediate features F i,1 ,F i,2 ,…,F i,L , the process is expressed as:

[0036] F i,j =LTAL i,j (Fi,j-1 ),j=1,2,…,L

[0037] Among them, LTAL i,j (·) represents the jth local Transformer attention layer in the i-th residual Transformer attention block. Then, a convolutional layer is added before the residual connection. The output of the residual Transformer attention block is expressed as:

[0038] F i,out =Conv i (F i,L )+F i,0

[0039] Among them, Conv i (·) is the convolutional layer in the i-th residual Transformer attention block.

[0040] According to some embodiments of the present invention, the DA 2 The output result of the Net model is convolved and fused with the image sequence to obtain a restored image, including the following steps:

[0041] The restored image is obtained using the following formula:

[0042]

[0043] in, is the final restored image, F f It's DA 2 The output of the Net model, X in is the image sequence, and Conv(·) is the convolution operation.

[0044] In a second aspect, the technical solution of the present invention provides a real-time ultra-high-definition image and video enhancement system, comprising:

[0045] The video frame extraction module is used to perform a video frame extraction operation on the source video to obtain the corresponding image sequence;

[0046] a text feature extraction module, communicatively connected to the video frame extraction module, configured to input the image sequence into a visual language model, and perform image text description on the image sequence through the visual language model to obtain text features corresponding to each image;

[0047] A color feature extraction module is connected to the video frame extraction module to construct a color retrieval enhancement gallery, and inputs the image sequence into the color retrieval enhancement gallery to retrieve the color features that need to be supplemented;

[0048] DA 2Net model, inputting the image sequence, the text features, and the color features into the D 2 Net model, and obtain the DA 2 Output of Net model;

[0049] Video enhancement module, with the DA 2 Net model communication connection, the DA 2 The output result of the Net model is convolved and fused with the image sequence to obtain a restored image, and an enhanced video is generated based on the restored image.

[0050] In a third aspect, the technical solution of the present invention provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the real-time enhancement method for ultra-high-definition images and videos as described in any one of the first aspects.

[0051] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, wherein the abstract drawing is identical to one of the drawings in the specification:

[0053] Figure 1 A flowchart of a method for real-time enhancement of ultra-high-definition images and videos provided by one embodiment of the present invention;

[0054] Figure 2 A flowchart of a method for real-time enhancement of ultra-high-definition images and videos provided by one embodiment of the present invention;

[0055] Figure 3 A flowchart of a method for real-time enhancement of ultra-high-definition images and videos provided by one embodiment of the present invention;

[0056] Figure 4 A flowchart of a method for real-time enhancement of ultra-high-definition images and videos provided by one embodiment of the present invention;

[0057] Figure 5 A flowchart of a method for real-time enhancement of ultra-high-definition images and videos provided by one embodiment of the present invention;

[0058] Figure 6 The DA of the real-time enhancement method for ultra-high-definition images and videos provided by one embodiment of the present invention 2 Net model structure diagram. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0060] It should be noted that although the system diagrams illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the system or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0061] Reference Figure 1 、 Figure 2 and Figure 6 , Figure 1 A flowchart of a method for real-time enhancement of ultra-high-definition images and videos provided by one embodiment of the present invention; Figure 2 A flowchart of a method for real-time enhancement of ultra-high-definition images and videos provided by one embodiment of the present invention; Figure 6 The DA of the real-time enhancement method for ultra-high-definition images and videos provided by one embodiment of the present invention 2 The structure diagram of the NET model. The method for real-time enhancement of ultra-high-definition images and videos includes but is not limited to the following steps, and the numbers of the following steps should not be understood as having a certain order:

[0062] Step S110, performing a video frame extraction operation on the source video to obtain a corresponding image sequence;

[0063] Step S120: input the image sequence into a visual language model, and use the visual language model to perform image text description on the image sequence to obtain the corresponding text features of each image;

[0064] Step S130: constructing a color retrieval enhancement gallery, inputting the image sequence into the color retrieval enhancement gallery for retrieval to obtain the color features that need to be supplemented;

[0065] Step S140: input the image sequence, text features, and color features into the D 2 Net model, and obtain DA 2 Output of Net model;

[0066] Step S150: 2 The output of the Net model is convolved and fused with the image sequence to obtain a restored image, and an enhanced video is generated based on the restored image.

[0067] In some embodiments, the method for real-time enhancement of ultra-high-definition images and videos includes the following steps: performing a video frame extraction operation on the source video to obtain a corresponding image sequence; inputting the image sequence into a visual language model, and performing an image text description on the image sequence through the visual language model to obtain the corresponding text features of each image; constructing a color retrieval enhancement gallery, inputting the image sequence into the color retrieval enhancement gallery to retrieve and obtain the color features that need to be supplemented; inputting the image sequence, text features, and color features into the D 2 Net model, and obtain DA 2 The output of the Net model; 2 The output of the Net model is convolved and fused with the image sequence to obtain a restored image, and an enhanced video is generated based on the restored image.

[0068] Step S110: Frame extraction is performed on the input 4K video to obtain a corresponding image sequence. A 4K resolution video is received and frame extraction is performed on the input 4K video. Specifically, each frame image in the video is extracted at a predetermined frame rate (e.g., 60 frames per second) to generate an image sequence. Each image is then input into the model constructed below.

[0069] Step S120: extract text features F from the input image t :

[0070] The image is input into the open source visual language model CogVLM, and the prompt "Describe the content of this image" is used to generate the corresponding text. The text information is then input into the text encoder of CLIP to obtain the corresponding text feature F for each image. t . The process can be written as:

[0071] F t =Encoder(VLM(X in ,prompt)),

[0072] Among them, F t is the text feature, Encoder(·) is the text encoder of CLIP, VLM(·) is the open source visual language model CogVLM, X in is the input picture, and prompt is the prompt for the visual language model.

[0073] Step S130: Construct a color retrieval enhancement gallery, input the original image to perform retrieval and obtain the color features F that need to be supplemented. c .

[0074] First, a Gaussian blur kernel is applied to the input image, which blurs texture details but preserves the overall color distribution. Then, the embedding model ResNet is used to extract features from the image. This process can be written as:

[0075] G(X in )=GaussianBlur(X in ,kernel size ,σ),

[0076] F image =E(G(X in )),

[0077] Among them, X in is the input image, F image is the output image feature, E(·) is the model for extracting image features (embedding model), and G(·) is the Gaussian blur function applied to the input image with a kernel size of 33×33 and a standard deviation of 5.

[0078] Secondly, a vector database is built to retrieve the required color vector information corresponding to the input image. Specifically, the images in the ImageNet dataset are converted into feature vectors using the above embedding model and stored in the vector database.

[0079] Then, a search strategy is established to query the vector database.

[0080] Given a query F image , construct a small classifier C (a multilayer perceptron) to query the index of the most needed color vector. Set a hyperparameter and select the k most needed vectors. The whole process can be written as:

[0081] F c =CRAG(F image ,k,C),

[0082] Among them, F c is the output of the Color Retrieval Enhanced Gallery, and CRAG is the Color Retrieval Enhanced Gallery.

[0083] Step S140: The original input image X in , supplementary color feature information F c , extracted text information F t Input DA 2 Net network: the supplementary color feature information F c , extracted text information F t Add and original input image X in Input to DA together 2 Net model, the process can be written as:

[0084] F in =F c +F t ,

[0085] F f =DA 2 Net(X in ,F in )

[0086] Among them, F f It's DA 2 Net network output.

[0087] in DA 2 The differential proxy attention mechanism is used to calculate the attention features in the local Transformer attention layer of the Net network: the differential proxy attention mechanism introduces a set of additional proxy tokens A into the traditional attention module, represented as a four-tuple (Q, A, K, V). The proxy token A first acts as a proxy for the query token Q, responsible for aggregating information from K and V, and then broadcasting it back to Q, which can be expressed as:

[0088] O A =Attn S (Q,A,Attn S (A,K,V))=σ(QA T )σ(AK T )V

[0089] in, are the query, key, and value matrices, is the proxy token, and σ(·) represents the Softmax function. As shown in the above formula, the proxy attention mechanism consists of two attention operations. First, the proxy token A is regarded as a query and attention calculation is performed between A, K and V to aggregate the proxy feature V from all values. A Then, in the second attention calculation, A is used as the key and V A As a value, it works together with the query matrix Q to broadcast the global information of the proxy feature to each query tag, and finally obtains the output O A .

[0090] In order to calculate the first attention score, a gating mechanism is combined with a differential operation, which can be expressed as:

[0091] λ1=Sigmoid(β1)

[0092] σ(AK T )=λ1·Softmax(AK T )-(1-λ1)·Softmax(Conv(AK T ))

[0093] Where β1 is a learnable gating parameter, Sigmoid(·) is the activation function, and Conv(·) is the convolution operation. By applying a differential operation to two independently calculated Softmax attention maps and using a gating mechanism, this method reduces attention noise and achieves adaptive attention, allowing the model to focus more on key information.

[0094] In the second attention computation, global information from the agent features is assigned to each query token, expressed as:

[0095] λ2=Sigmoid(β2)

[0096] σ(QA T )=λ2·Softmax(QA T )-(1-λ2)·Softmax(Conv(QA T ))

[0097] Among them, β1 is a learnable gating parameter. By combining the global information transfer in the second attention calculation, the model achieves a more robust and accurate attention distribution.

[0098] in DA 2 The residual Transformer attention block of the Net network uses a local Transformer attention layer to extract features. The local Transformer attention layer consists of differential proxy attention, layer normalization, and a multi-layer perceptron. Given an input local feature X, the entire process in the local Transformer attention layer can be written as:

[0099] X=DAA(LN(X))+X,

[0100] X=MLP(LN(X))+X,

[0101] Where DAA(·) is the differential agent attention mechanism, LN(·) is layer normalization, and MLP(·) is a multi-layer perceptron consisting of two fully connected layers with a GELU activation function in between.

[0102] in DA 2 Extract and fuse features in the residual Transformer attention block of the Net network: Given the input feature F of the i-th residual Transformer attention block i,0 , first use L local Transformer attention layers (LTAL) to extract the intermediate features F i,1 ,F i,2 ,…,F i,L , the process can be expressed as:

[0103] F i,j =LTALi,j (F i,j-1 ),j=1,2,…,L

[0104] Among them, LTAL i,j (·) represents the jth local Transformer attention layer in the i-th residual Transformer attention block. Then, a convolutional layer is added before the residual connection. The output of the residual Transformer attention block can be expressed as:

[0105] F i,out =Conv i (F i,L )+F i,0

[0106] Among them, Conv i (·) is the convolutional layer in the i-th residual Transformer attention block.

[0107] Step S150, DA 2 Net network output and input image X in The final image restoration result is obtained by fusion

[0108] The restored image is obtained using the following formula:

[0109]

[0110] in, is the final image restoration result, F f It's DA 2 The output of the Net network, X in is the input image, and Conv(·) is the convolution operation that reconstructs the video from the enhanced image sequence. The enhanced image sequence is arranged in chronological order, maintaining the original image resolution, and reconstructing the enhanced video at 60 frames per second, resulting in a high-quality video output.

[0111] Traditional enhancement methods fail to effectively handle dynamic scene changes in high-resolution videos, easily leading to timing inconsistencies, flickering, or color drift. Secondly, 4K videos typically contain a wealth of detailed information, placing extremely high demands on computing power. Traditional methods are prone to computational bottlenecks when processing large amounts of data. Finally, existing enhancement methods often ignore the semantic information in video content, especially during complex lighting changes or scene transitions, which can easily lead to unnatural or disjointed enhancement effects.

[0112] To address these challenges, we propose a novel approach. First, we design a Color Retrieval Enhancement Gallery (CRAG) to enhance the color information of video frames by extracting color priors from a large number of natural images. Specifically, we apply a large-scale Gaussian blur kernel to images from the ImageNet dataset to remove detailed textures. These processed images are then stored in a gallery for retrieval.

[0113] Next, within the embedded color semantic space, the k word vectors that most need to complement the input image's color information are searched. This approach effectively ensures the consistency of the generated image color information during inference, resulting in a more natural and harmonious enhanced image. This process applies color correction and enhancement to every frame of the video, ensuring color stability and visual consistency across the video stream.

[0114] Secondly, a text prior is introduced by using a visual language model to extract textual information from the input image. This text prior not only helps avoid hallucination effects in low dynamic range (LDR) image restoration but also accurately supplements video details based on contextual information. In 4K video enhancement, especially in situations with rapid scene changes or complex lighting conditions, the text prior provides effective semantic guidance for dynamic scenes, reducing distortion or unnatural artifacts that may occur during the enhancement process.

[0115] Finally, the present invention proposes a differential proxy attention mechanism, which integrates a set of additional proxy tokens A into the traditional attention module, represented as a four-tuple (Q, A, K, V). The proxy token A acts as a proxy for the query token Q, first aggregating information from K and V, and then passing it back to Q. This method enhances the effectiveness of information transfer while maintaining the global context modeling capability of the model. In particular, when processing 4K videos, the temporal consistency between video frames is crucial. Through the differential operation, a gating mechanism is introduced when calculating the attention score, and the differential operation is performed on the two independently calculated Softmax attention maps, thereby effectively reducing the noise in the timing and ensuring consistency and smoothness between frames. This method effectively alleviates the temporal inconsistency problem that may occur in the video enhancement process, avoiding the typical flickering effect and loss of details.

[0116] Since video content usually has high dynamic variability and complex spatiotemporal information, this method can ensure that each frame is accurately enhanced when processing 4K video through the effective fusion of frame information and the enhancement of multimodal information, while maintaining the overall visual smoothness and color consistency of the video.

[0117] From the perspective of information extraction and fusion, using a visual language model to describe image sequences and obtain text features leverages the semantic information of the image. Traditional image and video enhancement methods often focus solely on the visual features of the image itself. However, this method incorporates text features to understand the image content at a semantic level, providing richer information for enhancement and making the enhanced images and videos more consistent with human understanding and expectations of the content.

[0118] Building a color retrieval enhancement gallery to capture the color features needed for supplementation is an effective way to mine and utilize image color information. Color plays a key role in the visual quality of images and videos. By supplementing color features, we can significantly improve the color richness, vividness, and realism of images and videos, making the enhanced images more visually appealing.

[0119] The image sequence, text features and color features are input into the DA 2 Net model realizes the fusion of multiple features. This fusion method can give full play to the advantages of different features, comprehensively consider the content, semantics and color of the image, so that D 2 Net model outputs more accurate and higher-quality results, laying the foundation for the subsequent generation of high-quality restored images and enhanced videos.

[0120] From a processing perspective, extracting frames from the source video to generate an image sequence is a highly efficient approach. Decomposing the video into an image sequence allows for detailed processing of each frame, avoiding the computationally intensive and complex processing of the entire video. This also facilitates subsequent feature extraction and enhancement operations.

[0121] Model combination and convolution fusion using DA 2 Net model, and then convolves its output with the original image sequence. This approach combines the model's intelligent processing capabilities with the local feature extraction capabilities of the convolution operation. While preserving key information from the original image, it effectively enhances image detail and quality, ultimately generating high-quality restored images and enhanced videos.

[0122] This method enables real-time enhancement of ultra-high-definition images and videos, which is crucial for many practical applications. For example, in live streaming and video surveillance, real-time enhancement can provide a clearer, higher-quality visual experience, meeting user demands for real-time image quality.

[0123] This method has broad application prospects. Real-time enhancement of ultra-high-definition images and videos can be applied in multiple fields, such as film and television production, virtual reality, and medical imaging. In film and television production, it can enhance the visual effects of the film; in virtual reality, it can enhance the sense of immersion; and in medical imaging, it can help doctors observe lesions and other details more clearly, showing broad application prospects and commercial value.

[0124] Reference Figure 3 , Figure 3 A flowchart of a method for real-time enhancement of ultra-high-definition images and videos provided in accordance with one embodiment of the present invention; the method for real-time enhancement of ultra-high-definition images and videos includes but is not limited to the following steps:

[0125] Step S210, inputting the image sequence into the visual language model;

[0126] Step S220: inputting a prompt, and the visual language model performs a text description on the image sequence based on the prompt;

[0127] Step S230 : compile the text description of the image sequence by a text compiler to obtain text features.

[0128] The image sequence is input into the visual language model, and the visual language model is used to describe the image sequence in text to obtain the corresponding text features of each image. The steps include: inputting the image sequence into the visual language model; inputting prompts, and the visual language model uses the prompts to describe the image sequence in text; and compiling the text description of the image sequence through a text compiler to obtain text features. The image is input into the open source visual language model CogVLM, and the prompt "Describe the content of this image" is used to generate the corresponding text. The text information is then input into the text encoder of CLIP to obtain the corresponding text features F for each image. t . The process can be written as:

[0129] F t =Encoder(VLM(X in ,prompt)),

[0130] Among them, F t is the text feature, Encoder(·) is the text encoder of CLIP, VLM(·) is the open source visual language model CogVLM, X in is the input picture, and prompt is the prompt for the visual language model.

[0131] Feeding image sequences into a visual language model along with prompts provides a clear task orientation for the model's processing. Prompts guide the visual language model to focus on specific information within the image sequence, such as the subject, scene, and object attributes. This makes the model-generated text descriptions more targeted and relevant, better reflecting the key information of the image and providing more valuable semantic guidance for subsequent image enhancement.

[0132] By changing the prompts, we can flexibly adjust how the visual language model understands and describes image sequences. Different application scenarios and needs may require attention to different aspects of the image. For example, artistic creation may focus more on the style and emotional expression of the image, while security monitoring may focus more on the target object and behavior in the image. Using prompts can easily adapt to these different needs, making the model more versatile and adaptable.

[0133] The visual language model possesses powerful cross-modal understanding capabilities, enabling in-depth semantic analysis and understanding of image sequences. It can convert visual information in an image into natural language descriptions, mining the implicit semantic relationships and contextual information within the image. Compared to traditional methods based solely on visual features, this semantically-based text description provides richer and more abstract information, helping to better grasp the content and meaning of the image and providing a more comprehensive basis for image enhancement.

[0134] The text descriptions generated by the visual language model can present various image information in the form of natural language. Natural language has a rich vocabulary and grammatical structure, which can more accurately and meticulously describe the details and features of the image. This rich information can provide a multi-dimensional reference for subsequent image enhancement. For example, in tasks such as image restoration and super-resolution reconstruction, the information in the text description can be used to restore or enhance specific parts of the image, making the enhanced image more consistent with human cognition and expectations.

[0135] Text features are generated by compiling the text descriptions of image sequences through a text compiler. This process refines and standardizes the text information. The text compiler removes redundant information from the text descriptions, extracts key semantic features, and converts them into a format suitable for subsequent processing. The resulting text features are more concise and accurate, which helps improve the processing efficiency and accuracy of the model. They also facilitate integration with other types of features (such as image features and color features), providing more effective support for real-time enhancement of images and videos.

[0136] The compiled text features are presented in a structured form, which is easier to be used by subsequent models (such as D 2Net model). This structured representation enables the model to better understand and utilize textual information, thereby more effectively incorporating semantic information during image and video enhancement, improving the quality and accuracy of the enhancement effect.

[0137] Reference Figure 4 , Figure 4 A flowchart of a method for real-time enhancement of ultra-high-definition images and videos provided in accordance with one embodiment of the present invention; the method for real-time enhancement of ultra-high-definition images and videos includes but is not limited to the following steps:

[0138] Step S310 , applying a Gaussian blur kernel to the image sequence to blur texture details and preserve the overall color distribution;

[0139] Step S320, extracting image feature vectors from the image sequence using the embedding model ResNet;

[0140] Step S330: storing the image feature vector into a vector database to obtain a color retrieval enhancement gallery.

[0141] A Gaussian blur kernel is applied to the input image, which blurs the texture details but preserves the overall color distribution. Then, the embedding model ResNet is used to extract features from the image. This process can be written as:

[0142] G(X in )=GaussianBlur(X in ,kernel size ,σ),

[0143] F image =E(G(X in )),

[0144] Among them, X in is the input image, F image is the output image feature, E(·) is the model for extracting image features (embedding model), and G(·) is the Gaussian blur function applied to the input image with a kernel size of 33×33 and a standard deviation of 5.

[0145] A vector database is built to retrieve the required color vector information corresponding to the input image. The images in the ImageNet dataset are converted into feature vectors using the above embedding model and stored in the vector database, thus obtaining the color retrieval enhancement gallery.

[0146] Applying a Gaussian blur kernel to an image sequence blurs texture details while preserving the overall color distribution. This operation is highly targeted. In many image enhancement tasks, texture details can interfere with the extraction and analysis of color features. Gaussian blurring removes or weakens these unnecessary details, allowing subsequent processing to focus more on the image's color features, improving the accuracy and efficiency of color feature extraction. After Gaussian blurring, the image's complexity is reduced, reducing redundant information in the data. This not only helps speed up subsequent feature extraction and storage, but also reduces computing resource consumption, making the entire system more efficient and stable when processing large-scale image sequences.

[0147] This paper uses the ResNet embedding model to extract image feature vectors from image sequences. ResNet is a deep residual network with excellent feature extraction capabilities, effectively extracting rich and representative features from images. In the process of constructing a color retrieval enhancement gallery, it accurately captures the image's color features and other potential color-related features, providing more comprehensive and accurate information for subsequent color retrieval.

[0148] The present invention can store extracted image feature vectors in a vector database to create a color retrieval enhancement gallery, facilitating rapid retrieval and matching. The vector database, with its efficient storage and retrieval algorithms, can quickly find other vectors similar to the input image feature vector, enabling rapid retrieval of color features. This is crucial for real-time enhancement of ultra-high-definition images and videos, requiring high real-time performance. It can quickly acquire the necessary color features, improving the efficiency of the entire enhancement process.

[0149] This invention uses a vector database to store image feature vectors, making the color retrieval enhancement gallery highly scalable and flexible. New image feature vectors can be easily added to continuously enrich the gallery's content to accommodate diverse image and video data. Furthermore, by adjusting the retrieval algorithm and parameters of the vector database, the retrieval accuracy and range of color features can be flexibly controlled according to specific application requirements.

[0150] Reference Figure 5 , Figure 5 A flowchart of a method for real-time enhancement of ultra-high-definition images and videos provided in accordance with an embodiment of the present invention; the method for real-time enhancement of ultra-high-definition images and videos includes but is not limited to the following steps:

[0151] Step S410 , given an image feature vector for query, construct a classifier to query the index of the most needed color vector;

[0152] Step S420 : Select the most needed k word vectors, and search the color retrieval enhancement gallery based on the image feature vector and the classifier to obtain the color features that need to be supplemented.

[0153] An image sequence is input into a color retrieval enhancement gallery for retrieval to obtain color features that need to be supplemented, including the following steps: given an image feature vector for query, a classifier is constructed to query the index of the most needed color vector; the most needed k word vectors are selected, and the color features that need to be supplemented are retrieved from the color retrieval enhancement gallery based on the image feature vector and the classifier.

[0154] Establish a retrieval strategy to query the vector database: Given a query F image , we construct a small classifier C (a multilayer perceptron) to query the index of the most needed color vector. We set a hyperparameter to select the k most needed vectors. The whole process can be written as:

[0155] F c =CRAG(F image ,k,C),

[0156] Among them, F c is the output of the Color Retrieval Enhanced Gallery, and CRAG is the Color Retrieval Enhanced Gallery.

[0157] The present invention uses a given image feature vector for querying and constructs a classifier to retrieve the most relevant color vector index. This approach focuses the search on color vectors closely related to the current image features, accurately locating areas of the image where color may be missing or in need of enhancement. Compared to aimless search methods, this significantly improves search efficiency and accuracy, making the acquired color features more targeted and directly meeting the color information requirements of image enhancement.

[0158] The present invention provides flexibility by enabling the selection of the k most needed word vectors. By adjusting the value of k, the number of retrieved color features can be flexibly controlled based on specific image enhancement requirements and computational resource constraints. If richer, more detailed color information is required to achieve a more refined enhancement effect, the value of k can be appropriately increased; if computational resources are more sensitive, or only rough adjustments to image color are required, the value of k can be reduced. This adjustability allows the method to adapt to different application scenarios and requirements.

[0159] This invention enables searching within a color retrieval enhancement gallery based on image feature vectors and classifiers, fully leveraging the vast amount of image feature vector information stored within the gallery. The color retrieval enhancement gallery itself is meticulously constructed and contains a wealth of color feature data. This retrieval method can extract the color features that best match the current image from the gallery, providing powerful support for image enhancement. This process is equivalent to precise screening within a vast library of color features, greatly increasing the likelihood of obtaining the appropriate color features.

[0160] The present invention utilizes image feature vectors and classifiers for retrieval, ensuring a high correlation between the acquired color features and the original image features. This correlation allows the supplemented color features to be better integrated into the original image, enhancing the image's color while maintaining the overall consistency and harmony of the image. For example, when color-enhancing a scene in an ultra-high-definition video, the retrieved color features can match the image content of that scene, eliminating any abrupt or inharmonious color effects and thus improving the visual quality of the entire video.

[0161] This invention accurately captures the color features that need to be supplemented, providing critical information support for subsequent image and video enhancement. By combining these color features with other features (such as image features and text features), images and videos can be processed more comprehensively, significantly improving the real-time enhancement of ultra-high-definition images and videos. This achieves better performance in terms of color richness and vividness, as well as its consistency with the image content, providing users with a higher-quality visual experience.

[0162] This retrieval method is highly scalable and versatile. As the image feature vectors in the color retrieval enhancement gallery continue to be enriched and updated, the retrieved color features will become more comprehensive and accurate. Furthermore, this feature vector and classifier-based retrieval approach is not limited to specific image or video datasets but can be applied to a wide range of image and video enhancement tasks, demonstrating its broad potential.

[0163] In some embodiments, the method for real-time enhancement of ultra-high-definition images and videos includes the following steps: performing a video frame extraction operation on the source video to obtain a corresponding image sequence; inputting the image sequence into a visual language model, and performing an image text description on the image sequence through the visual language model to obtain the corresponding text features of each image; constructing a color retrieval enhancement gallery, inputting the image sequence into the color retrieval enhancement gallery to retrieve and obtain the color features that need to be supplemented; inputting the image sequence, text features, and color features into the D 2 Net model, and obtain DA 2 The output of the Net model; 2The output of the Net model is convolved and fused with the image sequence to obtain a restored image, and an enhanced video is generated based on the restored image.

[0164] DA 2 The Net model includes: multiple residual Transformer attention blocks and convolutional layers. The residual Transformer attention blocks are used to extract and fuse features. The residual Transformer attention blocks include multiple local Transformer attention layers and convolutional layers. The local Transformer attention layers include: differential proxy attention mechanism, layer normalization and multi-layer perceptron. The differential proxy attention mechanism is used to calculate attention features.

[0165] Differential Proxy Attention Mechanism: A set of additional proxy tokens A is introduced into the traditional attention module, represented as a four-tuple (Q, A, K, V). The proxy token A first acts as a proxy for the query token Q, responsible for aggregating information from K and V, and then broadcasting it back to Q, expressed as:

[0166] O A =Attn S (Q,A,Attn S (A,K,V))=σ(QA T )σ(AK T )V

[0167] in, are the query, key, and value matrices, is the proxy token and σ(·) represents the Softmax function.

[0168] Extracting and fusing features in the residual Transformer attention block includes the following steps:

[0169] Given the input feature F of the i-th residual Transformer attention block i,0 , first use L local Transformer attention layers (LTAL) to extract the intermediate features F i,1 ,F i,2 ,…,F i,L , the process is expressed as:

[0170] F i,j =LTAL i,j (F i,j-1 ),j=1,2,…,L

[0171] Among them, LTAL i,j(·) represents the jth local Transformer attention layer in the i-th residual Transformer attention block. Then, a convolutional layer is added before the residual connection. The output of the residual Transformer attention block is expressed as:

[0172] F i,out =Conv i (F i,L )+F i,0

[0173] Among them, Conv i (·) is the convolutional layer in the i-th residual Transformer attention block.

[0174] to DA 2 The output of the Net model is convolved and fused with the image sequence to obtain the restored image, including the following steps:

[0175] The restored image is obtained using the following formula:

[0176]

[0177] in, is the final restored image, F f It's DA 2 The output of the Net model, X in is an image sequence, and Conv(·) is a convolution operation.

[0178] Improved results:

[0179] First, we validate the effectiveness of our model in the image domain. Compared to existing image enhancement methods (such as HDRNet, ZeroDCE, RUAS, LCDPNet, RetinexFormer, Sagiri, CSEC, and CECF), our method is experimentally validated on the LCDP and Mobile-Spec datasets, which include both overexposed and underexposed images, as well as on the underwater SUID dataset.

[0180] The experiments were conducted on a single NVIDIA A100 80GB GPU. 2 In the Net, the number of residual Transformer attention blocks is set to 6, and each residual Transformer attention block contains 6 local Transformer attention layers. The embedding dimension is set to 66, and the number of attention heads is set to 6.

[0181] Three widely recognized metrics are used to evaluate performance: PSNR, SSIM, and LPIPS. PSNR and SSIM focus on evaluating the fidelity and structural integrity of the reconstructed image, while LPIPS evaluates perceptual quality by measuring the similarity of deep features, providing an evaluation that is more consistent with human visual quality.

[0182] Among them, PSNR (Peak Signal-to-Noise Ratio) is a common indicator for measuring image quality. It is mainly used to evaluate the difference between the processed image and the original image. The larger the value, the better the image quality. The calculation formula is:

[0183]

[0184] Where MSE stands for mean square error, which is used to measure the difference between two images at the pixel level. I(i, j) and K(i, j) represent the pixel values ​​of the generated fused image and the corresponding gold standard image (ground truth) at position (i, j), respectively. m and n represent the width and height of the image, respectively. MAX represents the maximum value of the image pixels.

[0185] SSIM (Structural Similarity Index Measure) is an indicator used to evaluate the quality of two images. It mainly measures the similarity of images by comparing their brightness, contrast, and structural information. The closer the value is to 1, the better the image quality.

[0186] The calculation formula of SSIM is:

[0187] c1=(k1L) 2 ;

[0188] c2=(k2L) 2 ;

[0189]

[0190] Among them, μ Y is the average value of the fused image Y output by the model, is the gold standard Y corresponding to Y gt The average value of It represents images Y and Y gt The covariance of is the variance of Y, It's Y gt variance; L is the dynamic range of pixel values, k1 and k2 represent preset hyperparameters, where k1 = 0.01, k2 = 0.03, and c1 and c2 represent smoothing parameters.

[0191] The LPIPS (Learned Perceptual Image Patch Similarity) metric is used to evaluate the perceptual similarity between two images. It is based on features extracted by a deep convolutional neural network, rather than simple pixel differences. A smaller value indicates better image quality.

[0192] The calculation formula for LPIPS is:

[0193]

[0194] Among them, x represents the fusion image output by the model, x0 represents the gold standard image corresponding to x, and f l (x) and f l (x0) represents the lth layer feature map extracted by a pre-trained VGG deep neural network. l (x) is the feature of image x at layer l, f l (x0) is the feature of image x0 at layer l. Represents the Euclidean distance (L2 norm) between feature maps. This is the feature map f l (x) and f l The difference metric between (x0) is used to measure the similarity between two images in the l-th layer features. l Represents the weighted coefficient of each layer’s feature map. Different network layers have different effects on perceived similarity, and the weight w l Used to reflect this, it is usually learned through training. It means summing the results of all layers l and comprehensively considering the feature differences at different levels.

[0195] Table 1 Results on LCDP dataset

[0196]

[0197] Table 2 Results on the Mobile-Spec dataset

[0198]

[0199] Table 3 Results on the SUID dataset

[0200]

[0201]

[0202] Table 1 lists the comparison results on the LCDP dataset, and all methods are retrained under the same settings. 2Net outperforms the second-best method, RetinexFormer, by approximately 0.36dB and 0.02dB higher in PSNR and SSIM, respectively, while reducing LPIPS by 0.03.

[0203] Experiments were conducted on the Mobile-Spec dataset. As shown in Table 2, this method achieved the best performance on all evaluation metrics. In addition, Table 3 shows that DA 2 Net can effectively correct the color of underwater images, while existing image enhancement methods perform poorly in underwater scenes. Specifically, compared with the second best method, our method improves by 4.52dB in PSNR, 0.05dB in SSIM, and reduces by 0.0027 in LPIPS.

[0204] Overall, this method shows excellent results in enhancing overexposed, underexposed and underwater images, significantly improving image quality in various complex scenes.

[0205] Furthermore, experimental evaluation was conducted on low-quality 4K videos. The results demonstrate that during the inference phase, this method can efficiently process ultra-high-resolution 4K videos. Specifically, the method achieves a frame rate of 68 frames per second (FPS) when processing 4K videos. Even with lower-quality videos, it significantly improves the visual quality and enhances image detail and color.

[0206] In one embodiment, the ultra-high-definition image and video real-time enhancement system includes: a video frame extraction module, which is used to perform a video frame extraction operation on the source video to obtain a corresponding image sequence; a text feature extraction module, which is in communication with the video frame extraction module, and is used to input the image sequence into a visual language model, and perform an image text description on the image sequence through the visual language model to obtain the corresponding text features of each image; a color feature extraction module, which is in communication with the video frame extraction module, constructs a color retrieval enhancement gallery, and inputs the image sequence into the color retrieval enhancement gallery for retrieval to obtain the color features that need to be supplemented; 2 Net model, inputting image sequences, text features, and color features into the D 2 Net model, and obtain DA 2 The output of the Net model; video enhancement module, and DA 2 Net model communication connection, to DA 2 The output of the Net model is convolved and fused with the image sequence to obtain a restored image, and an enhanced video is generated based on the restored image.

[0207] The processor and the memory may be connected via a bus or other means.

[0208] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0209] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate and may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0210] In addition, an embodiment of the present invention also provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are executed by a processor or controller, for example, by a processor in the above-mentioned terminal embodiment, so that the above-mentioned processor can execute the ultra-high-definition image and video real-time enhancement method in the above-mentioned embodiment.

[0211] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices, or can be used to store desired information and any other medium that can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0212] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above implementation. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

[0213] The specific embodiments of the present invention described above do not limit the scope of protection of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for real-time enhancement of ultra-high-definition images and videos, characterized in that: The following steps are involved: Perform a video frame extraction operation on the source video to obtain the corresponding image sequence; Inputting the image sequence into a visual language model, and performing image text description on the image sequence using the visual language model to obtain text features corresponding to each image; Constructing a color retrieval enhancement gallery, inputting the image sequence into the color retrieval enhancement gallery to retrieve and obtain the color features that need to be supplemented; The image sequence, the text features, and the color features are input into the D 2 Net model, and obtain the DA 2 Output of Net model; Regarding the DA 2 The output result of the Net model is convolved and fused with the image sequence to obtain a restored image, and an enhanced video is generated based on the restored image; Constructing a color retrieval enhancement gallery includes the following steps: Applying a Gaussian blur kernel to the image sequence to blur texture details and preserve overall color distribution; Extracting image feature vectors from the image sequence using an embedding model ResNet; Storing the image feature vector in a vector database to obtain the color retrieval enhancement gallery; Inputting the image sequence into the color retrieval enhancement gallery to retrieve and obtain the color features that need to be supplemented, including the steps of: Given a feature vector of the image for query, construct a classifier to query the index of the most desired color vector; The most needed k word vectors are selected, and based on the image feature vector and the classifier, the color retrieval enhancement gallery is searched to obtain the color features that need to be supplemented.

2. The method for real-time enhancement of ultra-high-definition images and videos according to claim 1, characterized in that: The image sequence is input into a visual language model, and image text description of the image sequence is performed by the visual language model to obtain text features corresponding to each image, including the steps of: inputting the image sequence into a visual language model; Input a prompt, and the visual language model performs a text description on the image sequence based on the prompt; The text description of the image sequence is compiled by a text compiler to obtain text features.

3. The method for real-time enhancement of ultra-high-definition images and videos according to claim 1, wherein: The DA 2 Net models include: Multiple residual Transformer attention blocks and convolutional layers. The residual Transformer attention blocks are used to extract and fuse features. The residual Transformer attention block includes multiple local Transformer attention layers and convolutional layers; The local Transformer attention layer includes: differential proxy attention mechanism, layer normalization and multi-layer perceptron, and the differential proxy attention mechanism is used to calculate attention features.

4. The method for real-time enhancement of ultra-high-definition images and videos according to claim 3, characterized in that: The differential proxy attention mechanism introduces a set of additional proxy tokens A into the traditional attention module, represented as a four-tuple (Q, A, K, V). The proxy token A first acts as a proxy for the query token Q, responsible for aggregating information from K and V, and then broadcasting it back to Q, expressed as: Among them, Q, K, are the query, key, and value matrices, is the proxy token, Represents the Softmax function.

5. The method for real-time enhancement of ultra-high-definition images and videos according to claim 3, characterized in that: Extracting and fusing features in the residual Transformer attention block includes the following steps: Given the input features of the i-th residual Transformer attention block , first use Local Transformer Attention Layer (LTAL) extracts intermediate features , ,…, , the process is expressed as: in, represents the jth local Transformer attention layer in the i-th residual Transformer attention block. Then, a convolutional layer is added before the residual connection. The output of the residual Transformer attention block is expressed as: in, is the convolutional layer in the i-th residual Transformer attention block.

6. The method for real-time enhancement of ultra-high-definition images and videos according to claim 1, characterized in that: Regarding the DA 2 The output result of the Net model is convolved and fused with the image sequence to obtain a restored image, including the following steps: The restored image is obtained using the following formula: in, is the final restored image, It's DA 2 The output of the Net model is: is the image sequence, is the convolution operation.

7. A real-time ultra-high-definition image and video enhancement system, applying the real-time ultra-high-definition image and video enhancement method according to any one of claims 1 to 6, characterized in that: include: The video frame extraction module is used to perform a video frame extraction operation on the source video to obtain the corresponding image sequence; a text feature extraction module, communicatively connected to the video frame extraction module, configured to input the image sequence into a visual language model, and perform image text description on the image sequence through the visual language model to obtain text features corresponding to each image; A color feature extraction module is connected to the video frame extraction module to construct a color retrieval enhancement gallery, and inputs the image sequence into the color retrieval enhancement gallery to retrieve the color features that need to be supplemented; DA 2 Net model, inputting the image sequence, the text features, and the color features into the D 2 Net model, and obtain the DA 2 Output of Net model; Video enhancement module, with the DA 2 Net model communication connection, the DA 2 The output result of the Net model is convolved and fused with the image sequence to obtain a restored image, and an enhanced video is generated based on the restored image.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the ultra-high-definition image and video real-time enhancement method according to any one of claims 1 to 6.

Images