Image enhancement method, video transmission method, model training method and electronic device

Abstract

This application discloses an image enhancement method, comprising: acquiring an image to be enhanced; performing enhancement processing on the image to be enhanced using an enhancement model to obtain enhancement features corresponding to each channel; determining the weights corresponding to the enhancement features of each channel, the weights representing the contribution of the enhancement features of the corresponding channel to the enhanced image; performing weighted processing on the enhancement features of the corresponding channel using the weights to obtain weighted enhancement features corresponding to each channel; and fusing the weighted enhancement features to obtain the enhanced image corresponding to the image to be enhanced. This application also provides a video transmission method, a training method for the enhancement model, an apparatus, an electronic device, and a computer-readable storage medium. The solution provided by this application can perform targeted enhancement of features of different channels of an image, thereby achieving more accurate and effective image enhancement, better avoiding the weakening of important features or the amplification of unimportant features, and improving the image enhancement effect.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an image enhancement method, a video transmission method, a model training method, an apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of digital imaging technology, image compression coding technology has been widely used in image storage and transmission. However, after decoding compressed images, image quality degradation often occurs, such as blockiness, blurring, and noise artifacts. These quality degradation problems seriously affect the visual effect of the image and its subsequent applications.

[0003] In related technologies, image enhancement algorithms can be used to process decoded images to improve their quality. However, in practical applications, different images typically require different enhancement focuses, while existing technologies usually perform uniform enhancement on the image. This approach may result in the weakening of important features or the amplification of unimportant features, thus affecting the image enhancement effect. Summary of the Invention

[0004] This application provides an image enhancement method, video transmission method, model training method, apparatus, electronic device, and computer-readable storage medium, which can target the enhancement of features in different channels of an image, thereby achieving more accurate and effective image enhancement, better avoiding the weakening of important features or the amplification of unimportant features, and improving the image enhancement effect. The specific solution is as follows: In a first aspect, this application provides an image enhancement method, the method comprising: Obtain the image to be enhanced; The image to be enhanced is enhanced by a pre-trained enhancement model to obtain enhancement features corresponding to each channel. The enhancement features of different channels are used to represent the enhanced image from different dimensions. Determine the weights corresponding to the enhancement features of each channel, whereby the weights represent the degree of contribution of the enhancement features of the corresponding channel to the enhanced image. The enhanced features of the corresponding channels are weighted using the weights to obtain the weighted enhanced features for each channel; The weighted enhancement features are fused to obtain the enhanced image corresponding to the image to be enhanced.

[0005] Secondly, this application provides a method for enhancing the training of a model, including: Obtain training samples, which include the sample image to be enhanced and the corresponding original sample image; The input information of the first training model is determined based on the sample image to be enhanced, and the input information is input into the training enhancement unit of the first training model to obtain the sample enhancement features corresponding to each channel. The sample weights corresponding to the sample enhancement features of each channel are determined by the training weight determination unit of the first training model. The sample enhancement features of the corresponding channels are weighted using the sample weights to obtain the weighted enhancement features of each channel. The weighted enhancement features of each sample are fused to obtain the enhanced image of the sample corresponding to the image to be enhanced. Based on the difference between the augmented image and the original image, the model parameters of the first model to be trained are adjusted to obtain the trained augmented model.

[0006] Thirdly, this application also provides a video transmission method, the method comprising: The sending end compresses and encodes each video frame of the video to be sent to obtain a compressed bitstream, and then sends the compressed bitstream to the receiving end; The receiving end receives the compressed bitstream, decodes the compressed bitstream to obtain the video frame to be enhanced, and uses the image enhancement method described in any one of the first aspects to enhance the video frame to obtain the enhanced video frame.

[0007] Fourthly, this application also provides an electronic device, comprising: a processor, a memory, and computer program instructions stored in the memory and executable on the processor; wherein the processor executes the computer program instructions to implement the method as described in any one of the first to third aspects.

[0008] Fifthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in any one of the first to third aspects.

[0009] Sixthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method as described in any one of the first to third aspects.

[0010] Compared with the prior art, this application has the following advantages: The image enhancement method provided in this application involves acquiring an image to be enhanced and performing enhancement processing on the image using a pre-trained enhancement model to obtain enhancement features corresponding to each channel. These enhancement features for different channels represent the enhanced image from different dimensions. The method then determines the weights corresponding to the enhancement features of each channel. Since these weights represent the contribution of the enhancement features of the corresponding channel to the enhanced image (a higher weight indicates a greater contribution), the weights of the features of different channels can be dynamically adjusted according to the actual situation of the image and enhancement requirements. The determined weights are then used to weight the enhancement features of the corresponding channels to obtain weighted enhancement features for each channel. Finally, these weighted enhancement features are fused to obtain the enhanced image corresponding to the image to be enhanced. This method enables differentiated enhancement processing of features in different dimensions of the image. For example, channels containing important details are given higher weights to highlight their performance in the enhanced image, while channels with more noise or less information are given lower weights to reduce their negative impact on the overall image quality. This targeted weighted processing method allows the enhancement model to focus more on the key features of the image, effectively avoiding the problems that may occur in traditional uniform enhancement methods, such as the weakening of important features or the amplification of unimportant features. This significantly improves the accuracy and effectiveness of image enhancement and enhances the visual effect of the image. Attached Figure Description

[0011] Figure 1 This is a schematic diagram illustrating the application scenario of the solution provided in this application.

[0012] Figure 2 This is a flowchart illustrating an example of an image enhancement method provided in an embodiment of this application.

[0013] Figure 3 This is a schematic diagram of the model structure of the enhancement model in the image enhancement method provided in the embodiments of this application.

[0014] Figure 4 This is a flowchart illustrating another example of the image enhancement method provided in the embodiments of this application.

[0015] Figure 5 This is a schematic flowchart of an example of the training method for the augmented model provided in the embodiments of this application.

[0016] Figure 6 This is another flowchart illustrating the training method for the augmented model provided in the embodiments of this application.

[0017] Figure 7 This is a structural block diagram of an example of the image enhancement device provided in the embodiments of this application.

[0018] Figure 8 This is a structural block diagram of the electronic device provided in this application. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions of this application, the application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. However, this application can be implemented in many other ways different from those described below. Therefore, based on the embodiments provided in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0020] It should be noted that the terms "first," "second," "third," etc., in the claims, specification, and drawings of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. Such data are interchangeable where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown or described in this application. Furthermore, the terms "comprising," "having," and their variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0021] It should be understood that in the embodiments of this application, "at least one" means one or more, and "more than one" means two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the related objects before and after it are in an "or" relationship. "Contains A, B and / or C" means containing any one, two, or three of A, B, and C.

[0022] It should be understood that in the embodiments of this application, "B corresponding to A", "B corresponding to A", "A corresponds to B" or "B corresponds to A" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0023] To facilitate understanding of the various embodiments of this application, the application background of the embodiments will be explained.

[0024] With the rapid development of digital imaging technology, image compression coding technology has been widely used in image storage and transmission. However, after decoding compressed images, image quality degradation often occurs, such as blockiness, blurring, and noise artifacts. These quality degradation problems seriously affect the visual effect of the image and its subsequent applications.

[0025] In related technologies, image enhancement algorithms can be used to process decoded images to improve their quality. These technologies typically employ uniform enhancement, applying the same enhancement strategy to all image features. For example, all types of features (such as edges, textures, and smooth areas) are subjected to the same intensity of sharpening, denoising, or contrast adjustment. This uniform enhancement approach fails to consider the varying importance of different features within the image. For instance, features in different channels (such as luminance, chroma, or different frequency channels) typically contribute differently to the overall visual effect. Uniform enhancement struggles to accurately control these different channel features, potentially resulting in the weakening of important features (such as critical edges and texture information) while unimportant features (such as noise and artifacts) are unnecessarily amplified. This ultimately affects the overall effectiveness and practicality of image enhancement, thus impacting the overall enhancement outcome.

[0026] To address the above problems, embodiments of this application provide an image enhancement method, a video transmission method, a model training method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The aim is to enable targeted enhancement of features in different channels of an image, thereby achieving more precise and effective image enhancement, better avoiding the weakening of important features or the amplification of unimportant features, and improving the image enhancement effect.

[0027] The image enhancement method provided in this application can be applied to image enhancement in various professional fields. Specifically, it can be used to enhance compressed and decoded video frames during video transmission, and to enhance compressed and decoded images of ordinary pictures. In the field of video transmission, compressed video streams can reduce data volume and improve transmission efficiency during transmission, but decoded video frames may suffer from quality degradation. The image enhancement method of this application can enhance these video frames, making the video picture clearer and more natural, and improving the user's viewing experience. In terms of ordinary image processing, whether it is a personal photograph or an image downloaded from the internet, compression and subsequent decoding may result in blurring, blockiness, and other problems. The method of this application can also enhance these images, restoring the true details and colors of the image. It can also be applied to image enhancement in other fields, which are not specifically limited in this application.

[0028] To facilitate understanding of the method embodiments of this application, their application scenarios are described. Please refer to... Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of the solution provided in the embodiments of this application. This application scenario is merely an illustrative example and is not intended to limit the specific application scenario. Figure 1 As shown, in this application scenario, a server 102 and a client 101 are provided. In this embodiment, the client 101 and the server 102 establish a connection through network communication to transmit data.

[0029] Server 102 possesses high computing power. Server 102 can be a server, featuring high-speed central processing unit (CPU) computing power, long-term reliable operation, powerful input / output (I / O) external data throughput, and better scalability. Server 102 can be a single server or a server cluster. Server 102 is used to send compressed image or video streams to client 101. Server 102 can also provide other specific services to client 101, such as user information access, website access, and application access, which are not specifically limited in this application.

[0030] Client 101 can be an electronic device with display and data processing capabilities, such as a mobile phone, tablet, smartwatch, desktop computer, smart TV, VR device, in-vehicle device, wearable device, or laptop. Client 101 is used to receive compressed image or video streams sent by server 102 and decode them. After decoding, client 101 can use the image enhancement method provided in this application to enhance the decoded image or video frames. Client 101 can also be used to send access requests, interactive information, etc., to server 102, so that server 102 can send corresponding request data to client 101 for display.

[0031] The client 101 and the server 102 can communicate using various communication systems, such as wired or wireless communication systems.

[0032] In other application scenarios, server 102 can also obtain the image to be enhanced sent by client 101, enhance it using the image enhancement method provided in this application, and then return it to client 101. This approach is more suitable when the processing power of client 101 devices is limited, as it can fully utilize the powerful computing capabilities of server 102 to provide high-quality enhanced images to client 101. The solution provided in this application can also be applied to other application scenarios, and this application does not specifically limit it.

[0033] Example 1 The first embodiment of this application provides an image enhancement method that can be applied to electronic devices, such as servers, desktop computers, laptops, mobile phones, tablets, smartwatches, smart TVs, VR devices, in-vehicle devices, wearable devices, encoding terminals, decoding terminals, and other electronic devices with data processing functions.

[0034] like Figure 2 As shown, the image enhancement method provided in the first embodiment of this application includes the following steps S110 to S150.

[0035] Step S110: Obtain the image to be enhanced.

[0036] The image to be enhanced is an image or video frame obtained by decoding a compressed image or video stream after compression encoding. The compressed image or video stream is obtained by compressing and encoding the original image or original video frame.

[0037] The image to be enhanced can be any video frame from the video to be enhanced, or it can be an image to be enhanced. Electronic devices can decode compressed and encoded images to obtain the image to be enhanced, or they can receive the image to be enhanced transmitted from other devices after decoding.

[0038] Step S120: Enhance the image to be enhanced using a pre-trained enhancement model to obtain enhancement features corresponding to each channel. The enhancement features of different channels are used to represent the enhanced image from different dimensions.

[0039] In this embodiment, the enhancement model is used to perform multi-channel feature extraction and enhancement on the image to be enhanced, so as to obtain enhancement features of different dimensions, that is, enhancement features of different channels. Specifically, the enhancement model can generate an enhancement feature map corresponding to the image to be enhanced, which includes enhancement features corresponding to different channels.

[0040] The enhancement model can specifically decompose the image to be enhanced into multiple different channels. These channels can include, for example, chroma channels, luminance channels, frequency channels, edge feature channels, texture feature channels, smoothing region feature channels, etc., but are not limited to these. Those skilled in the art can specify which feature channels to include based on actual needs. Subsequently, the enhancement model performs enhancement processing on each decomposed channel separately. For example, for the luminance channel, contrast enhancement and detail enhancement can be performed; for the chroma channel, color fidelity preservation and noise suppression can be performed; for high-frequency channels (which typically contain edge and detail information of the image), sharpening enhancement can be performed; and for low-frequency channels (which typically contain the overall contour and luminance information of the image), smoothing and denoising can be performed. Through this independent enhancement processing of each channel, the features of each channel can be enhanced to suit its characteristics, thus obtaining the enhancement features corresponding to each channel. These enhancement features, from different dimensions, collectively constitute the latent representation of the enhanced image.

[0041] In one specific embodiment, the image to be enhanced can be input into the enhancement model to obtain the enhancement features corresponding to each channel. Alternatively, the image to be enhanced can be preprocessed (such as resizing, normalization, etc.) before being input into the enhancement model to obtain the enhancement features corresponding to each channel, so as to ensure that the input data meets the processing requirements of the enhancement unit.

[0042] The enhancement model can adopt deep learning model structures such as convolutional neural networks (CNN) and Transformer. Through the processing of multiple layers of the network, it gradually extracts the deep features of the image to be enhanced in different channels, enhances these features, and outputs the enhanced features of each channel.

[0043] The enhancement model is pre-trained, specifically through supervised training, semi-supervised training, or other training methods. For example, a training sample set can be obtained, containing a large number of original high-quality images and corresponding low-quality images that have undergone compression encoding and decoding, resulting in quality degradation. During training, the low-quality images are used as input to the enhancement model, and the original high-quality images are used as supervision signals. The model parameters are adjusted by calculating the difference between the enhanced image output by the model and the original high-quality image, thus obtaining the enhancement model. The training method of the enhancement model will be described in detail later in this application.

[0044] Step S130: Determine the weights corresponding to the enhancement features of each channel, whereby the weights represent the degree of contribution of the enhancement features of the corresponding channel to the enhanced image.

[0045] Specifically, channel weight rules can be pre-defined, including weight allocation schemes for different types of images. For example, for landscape images, the weight of high-frequency channels (texture, edges) can be higher than that of low-frequency channels; in night scene images, the weight of the luminance channel is higher than that of the chroma channel. Electronic devices can determine the weights of the enhancement features of each channel in the image to be enhanced according to the above channel weight rules. Alternatively, a weight prediction model can be trained, taking the enhancement features of each channel of the image to be enhanced and the image itself as input, and predicting the weights corresponding to the enhancement features of each channel. For example, the weight prediction model can be a lightweight neural network that evaluates the importance of the enhancement features of each channel and outputs the weight value of each channel. The weight prediction model can be optimized by training sample images in the training sample set and their corresponding manually labeled weights, or by joint training with the enhancement model, so that it can accurately predict the contribution of each channel enhancement feature in different scenes and different types of images to be enhanced. For the specific training process, please refer to the training of the enhancement unit in the following section.

[0046] Optionally, the weight of each channel enhancement feature can be determined based on at least one channel feature factor, such as the importance of the enhancement feature corresponding to the channel, feature sharpness, noise level, and correlation with the overall visual effect of the image. For example, high-frequency channel enhancement features containing key edge and texture information can be assigned higher weights because they have a greater impact on the image detail representation; while smooth region feature channels with relatively obvious noise, or certain chroma channels that contribute less to the overall visual effect, can be assigned lower weights, thereby highlighting the role of important feature channels and suppressing unimportant or easily disturbing channel features.

[0047] Step S140: Use the weights to weight the enhancement features of the corresponding channels to obtain the weighted enhancement features of each channel.

[0048] When weighting the enhancement features of each channel, the enhancement feature of each channel can be multiplied by its corresponding weight to obtain the weighted enhancement feature of that channel. For example, if the enhancement feature of a high-frequency channel is a matrix-like feature map with a weight of 0.8, then multiplying each element of the feature map by 0.8 will yield the weighted enhancement feature of that high-frequency channel. This weighting operation ensures that the enhancement features of channels with higher weights have a greater proportion in the subsequent synthesis of the enhanced image, thus better preserving and highlighting feature information that contributes significantly to the overall visual effect of the image. Conversely, the enhancement features of channels with lower weights have their influence weakened, preventing unimportant features or noise from being over-amplified.

[0049] Step S150: Perform fusion processing on each of the weighted enhancement features to obtain the enhanced image corresponding to the image to be enhanced.

[0050] Specifically, the weighted enhancement features of each channel of the image to be enhanced can be fused to obtain a fused feature. Based on this fused feature, the enhanced image corresponding to the image to be enhanced can be determined. The fusion process can employ various methods. For example, the weighted enhancement features of each channel can be added element-wise, or the weighted enhancement features can be integrated through a convolutional layer, or the weighted enhancement features of each channel can be concatenated along the channel dimension to obtain the fused feature. The enhanced image is then obtained based on the fused feature. Specifically, the fused feature can be upsampled or subjected to other mapping operations to convert it into an image of the same size as the image to be enhanced. For example, if the enhancement features of each channel output by the enhancement model are in the form of feature maps, after weighting, these weighted feature maps can be concatenated along the channel dimension, then feature fusion can be performed through one or more convolutional layers, and finally, the enhanced image can be generated through an output layer. This fusion method can make full use of the different dimensional information carried by the weighted enhancement features of each channel, and integrate them into a visually superior image. It effectively avoids the feature imbalance problem that may be caused by uniform enhancement, and makes the enhanced image more targeted in terms of detail, color, contrast and other aspects.

[0051] In one specific embodiment, step S150 can also be implemented by the following steps: fusing the weighted enhancement features to obtain fused features, and fusing the fused features with the image to be enhanced to obtain the enhanced image corresponding to the image to be enhanced. This preserves the intact and valid information in the image to be enhanced while introducing enhanced features, further improving the realism and naturalness of the enhanced image. For example, the enhanced image can be generated by element-wise addition or weighted addition of the fused features with the original features obtained from preprocessing (such as feature extraction) of the image to be enhanced. This can enhance image quality while reducing distortion caused by over-enhancement.

[0052] In one embodiment, the enhancement model may include an enhancement unit and a weight determination unit. The enhancement unit is used to perform multi-channel feature extraction and enhancement on the image to be enhanced, so as to obtain the enhancement features corresponding to each channel. The weight determination unit may be connected in series after the enhancement unit, and the weight determination unit is used to determine the weights corresponding to the enhancement features of each channel.

[0053] The augmentation unit can be a neural network structure, such as a feature processing layer containing convolutional layers, normalization layers, activation function layers, or other intelligent network structures. The weight determination unit can contain an attention mechanism module to assign corresponding weights to different channels. Through the collaborative work of the augmentation unit and the weight determination unit, the augmentation model can more accurately perform channel-specific augmentation and weight allocation on the image to be augmented, providing a foundation for subsequent weighted fusion and the generation of the augmented image.

[0054] Correspondingly, step S120 can be implemented as follows: step S121. Step S130 can be implemented as follows: step S131.

[0055] Step S121: Enhance the image to be enhanced by the enhancement unit of the pre-trained enhancement model to obtain the enhancement features corresponding to each channel of the image to be enhanced.

[0056] Specifically, the image to be enhanced can be input into the enhancement unit, or the image to be enhanced can be preprocessed such as normalization and size adjustment before being input into the enhancement unit. After the enhancement unit performs enhancement processing on the input information, it can obtain the enhancement features corresponding to each channel.

[0057] Step S131: Determine the weights corresponding to the enhancement features of each channel through the weight determination unit of the pre-trained enhancement model.

[0058] Specifically, the enhancement features of each channel can be input into the weight determination unit. The weight determination unit analyzes these enhancement features, for example, by using an attention mechanism to calculate the importance score of each channel's feature, and outputs this importance score as the weight corresponding to the enhancement feature of that channel. For instance, the weight determination unit analyzes the enhancement features of each channel, such as calculating the information entropy, gradient magnitude, and similarity to the original image of the feature map, to obtain the weight of each channel's enhancement feature. For example, enhancement features of high-frequency channels typically have larger gradient magnitudes, contain rich edge details, and have relatively high information entropy; therefore, they can be judged to contribute significantly to the image enhancement effect and are thus assigned higher weights. Conversely, for some channels with more noise, the stability and reliability of their features are poor, and their information entropy may be abnormal or their gradient magnitudes chaotic. The weight determination unit usually assigns them lower weights to avoid these noisy features negatively impacting the final enhanced image.

[0059] Alternatively, the weight determination unit can combine the image information of the image to be enhanced to determine the weights corresponding to the enhancement features of each channel, making the weight allocation more in line with the actual enhancement needs of the image. For example, when the image to be enhanced is a text image, the weight determination unit can identify the edge channel enhancement features of the text region and assign them higher weights to highlight the clarity of the text; when the image to be enhanced is a portrait, the relevant feature channels of the face region (such as the skin color channel and the facial feature edge channel) can be given higher weights to prioritize the enhancement effect of the subject.

[0060] In this embodiment, by incorporating an enhancement unit and a weight determination unit into the enhancement model, the image enhancement process becomes more integrated and intelligent. The enhancement unit extracts and enhances multi-channel features from the image to be enhanced, ensuring that image features of different dimensions are enhanced. The weight determination unit assigns corresponding weights to the enhancement features of each channel, achieving close collaboration between feature enhancement and weight allocation. This structural design enables feature enhancement and weight determination to be achieved simultaneously through a single enhancement model, simplifying the overall image enhancement process and reducing the complexity of model deployment and application. Furthermore, the enhancement unit and weight determination unit can be jointly optimized during the same training process, allowing the weight determination to better adapt to the enhancement features output by the enhancement unit, improving the accuracy of weight allocation and the overall coordination of the enhancement effect.

[0061] In one specific embodiment, such as Figure 3 As shown, the enhancement model may include multiple enhancement modules connected in series. Each enhancement module includes enhancement units and weight determination units connected in series. That is, enhancement units and weight determination units are set in series within each enhancement module, and the output of the previous enhancement module serves as the input of the next enhancement module.

[0062] Correspondingly, step S121 can obtain the enhancement features corresponding to each channel of the image to be enhanced by following step S121a.

[0063] Step S121a: For each enhancement module, determine the input data of the enhancement module based on the output data of the previous enhancement module, and perform enhancement processing on the input data through the enhancement unit of the enhancement module to obtain the enhancement features of each channel corresponding to the enhancement module. The input data of the first enhancement module is determined based on the image to be enhanced.

[0064] In other words, for each enhancement module of the enhancement model, the enhancement features of each channel of the image to be enhanced corresponding to that module will be obtained. The input data of the first enhancement module can be the image to be enhanced mentioned above, or it can be the data after preprocessing the image to be enhanced (such as resizing, normalization, etc.).

[0065] Steps S131 and S140 can be implemented as a whole by following steps S141 to S142.

[0066] Step S141: For each enhancement module, the weights corresponding to the enhancement features of each channel of the enhancement module are determined by the weight determination unit of the enhancement module. The corresponding enhancement features are weighted using the weights to obtain the weighted features of each channel of the enhancement module. The weighted enhancement features of each channel of the enhancement module are determined based on the weighted features.

[0067] The weighted enhancement features of each channel corresponding to the enhancement module are the output data of that enhancement module.

[0068] When determining the weighted enhancement features of each channel corresponding to the enhancement module based on the weighted features, the weighted features of each channel corresponding to the enhancement module can be determined as the weighted enhancement features of each channel corresponding to the enhancement module.

[0069] When determining the weighted enhancement features for each channel of an enhancement module based on the weighted features, the weighted enhancement features for each channel of the enhancement module can also be obtained by combining the weighted features of each channel with the input data of the enhancement module. Specifically, the weighted enhancement features for each channel of the enhancement module can be combined with the input data of the enhancement module to obtain the weighted enhancement features for each channel of the enhancement module. For example, the weighted features for each channel of the enhancement module can be weighted based on preset weight values ​​and then combined with the input data of the enhancement module to obtain the weighted enhancement features for each channel of the enhancement module. In this way, the basic information in the input data can be preserved while incorporating the enhanced features, allowing the subsequent enhancement module to optimize based on more complete information, avoiding the loss of the original image's basic structure due to over-enhancement, and improving the accuracy of the enhanced image.

[0070] For example, the weighted enhancement features of each channel corresponding to any enhancement module can be determined by the following formula (1).

[0071] (1) in, This represents the output data of the enhancement module, i.e., the weighted enhancement features of each channel; This represents the weighted features of each channel corresponding to the enhancement module, and is also the output data of the last enhancement sub-unit in the enhancement unit of the enhancement module; This represents the input data for the enhancement module; The preset weight value can be set, for example, to 0.2.

[0072] Step S142: Determine the weighted enhancement features of each channel corresponding to the last enhancement module as the weighted enhancement features of each channel corresponding to the image to be enhanced.

[0073] For example, suppose the enhancement model contains three enhancement modules connected in series. The input data for the first enhancement module is the preprocessed image data of the image to be enhanced. The enhancement unit of the first enhancement module processes the input data to obtain the enhancement features for each channel of the module. Then, the weight determination unit of the first enhancement module assigns corresponding weights to the enhancement features of each channel. The enhancement features of each channel are then multiplied by their corresponding weights to obtain weighted features. Based on these weighted features, the output data of the first enhancement module is obtained, which serves as the input data for the second enhancement module. This process continues until the output data of the third enhancement module is obtained, which is the weighted enhancement features for each channel of the last enhancement module. Through this multi-module cascade approach, image features can be finely adjusted in each round of enhancement, and the weight allocation can dynamically adapt to feature representations of different depths, resulting in a progressively improving enhancement effect and enhancing the layering, realism, and detail richness of the output enhanced image.

[0074] In one specific embodiment, such as Figure 3 As shown, the enhancement unit includes multiple enhancement sub-units connected in series. In step S121a, the enhancement features of each channel corresponding to the enhancement module can be obtained by following steps A to B.

[0075] Step A: For each enhancement subunit of the enhancement unit of the enhancement module, the output data of the previous enhancement subunit is determined as the input data of the enhancement subunit. The input data is enhanced by the enhancement subunit to obtain the enhancement features of each channel corresponding to the enhancement subunit. The input data of the first enhancement subunit is the output data of the previous enhancement module. Step B: Determine the enhancement features of each enhancement channel output by the last enhancement subunit of the enhancement unit as the enhancement features of each channel corresponding to the enhancement module.

[0076] like Figure 3 As shown, the enhancement subunit can include multiple convolutional layers connected in series. The convolutional layers of the enhancement subunit can be connected by dense residual connections to fuse the features of each convolutional layer. This allows each convolutional layer to receive the output features of the preceding convolutional layers, effectively alleviating the gradient vanishing problem in deep networks and enhancing the reusability of features.

[0077] This embodiment sets up multiple enhancement sub-units, enabling them to perform hierarchical and progressive feature enhancement processing on the input data. Each enhancement sub-unit can focus on a specific level or type of feature enhancement. This hierarchical enhancement approach allows each sub-unit to gradually improve the richness and representational power of the features, avoiding the problems of insufficient feature extraction or one-sided enhancement effects that occur when a single enhancement sub-unit processes complex images. Simultaneously, the cascaded structure of multiple enhancement sub-units allows for iterative optimization of features during the transfer process. The enhanced features output by the previous enhancement sub-unit can serve as the input for the next, allowing subsequent enhancement sub-units to perform deeper processing based on the enhanced features. This effectively improves the overall feature enhancement performance of the enhancement unit, providing more accurate multi-channel enhanced features for subsequent weight determination and weighted fusion.

[0078] In one specific embodiment, the weight determination unit may include a pooling module and a one-dimensional convolution module. The pooling module is used to perform dimensionality reduction processing on the enhancement features of each channel to extract a one-dimensional representation feature of the enhancement features of each channel. This one-dimensional representation feature can effectively summarize the overall information of the enhancement features of the corresponding channel.

[0079] The one-dimensional convolutional module is a one-dimensional convolutional structure. Its input is the one-dimensional representation feature output by the pooling module. By performing convolution operations in a single dimension, it can effectively capture the sequence correlation and local dependencies of the enhancement features of each channel in that dimension. The one-dimensional convolutional module is used to perform convolution operations on the channel statistics output by the pooling module to generate the weights corresponding to the enhancement features of each channel. The one-dimensional convolutional module can contain one or more convolutional kernels. By performing convolution operations on the pooled channel feature vectors, it captures the correlation between features of different channels and outputs weight values ​​equal to the number of channels. For example, if the pooling yields statistics for N channels, forming a 1×N vector, the one-dimensional convolutional module can use a 1×k convolutional kernel to convolve this vector. After processing by the activation function, it obtains N weight values, each weight value corresponding to the enhancement feature of one channel.

[0080] Correspondingly, step S131 above can be followed by steps S131a~S131b to determine the weights corresponding to the enhancement features of each channel.

[0081] Step S131a: The enhanced features of each channel are pooled by the pooling module of the weight determination unit of the enhancement model, so as to pool the enhanced features of each channel into one-dimensional features respectively.

[0082] like Figure 3As shown, if the enhancement features of each channel are feature maps of size H×W×C (where H is the height, W is the width, and C is the number of channels), the pooling module can use global average pooling, global max pooling, or other pooling methods to reduce the dimensionality of the H×W dimension features of each channel. For example, when using global average pooling, the average of all pixel values ​​in each channel is taken to obtain a 1×1×C feature vector, that is, each channel corresponds to a one-dimensional value, thus transforming the two-dimensional enhancement features of each channel into a one-dimensional representation feature. This dimensionality reduction method can effectively preserve the overall statistical information of the channel features while significantly reducing the amount of computation.

[0083] For example, based on Figure 3 For example, the pooling module can pool the enhanced feature maps of each channel into one-dimensional features using the following formula (2).

[0084] (2) in, Let represent the one-dimensional feature value of the c-th channel after global average pooling, where W and H represent the width and height of the feature map, respectively. This represents the pixel value at position (i,j) in the enhanced feature map of the c-th channel.

[0085] Step S131b: The weights of the one-dimensional features corresponding to each channel are determined by the one-dimensional convolution module of the weight determination unit of the enhancement model, and are used as the weights of the enhancement features of the corresponding channels.

[0086] Specifically, the one-dimensional features can be convolved using a one-dimensional convolution module to obtain the weights corresponding to the enhanced features of each channel. For example, ... Figure 3 As shown, the 1×1×C one-dimensional feature vector output by the pooling module can be input into the one-dimensional convolution module. The one-dimensional convolution module performs sliding convolution on this one-dimensional array using a convolution kernel to extract the dependencies between the enhanced features of different channels, and then passes them through an activation function. (Such as the Sigmoid function) maps the output values ​​to the range of 0 to 1, ultimately obtaining a number of weight values ​​equal to the number of channels C. Each weight value corresponds to the enhanced feature of one channel. Multiplying the weight values ​​by the enhanced features of the corresponding channels yields the weighted enhanced features for each channel.

[0087] The convolution sum of the above one-dimensional convolution module can be determined based on the number of channels corresponding to the enhanced features. Specifically, the number of convolution kernels can be positively correlated with the number of channels corresponding to the enhanced features, that is, the more channels there are, the more convolution kernels are required to capture the complex relationships between multi-channel features more fully. Specifically, the convolution kernel of the one-dimensional convolution can be determined by the following formula (3).

[0088] (3) in and These are hyperparameters; the actual selection can be... and , To select the nearest odd number, C represents the number of channels corresponding to the enhanced feature.

[0089] For example, the weights of the enhancement features of each channel can be determined by the following formula (4).

[0090] (4) in For activation functions, such as the Sigmoid function, Conv1D For one-dimensional convolution, the kernel size is calculated using formula (3). s is a weight vector containing C elements, where each element s c The weight values ​​for the enhanced features corresponding to the c-th channel. A simple one-dimensional convolution operation is performed on the 1×1×C one-dimensional features, and the output is also a 1×1×C feature.

[0091] This embodiment uses a weight determination method based on pooling and one-dimensional convolution. Pooling efficiently extracts global statistical information from each channel, while one-dimensional convolution captures the correlation between channels, resulting in a more reasonable weight allocation that accurately reflects the contribution of each channel's enhancement features to the final image enhancement effect. Furthermore, pooling the enhancement features into one-dimensional features and determining the weights through one-dimensional convolution, with the entire process performed in one-dimensional space, significantly reduces computational complexity and improves the efficiency of weight determination. This allows the enhancement model to meet real-time processing requirements while ensuring accurate weight allocation.

[0092] The image enhancement method provided in this application involves acquiring an image to be enhanced and performing enhancement processing on the image using a pre-trained enhancement model to obtain enhancement features corresponding to each channel. These enhancement features for different channels represent the enhanced image from different dimensions. The method then determines the weights corresponding to the enhancement features of each channel. Since these weights represent the contribution of the enhancement features of the corresponding channel to the enhanced image (a higher weight indicates a greater contribution), the weights of the features of different channels can be dynamically adjusted according to the actual situation of the image and enhancement requirements. The determined weights are then used to weight the enhancement features of the corresponding channels to obtain weighted enhancement features for each channel. Finally, these weighted enhancement features are fused to obtain the enhanced image corresponding to the image to be enhanced. This method enables differentiated enhancement processing of features in different dimensions of the image. For example, channels containing important details are given higher weights to highlight their performance in the enhanced image, while channels with more noise or less information are given lower weights to reduce their negative impact on the overall image quality. This targeted weighted processing method allows the enhancement model to focus more on the key features of the image, effectively avoiding the problems that may occur in traditional uniform enhancement methods, such as the weakening of important features or the amplification of unimportant features. This significantly improves the accuracy and effectiveness of image enhancement and enhances the visual effect of the image.

[0093] In one embodiment, prior to step S121, the image enhancement method may further include the following steps S120a to S120b.

[0094] Step S120a: Obtain the compression quantization parameters corresponding to the image to be enhanced.

[0095] The quantization parameter (QP) is a parameter used in video encoding to control the compression quality. The larger the QP value, the stronger the quantization and the higher the data compression rate, but the more severe the loss of image quality. Conversely, the smaller the QP value, the better the image quality is preserved, but the larger the file size.

[0096] In this embodiment, the image to be enhanced is a video frame to be enhanced. The electronic device can parse the QP value corresponding to the video frame to be enhanced from the video bitstream.

[0097] Step S120b: Select a target augmentation model from the pre-trained augmentation models that matches the parameter range to which the compression quantization parameters belong, wherein each augmentation model has its own applicable compression parameter range, and the augmentation model is trained based on sample data corresponding to the applicable compression parameter range.

[0098] Correspondingly, step S121 can be implemented by the following steps: enhancing the image to be enhanced by the enhancement unit of the target enhancement model to obtain the enhancement features corresponding to each channel of the image to be enhanced.

[0099] Step S131 can be implemented by the following steps: determining the weights corresponding to the enhancement features of each channel through the weight determination unit of the target enhancement model.

[0100] In other words, this embodiment trains targeted enhancement models for different QP parameter ranges. For example, QP values ​​can be divided into low QP ranges (e.g., 0-15), medium QP ranges (e.g., 16-30), and high QP ranges (e.g., 31-51), with each parameter range corresponding to an independently trained enhancement model. During training, these models use distorted images caused by compression in the corresponding QP range as input samples and the original uncompressed image as a label for supervised learning. In this way, each enhancement model can focus on learning the distortion features under a specific QP range, so that during inference, the electronic device can call the matching target enhancement model based on the parameter range to which the QP value of the video frame to be enhanced belongs. This method of adaptively selecting models based on QP parameters avoids the problem of insufficient generalization ability of a single model when dealing with different degrees of distortion, enabling the enhancement model to more accurately optimize for the compression distortion type and degree of the current video frame, further improving the quality and realism of the enhanced image.

[0101] In one embodiment, the enhancement model may include a luminance enhancement sub-model and a chrominance enhancement sub-model. Correspondingly, step S121 can be implemented according to the following steps S121b~S121c.

[0102] The specific model structures of the luminance enhancement sub-model and the chrominance enhancement sub-model are the same as those of the enhancement model described above, and will not be detailed here. The luminance enhancement sub-model is used to enhance the luminance channel features of the image, while the chrominance enhancement sub-model is used to enhance the chrominance channel features of the image.

[0103] Step S121b: The brightness component map of the image to be enhanced is enhanced by the enhancement unit of the brightness enhancement sub-model of the pre-trained enhancement model to obtain the enhanced brightness features corresponding to each channel.

[0104] The image to be enhanced includes a luminance component map and a chrominance component map. The luminance component map (such as the Y channel) is used to reflect the brightness and darkness information of the image, while the chrominance component map (such as the Cb and Cr channels) is used to characterize the color information of the image.

[0105] Step S121c: The chroma component map of the image to be enhanced is enhanced by the enhancement unit of the chroma enhancement sub-model of the enhancement model to obtain the enhanced chroma features corresponding to each channel.

[0106] The above step S131 can be followed by the steps S131c~S131d to determine the weights corresponding to the enhancement features of each channel.

[0107] Step S131c: The weight determination unit of the brightness enhancement sub-model determines the brightness weight corresponding to the brightness enhancement feature of each channel.

[0108] Step S131d: Determine the chromaticity weights corresponding to the enhanced chromaticity features of each channel through the weight determination unit of the chromaticity enhancement sub-model.

[0109] When enhancing the luminance and chrominance component maps, the methods used by the luminance enhancement sub-model to enhance the luminance classification map and by the chrominance enhancement sub-model to enhance the chrominance component map are consistent with the overall image enhancement process of the aforementioned enhancement model. For example, both methods involve layered and progressive feature enhancement through their respective enhancement units, followed by weight determination units that determine the weights of the enhanced features for each channel and perform weighted fusion. The specific enhancement process will not be repeated here.

[0110] Step S140 can be implemented by following steps S143~S144.

[0111] Step S143: Based on the brightness weight, the enhanced brightness features of the corresponding channels are weighted to obtain the weighted brightness features of each channel.

[0112] Step S144: Based on the chromaticity weight, the enhanced chromaticity features of the corresponding channels are weighted to obtain the weighted chromaticity features of each channel.

[0113] Step S150 can be performed by following steps S151 to S153 to determine the enhanced image corresponding to the image to be enhanced.

[0114] Step S151: Perform fusion processing on each of the weighted brightness features to obtain the enhanced brightness component map corresponding to the image to be enhanced.

[0115] Step S152: Perform fusion processing on each of the weighted chromaticity features to obtain the enhanced chromaticity component map corresponding to the image to be enhanced.

[0116] The execution process of steps S151 to S152 can be referred to step S150 above, and will not be described in detail here.

[0117] Step S153: Fuse the enhanced luminance component map and the enhanced chrominance component map to obtain the enhanced image corresponding to the image to be enhanced.

[0118] Specifically, the enhanced luminance component map and the enhanced chrominance component map can be merged into a complete enhanced image.

[0119] This implementation method, which involves processing and then fusing different channels, can perform refined enhancements on the different characteristics of brightness and chroma, avoiding mutual interference between brightness and chroma features, and further improving the overall effect of image enhancement, as well as color reproduction and detail clarity.

[0120] In one specific embodiment, when the image to be enhanced is a video frame to be enhanced, the following steps S120c to S120d may be included before step S121.

[0121] Step S120c: Determine the first encoding process information related to image brightness corresponding to the video frame to be enhanced during the encoding or decoding process.

[0122] Step S120d: Determine the second encoding process information related to image chroma corresponding to the video frame to be enhanced during the encoding or decoding process.

[0123] Because a simulated decoding and reconstruction process is performed during the encoding process, and this simulated decoding and reconstruction process is consistent with the decoding process at the decoding end, both the encoding and decoding processes generate encoding process information, and the encoding process information generated during the encoding and decoding processes is the same.

[0124] The aforementioned first encoding process information is related to luminance, that is, the first encoding process information can reflect the processing and distortion of the luminance component in the encoding or decoding process. Specifically, the first encoding process information may include at least one of the following in the encoding or decoding process: the prediction frame, the frame before loop filtering, the frame after loop filtering, the coding unit (CU) partition map, the CU depth map, and the filter intensity map corresponding to the video frame to be enhanced.

[0125] Among them, the prediction frame records the motion compensation information between the current frame and the reference frame, which can reflect the motion distortion of the luminance component in the time domain; the frame before and after the loop filter can reflect the impact of the loop filter operation on the smoothness of the luminance component; the CU partition map and CU depth map show the spatial partitioning strategy of the luminance component during encoding; and the filter intensity map can reflect the processing intensity of the luminance component in the loop filter process.

[0126] The aforementioned second encoding process information is related to chroma, meaning it reflects the processing and distortion experienced by the chroma components during encoding or decoding. Specifically, the second encoding process information may include at least one of the following: the prediction frame, the frame before loop filtering, the frame after loop filtering, and the filter intensity map corresponding to the video frame to be enhanced during encoding or decoding. The prediction frame reflects the motion compensation result of color information in the time domain; the frames before and after loop filtering demonstrate the adjustment of color smoothness by the filtering operation; and the filter intensity map reflects the processing intensity of the chroma components in the loop filtering.

[0127] Correspondingly, step S121b above can obtain the enhanced brightness features corresponding to each channel by following these steps: based on the information from the first encoding process, and through the enhancement unit of the brightness enhancement sub-model of the pre-trained enhancement model, the brightness component map of the image to be enhanced is enhanced to obtain the enhanced brightness features corresponding to each channel.

[0128] The above step S121c can obtain the enhanced chroma features corresponding to each channel according to the following steps: based on the second encoding process information, and through the enhancement unit of the chroma enhancement sub-model of the enhancement model, the chroma component map of the image to be enhanced is enhanced to obtain the enhanced chroma features corresponding to each channel.

[0129] Specifically, the information from the first encoding process and the luminance component map of the image to be enhanced can be concatenated and input into the enhancement unit of the luminance enhancement sub-model. This allows the enhancement unit to learn the features of the luminance component map itself while combining the luminance distortion information indicated by the first encoding process information, thus more accurately capturing and repairing luminance-related distortions. Similarly, the information from the second encoding process and the chrominance component map of the image to be enhanced can be concatenated and input into the enhancement unit of the chrominance enhancement sub-model. This allows the enhancement unit to utilize the chrominance distortion information indicated by the second encoding process information to perform targeted enhancement of the chrominance component map.

[0130] This embodiment integrates encoding process information into the feature processing of the enhancement model, enabling the model to gain a deeper understanding of the sources and characteristics of image distortion. This allows for more targeted and effective enhancement, further improving the quality of the enhanced image. Furthermore, this application determines first and second encoding process information for luminance and chrominance respectively, allowing the luminance enhancement sub-model and chrominance enhancement sub-model to utilize their respective encoding process information for enhancement processing. This avoids cross-interference between luminance and chrominance encoding process information, further improving the accuracy of luminance and chrominance component enhancement.

[0131] The image enhancement method of this application is described below by way of example, such as Figure 4As shown, the image enhancement method in this example includes the following steps a~f.

[0132] Step a: The decoding end acquires the reconstructed frame as the image to be enhanced, and acquires the encoding (or decoding) process information generated during the decoding process of the reconstructed frame.

[0133] Step b: Select the target enhancement model for the parameter range to which the image to be enhanced belongs based on the compression parameters of the image to be enhanced.

[0134] For example, such as Figure 4 As shown, the target augmentation model can be selected from the augmentation models of 7 parameter segments.

[0135] Step c: Divide the image to be enhanced into a luminance component (Y channel) map and a chrominance component (UV channel) map.

[0136] Step d: Invoke the target augmentation model, which includes a luma sub-model and a chroma sub-model.

[0137] Step e: Perform feature preprocessing on the luminance component map and the first encoding process information, and perform feature preprocessing on the chrominance component map and the second encoding process information.

[0138] The first encoding process information includes the predicted frame corresponding to the reconstructed frame, the frame before loop filtering, the frame after loop filtering, the coding unit (CU) partitioning map, the CU depth map, and the filter intensity map. The second encoding process information includes the predicted frame corresponding to the reconstructed frame, the frame before loop filtering, the frame after loop filtering, and the filter intensity map. Feature preprocessing may include operations such as feature alignment, dimensionality unification, and concatenation.

[0139] Step f: Input the preprocessed data into the corresponding sub-models for image enhancement to obtain the enhanced image.

[0140] The specific process of each step in this example can be found in the detailed description above, and will not be repeated here.

[0141] Example 2 The second embodiment of this application also provides a video transmission method, which is applied to a video transmission system including a transmitting end and a receiving end, and the method includes the following steps S210~S220.

[0142] Step S210: The sending end compresses and encodes each video frame of the video to be sent to obtain a compressed bitstream, and sends the compressed bitstream to the receiving end.

[0143] When performing compression encoding, the sending end can use relevant video coding standards, such as H.264, H.265, AV1, etc. The sending end can choose the appropriate coding standard to compress video frames according to specific circumstances.

[0144] Step S220: The receiving end receives the compressed bitstream, decodes the compressed bitstream to obtain the video frame to be enhanced, and uses the image enhancement method described in any one of the first embodiments to enhance the video frame to be enhanced, thereby obtaining the enhanced video frame.

[0145] The receiving end can use the same decoding standard as the sending end during encoding to correctly reconstruct the video frame. Processing the decoded video frame to be enhanced according to the image enhancement method of the first embodiment can effectively improve the quality problems caused by compression encoding, such as reducing blur and eliminating artifacts. The specific enhancement process can be found in the first embodiment and will not be detailed here.

[0146] Example 3 The third embodiment of this application also provides a method for training an enhanced model. This method is applied to an electronic device, which may be a server, desktop computer, laptop computer, mobile phone, tablet computer, smartwatch, smart TV, VR device, in-vehicle device, wearable device, or other electronic device with data processing capabilities. Figure 5 As shown, the training method includes the following steps S310 to S360.

[0147] Step S310: Obtain training samples, which include the sample image to be enhanced and the corresponding original sample image.

[0148] Training samples can be pre-stored samples, samples downloaded from the network, or samples received from other devices, etc., and this application does not specifically limit them.

[0149] The original sample image refers to the original image that has not undergone compression encoding. The sample image to be enhanced is the image obtained by compressing and encoding the original sample image and then decoding it.

[0150] Step S320: Determine the input information of the first training model based on the sample image to be enhanced, and input the input information into the training enhancement unit of the first training model to obtain the sample enhancement features corresponding to each channel.

[0151] Step S330: Determine the sample weights corresponding to the sample enhancement features of each channel through the training weight determination unit of the first training model.

[0152] Step S340: Use the sample weights to weight the sample enhancement features of the corresponding channels to obtain the weighted enhancement features of each channel.

[0153] Step S350: Perform fusion processing on the weighted enhancement features of each sample to obtain the enhanced image of the sample corresponding to the image to be enhanced.

[0154] The execution process of step S320 is similar to that of step S121 in the first embodiment, the execution process of step S330 is similar to that of step S131 in the first embodiment, the execution process of step S340 is similar to that of step S140 in the first embodiment, and the execution process of step S350 is similar to that of step S150 in the first embodiment. The specific process can be referred to the first embodiment, and will not be described in detail here.

[0155] Step S360: Based on the difference between the enhanced image of the sample and the original image of the sample, adjust the model parameters of the first model to be trained to obtain the trained enhanced model.

[0156] Specifically, the difference between the augmented image and the original image can be measured by calculating the loss function value. Common loss functions include the mean squared error loss (MSE) and the structural similarity index loss (SSIM). Taking the mean squared error loss function as an example, it calculates the average of the sum of squares of the differences between corresponding pixel values ​​in the augmented image and the original image. A larger loss function value indicates a greater difference between the augmented image and the original image, indicating poor model performance; conversely, a smaller loss function value indicates better model performance.

[0157] After obtaining the loss function value, optimization algorithms such as gradient descent can be used to adjust the parameters of each unit in the first model to be trained. The gradient descent algorithm calculates the gradient of the loss function with respect to the model parameters and updates the model parameters in the opposite direction of the gradient, so that the loss function value gradually decreases.

[0158] After multiple iterations of training, the model parameters are continuously adjusted until the model meets the preset training conditions, such as the loss function value being less than a preset threshold or the number of training rounds reaching the upper limit. The model obtained at this point is the enhanced model after training.

[0159] In one embodiment, prior to step S320, the training method may further include the following steps S320a to S320c.

[0160] Step S320a: Determine the input information of the second training model based on the sample image to be enhanced, and input the input information into the second training model to obtain the second sample enhanced image. The second training model includes training enhancement units.

[0161] The second model to be trained does not contain a weight determination unit, but only a boosting unit. The execution process of step S320a is similar to that of step S320, and will not be described in detail here.

[0162] Step S320b: Based on the difference between the second sample enhanced image and the corresponding original sample image, adjust the parameters of the second training model to obtain the enhancement model to be optimized, wherein the enhancement model to be optimized includes enhancement units to be optimized.

[0163] Step S320c: Embed a weight determination unit after the enhancement unit of the model to be optimized to obtain the first model to be trained.

[0164] This implementation first trains a second model containing only augmentation units. Then, it adjusts parameters using the difference between the augmented image and the original image to obtain an optimized augmentation model containing only augmentation units. Finally, it embeds a weight determination unit after the optimized augmentation units to form a first model. This allows the augmentation units to focus on feature extraction and basic augmentation capabilities during the initial training, laying a solid foundation for the subsequent integration of the weight determination unit, thereby improving the overall training efficiency and the performance of the final augmented model. This phased training approach, optimizing basic augmentation functions first and then introducing a weight mechanism for joint optimization, helps the model converge more stably to a better parameter space. Furthermore, since the first stage has already adjusted the parameters of the augmentation units to possess a certain level of basic augmentation capability, after embedding the weight determination unit in the second stage, the model can further learn how to dynamically allocate the weights of the augmentation features in each channel, building upon its existing good feature extraction capabilities. This results in faster convergence and higher training efficiency in the second stage.

[0165] In one specific embodiment, the training method can be used to train augmentation models corresponding to different compression parameter segments. Correspondingly, the above step S310 can be implemented according to the following step S311.

[0166] Step S311: Obtain each group of training samples corresponding to different compression parameter segments. Each group of training samples includes the sample image to be enhanced and the corresponding original sample image. The compression parameters corresponding to the sample image to be enhanced in each group of training samples are located within the compression parameter segment corresponding to that group of training samples.

[0167] Correspondingly, for each compression parameter segment, steps S320~S360 are performed using training samples corresponding to that compression parameter segment to obtain an augmented model that matches each compression parameter segment.

[0168] This embodiment can train enhancement models corresponding to each parameter segment according to the characteristics of different compression parameter segments, so that each enhancement model can more accurately adapt to the image distortion situation within its corresponding compression parameter range.

[0169] In one specific embodiment, step S320 can obtain the sample enhancement features corresponding to each channel according to the following steps S321~S322.

[0170] Step S321: Determine the brightness input information of the first brightness model to be trained based on the sample brightness component image of the sample image to be enhanced, and input the brightness input information into the brightness enhancement unit of the first brightness model to be trained to obtain the sample brightness enhancement features corresponding to each channel.

[0171] Step S322: Determine the chromaticity input information of the first chromaticity model to be trained based on the sample chromaticity component image of the sample image to be enhanced, and input the chromaticity input information into the chromaticity enhancement unit of the first chromaticity model to be trained, to obtain the sample chromaticity enhancement features corresponding to each channel.

[0172] Step S330 can be performed by determining the sample weights corresponding to the sample enhancement features of each channel according to the following steps S331~S332.

[0173] Step S331: Determine the sample brightness weights corresponding to the sample brightness enhancement features of each channel through the training brightness weight determination unit of the first training brightness model.

[0174] Step S332: Determine the sample chromaticity weights corresponding to the sample chromaticity enhancement features of each channel through the chromaticity weight determination unit of the first chromaticity model to be trained.

[0175] Step S340 can be performed by following steps S341~S342 to obtain the weighted enhanced features of the samples corresponding to each channel.

[0176] Step S341: Based on the sample brightness weight, the sample brightness enhancement features of the corresponding channel are weighted to obtain the sample weighted brightness features of each channel.

[0177] Step S342: Based on the sample chromaticity weight, the sample chromaticity enhancement features of the corresponding channels are weighted to obtain the sample weighted chromaticity features corresponding to each channel.

[0178] Step S350 can be performed by following steps S351~S352 to determine the sample enhanced image corresponding to the sample image to be enhanced.

[0179] Step S351: Perform fusion processing on the weighted brightness features of each sample to obtain the enhanced brightness component image of the sample image to be enhanced.

[0180] Step S352: Perform fusion processing on the weighted chromaticity features of each sample to obtain the enhanced chromaticity component image of the sample image to be enhanced.

[0181] Step S360 can be obtained by following the steps S361~S363 to obtain the trained augmented model.

[0182] Step S361: Based on the difference between the enhanced luminance component image of the sample and the original luminance component image of the original sample image, adjust the model parameters of the first luminance model to be trained to obtain the trained luminance enhancement sub-model.

[0183] Step S362: Based on the difference between the enhanced chroma component image and the original chroma component image of the original sample image, adjust the model parameters of the first chroma model to be trained to obtain the trained chroma enhancement sub-model.

[0184] Step S363: Combine the trained luminance enhancement sub-model and the trained chrominance enhancement sub-model to obtain the trained enhancement model.

[0185] The training method described in this embodiment allows for the construction of independent training processes for the luminance and chrominance components. This enables the luminance enhancement sub-model and the chrominance enhancement sub-model to focus on learning the enhancement rules and weight allocation strategies for their respective components. This training method avoids mutual interference between luminance and chrominance features during training and allows for more precise optimization of their respective model parameters. The final combined enhancement model, when processing image enhancement tasks, can perform targeted enhancement processing on the luminance and chrominance components through each sub-model, thereby further improving the overall quality of the enhanced image.

[0186] The training method of the augmented model of this application is described below by way of example. Figure 6 As shown, the model training method in this example includes the following steps S10~S50.

[0187] Step S10: Obtain training samples, which include the sample image to be enhanced, the original sample image, and the sample encoding process information corresponding to the sample image to be enhanced.

[0188] The sample encoding process information includes the sample prediction frame generated during the compression encoding process of the original sample image (or generated during the decoding process to obtain the sample image to be enhanced), the sample loop filtering pre-frame, the sample loop filtering post-frame, the sample CU partition map, the sample CU depth map, and the sample filtering intensity map.

[0189] Step S20: The sample image to be enhanced and the information of each first encoding process are concatenated in the feature channel to obtain the brightness input information of the brightness sub-model to be trained.

[0190] Step S30: Concatenate the sample image to be enhanced and the information of each second encoding process in the feature channel to obtain the chromaticity input information of the chromaticity sub-model to be trained.

[0191] The first encoding process information includes sample prediction frames, sample loop filtering pre-frames, sample loop filtering post-frames, sample CU partitioning map, sample CU depth map, and sample filtering intensity map; the second encoding process information includes sample prediction frames, sample loop filtering pre-frames, and sample loop filtering post-frames.

[0192] Step S40: For any quantization parameter segment, input the brightness input information corresponding to the quantization parameter segment into the brightness sub-model to be trained to train the model and obtain the brightness enhancement sub-model corresponding to the parameter segment.

[0193] Step S50: For any quantization parameter segment, input the chromaticity input information corresponding to the quantization parameter segment into the chromaticity sub-model to be trained to train the model and obtain the chromaticity enhancement sub-model corresponding to the parameter segment.

[0194] Example 4 The fourth embodiment of this application also provides an image enhancement apparatus corresponding to the image enhancement method embodiment provided in the first embodiment. Since the apparatus embodiment is basically similar to the method embodiment, it is described simply. For details of the relevant technical features and their effects, please refer to the corresponding descriptions of the image enhancement method embodiments provided above. Figure 7 As shown, the image enhancement device provided in this embodiment includes: Image acquisition module 410 is used to acquire the image to be enhanced; The feature enhancement module 420 is used to enhance the image to be enhanced by a pre-trained enhancement model to obtain enhancement features corresponding to each channel. The enhancement features of different channels are used to represent the enhanced image from different dimensions. The weight determination module 430 is used to determine the weights corresponding to the enhancement features of each channel, wherein the weights are used to represent the degree of contribution of the enhancement features of the corresponding channel to the enhanced image. The weighted processing unit 440 is used to perform weighted processing on the enhancement features of the corresponding channels using the weights to obtain the weighted enhancement features corresponding to each channel; The image generation unit 450 is used to perform fusion processing on each of the weighted enhancement features to obtain the enhanced image corresponding to the image to be enhanced.

[0195] Example 5 The fifth embodiment of this application also provides an electronic device embodiment corresponding to the image enhancement method provided in the first embodiment. The following description of the electronic device embodiment is merely illustrative. The electronic device embodiment is as follows: Please refer to Figure 8 Understanding the above electronic devices, Figure 8 This is a schematic diagram of an electronic device. The electronic device provided in this embodiment includes: a processor 1001, a memory 1002, a communication bus 1003, and a communication interface 1004; The memory 1002 is used to store computer instructions for data processing. When these computer instructions are read and executed by the processor 1001, the following steps are performed: Obtain the image to be enhanced; The image to be enhanced is enhanced by a pre-trained enhancement model to obtain enhancement features corresponding to each channel. The enhancement features of different channels are used to represent the enhanced image from different dimensions. Determine the weights corresponding to the enhancement features of each channel, whereby the weights represent the degree of contribution of the enhancement features of the corresponding channel to the enhanced image. The enhanced features of the corresponding channels are weighted using the weights to obtain the weighted enhanced features for each channel; The weighted enhancement features are fused to obtain the enhanced image corresponding to the image to be enhanced.

[0196] The sixth embodiment of this application also provides a computer-readable storage medium for implementing the methods of any one of the first to third embodiments. The embodiments of the computer-readable storage medium provided in this application are described in a relatively simple manner; relevant parts can be found in the corresponding descriptions of the above method embodiments. The embodiments described below are merely illustrative.

[0197] The computer-readable storage medium provided in this embodiment stores computer instructions, which, when executed by a processor, implement the steps of any one of the first to third embodiments.

[0198] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0199] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0200] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined in this application, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0201] 2. Those skilled in the art will understand that embodiments of this application can provide methods, systems, or computer program products. Therefore, embodiments of this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0202] 3. This application embodiment may involve the use of user data. In practical applications, user-specific personal data may be used within the scope permitted by applicable laws and regulations of the country in which the application is located (e.g., with the user's explicit consent and effective notification to the user, etc.). Furthermore, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0203] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. An image enhancement method, characterized in that, The method includes: Obtain the image to be enhanced; The image to be enhanced is enhanced by a pre-trained enhancement model to obtain enhancement features corresponding to each channel. The enhancement features of different channels are used to represent the enhanced image from different dimensions. Determine the weights corresponding to the enhancement features of each channel, whereby the weights represent the degree of contribution of the enhancement features of the corresponding channel to the enhanced image. The enhanced features of the corresponding channels are weighted using the weights to obtain the weighted enhanced features for each channel; The weighted enhancement features are fused to obtain the enhanced image corresponding to the image to be enhanced.

2. The image enhancement method according to claim 1, characterized in that, The enhancement model includes an enhancement unit and a weight determination unit; the enhancement process of the image to be enhanced using the pre-trained enhancement model includes: The image to be enhanced is enhanced using the enhancement units of a pre-trained enhancement model; Determining the weights corresponding to the enhancement features of each channel includes: The weight determination unit determines the weights corresponding to the enhancement features of each channel.

3. The image enhancement method according to claim 2, characterized in that, The enhancement model includes multiple enhancement modules connected in series, and each enhancement module includes the enhancement unit and the weight determination unit connected in series. The enhancement unit of the pre-trained enhancement model performs enhancement processing on the image to be enhanced, obtaining enhancement features corresponding to each channel, including: For each enhancement module, the input data of the enhancement module is determined based on the output data of the previous enhancement module. The enhancement unit of the enhancement module performs enhancement processing on the input data to obtain the enhancement features of each channel corresponding to the enhancement module. The input data of the first enhancement module is determined based on the image to be enhanced. The step of determining the weights corresponding to the enhancement features of each channel through the weight determination unit of the enhancement model, and then using the weights to perform weighted processing on the enhancement features of the corresponding channels to obtain the weighted enhancement features corresponding to each channel includes: For each enhancement module, the weights corresponding to the enhancement features of each channel of the enhancement module are determined by the weight determination unit of the enhancement module. The corresponding enhancement features are weighted using the weights to obtain the weighted features of each channel of the enhancement module. The weighted enhancement features of each channel of the enhancement module are then determined based on the weighted features. The weighted enhancement features of each channel corresponding to the last enhancement module are determined as the weighted enhancement features of each channel corresponding to the image to be enhanced.

4. The image enhancement method according to claim 3, characterized in that, The enhancement unit includes multiple enhancement sub-units connected in series; The enhancement unit of the enhancement module performs enhancement processing on the input data to obtain the enhancement features of each channel corresponding to the enhancement module, including: For each enhancement subunit of the enhancement unit of the enhancement module, the output data of the previous enhancement subunit is determined as the input data of the enhancement subunit. The input data is enhanced by the enhancement subunit to obtain the enhancement features of each channel corresponding to the enhancement subunit. The input data of the first enhancement subunit is the output data of the previous enhancement module. The enhancement features of each enhancement channel output by the last enhancement subunit of the enhancement unit are determined as the enhancement features of each channel corresponding to the enhancement module.

5. The image enhancement method according to claim 3, characterized in that, The step of determining the weighted enhancement features of each channel corresponding to the enhancement module based on the weighted features includes: Based on the weighted features and combined with the input data of the enhancement module, the weighted enhancement features of each channel corresponding to the enhancement module are obtained.

6. The image enhancement method according to claim 1, characterized in that, The step of fusing the weighted enhancement features to obtain the enhanced image corresponding to the image to be enhanced includes: The weighted enhanced features are then fused to obtain the fused features. The fusion feature is fused with the image to be enhanced to obtain the enhanced image corresponding to the image to be enhanced.

7. The image enhancement method according to claim 2, characterized in that, The weight determination unit includes a pooling module and a one-dimensional convolution module; The step of determining the weights corresponding to the enhancement features of each channel by the weight determination unit of the enhancement model includes: The pooling module of the weight determination unit of the enhancement model pools the enhancement features of each channel to pool the enhancement features of each channel into one-dimensional features. The weights of the one-dimensional features corresponding to each channel are determined by the one-dimensional convolution module, and these weights are used as the weights of the enhanced features of the corresponding channels.

8. The image enhancement method according to claim 2, characterized in that, Before the enhancement processing of the image to be enhanced is performed by the enhancement unit of the pre-trained enhancement model, the method further includes: Obtain the compression quantization parameters corresponding to the image to be enhanced; Select a target augmentation model from the pre-trained augmentation models that matches the parameter range to which the compression quantization parameters belong, wherein each augmentation model has its own applicable compression parameter range, and the augmentation model is trained based on sample data corresponding to the applicable compression parameter range; The enhancement process performed on the image to be enhanced by the enhancement unit of the pre-trained enhancement model includes: The image to be enhanced is enhanced by the enhancement unit of the target enhancement model; The step of determining the weights corresponding to the enhancement features of each channel by the weight determination unit includes: The weights corresponding to the enhancement features of each channel are determined by the weight determination unit of the target enhancement model.

9. The image enhancement method according to claim 2, characterized in that, The enhancement model includes a luminance enhancement sub-model and a chrominance enhancement sub-model; The enhancement unit of the pre-trained enhancement model performs enhancement processing on the image to be enhanced, obtaining enhancement features corresponding to each channel, including: The brightness component map of the image to be enhanced is enhanced by the enhancement unit of the brightness enhancement sub-model of the pre-trained enhancement model, so as to obtain the enhanced brightness features corresponding to each channel. The enhancement unit of the chroma enhancement sub-model of the enhancement model enhances the chroma component map of the image to be enhanced, and obtains the enhanced chroma features corresponding to each channel. The step of determining the weights corresponding to the enhancement features of each channel by the weight determination unit of the enhancement model includes: The brightness weights corresponding to the enhanced brightness features of each channel are determined by the weight determination unit of the brightness enhancement sub-model. The weight determination unit of the chroma enhancement sub-model determines the chroma weights corresponding to the enhanced chroma features of each channel. The step of weighting the enhanced features of the corresponding channels using the weights to obtain the weighted enhanced features for each channel includes: The enhanced brightness features of the corresponding channels are weighted based on the brightness weights to obtain the weighted brightness features of each channel. The enhanced chromaticity features of the corresponding channels are weighted based on the chromaticity weights to obtain the weighted chromaticity features of each channel. The step of fusing the weighted enhancement features to obtain the enhanced image corresponding to the image to be enhanced includes: The weighted brightness features are fused to obtain the enhanced brightness component map corresponding to the image to be enhanced. The weighted chromaticity features are fused to obtain the enhanced chromaticity component map corresponding to the image to be enhanced. The enhanced luminance component image and the enhanced chrominance component image are fused to obtain the enhanced image corresponding to the image to be enhanced.

10. The image enhancement method according to claim 9, characterized in that, The image to be enhanced is a video frame to be enhanced; Before the enhancement processing of the image to be enhanced is performed by the enhancement unit of the pre-trained enhancement model, the method further includes: Determine the first encoding process information related to image brightness corresponding to the video frame to be enhanced during the encoding or decoding process; Determine the second encoding process information related to image chroma corresponding to the video frame to be enhanced during the encoding or decoding process; The enhancement unit of the pre-trained enhancement model's brightness enhancement sub-model enhances the brightness component map of the image to be enhanced, obtaining the enhanced brightness features corresponding to each channel, including: Based on the information from the first encoding process, and through the enhancement unit of the brightness enhancement sub-model of the pre-trained enhancement model, the brightness component map of the image to be enhanced is enhanced to obtain the enhanced brightness features corresponding to each channel. The enhancement unit of the chroma enhancement sub-model of the enhancement model performs enhancement processing on the chroma component map of the image to be enhanced, obtaining the enhanced chroma features corresponding to each channel, including: Based on the information from the second encoding process, and through the enhancement unit of the chroma enhancement sub-model of the enhancement model, the chroma component map of the image to be enhanced is enhanced to obtain the enhanced chroma features corresponding to each channel.

11. The image enhancement method according to claim 10, characterized in that, The first encoding process information includes at least one of the following during the encoding or decoding process: the prediction frame, the frame before loop filtering, the frame after loop filtering, the coding unit (CU) partition map, the CU depth map, and the filter intensity map corresponding to the video frame to be enhanced. The second encoding process information includes at least one of the following during the encoding or decoding process: the prediction frame, the frame before loop filtering, the frame after loop filtering, and the filter intensity map corresponding to the video frame to be enhanced.

12. A method for training an augmentation model, characterized in that, include: Obtain training samples, which include the sample image to be enhanced and the corresponding original sample image; The input information of the first training model is determined based on the sample image to be enhanced, and the input information is input into the training enhancement unit of the first training model to obtain the sample enhancement features corresponding to each channel. The sample weights corresponding to the sample enhancement features of each channel are determined by the training weight determination unit of the first training model. The sample enhancement features of the corresponding channels are weighted using the sample weights to obtain the weighted enhancement features of each channel. The weighted enhancement features of each sample are fused to obtain the enhanced image of the sample corresponding to the image to be enhanced. Based on the difference between the augmented image and the original image, the model parameters of the first model to be trained are adjusted to obtain the trained augmented model.

13. The training method according to claim 11, characterized in that, Before determining the input information of the first model to be trained based on the sample image to be enhanced, the method further includes: The input information of the second training model is determined based on the sample image to be enhanced, and the input information is input into the second training model to obtain the second sample enhanced image. The second training model includes an enhancement unit to be trained. Based on the difference between the second sample enhanced image and the corresponding original sample image, the parameters of the second model to be trained are adjusted to obtain an enhanced model to be optimized, wherein the enhanced model to be optimized includes an enhanced unit to be optimized. An optimization and enhancement unit is embedded after the optimization and enhancement unit of the model to be optimized to obtain the first model to be trained.

14. The training method according to claim 11, characterized in that, The training method is used to train augmentation models corresponding to different compression parameter segments; The acquisition of training samples includes: Obtain each set of training samples corresponding to different compression parameter segments. Each set of training samples includes the sample image to be enhanced and the corresponding original sample image. The compression parameters corresponding to the sample image to be enhanced in each set of training samples are located within the compression parameter segment corresponding to that set of training samples. For each compression parameter segment, the step following the determination of the input information of the first model to be trained based on the sample image to be enhanced is performed using training samples corresponding to that compression parameter segment, in order to obtain an enhancement model that matches that compression parameter segment.

15. A video transmission method, characterized in that, The method includes: The sending end compresses and encodes each video frame of the video to be sent to obtain a compressed bitstream, and then sends the compressed bitstream to the receiving end; The receiving end receives the compressed bitstream, decodes the compressed bitstream to obtain the video frame to be enhanced, and uses the image enhancement method according to any one of claims 1 to 11 to enhance the video frame to be enhanced, thereby obtaining the enhanced video frame.

16. An electronic device, characterized in that, include: Processor, memory, and computer program instructions stored in said memory and executable on the processor; When the processor executes the computer program instructions, it implements the method as described in any one of claims 1-15.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-15.

Images