Video enhancement method, apparatus, device, and storage medium

Abstract

The application relates to the technical field of image enhancement, and discloses a video enhancement method, device, equipment and storage medium, the method comprising the following steps: acquiring a source image, pre-processing the source image to obtain a low-resolution image; performing down-sampling on the low-resolution image to obtain a down-sampled image; sampling the down-sampled image through a local feature extraction network and a global feature extraction network to obtain local features and global features; integrating the local features and the global features to construct a bilateral grid coefficient matrix through global average pooling; processing the source image to obtain a guide image; processing the bilateral grid coefficient matrix to obtain an affine matrix, and processing the guide image through the affine matrix to obtain an affine matrix enhanced image. The model algorithm is acquired in a deep learning mode, so that the model algorithm has stronger generalization capability, can cope with video enhancement in different environments, and has low hardware requirements.

Description

Technical Field

[0001] This invention relates to the field of computer networks and communication technology, and in particular to a video enhancement method, apparatus, device, and storage medium. Background Technology

[0002] Today, with the explosive growth of video content, users' demand for higher quality, clearer, and more immersive video experiences is constantly increasing. The continuous development and application of video enhancement technologies not only meet users' needs for high-quality video but also drive innovation and development in multiple fields. Advances in this technology are often closely related to the continuous development of related technologies such as computer vision and artificial intelligence, providing richer and better viewing experiences.

[0003] In low light or backlight conditions, camera equipment is susceptible to interference, resulting in images with low brightness, low contrast, high noise, artifacts, and color distortion due to insufficient light, negatively impacting the human visual experience. Image enhancement algorithms aim to improve image contrast, saturation, and remove noise to make people or objects in the image more prominent and salient, facilitating subsequent recognition and processing. Currently, real-time video enhancement methods are mainly divided into two categories: traditional image enhancement methods, such as histogram equalization or Gamma transformation, which require manual parameter adjustment and have poor adaptability; and deep learning-based methods, which, while not requiring parameter tuning, demand higher hardware requirements.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a video enhancement method that addresses the technical problems of poor adaptability and high hardware requirements associated with manually adjusting parameters.

[0006] To achieve the above objectives, the present invention provides a video enhancement method, apparatus, device, and storage medium, the method comprising the following steps:

[0007] Acquire a source image, and preprocess the source image to obtain a low-resolution image;

[0008] The low-resolution image is downsampled to obtain the downsampled image;

[0009] The downsampled image is sampled using a local feature extraction network and a global feature extraction network to obtain local and global features;

[0010] The local and global features are integrated and then a two-sided grid coefficient matrix is ​​constructed using global average pooling.

[0011] A guide map is obtained by processing the source image;

[0012] An affine matrix is ​​obtained by processing the bilateral grid coefficient matrix, and the guide map is processed using the affine matrix to obtain an affine matrix enhanced image.

[0013] Optionally, the step of sampling the downsampled image through a local feature extraction network and a global feature extraction network to obtain local and global features includes:

[0014] The downsampled image is fused through convolution using an SK network to obtain the first feature information;

[0015] The first feature information is subjected to global average pooling through a CA network to obtain local features;

[0016] The downsampled image is used to learn global features by employing a preset number of convolutional layers and a preset number of fully connected layers.

[0017] Optionally, the step of convolutionally fusing the downsampled image through an SK network to obtain the first feature information includes:

[0018] The downsampled images are then subjected to convolution filtering by adding a preset number of convolutions to obtain convolution results at different scales.

[0019] The convolution results at different scales are fused to obtain a preset number of feature maps;

[0020] After summing the preset number of feature maps, global average pooling is performed to obtain the second feature information;

[0021] The second feature information is passed through a fully connected layer to obtain a preset number of feature maps;

[0022] The preset number of feature maps are processed by the Softmax function to obtain a weighted feature layer;

[0023] The first feature information is obtained by calculating the weighted feature layer.

[0024] Optionally, the step of performing global average pooling on the first feature information through a CA network to obtain local features includes:

[0025] The first feature information is subjected to global flat pooling to obtain a feature vector of a preset size;

[0026] Weights are learned from the preset-size feature vector through a preset convolution;

[0027] The results of weight learning are mapped onto a set containing preset elements to obtain a weight vector;

[0028] Local features are obtained by channel-weighting the first feature information using the weight vector. Optionally, the step of integrating the local features and the global features and constructing a two-sided grid coefficient matrix using global average pooling includes:

[0029] The global features and the local features are integrated using a weighted summation strategy to obtain integrated feature information.

[0030] The integrated feature information is subjected to global average pooling to obtain the bilateral grid coefficient matrix.

[0031] Optionally, obtaining the guide map by processing the source image includes:

[0032] The source image is separated into an RGB three-channel image;

[0033] The RGB three-channel images are convolutionally fused using an SK network to obtain the guide map of the RGB three-channel images.

[0034] Optionally, the step of obtaining an affine matrix by processing the bilateral grid coefficient matrix, and then processing the guide map using the affine matrix to obtain an affine matrix-enhanced image, includes:

[0035] The grid data of the bilateral grid coefficient matrix is ​​inserted into the guide graph using trilinear interpolation, and an affine matrix is ​​obtained using a convolutional layer;

[0036] The affine matrix is ​​sliced ​​into a predetermined number of three-dimensional coefficient matrices;

[0037] Map the three-dimensional coefficient matrix one-to-one with the guide map;

[0038] The affine matrix-enhanced image is obtained by fully fitting the three-dimensional coefficient matrix and the color information of the guide map using a residual network.

[0039] Optionally, the preprocessing of the source image to obtain a low-resolution image includes:

[0040] The source image is transmitted through the Unet network;

[0041] Obtain fixed weights, a preset number of feature maps of the Unet network, the preset number of convolutional kernels, and the preset number of bias values;

[0042] The activation function is used to calculate the fixed weights, the preset number of feature maps of the Unet network, the preset number of convolution kernels, and the preset number of biases to obtain the next layer feature map of the preset number of feature maps, until a low-resolution image is obtained.

[0043] Furthermore, to achieve the above objectives, the present invention also proposes a video enhancement device, the video enhancement device comprising:

[0044] The image processing module is used to acquire a source image, preprocess the source image to obtain a low-resolution image, and downsample the low-resolution image to obtain a downsampled image.

[0045] The matrix construction module is used to sample the downsampled image through a local feature extraction network and a global feature extraction network to obtain local features and global features; and to integrate the local features and the global features to construct a bilateral grid coefficient matrix through global average pooling.

[0046] The image output module is used to obtain a guide map by processing the source image; to obtain an affine matrix by processing the bilateral grid coefficient matrix; and to process the guide map using the affine matrix to obtain an affine matrix enhanced image.

[0047] Furthermore, to achieve the above objectives, the present invention also proposes a video enhancement device, the device comprising: a memory, a processor, and a video enhancement program stored in the memory and executable on the processor, the video enhancement program being configured to implement the steps of the video enhancement method as described above.

[0048] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a video enhancement program, which, when executed by a processor, implements the steps of the video enhancement method as described above.

[0049] This invention acquires a source image, preprocesses it to obtain a low-resolution image, downsamples the low-resolution image to obtain a downsampled image, samples the downsampled image using a local feature extraction network and a global feature extraction network to obtain local and global features, integrates the local and global features and constructs a bilateral grid coefficient matrix using global average pooling, processes the source image to obtain a guide map, processes the bilateral grid coefficient matrix to obtain an affine matrix, and processes the guide map using the affine matrix to obtain an affine matrix-enhanced image. This achieves real-time improvement of video contrast and optimization of video image quality without additional parameter tuning, while reducing the hardware requirements for strategy execution. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the structure of the video enhancement device in the hardware operating environment involved in the embodiments of the present invention;

[0051] Figure 2This is a flowchart illustrating the first embodiment of the video enhancement method of the present invention;

[0052] Figure 3 This is a schematic diagram of the overall network structure of the first embodiment of the video enhancement method of the present invention;

[0053] Figure 4 This is a flowchart illustrating the second embodiment of the video enhancement method of the present invention;

[0054] Figure 5 This is a schematic diagram of the SK structure in the second embodiment of the video enhancement method of the present invention;

[0055] Figure 6 This is a schematic diagram of the CA structure in the second embodiment of the video enhancement method of the present invention;

[0056] Figure 7 This is a flowchart illustrating the third embodiment of the video enhancement method of the present invention;

[0057] Figure 8 This is a schematic diagram of the residual network structure in the third embodiment of the video enhancement method of the present invention;

[0058] Figure 9 This is a flowchart illustrating the fourth embodiment of the video enhancement method of the present invention;

[0059] Figure 10 This is a schematic diagram of the Unet network structure in the fourth embodiment of the video enhancement method of the present invention;

[0060] Figure 11 This is a structural block diagram of the first embodiment of the video enhancement device of the present invention.

[0061] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0062] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0063] Reference Figure 1 , Figure 1 This is a schematic diagram of the video enhancement device structure in the hardware operating environment involved in the embodiments of the present invention.

[0064] like Figure 1As shown, the video enhancement device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0065] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the video enhancement device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0066] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and the 123456 program.

[0067] exist Figure 1 In the video enhancement device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the video enhancement device of the present invention can be set in the video enhancement device, and the video enhancement device calls the video enhancement program stored in the memory 1005 through the processor 1001 and executes the video enhancement method provided in the embodiment of the present invention.

[0068] This invention provides a video enhancement method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the video enhancement method of the present invention.

[0069] In this embodiment, the video enhancement method includes the following steps:

[0070] Step S10: Obtain the source image, preprocess the source image to obtain a low-resolution image.

[0071] It should be noted that the executing entity of the method in this embodiment can be a terminal device with data processing and program execution functions, such as a smart computer or desktop computer, or an electronic device with the same or similar functions, such as the aforementioned video enhancement device. The following description uses a video enhancement device as an example to illustrate this embodiment and the subsequent embodiments.

[0072] Understandably, low-resolution images refer to images with relatively few details and a relatively small number of pixels, thus appearing visually coarser or blurrier.

[0073] It should be understood that preprocessing refers to reducing the resolution of the source image through a U-Net network. U-Net is a deep learning neural network architecture, which is an encoder-decoder structure, where the encoder is used to capture the features of the image, and the decoder is used to map these features back to the segmentation of the original image.

[0074] In the specific implementation, the source image is input, and the resolution of the source image is reduced through preprocessing.

[0075] Step S20: Downsample the low-resolution image to obtain the downsampled image.

[0076] It's important to note that downsampling is a process of reducing the sampling density or resolution of an image or signal. This typically involves reducing the number of sampling points or pixels in a signal or image to reduce the amount of data or computational burden while preserving key information. The purpose of downsampling is to reduce image size, decrease network parameter computation, and speed up network operation.

[0077] In the actual implementation, the low-resolution image is downsampled through multiple convolutional layers to obtain a scaled-down image.

[0078] Step S30: Sample the downsampled image using a local feature extraction network and a global feature extraction network to obtain local and global features.

[0079] It should be noted that the local feature extraction network consists of an SK structure, a 3x3 convolutional layer, and a CA structure, while the global feature extraction network uses multiple convolutional layers and fully connected layers to learn global feature vectors.

[0080] In the specific implementation, local features are obtained by sampling the downsampled image using a local feature extraction network, and global features are obtained by sampling the downsampled image using a global feature extraction network.

[0081] Step S40: After integrating the local features and the global features, construct a two-sided grid coefficient matrix through global average pooling.

[0082] It's important to note that global average pooling is a pooling operation typically used in the last layer of convolutional neural networks in deep learning. Unlike traditional average pooling, global average pooling does not use a pooling window; instead, it averages all values ​​across the entire feature map. One advantage of global average pooling is its insensitivity to the spatial size of the input, thus adapting to input images of varying sizes. Furthermore, it reduces the number of parameters because there is no pooling window size to learn, thereby reducing the risk of overfitting.

[0083] Understandably, the bilateral grid coefficient matrix is ​​a matrix used to calculate the weights of a bilateral filter. Bilateral filtering is an image processing technique that aims to preserve edge information while smoothing image texture. This technique uses a bilateral filter with weights consisting of spatial distance and gray-level similarity. The weights of the bilateral filter can be calculated using a bilateral grid coefficient matrix.

[0084] In the specific implementation, a weighted summation strategy is adopted to integrate the global and local feature information of the source image, and then a 12x16x16x16 bilateral grid coefficient matrix is ​​obtained through global average pooling.

[0085] Step S50: Obtain the guide map by processing the source image.

[0086] It should be noted that the guide image here refers to the guide image of an RGB three-channel image. A guide image is usually an auxiliary image used to guide a certain image processing task. The guide image can be a grayscale image or a three-channel image with the same size as the original RGB image.

[0087] Furthermore, the source image is separated into RGB three-channel images; the RGB three-channel images are then convolved and fused using the SK network to obtain the guide map of the RGB three-channel images.

[0088] In the specific implementation, the source image is separated into RGB three-channel images, and then the SK operation is performed on the RGB three-channel images respectively. The three-channel guide maps are then fused by convolution through the SK network to obtain the guide maps of the RGB three-channel images.

[0089] Step S60: Obtain the affine matrix by processing the bilateral grid coefficient matrix, and process the guide map using the affine matrix to obtain the affine matrix enhanced image.

[0090] It should be noted that an affine matrix is ​​a linear transformation matrix used to describe affine transformations in two-dimensional or three-dimensional space. Affine matrices can be used to perform various geometric transformations on images, thereby achieving image enhancement and processing.

[0091] Furthermore, the grid data of the bilateral grid coefficient matrix is ​​inserted into the guide map using trilinear interpolation, and the affine matrix is ​​obtained using a convolutional layer; the affine matrix is ​​sliced ​​into a preset number of three-dimensional coefficient matrices; the three-dimensional coefficient matrices are mapped one-to-one with the guide map; and the color information of the three-dimensional coefficient matrix and the guide map is fully fitted by a residual network to obtain an affine matrix enhanced image.

[0092] like Figure 3 As shown, Figure 3 This is a schematic diagram of the network structure of the video enhancement method of the present invention. Full-res input: full-resolution input; Full-res output: full-resolution output; Low-res input: low-resolution input; Downsample: downsampling; Local feature: local feature; Global feature: global feature; Feature extraction: feature extraction; Fusion: fusion (weighted summation of global and local features); Bliateral grid of coefficient: bilateral grid coefficients (matrix); Guidance map: guidance map; Slicing: slicing operation; Sliced ​​coefficient: coefficients after slicing (matrix); Apply coefficients: affine coefficients (matrix). First, the full-resolution input image is converted into a low-resolution input. Through feature extraction, including downsampling, local feature extraction (SK network and CA network), and global feature extraction, global and local features are obtained. The global and local features are weighted and summed to obtain bilateral grid coefficients (matrix). The full-resolution input image is then sliced ​​with the bilateral grid coefficients (matrix) after passing through a guide map to obtain sliced ​​coefficients (matrix). The sliced ​​coefficients (matrix) and the full-resolution input image are then processed through affine coefficients (matrix) to obtain the full-resolution output image.

[0093] In the specific implementation, trilinear interpolation is used to insert grid data into the guide map, a convolutional layer is used to obtain the affine matrix, the affine matrix is ​​then sliced ​​into multiple three-dimensional coefficient matrices, and mapped one by one with the guide maps of different color channels. Finally, the color information is fully fitted through a residual network to obtain the image after affine matrix enhancement.

[0094] This embodiment acquires a source image, preprocesses it to obtain a low-resolution image, downsamples the low-resolution image to obtain a downsampled image, samples the downsampled image using local and global feature extraction networks to obtain local and global features, integrates the local and global features and constructs a bilateral grid coefficient matrix using global average pooling, processes the source image to obtain a guidance map, processes the bilateral grid coefficient matrix to obtain an affine matrix, and processes the guidance map using the affine matrix to obtain an affine matrix-enhanced image. This achieves real-time improvement of video contrast and optimization of video image quality without additional parameter tuning, while reducing the hardware requirements for strategy operation.

[0095] refer to Figure 4 , Figure 4 This is a flowchart illustrating the second embodiment of the video enhancement method of the present invention.

[0096] Based on the first embodiment described above, in this embodiment, step S30 may include:

[0097] Step S301: The downsampled image is fused by convolution through the SK network to obtain the first feature information.

[0098] It's important to note that convolutional fusion is an operation in deep learning that combines feature maps from different levels within a convolutional neural network. The purpose of this fusion is to improve the model's expressive power and performance by merging information from different levels.

[0099] Furthermore, the downsampled images are subjected to convolution filtering with a predetermined number of convolutions to obtain convolution results at different scales. The convolution results at different scales are then fused to obtain a predetermined number of feature maps. The predetermined number of feature maps are summed and then subjected to global average pooling to obtain second feature information. The second feature information is then passed through a fully connected layer to obtain a predetermined number of feature maps. The predetermined number of feature maps are then passed through a Softmax function to obtain a weighted feature layer. The weighted feature layer is then used to calculate the first feature information.

[0100] It should be noted that 3x3 and 5x5 convolutions are added to the SK structure for convolution filtering.

[0101] like Figure 5 As shown, Figure 5This is a schematic diagram of the SK structure in the second embodiment of the video enhancement method of the present invention. Multi-scaleconv: multi-scale convolution operation; Attention weight: attention weight; Globel Avgpool: global average pooling; Full connection: fully connected layer; Softmax: Softmax function. The two feature maps are summed element-wise through multi-scale convolution operation. The sum is then processed by attention weights, including first performing global average pooling, then entering the fully connected layer, and finally processing by the Softmax function to obtain weighted feature layers A and B, thus obtaining the first feature information.

[0102] In the specific implementation, during the fusion stage, the two feature maps are summed element-wise as follows:

[0103] I = I1 + I2

[0104] Then, global average pooling is performed to obtain S, the feature map dimension is transformed from [CxHxW] to [Cx1x1], and then it enters the fully connected layer to obtain Z:

[0105]

[0106] Z=δ(β(W s ))

[0107] Where β is the normalization layer, W s These are the weights of the fully connected layer. After the first two stages, the weight information from different scale spaces is subjected to Softmax to obtain weight feature layers A and B. The final output is:

[0108] Output = AU1 + BU2, A + B = 1

[0109] The output is the first feature information.

[0110] Step S302: Perform global average pooling on the first feature information through the CA network to obtain local features.

[0111] Furthermore, the first feature information is subjected to global average pooling to obtain a feature vector of a preset size; the preset size feature vector is subjected to weight learning through a preset convolution; the result of weight learning is mapped to a set containing preset elements to obtain a weight vector; the first feature information is subjected to channel weighting through the weight vector to obtain local features.

[0112] like Figure 6 As shown, Figure 6This is a schematic diagram of the CA structure in the second embodiment of the video enhancement method of the present invention. GlobelAvgpool: global average pooling, 1x1 convolution: convolutional filtering with a kernel size of 1x1. First, global average pooling is performed on the input to obtain a feature vector of a preset size. Then, the feature vector is weighted using 1x1 convolution to learn weights. Finally, the input features are channel-weighted using the weight vector to obtain local features.

[0113] In the specific implementation, firstly, global average pooling is performed on the input of size [CxHxW] to obtain a feature vector Vec of size [Cx1x1]; then, 1x1 convolution is used to learn the weights of the feature vector, and the Sigmoid function is used to map the result to (0,1) to obtain the weight vector VexW; finally, the weight vector is used to perform channel weighting on the input features to obtain local features.

[0114] The structure operation process is as follows:

[0115]

[0116] VecW(I input ) = F Sig [Vec(I input )]

[0117] I Output =I input ⊙VecW

[0118] Step S303: Use a preset number of convolutional layers and a preset number of fully connected layers to learn global features from the downsampled image.

[0119] In its implementation, the global feature extraction network uses multiple convolutional layers and fully connected layers to learn global feature vectors and obtain global features.

[0120] This embodiment uses two asymmetric path networks to extract local and global features, and obtains the model algorithm through deep learning to make it have stronger generalization ability.

[0121] refer to Figure 7 , Figure 7 This is a flowchart illustrating the third embodiment of the video enhancement method of the present invention.

[0122] Based on the above embodiments, in this embodiment, step S60 may include:

[0123] Step S601: The grid data of the bilateral grid coefficient matrix is ​​inserted into the guide graph using trilinear interpolation, and the affine matrix is ​​obtained using a convolutional layer.

[0124] It should be noted that trilinear interpolation is a method of interpolation in three-dimensional space. It is typically used to estimate the value at a point on a discrete three-dimensional data grid.

[0125] Convolutional layers are a commonly used neural network layer in deep learning, used to extract features from input data.

[0126] In the specific implementation, the grid data of the bilateral grid coefficient matrix is ​​inserted into the guide graph using trilinear interpolation, and the resulting feature image is passed through a convolutional layer to obtain an affine matrix.

[0127] Step S602: Slice the affine matrix into a preset number of three-dimensional coefficient matrices.

[0128] It's important to note that an affine matrix is ​​a linear transformation matrix used to represent affine transformations, including translation, rotation, scaling, and shearing. In two-dimensional space, an affine transformation can be represented by a 2x2 matrix, called the affine matrix. In three-dimensional space, the corresponding matrix is ​​3x3.

[0129] In the specific implementation, the affine matrix is ​​divided into multiple three-dimensional coefficient matrices.

[0130] Step S603: Map the three-dimensional coefficient matrix to the guide map one by one.

[0131] It should be noted that a one-to-one mapping is a mathematical concept that describes the relationship between two sets. In a one-to-one mapping, each element has a unique corresponding element, and each element is not mapped to multiple elements.

[0132] In the specific implementation, the elements on the three-dimensional coefficient matrix are mapped one-to-one with the elements on the guide graph.

[0133] Step S604: By fully fitting the color information of the three-dimensional coefficient matrix and the guide map through the residual network, an affine matrix enhanced image is obtained.

[0134] It should be noted that residual networks are a type of deep neural network architecture.

[0135] like Figure 8 As shown, Figure 8This is a schematic diagram of the residual network structure in the third embodiment of the video enhancement method of the present invention. 3x3Conv,BN,PRelu => 3x3. Conv represents convolutional filtering with a kernel size of 3x3; BN represents batch normalization; PRelu represents the PRelu activation function. The input consists of a three-dimensional coefficient matrix and the color information f(x) from the guide map. The main path passes f(x) through a convolutional filter with a kernel size of 3x3, and then performs batch normalization on the filtered image. The jump path adds f(x) to the output of the main path. The outputs of these two paths are then processed by the PRelu activation function to obtain the enhanced image.

[0136] In the specific implementation, the color information of the three-dimensional coefficient matrix and the guide map are input into the residual network, and the three-dimensional coefficient matrix and the color information of the guide map are fully fitted to obtain the enhanced image.

[0137] Furthermore, after the network outputs, the model enters the training phase. Each batch of training data is fed into the model, and predicted values ​​are output through forward propagation. Then, the loss function calculates the difference between the predicted and true values, which is the loss value. After obtaining the loss value, the model updates its parameters through backpropagation to reduce the loss between the true and predicted values, making the model's predicted values ​​closer to the true values, thus achieving the learning objective.

[0138] The network iterative update loss function is set as follows:

[0139] Loss = L L1 +L MSSlM +L VGGLoss

[0140] Among them, L L1 L MSSIM and L VGGLoss These are L1 loss, multi-scale structural similarity loss, and perceptual loss, respectively. L1 L MSSIM and L VGGLoss The formula is expressed as:

[0141]

[0142]

[0143] L MSSIMM =1-MSSIM

[0144]

[0145] This embodiment uses trilinear interpolation to insert the grid data of the bilateral grid coefficient matrix into the guide map, and uses a convolutional layer to obtain the affine matrix; the affine matrix is ​​sliced ​​into a preset number of three-dimensional coefficient matrices; the three-dimensional coefficient matrices are mapped one-to-one with the guide map; and the color information of the three-dimensional coefficient matrix and the guide map is fully fitted by a residual network to obtain the affine matrix-enhanced image. No additional parameter tuning is required, and video contrast is improved in real time, optimizing video image quality.

[0146] refer to Figure 9 , Figure 9 This is a flowchart illustrating the fourth embodiment of the video enhancement method of the present invention.

[0147] Based on the above embodiments, in this embodiment, step S10 may include:

[0148] Step S101: Pass the source image through the Unet network.

[0149] It's important to note that Unet is a deep learning network architecture for image segmentation. Unet is primarily used for semantic segmentation tasks, where the goal is to segment images into different semantic regions, such as segmenting organs in medical images or other objects within an image. The network structure has an encoder-decoder form, with skip connections between the encoder and decoder at different levels to preserve more spatial information.

[0150] like Figure 10 As shown, Figure 10 This is a schematic diagram of the Unet network structure in the fourth embodiment of the video enhancement method of the present invention. Unet contains 8 convolutional layers, each with a 3x3 kernel, a stride of 1, and a padding of 1.

[0151] Step S102: Obtain the feature maps of the Unet network with fixed weights and a preset number of layers, the convolution kernels of the preset number of layers, and the bias values ​​of the preset number of layers.

[0152] It should be noted that fixed weights typically refer to keeping the weights of certain neural network layers constant during training. This can be done at different stages of training or throughout the entire training process; in this strategy, the fixed weight 'a' is set to 0.25.

[0153] Z n The feature map of the preset number of layers of the Unet network, W n and b n These represent the kernel size and the bias value for the preset number of layers, respectively.

[0154] In the specific implementation, a fixed weight 'a' is obtained and set to 0.25, and the feature map Z of the preset number of layers of the Unet network is obtained. n , Preset number of convolutional kernels W n The bias value b of the preset number of layersn .

[0155] Step S103: Calculate the fixed weights, the preset number of feature maps of the Unet network, the preset number of convolution kernels, and the preset number of biases using the activation function to obtain the next layer feature map of the preset number of feature maps, until a low-resolution image is obtained.

[0156] It should be noted that the activation function is PReLU, an improvement on the traditional ReLU. Unlike ReLU, PReLU introduces learnable parameters, making the slope of the activation function no longer fixed on the negative half-axis. The PReLU function formula is expressed as:

[0157] PRelu = max(0,y) + a·min(0,y)

[0158] Understandably, the formula used in the calculation is:

[0159] Z n =PRelu(W n ·F n-1 +b n )

[0160] Where Zn represents the preset number of feature maps, PRelu is the activation function, Wn and bn represent the preset number of convolution kernels and the preset number of bias values, respectively, Fn-1 represents the next layer of feature maps after the preset number of feature maps, · represents convolution, a is a fixed weight of 0.25, and y represents the specific value in the feature map.

[0161] In the specific implementation, the fixed weight 'a' and the preset layer feature map Z of the Unet network are used. n , Preset number of convolutional kernels W n The bias value b of the preset number of layers n The formula is input to calculate and reduce the resolution of the source image, and then the images are uniformly adjusted to a size of 256x256.

[0162] In this embodiment, the source image is preprocessed to reduce its resolution through a Unet network, resulting in an image size that meets the target requirements.

[0163] Furthermore, embodiments of the present invention also propose a storage medium storing a video enhancement program, which, when executed by a processor, implements the steps of the video enhancement method described above.

[0164] Furthermore, the present invention also proposes a video enhancement device, the device comprising: a memory, a processor, and a video enhancement program stored in the memory and executable on the processor, the video enhancement program being configured to implement the steps of the video enhancement method as described above.

[0165] Reference Figure 11 , Figure 11 This is a structural block diagram of the first embodiment of the video enhancement device of the present invention.

[0166] like Figure 11 As shown, the video enhancement device proposed in this embodiment of the invention includes:

[0167] Image processing module 10 is used to acquire a source image, preprocess the source image, and obtain a low-resolution image;

[0168] The image processing module 10 is also used to downsample the low-resolution image to obtain the downsampled image;

[0169] The matrix construction module 20 is used to sample the downsampled image through a local feature extraction network and a global feature extraction network to obtain local features and global features;

[0170] The matrix construction module 20 is also used to integrate the local features and the global features and then construct a two-sided grid coefficient matrix through global average pooling;

[0171] Image output module 30 is used to obtain a guide map by processing the source image;

[0172] The image output module 30 is also used to obtain an affine matrix by processing the bilateral grid coefficient matrix, and to process the guide map by the affine matrix to obtain an affine matrix enhanced image.

[0173] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0174] In this embodiment, the image processing module 10 is used to acquire a source image, preprocess the source image to obtain a low-resolution image, and downsample the low-resolution image to obtain a downsampled image. The matrix construction module 20 is used to sample the downsampled image through a local feature extraction network and a global feature extraction network to obtain local features and global features; the local features and global features are integrated and then constructed using global average pooling to construct a bilateral grid coefficient matrix. The image output module 30 is used to obtain a guide map by processing the source image; an affine matrix is ​​obtained by processing the bilateral grid coefficient matrix; and the guide map is processed using the affine matrix to obtain an affine matrix-enhanced image. The model algorithm is obtained through deep learning, giving it stronger generalization ability, enabling it to handle video enhancement in different environments, and requiring low hardware specifications.

[0175] In one embodiment, the image processing module 10 is further configured to obtain fixed weights, a preset number of feature maps of the Unet network, a preset number of convolutional kernels, and a preset number of bias values ​​from the source image through the Unet network;

[0176] The activation function is used to calculate the fixed weights, the preset number of feature maps of the Unet network, the preset number of convolution kernels, and the preset number of biases to obtain the next layer feature map of the preset number of feature maps, until a low-resolution image is obtained.

[0177] In one embodiment, the matrix construction module 20 is further configured to perform convolutional fusion on the downsampled image through an SK network to obtain first feature information;

[0178] The first feature information is subjected to global average pooling through a CA network to obtain local features;

[0179] Global features are obtained by learning from the downsampled image using a preset number of convolutional layers and a preset number of fully connected layers.

[0180] In one embodiment, the matrix construction module 20 is further configured to add a preset number of convolutions to the downsampled image for convolution filtering, thereby obtaining convolution results at different scales;

[0181] The convolution results at different scales are fused to obtain a predetermined number of feature maps;

[0182] After summing the preset number of feature maps, global average pooling is performed to obtain the second feature information;

[0183] The second feature information is passed through a fully connected layer to obtain a predetermined number of feature maps;

[0184] The preset number of feature maps are processed by the Softmax function to obtain the weighted feature layer;

[0185] The first feature information is obtained by calculating the weighted feature layer.

[0186] In one embodiment, the matrix construction module 20 is further configured to perform global flat pooling on the first feature information to obtain a feature vector of a preset size;

[0187] Weights are learned from a feature vector of a preset size through a preset convolution;

[0188] The results of weight learning are mapped onto a set containing preset elements to obtain a weight vector;

[0189] Local features are obtained by channel-weighting the first feature information using a weight vector.

[0190] In one embodiment, the matrix construction module 20 is further configured to integrate global features and local features using a weighted summation strategy to obtain integrated feature information;

[0191] Global average pooling is performed on the integrated feature information to obtain a two-sided grid coefficient matrix.

[0192] In one embodiment, the image output module 30 is further configured to separate the source image into an RGB three-channel image;

[0193] The three-channel images are convolutionally fused using the SK network to obtain the guide map of the RBG three-channel image.

[0194] In one embodiment, the image output module 30 is further configured to insert the grid data of the bilateral grid coefficient matrix into the guide graph using trilinear interpolation, and to obtain an affine matrix using a convolutional layer;

[0195] Slice the affine matrix into a predetermined number of three-dimensional coefficient matrices;

[0196] Map the three-dimensional coefficient matrix one-to-one with the guide map;

[0197] By fully fitting the color information of the three-dimensional coefficient matrix and the guide map through the residual network, an affine matrix-enhanced image is obtained.

[0198] Other embodiments or specific implementations of the video enhancement device of the present invention can be found in the above-described method embodiments, and will not be repeated here.

[0199] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0200] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0201] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0202] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A video enhancement method, characterized in that, The video enhancement method includes the following steps: Acquire a source image, and preprocess the source image to obtain a low-resolution image; The low-resolution image is downsampled to obtain the downsampled image; The downsampled image is sampled using a local feature extraction network and a global feature extraction network to obtain local and global features; The local and global features are integrated and then a two-sided grid coefficient matrix is ​​constructed using global average pooling. A guide map is obtained by processing the source image; An affine matrix is ​​obtained by processing the bilateral grid coefficient matrix, and the guide map is processed using the affine matrix to obtain an affine matrix enhanced image. The step of sampling the downsampled image through a local feature extraction network and a global feature extraction network to obtain local and global features includes: The downsampled image is fused through convolution using an SK network to obtain the first feature information; The first feature information is subjected to global average pooling through a CA network to obtain local features; The downsampled image is learned using a preset number of convolutional layers and a preset number of fully connected layers to obtain global features; The step of performing global average pooling on the first feature information through a CA network to obtain local features includes: The first feature information is subjected to global average pooling to obtain a feature vector of a preset size; Weights are learned from the preset-size feature vector through a preset convolution; The results of weight learning are mapped onto a set containing preset elements to obtain a weight vector; The first feature information is channel-weighted using the weight vector to obtain local features.

2. The video enhancement method as described in claim 1, characterized in that, The step of convolutionally fusing the downsampled image through an SK network to obtain the first feature information includes: The downsampled images are then subjected to convolution filtering by adding a preset number of convolutions to obtain convolution results at different scales. The convolution results at different scales are fused to obtain a preset number of feature maps; After summing the preset number of feature maps, global average pooling is performed to obtain the second feature information; The second feature information is passed through a fully connected layer to obtain a preset number of feature maps; The preset number of feature maps are processed by the Softmax function to obtain a weighted feature layer; The first feature information is obtained by calculating the weighted feature layer.

3. The video enhancement method as described in claim 1, characterized in that, The step of integrating the local features and the global features and then constructing a two-sided grid coefficient matrix through global average pooling includes: The global features and the local features are integrated using a weighted summation strategy to obtain integrated feature information. The integrated feature information is subjected to global average pooling to obtain the bilateral grid coefficient matrix.

4. The video enhancement method as described in claim 1, characterized in that, The process of obtaining the guide map by processing the source image includes: The source image is separated into an RGB three-channel image; The RGB three-channel images are convolutionally fused using an SK network to obtain the guide map of the RGB three-channel images.

5. The video enhancement method as described in claim 1, characterized in that, The step of obtaining an affine matrix by processing the bilateral grid coefficient matrix, and then processing the guide map using the affine matrix to obtain an affine matrix-enhanced image includes: The grid data of the bilateral grid coefficient matrix is ​​inserted into the guide graph using trilinear interpolation, and an affine matrix is ​​obtained using a convolutional layer; The affine matrix is ​​sliced ​​into a predetermined number of three-dimensional coefficient matrices; Map the three-dimensional coefficient matrix one-to-one with the guide map; The affine matrix-enhanced image is obtained by fully fitting the three-dimensional coefficient matrix and the color information of the guide map using a residual network.

6. The video enhancement method as described in claim 1, characterized in that, The preprocessing of the source image to obtain a low-resolution image includes: The source image is processed through a Unet network to obtain fixed weights, a preset number of feature maps of the Unet network, a preset number of convolutional kernels, and a preset number of bias values; The activation function is used to calculate the fixed weights, the preset number of feature maps of the Unet network, the preset number of convolution kernels, and the preset number of biases to obtain the next layer feature map of the preset number of feature maps, until a low-resolution image is obtained.

7. A video enhancement device, characterized in that, The device includes: The image processing module is used to acquire a source image, preprocess the source image to obtain a low-resolution image, and downsample the low-resolution image to obtain a downsampled image. The matrix construction module is used to sample the downsampled image through a local feature extraction network and a global feature extraction network to obtain local features and global features; and to integrate the local features and the global features to construct a bilateral grid coefficient matrix through global average pooling. An image output module is used to obtain a guide map by processing the source image; to obtain an affine matrix by processing the bilateral grid coefficient matrix; and to process the guide map using the affine matrix to obtain an affine matrix enhanced image. The matrix construction module is further configured to: perform convolutional fusion on the downsampled image using an SK network to obtain first feature information; perform global average pooling on the first feature information using a CA network to obtain local features; learn global features on the downsampled image using a preset number of convolutional layers and a preset number of fully connected layers; perform global average pooling on the first feature information to obtain a feature vector of a preset size; learn weights on the feature vector of the preset size using a preset convolution; map the result of the weight learning to a set containing preset elements to obtain a weight vector; and perform channel weighting on the first feature information using the weight vector to obtain local features.

8. A storage medium, characterized in that, The storage medium stores a video enhancement program, which, when executed by a processor, implements the steps of the video enhancement method as described in any one of claims 1 to 6.

Images