A method for image enhancement based on video IP frames

Abstract

This application relates to the field of image data processing technology, and in particular to an image enhancement method based on video IP frames. The method includes: acquiring a target video V; parsing V to obtain a list L of frame numbers corresponding to keyframes in V; sequentially traversing V, and if q matches a certain id in L... m If they are equal, then v is obtained using the first sub-model in the trained target neural network model. q Transformation matrix T q and T q The latest transformation matrix T is determined. new , will T q As input to the second sub-model in the trained target neural network model, the output of the second sub-model in the trained target neural network model is used as v. q The corresponding enhanced image; otherwise, T new As input to the second sub-model in the trained target neural network model, the output of the second sub-model in the trained target neural network model is used as v. q The corresponding enhanced image. This invention improves the efficiency of image enhancement for videos.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to an image enhancement method based on video IP frames. Background Technology

[0002] Several image enhancement algorithms have been disclosed in the prior art. For example, the contrast-limited adaptive histogram equalization (CLAH) algorithm is a relatively effective image enhancement algorithm. However, this algorithm runs on the CPU, which requires CPU resources. Moreover, the CPU is a serial processing mechanism, and running the CLAH algorithm on the CPU to enhance video is slow, making it difficult to meet users' high-efficiency requirements for video image enhancement. How to improve the efficiency of video image enhancement is an urgent technical problem to be solved. Summary of the Invention

[0003] The purpose of this invention is to provide an image enhancement method based on video IP frames to improve the efficiency of video image enhancement.

[0004] According to the present invention, an image enhancement method based on video IP frames is provided, comprising the following steps:

[0005] S10, Obtain the target video V, V = (v1, v2, ..., v q ,…,v Q ), v q Let V be the q-th frame of images included, where q ranges from 1 to Q, and Q is the number of images included in V. q For v q+1 The previous frame image, v q+1 V is the (q+1)th frame of the image; v q For keyframes or forward predictive coded frames in V.

[0006] S20, parse V and obtain the list L of frames corresponding to the keyframes in V, L = (id1, id2, ..., id...). m ,…,id M ), id m Let m be the frame number corresponding to the m-th keyframe in V, where m ranges from 1 to M, and M is the number of keyframes included in V.

[0007] S30, sequentially traverse V, if q matches a certain id in L... m If they are equal, proceed to S40; otherwise, proceed to S50.

[0008] S40, obtain v using the first sub-model in the trained target neural network model. q Transformation matrix T q and T qdetermining the latest conversion matrix T new .

[0009] S41, T q is taken as the input of the second sub-model in the trained target neural network model, and the output of the second sub-model in the trained target neural network model is taken as v q the corresponding enhanced image.

[0010] S50, T new .

[0011] S51, T new is taken as the input of the second sub-model in the trained target neural network model, and the output of the second sub-model in the trained target neural network model is taken as v q the corresponding enhanced image.

[0012] Compared with the prior art, the present application has at least the following beneficial effects:

[0013] In the image enhancement processing of each frame image in the target video, the type of each frame in the target video is analyzed, and the frame number list corresponding to the key frame in the target video is obtained. On this basis, the image enhancement processing is performed according to the order of the frame number of each frame in the target video from small to large. Thus, the conversion matrix corresponding to the key frame can be obtained first, which is determined as the latest conversion matrix. In the subsequent image enhancement of the forward prediction coding frame, the conversion matrix corresponding to the corresponding key frame (i.e. the latest conversion matrix) is a known value, which can be directly taken as the input of the second sub-model. Thus, in the image enhancement processing of the forward prediction coding frame, the process of obtaining the conversion matrix corresponding to the forward prediction coding frame by using the first sub-model can be omitted, the time for obtaining the enhanced image can be saved, and the efficiency of image enhancement of the video is improved. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0015] Figure 1 The flowchart of the image enhancement method based on video IP frame provided by the embodiments of the present application. DETAILED DESCRIPTION

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1

[0018] According to this embodiment, as Figure 1 As shown, an image enhancement method based on video IP frames is provided, including the following steps:

[0019] S10, Obtain the target video V, V = (v1, v2, ..., v q ,…,v Q ), v q Let V be the q-th frame of images included, where q ranges from 1 to Q, and Q is the number of images included in V. q For v q+1 The previous frame image, v q+1 V is the (q+1)th frame of the image; v q For keyframes or forward predictive coded frames in V.

[0020] In this embodiment, if v q If v is the first frame image in the target video, then q = 1; if v q If the second frame is the image in the target video, then q = 2; and so on. In this embodiment, the frame number of each image in the target video is determined according to the chronological order of the corresponding time frames; that is, the earlier the time of the image, the smaller its corresponding frame number.

[0021] S20, parse V and obtain the list of frame numbers L corresponding to the keyframes in V, L = (id1, id2, ..., id...). m ,…,id M ), id m Let m be the frame number corresponding to the m-th keyframe in V, where m ranges from 1 to M, and M is the number of keyframes included in V.

[0022] Those skilled in the art will understand that any method in the prior art for parsing keyframes and forward predictive coded frames in a video falls within the protection scope of this invention. Optionally, ffmpeg can be used to parse the target video to obtain a list of frame numbers corresponding to the keyframes in the target video.

[0023] S30, sequentially traverse V, if q matches a certain id in L... m If they are equal, proceed to S40; otherwise, proceed to S50.

[0024] S40, obtain v using the first sub-model in the trained target neural network model. q Transformation matrix T q and T q The latest transformation matrix T is determined. new .

[0025] S41, T q As input to the second sub-model in the trained target neural network model, the output of the second sub-model in the trained target neural network model is used as v. q The corresponding enhanced image.

[0026] S50, obtain T new .

[0027] S51, T new As input to the second sub-model in the trained target neural network model, the output of the second sub-model in the trained target neural network model is used as v. q The corresponding enhanced image.

[0028] For example, if the target video contains 100 frames, arranged chronologically, with frames 1, 10, 20, 30, 40, 50, 60, 70, 80, and 90 being keyframes, then L = (1, 10, 20, 30, 40, 50, 60, 70, 80, 90); the others are forward predictive coded frames. Specifically, frames 2-9 are the forward predictive coded frames corresponding to frame 1, frames 11-19 are the forward predictive coded frames corresponding to frame 10, frames 21-29 are the forward predictive coded frames corresponding to frame 20, and so on. If v q If the number of frames in V is 10, then 10 is equal to 10 in L. Therefore, execute S40, using the first sub-model in the trained target neural network model to obtain v. 10 Transformation matrix T 10 and T 10 The latest transformation matrix T is determined. new Execute S41; if v q If the number of frames in V is 11, then 11 is not equal to the number of frames in L. Therefore, execute S50 to obtain T. new (i.e. T) 10 ), execute S51. If v q If the number of frames in V is 20, then 20 is equal to 20 in L. Therefore, execute S40, using the first sub-model in the trained target neural network model to obtain v. 20 Transformation matrix T 20 and T 20 The latest transformation matrix T is determined. new Execute S41; if v qIf the frame number in V is 25, and 25 is not equal to each frame number in L, then S50 is performed to obtain T new (i.e., T 20 ), and S51 is performed.

[0029] In the image enhancement processing of each frame image in the target video, the type of each frame in the target video is analyzed, and a frame number list corresponding to the key frame in the target video is obtained. On this basis, the image enhancement processing is performed according to the frame number of each frame in the target video from small to large. Therefore, the conversion matrix corresponding to the key frame can be obtained first, which is determined as the latest conversion matrix. When the image enhancement of the forward prediction coding frame is performed subsequently, the conversion matrix corresponding to the key frame (i.e., the latest conversion matrix) is a known value, which can be directly used as the input of the second sub-model. Therefore, in the process of image enhancement processing of the forward prediction coding frame, the process of obtaining the conversion matrix corresponding to the forward prediction coding frame by using the first sub-model can be omitted, the time for obtaining the enhanced image can be saved, and the efficiency of image enhancement of the video is improved.

[0030] Specifically, the target neural network model in the embodiment includes a first sub-model and a second sub-model, the output of the first sub-model is the input of the second sub-model; the first sub-model is used to obtain the conversion matrix of the input image, and the first sub-model includes, in sequence according to the direction of information transmission, four convolution layers, a two-dimensional adaptive average pooling layer, a flatten layer, a first linear layer, a relu activation layer, a second linear layer, and a sigmoid activation layer; the output of the sigmoid activation layer is the conversion matrix of the input image; the second sub-model is used to obtain the enhanced image corresponding to the input image according to the output of the first sub-model, and the second sub-model includes a convolution operation module and a summation operation module, the input of the convolution operation module is the input image and the output of the first sub-model, the input of the summation operation module is the input image and the output of the convolution operation module, and the output of the summation operation module is the enhanced image corresponding to the input image.

[0031] Specifically, each convolution layer includes a convolution processing module, a batch normalization operation processing module, and a relu activation processing module connected in sequence.

[0032] Specifically, when the target neural network model is a neural network model for obtaining an enhanced image of a single-channel image, the convolution processing module includes one 3x3 convolution kernel, and the conversion matrix of the input image is one 3x3 matrix; when the target neural network model is a neural network model for obtaining an enhanced image of a 3-channel image, the convolution processing module includes three 3x3 convolution kernels, and the conversion matrix of the input image is three 3x3 matrices, wherein the first 3x3 matrix is used for convolution with the first channel image of the input image, the second 3x3 matrix is used for convolution with the second channel image of the input image, and the third 3x3 matrix is used for convolution with the third channel image of the input image.

[0033] As the first specific implementation, when the target neural network model is a neural network model for obtaining an enhanced image of a single-channel image, according to the direction of information transmission, the parameters of the convolution kernel included in the convolution processing module of the first convolution layer are (1, 32, 3, 1), the first 1 indicates that the input is a single channel and the output is 32 channels, the convolution kernel size is 3x3, and the second 1 indicates that the step is 1; the parameters of the convolution kernel included in the convolution processing module of the second convolution layer are (32, 64, 3, 1), 32 indicates that the input is 32 channels, 64 indicates that the output is 64 channels, the convolution kernel size is 3x3, and 1 indicates that the step is 1; the parameters of the convolution kernel included in the convolution processing module of the third convolution layer are (64, 128, 3, 1), 64 indicates that the input is 64 channels, 128 indicates that the output is 128 channels, the convolution kernel size is 3x3, and 1 indicates that the step is 1; the parameters of the convolution kernel included in the convolution processing module of the fourth convolution layer are (128, 256, 3, 1), 128 indicates that the input is 128 channels, 256 indicates that the output is 256 channels, the convolution kernel size is 3x3, and 1 indicates that the step is 1; the parameters of the first linear layer are (256, 128), 256 indicates that the input size is 256, and 128 indicates that the output size is 128; the parameters of the second linear layer are (128, 9), 128 indicates that the input size is 128, and 9 indicates that the output size is 9.

[0034] As the second specific embodiment, when the target neural network model is a neural network model for obtaining an enhanced image of a 3-channel image, according to the direction of information transmission, the parameters of the convolution kernel included in the convolution processing module of the first convolution layer are (3, 32, 3, 1), the first 3 indicates that the input is 3 channels and the output is 32 channels, the second 3 indicates that the size of the convolution kernel is 3*3, and 1 indicates that the step is 1; the parameters of the convolution kernel included in the convolution processing module of the second convolution layer are (32, 64, 3, 1), 32 indicates that the input is 32 channels, 64 indicates that the output is 64 channels, the size of the convolution kernel is 3*3, and 1 indicates that the step is 1; the parameters of the convolution kernel included in the convolution processing module of the third convolution layer are (64, 128, 3, 1), 64 indicates that the input is 64 channels, 128 indicates that the output is 128 channels, the size of the convolution kernel is 3*3, and 1 indicates that the step is 1; the parameters of the convolution kernel included in the convolution processing module of the fourth convolution layer are (128, 256, 3, 1), 128 indicates that the input is 128 channels, 256 indicates that the output is 256 channels, the size of the convolution kernel is 3*3, and 1 indicates that the step is 1; the parameters of the first linear layer are (256, 128), 256 indicates that the input size is 256, and 128 indicates that the output size is 128; the parameters of the second linear layer are (128, 27), 128 indicates that the input size is 128, and 27 indicates that the output size is 27.

[0035] In this embodiment, the convolution operation module is used for padding processing of the input image, and the output of the first sub-model is used for convolution operation on the input image after padding processing, and the summation operation module is used for summation operation on the input image and the convolution result output by the convolution operation module.

[0036] In this embodiment, when the target neural network model is a neural network model for obtaining an enhanced image of a single-channel image, the conversion matrix of the input image is a 3*3 matrix; when the target neural network model is a neural network model for obtaining an enhanced image of a 3-channel image, the conversion matrix of the input image is three 3*3 matrices; accordingly, the padding size is 1 when padding processing is performed on the input image, specifically, a circle of 0 is filled outside the input image, so that the convolution result output by the convolution operation module is equal to the size of the input image, and summation operation can be performed.

[0037] Specifically, the training process of the target neural network model includes the following steps:

[0038] S1, obtaining an image sample list A, A=(a1, a2, …, an, …, aN), an is the nth image sample, the value range of n is 1 to N, N is the number of image samples; each a n ,…,a N is a 3*3 matrix, and the value of each element in the matrix is a real number. n ​n The number of channels and the channel type corresponding to the image in the target video are the same.

[0039] S2, obtaining an image sample label list B=(b1, b2, …, b n ,…,b N ), b n is an image after image enhancement on a n .

[0040] In the embodiment, b n is the corresponding label of a n .

[0041] Those skilled in the art know that any method for enhancing the image in the prior art falls within the protection scope of the present application; optionally, b n is an image after image enhancement on a n using the clahe algorithm.

[0042] S3, training the target neural network model using A and B.

[0043] Those skilled in the art know that any supervised training method in the prior art falls within the protection scope of the present application.

[0044] Optionally, the image enhancement method based on the video IP frame in the embodiment is executed on a CPU or a GPU.

[0045] Preferably, the image enhancement method based on the video IP frame in the embodiment is executed on a GPU, so that the operation of image enhancement processing on the video no longer occupies the CPU resource, and more CPU resources can be saved for the user; moreover, the GPU is for parallel computing, and the time for image enhancement processing on multiple videos is shorter, which can improve the efficiency of image enhancement processing on multiple videos and meet the high efficiency requirement of the user for image enhancement on the video.

[0046] The embodiment utilizes the trained target neural network model to realize enhancement processing on an image. The target neural network model includes a first sub-model and a second sub-model. The second sub-model includes a convolution operation module and a summation operation module. The output of the convolution operation module represents the difference between the input image and the corresponding enhanced image. The input of the convolution operation module is the input image and the output of the first sub-model (i.e., the conversion matrix of the input image). The conversion matrix of the input image is obtained by the first sub-model. The first sub-model includes the structure of a neural network. It can be seen that, instead of directly learning how to obtain the corresponding enhanced image based on the input image by using the neural network, the neural network is indirectly used to learn the difference between the input image and the corresponding enhanced image. Since the change interval of the difference between the input image and the enhanced image is relatively small, it is relatively easy to learn. Therefore, the target neural network model is more likely to converge in the process of training the target neural network model, the length of the corresponding training process is relatively short, and the accuracy of the enhanced image obtained by using the trained target neural network model is also relatively high.

[0047] Moreover, the structure of the first sub-model included in the target neural network model in the embodiment is relatively simple. It is a lightweight neural network. The time required for obtaining the enhanced image corresponding to the video by using the trained target neural network model is relatively short. The speed of image enhancement processing on the video is improved.

[0048] Embodiment Two

[0049] Compared with the first embodiment, the difference between the first embodiment and the second embodiment includes that the first sub-model of the second embodiment includes, in sequence according to the direction of information transmission, four convolution layers in cascade, a two-dimensional adaptive average pooling layer, a flatten layer, a first linear layer, a relu activation layer, a second linear layer, a sigmoid activation layer, and an update module. The update module is used to update the conversion matrix of the input image according to the parameter adjustment requirement information input by the user, to obtain the updated conversion matrix of the input image.

[0050] The first sub-model in the embodiment further includes an update module. The output of the corresponding first sub-model is the updated conversion matrix of the input image. The input of the corresponding second sub-model is the updated conversion matrix of the input image, i.e., the conversion matrix T q of the input image v q is the updated conversion matrix.

[0051] In the process of training the target neural network model, the user does not input, the update module does not update the conversion matrix of the input image, the output of the first sub-model is the conversion matrix of the input image, and the input of the second sub-model is the conversion matrix of the input image.

[0052] Specifically, when the target neural network model is a neural network model for obtaining an enhanced image of a single-channel image, the conversion matrix of the input image is a 3x3 matrix; the updating of the conversion matrix of the input image according to the parameter adjustment requirement information input by the user comprises:

[0053] S100, obtaining a conversion matrix T0 of an input image, T0=[e 0 1,1 ,e 0 1,2 ,e 0 1,3 ;e 0 2,1 ,e 0 2,2 ,e 0 2,3 ;e 0 3,1 ,e 0 3,2 ,e 0 3,3 ], e 0 x,y is an element in the xth row and yth column of T0, x=1, 2, 3, y=1, 2, 3.

[0054] S200, obtaining a first target adjustment value w1; if the parameter adjustment requirement information input by the user is information indicating an increase in VMAF, then w1=e 0 2,2 +Δw1, Δw1 is a first parameter adjustment amplitude, 0<Δw1≤f max -e 0 2,2 , f max is a maximum value of a preset value range F' of the conversion matrix; if the parameter adjustment requirement information input by the user is information indicating an increase in PSNR, then w1=e 0 2,2 -Δw2, Δw2 is a second parameter adjustment amplitude, 0<Δw2≤e 0 2,2 -f min , f min is a minimum value of F'.

[0055] Those skilled in the art know that video quality multi-method evaluation fusion (VMAF) and peak signal-to-noise ratio (PSNR) are two relatively common parameters for evaluating video / image quality.

[0056] In this embodiment, Δw1 and Δw2 are empirical values, and optionally, Δw1=(f max -e 0 2,2 ) / 2, Δw2=(e0 2,2 -f min ) / 2.

[0057] S300, the [e 0 1,1 ,e 0 1,2 ,e 0 1,3 ;e 0 2,1 ,w1,e 0 2,3 ;e 0 3,1 ,e 0 3,2 ,e 0 3,3 ] is determined as the updated conversion matrix of the input image.

[0058] When the target neural network model is a neural network model for obtaining an enhanced image of a 3-channel image, the conversion matrix of the input image is three 3x3 matrices; the updating of the conversion matrix of the input image according to the parameter adjustment requirement information input by the user comprises:

[0059] S101, obtaining the conversion matrix T0 of the input image, T0=(T 1 0,T 2 0,T 3 0), T k 0 is the kth conversion matrix corresponding to the input image, k=1, 2, 3; T k 0=[e k,0 1,1 ,e k,0 1,2 ,e k,0 1,3 ;e k,0 2,1 ,e k,0 2,2 ,e k,0 2,3 ;e k,0 3,1 ,e k,0 3,2 ,e k ,0 3,3 ], e k,0 x,y is the element of the xth row and yth column of T k 0.

[0060] S102, obtaining a second target adjustment value w2, w2=(w 2,1 ,w 2,2 ,w 2,3), w 2,k for e k,0 2,2 corresponding target adjustment value; if the parameter adjustment requirement information input by the user is information indicating to increase the VMAF, then w 2,k for e k,0 2,2 + Δw3, Δw3 is a third parameter adjustment amplitude, 0 < Δw3 ≤ f max -max(e 1,0 2,2 for e 2,0 2,2 for e 3,0 2,2 ), f max is a maximum value of a preset value range F' of the conversion matrix, max() is to take the maximum value; if the parameter adjustment requirement information input by the user is information indicating to increase the PSNR, then w 2,k for e k,0 2,2 - Δw4, Δw4 is a fourth parameter adjustment amplitude, 0 < Δw4 ≤ min(e 1,0 2,2 for e 2,0 2,2 for e 3 ,0 2,2 )- f min , f min is a minimum value of F', min() is to take the minimum value.

[0061] In the embodiment, Δw3 and Δw4 are empirical values, and optionally, Δw3 = (f max -max(e 1,0 2,2 for e 2,0 2,2 for e 3 ,0 2,2 )) / 2, and Δw4 = (min(e 1,0 2,2 for e 2,0 2,2 for e 3,0 2,2 )- f min ) / 2.

[0062] In the embodiment, if the parameter adjustment requirement information input by the user is information indicating to increase the VMAF and f max -max(e 1,0 2,2 for e 2,0 2,2 for e 3,0 2,2If ) = 0, then Δw3 = 0, and the user interface displays the preset information indicating that the increase in VMAF should not be performed; if the user-input parameter adjustment requirement information is the information indicating the increase in PSNR and min(e 1,0 2,2 ,e 2 ,0 2,2 ,e 3,0 2,2 )-f min If Δw4 = 0, then Δw4 = 0, and a preset message indicating that increasing PSNR should not be performed will be displayed on the user interface.

[0063] S103, will ([e 1,0 1,1 ,e 1,0 1,2 ,e 1,0 1,3 ;e 1,0 2,1 ,w 2,1 ,e 1,0 2,3 ;e 1,0 3,1 ,e 1,0 3,2 ,e 1,0 3,3 ],[e 2,0 1,1 ,e 2,0 1,2 ,e 2,0 1,3 ;e 2,0 2,1 ,w 2,2 ,e 2,0 2,3 ;e 2,0 3,1 ,e 2,0 3,2 ,e 2,0 3,3 ],[e 3,0 1,1 ,e 3,0 1,2 ,e 3,0 1,3 ;e 3,0 2,1 ,w 2,3 ,e 3,0 2,3 ;e 3,0 3,1 ,e 3,0 3,2 ,e 3,0 3,3 The updated transformation matrix of the input image is determined.

[0064] Optionally, the process of obtaining F' includes:

[0065] S110, Obtain the test image set C, C = {c1, c2, ..., c...} r ,…,c R}, c r Let C be the r-th test image, where r ranges from 1 to R, and R is the number of test images included in C. r The number and type of channels are the same as those of the images in the target video.

[0066] In this embodiment, each c r The image is from the same application scenario as the image in the target video.

[0067] In this embodiment, R is an empirical value, and R = 1000 is optional.

[0068] S210, iterate through C, and... r The input is fed into the trained target neural network model to obtain c. r The corresponding transformation matrix T' r and T' r Append to the preset set of transformation matrices T' to obtain T' = {T'1, T'2, ..., T' r ,…,T' R}, T' is initialized to an empty set.

[0069] S310, Obtain the range of values ​​F of the elements in the transformation matrix according to T', F = (f1, f2, ..., f h ,…,f H ), f h Let f be the range of values ​​for the h-th element in the transformation matrix. h =[f h,1 ,f h,2 ], f h,1 f h The minimum value of f h,2 f h The maximum value of f h,1 =min(D h ), f h,2 =max(D h ), D h To make each T' in T' r The corresponding h-th element is appended to the preset h-th element set D' h The resulting set, D' h The initialization is an empty set, min() is to get the minimum value, max() is to get the maximum value, h is in the range of 1 to H, and H is the number of elements in the transformation matrix.

[0070] In this embodiment, when the conversion matrix of the input image is one 3x3 matrix, the number of elements in the conversion matrix is 9; when the conversion matrix of the input image is three 3x3 matrices, the number of elements in the conversion matrix is 27.

[0071] S410, F' is obtained, F' = [f min ,f max ], f min = min(f 1,1 ,f 2,1 ,…,f h,1 ,…,f H,1 ), f max = max(f 1,2 ,f 2,2 ,…,f h,2 ,…,f H,2 ).

[0072] According to S110-S410 of this embodiment, the overall value range corresponding to all elements in the conversion matrix can be obtained, and the overall value range is taken as the value range F' of the conversion matrix to limit the amplitude of adjustment of the intermediate elements in the conversion matrix, so that the element value after adjustment of the intermediate elements in the conversion matrix can be prevented from exceeding the overall value range corresponding to all elements in the conversion matrix before adjustment.

[0073] As a specific implementation, two buttons are set on the user interface, the first button corresponds to increasing VMAF, and the second button corresponds to increasing PSNR; if it is obtained that the user clicks the first button, it is determined that the parameter adjustment requirement information input by the user is information indicating to increase VMAF; if it is obtained that the user clicks the second button, it is determined that the parameter adjustment requirement information input by the user is information indicating to increase PSNR.

[0074] In addition to the advantages of Embodiment One, this embodiment can obtain a video with relatively high VMAF or relatively high PSNR according to the user's demand; specifically, the updating module of this embodiment is configured to update the conversion matrix of the input image according to the parameter adjustment requirement information input by the user, wherein the parameter adjustment requirement information input by the user can reflect the user's demand to obtain an enhanced image with relatively high VMAF or relatively high PSNR; after the updating of the conversion matrix of the input image by the updating module, the VMAF or PSNR of the enhanced image output by the second sub-model is improved relative to the case where the conversion matrix of the input image is not updated, thereby meeting the user's demand to obtain an enhanced image with relatively high VMAF or relatively high PSNR.

[0075] While certain specific embodiments of the application have been described in detail herein for the purposes of exemplification, numerous other variations and modifications will be apparent to persons skilled in the art. Alterations and modifications of detail can be made by those skilled in the art, having the benefit of the above description, without departing from the spirit and scope of the application. It is intended that all such alterations and modifications be included within the scope of the application whose limits are to be determined only by the appended claims.

Claims

1. An image enhancement method based on video IP frames, characterized in that, Includes the following steps: S10, Obtain the target video V, V=(v1,v2,…,v…) q ,…,v Q ), v q Let V be the q-th frame of images included, where q ranges from 1 to Q, and Q is the number of images included in V. q For v q+1 The previous frame image, v q+1 V is the (q+1)th frame of the image; v q For keyframes or forward predictive coded frames in V; S20, parse V and obtain the list L of frames corresponding to the keyframes in V, L=(id1,id2,…,id…). m ,…,id M ), id m Let m be the frame number corresponding to the m-th keyframe in V, where m ranges from 1 to M, and M is the number of keyframes included in V. S30, sequentially traverse V, if q matches a certain id in L... m If they are equal, proceed to S40; otherwise, proceed to S50. S40, obtain v using the first sub-model in the trained target neural network model. q Transformation matrix T q and T q The latest transformation matrix T is determined. new When each v in V q When the image is a single-channel image, the transformation matrix of the input image is a 3×3 matrix; when each v in V q When the image is a 3-channel image, the transformation matrix of the input image is three 3×3 matrices; S41, T q As input to the second sub-model in the trained target neural network model, the output of the second sub-model in the trained target neural network model is used as v. q The corresponding enhanced image; S50, obtain T new ; S51, T new As input to the second sub-model in the trained target neural network model, the output of the second sub-model in the trained target neural network model is used as v. q The corresponding enhanced image; The training process of the target neural network model includes the following steps: S1, obtaining the image sample list A; S2, obtain the image sample label list B; S3, use A and B to train the target neural network model.

2. The image enhancement method based on video IP frames according to claim 1, characterized in that, Use ffmpeg to parse V.

3. The image enhancement method based on video IP frames according to claim 1, characterized in that, The target neural network model includes a first sub-model and a second sub-model. The output of the first sub-model is the input of the second sub-model. The first sub-model is used to obtain the transformation matrix of the input image. The first sub-model includes, in sequence according to the direction of information transmission, four cascaded convolutional layers, a two-dimensional adaptive average pooling layer, a flatten layer, a first linear layer, a ReLU activation layer, a second linear layer, and a sigmoid activation layer. The output of the sigmoid activation layer is the transformation matrix of the input image. The second sub-model is used to obtain the enhanced image corresponding to the input image based on the output of the first sub-model. The second sub-model includes a convolution operation module and a summation operation module. The input of the convolution operation module is the input image and the output of the first sub-model. The input of the summation operation module is the input image and the output of the convolution operation module. The output of the summation operation module is the enhanced image corresponding to the input image.

4. The image enhancement method based on video IP frames according to claim 3, characterized in that, A=(a1,a2,…,a n ,…,a N ), a n Let n be the nth image sample, where n ranges from 1 to N, and N is the number of image samples; each a n The number and type of channels are the same as those in the image in V; B=(b1,b2,…,b n ,…,b N ), b n For a n The image after image enhancement.

5. The image enhancement method based on video IP frames according to claim 3, characterized in that, When each v in V q When the image is a single-channel image, the target neural network model is a neural network model for obtaining an enhanced image of the single-channel image, and each convolutional layer includes one 3×3 convolutional kernel.

6. The image enhancement method based on video IP frames according to claim 3, characterized in that, When each v in V q When the image is a 3-channel image, the target neural network model is a neural network model for acquiring enhanced images of the 3-channel image, and each convolutional layer includes 3 3×3 convolutional kernels.

7. The image enhancement method based on video IP frames according to claim 4, characterized in that, b n To use the Clahe algorithm to analyze a n The image after image enhancement.

Images