A method and system for video sharpness enhancement

Abstract

The present application relates to the technical field of resolution enhancement, in particular to a video definition enhancement method and system, in the present application, through the window accumulation of adjacent frame gray difference and coordinate difference, the minimum displacement is selected and the mapping is updated, the boundary neighborhood weighted correction is corrected, the cross-frame misplacement is reduced and the motion area connection is ensured, three frame gray sequences are captured to calculate the difference value, the replacement value is written according to the monotony and amplitude conditions, the replacement value and the aligned frame difference are input into the Fourier transform, the residual field is decomposed into amplitude and phase distribution in the frequency domain, the abnormal block is removed through the amplitude and phase consistency when the frame is superimposed, the absolute value of the neighborhood block row difference and column difference in the residual iterative texture matrix is accumulated as the energy, the energy and the center difference are combined with the threshold value to output the mark, the texture area and the edge area are distinguished; the brightness is updated according to the weight mean value for the low-frequency position, the maximum difference and the direction difference are combined for the high-frequency position, the edge gradient is extracted through the convolutional neural network and the correction value is generated, the edge step and the blur are suppressed, and the high-frequency detail continuity is maintained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of resolution enhancement, in particular to a video definition enhancement method and system. BACKGROUND

[0002] The technical field of resolution enhancement aims to convert low-resolution video or image signals into high-resolution signals through mathematical modeling and signal processing methods, enhance the performance of picture details and structural information, improve the sharpness of edge contours, restore texture details, suppress blur and artifacts, and optimize visual quality without increasing the bandwidth of the original data.

[0003] The purpose of a video definition enhancement method is to process low-resolution or limited-quality video sources to generate high-resolution and more complete detail video output, improve the problems of blur, distortion and detail loss caused by limitations of acquisition equipment, compression encoding or insufficient transmission bandwidth, and make the video present a clearer and more natural visual effect in the process of playing and displaying by enhancing the picture resolution and restoring the texture structure, so as to meet the requirements of video quality in ultra-high-definition display, professional monitoring and diagnosis, and high-quality content production scenes.

[0004] In the prior art, when enlarging and sharpening low-resolution sequences, the cross-frame misalignment is processed as static pixels, motion objects and boundaries produce trailing and ghosting, subtitles scrolling areas appear broken, under simple interpolation and time average operation, compressed noise and high-frequency texture are enlarged at the same time, moire and jaggies are formed in fine grid and diagonal line areas, block noise appears in low-texture areas, in frequency domain processing, there is a lack of amplitude and phase constraints, periodic interference and encoding artifacts are mixed with real textures, edges produce halos and overshoot, under the unified processing of high and low frequencies, low-frequency brightness drift accumulates to cause flicker, texture repeatedly appears in static pictures, details disappear in long-range areas, near-range edges are broken, and small target recognition in monitoring and medical imaging scenes appears misjudgment. SUMMARY

[0005] The purpose of the present application is to solve the shortcomings in the prior art and propose a video definition enhancement method and system.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows: a video definition enhancement method, comprising the following steps:

[0007] S1: based on a low-resolution video frame sequence, establishing frame index and sorting, extracting pixel gray sequence, calculating adjacent frame gray difference and coordinate difference, window cumulative difference and selecting minimum displacement, updating pixel mapping and weighting boundary neighborhood, and generating time sequence alignment matrix;

[0008] S2: Based on the timing alignment matrix, three frames of gray sequence are captured and the difference is calculated, the monotony or fluctuation amplitude is judged, the current gray or mean value is replaced, the Fourier transform is adopted, the residual field is calculated by replacing the difference of the alignment frame, the residual field is superimposed and the true and false judgment is performed to screen out abnormal blocks, and the residual iterative texture matrix is obtained;

[0009] S3: Based on the residual iterative texture matrix, the pixel neighborhood block is extracted and the row difference and column difference are calculated, the energy value is obtained by accumulating the absolute difference, the energy value is compared with the center difference and the threshold value is combined to output the flag, the flag is high frequency or low frequency and the index is recorded, and the frequency domain marking matrix is generated.

[0010] S4: Based on the frequency domain marking matrix, the mean value of the low frequency area is updated according to the weight, the convolutional neural network is adopted, the high frequency area is corrected according to the maximum difference and the direction difference, the low frequency and the high frequency are combined and added point by point with the residual iterative texture matrix, and the texture compensation matrix is obtained.

[0011] S5: Based on the texture compensation matrix, the gray sequence is established on the time axis and the difference is calculated, the difference is compared with the threshold value to select the weighted average or the center frame, the pixel synthesis is performed and written into the buffer, and the high-definition sequence frame result is established.

[0012] As a further scheme of the application, the timing alignment matrix includes pixel position index, gray difference value distribution, neighborhood weighting parameter, the residual iterative texture matrix includes residual value field, judgment mark set, texture enhancement block, the frequency domain marking matrix includes high frequency block index, low frequency block index, energy difference value record, the texture compensation matrix includes brightness update value, edge correction value, synthesis pixel value, and the high-definition sequence frame result includes frame index sequence, pixel gray set and time sequence smoothing output set.

[0013] As a further scheme of the application, the specific steps of the residual iterative texture matrix are:

[0014] Based on the low-resolution video frame sequence, the frame index is established and sorted, the pixel gray sequence is extracted, the gray difference and coordinate difference of adjacent frames are calculated, the window cumulative difference is selected and the minimum displacement is selected, the pixel mapping is updated and the weighted boundary neighborhood is generated, and the timing alignment matrix is generated.

[0015] Based on the timing alignment matrix, the mapping relationship of the frame index is called, the pixel position is compared point by point, the mapping index is recorded according to the difference value size, and the residual iterative texture matrix is generated.

[0016] As a further scheme of the application, the specific steps of generating the residual iterative texture matrix are:

[0017] Based on the time sequence alignment matrix, three frames of gray scale sequences are captured and the difference is calculated to judge the monotony or fluctuation amplitude of the gray scale change, the current gray scale or the average value is selected as the replacement value, and a gray scale replacement matrix is generated;

[0018] Based on the gray scale replacement matrix, a Fourier transform is used to calculate the residual field formed by the difference between the replacement value and the aligned frame, the residual field is added frame by frame, and a true or false judgment is performed to screen out abnormal blocks, and a residual judgment matrix is generated.

[0019] Based on the residual judgment matrix, the texture data is reconstructed according to the index and the residual information is accumulated, the residual distribution is recorded and the texture matrix is output, and the residual iterative texture matrix is obtained.

[0020] As a further scheme of the present application, the Fourier transform first receives pixel data in the gray scale replacement matrix as input, converts the discrete signal represented by the gray scale value of each pixel point in the time domain into a frequency domain representation, obtains the amplitude and phase of each frequency component, and performs a discrete Fourier transform on the input pixel gray scale sequence to decompose it into a superposition form of a finite number of sine and cosine basis functions. Then, the difference signal between the replacement value and the aligned frame is mapped in the frequency domain, the residual intensity under the frequency distribution is calculated, the residual field is formed, and after frame-by-frame superposition, the true or false judgment is performed using the amplitude threshold value and the phase consistency condition to remove abnormal blocks in the frequency domain features and generate the residual judgment matrix.

[0021] As a further scheme of the present application, the specific steps for generating the texture compensation matrix are:

[0022] Based on the residual iterative texture matrix, the pixel neighborhood blocks are extracted and the row difference and column difference are calculated, the absolute difference is accumulated to obtain the energy value, the energy value is compared with the center difference and combined with the threshold to generate a flag, the index is recorded and a frequency domain marker matrix is established.

[0023] Based on the frequency domain marker matrix, the neighborhood blocks corresponding to the marker region are selected, the center difference and the energy value are called according to the index, the high frequency or low frequency position is distinguished, and the marker is written to generate the texture compensation matrix.

[0024] As a further scheme of the present application, the specific steps for generating the texture compensation matrix are:

[0025] Based on the frequency domain marker matrix, the average value of the low frequency region is updated according to the weight ratio, and the correction result is written into the matrix unit to generate a low frequency brightness matrix.

[0026] Based on the low frequency brightness matrix, a convolutional neural network is used to calculate the difference value of the high frequency region and combine the directional difference to correct the edge, and the edge correction value and the original high frequency region are merged to generate a high frequency correction matrix.

[0027] Based on the high frequency correction matrix, residual iterative texture matrix is called and added point by point, low frequency and high frequency matrix are combined and written into output frame, texture compensation matrix is obtained.

[0028] As a further scheme of the application, the convolutional neural network receives the high-frequency region pixel block marked by the low-frequency brightness matrix as input, sequentially performs convolution operation through multiple convolution units, each convolution unit uses a preset convolution kernel to extract directional gradient and texture boundary feature, after convolution operation, a pooling unit is used to down-sample feature mapping, reduce data dimension and retain main edge feature distribution, after convolution and pooling processing, a full connection unit is used to calculate difference value amplitude and direction difference of the high-frequency region, and generate edge correction value, the edge correction value is merged with the original high-frequency region point by point to form a high-frequency correction matrix.

[0029] As a further scheme of the application, the specific steps for generating the high-definition sequence frame result are:

[0030] Based on the texture compensation matrix, a gray sequence is established on the time axis and the adjacent difference is calculated, the difference value and the threshold value relationship are judged, the weighted average or the center frame is selected for pixel synthesis, and a pixel synthesis buffer is generated;

[0031] Based on the pixel synthesis buffer, the gray data is written frame by frame and the time sequence is maintained, the synthesis frame is updated according to the index and the output is established, and the high-definition sequence frame result is obtained.

[0032] A video definition enhancement system for executing the above-mentioned video definition enhancement method, the system comprises:

[0033] A time sequence alignment module: based on the low-resolution video frame sequence, an index sequence is established, a pixel gray sequence is extracted, the gray difference and the coordinate difference between adjacent frames are calculated, the minimum displacement is selected by window accumulation, the mapping is updated and the boundary is weighted, and a time sequence alignment matrix is obtained.

[0034] A residual construction module: based on the time sequence alignment matrix, the difference value is calculated by grabbing three gray sequences, the monotonicity or fluctuation amplitude is judged, and the replacement matrix is written, the Fourier transform is performed on the replacement value and the aligned frame difference to generate a residual field, the residual iterative texture matrix is obtained by frame superposition and through amplitude and phase determination of abnormal blocks.

[0035] A frequency domain marking module: based on the residual iterative texture matrix, a neighborhood block is extracted, the row difference and the column difference are calculated and the absolute value is accumulated, the energy value and the center difference are compared combined with a threshold to output a mark, and a frequency domain marking matrix is obtained.

[0036] A texture compensation module: based on the frequency domain marking matrix, the brightness is updated according to the weight mean in the low-frequency region, the difference value and the direction difference in the high-frequency region are called and the edge is corrected by the convolutional neural network, and the result is written into the matrix to obtain a texture compensation matrix.

[0037] Sequence synthesis module: based on the texture compensation matrix, the time axis gray sequence is calculated, the adjacent difference value is compared, the weighted average or the center frame is selected and written into the buffer, the index output is updated frame by frame, and the high-definition sequence frame result is obtained.

[0038] Compared with the prior art, the advantages and positive effects of the present application are that:

[0039] In the present application, by window accumulation of adjacent frame gray difference and coordinate difference, the minimum displacement is selected and the mapping is updated, the boundary neighborhood weighted correction is corrected, the cross-frame misplacement is reduced and the motion region is connected, and the three-frame gray sequence is calculated, the difference value is written according to the monotonicity and amplitude condition, the replacement value is input into the Fourier transform, the residual field is decomposed in the frequency domain, the abnormal block is removed according to the amplitude and phase consistency, and the noise and periodic artifacts are effectively isolated.

[0040] In the present application, the absolute value of the neighborhood block row difference and column difference in the residual iterative texture matrix is accumulated as energy, the energy and the center difference are combined with the threshold value to output the flag, and the texture area and the edge area are distinguished; the brightness is updated according to the weighted mean value for the low-frequency position, and the edge gradient is extracted and the correction value is generated by the convolutional neural network for the high-frequency position combined with the maximum difference and the direction difference, so that the edge step and the blur are suppressed, and the high-frequency details are continuously maintained.

[0041] In the present application, the low-frequency and high-frequency results are combined and added point by point with the residual matrix to obtain the texture compensation matrix, the gray sequence is established on the time axis and the difference value is compared, the weighted average or the center frame is written into the buffer according to the threshold value, the time sequence flicker is suppressed, and the high-definition sequence frame presents details in continuous playing and complete edges. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The present application is a workflow schematic diagram. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application. Example 1

[0044] Please refer to Figure 1 The present application provides a technical scheme: a video definition improvement method, comprising the following steps:

[0045] S1: Based on the low-resolution video frame sequence, establish frame index and order, extract pixel gray sequence, calculate adjacent frame gray difference and coordinate difference, window cumulative difference and select the minimum displacement, update pixel mapping and weighted boundary neighborhood, generate time alignment matrix;

[0046] S2: Based on the time alignment matrix, capture three frame gray sequences and calculate the difference, judge the monotony or fluctuation amplitude, select the current gray or mean value instead, use Fourier transform, calculate the residual field formed by the difference between the replacement value and the alignment frame, superimpose the residual field and perform true or false judgment to exclude abnormal blocks, and obtain the residual iterative texture matrix;

[0047] S3: Based on the residual iterative texture matrix, extract the pixel neighborhood block and calculate the row difference and column difference, accumulate the absolute difference to obtain the energy value, compare the energy value with the center difference and combine the threshold value to output the flag, the flag is high frequency or low frequency and the index is recorded, and the frequency domain marker matrix is generated;

[0048] S4: Based on the frequency domain marker matrix, update the brightness by weighted mean in the low frequency area, correct the edge in the high frequency area by maximum difference and direction difference using convolutional neural network, combine the low frequency and high frequency and add them point by point with the residual iterative texture matrix to obtain the texture compensation matrix;

[0049] S5: Based on the texture compensation matrix, establish the gray sequence on the time axis and calculate the difference, judge the difference and threshold relationship to select the weighted average or center frame, perform pixel synthesis and write into the buffer to establish the high-definition sequence frame result.

[0050] The time alignment matrix includes pixel position index, gray difference value distribution and neighborhood weighting parameter, the residual iterative texture matrix includes residual value field, judgment marker set and texture enhancement block, the frequency domain marker matrix includes high frequency block index, low frequency block index and energy difference value record, the texture compensation matrix includes brightness update value, edge correction value and synthesis pixel value, and the high-definition sequence frame result includes frame index sequence, pixel gray set and time sequence smoothing output set.

[0051] The specific steps of the residual iterative texture matrix are:

[0052] Based on the low-resolution video frame sequence, establish frame index and order, extract pixel gray sequence, calculate adjacent frame gray difference and coordinate difference, window cumulative difference and select the minimum displacement, update pixel mapping and weighted boundary neighborhood, generate time alignment matrix;

[0053] Based on the time alignment matrix, call the mapping relationship of frame index, compare the pixel position point by point, filter and record the mapping index according to the difference value size, and generate the residual iterative texture matrix;

[0054] Based on the low-resolution video frame sequence, the input frame is indexed and sorted using a quicksort algorithm, the median is selected as the pivot, the left and right pointers are exchanged, the index sequence is recursively output and the frame array is rearranged, the three-channel pixel matrix is converted to a two-dimensional gray matrix point by point using the gray scale conversion method, the pixel-by-pixel difference operation is performed on adjacent frames to generate a gray difference matrix, and the horizontal and vertical coordinates of the same position pixel are calculated to generate a coordinate difference matrix. The difference is accumulated in a sliding window of 5 frames and the result is saved. The minimum value finding step records the displacement index, updates the mapping table, and performs weighted processing in the boundary area using a 3x3 weight matrix. Finally, the timing alignment matrix is output.

[0055] Based on the timing alignment matrix, the difference value matrix is generated by comparing the gray values point by point using the mapping relationship, the ascending order is sorted and the threshold is set to the top 10% of the difference value position, the mapping index is filtered and recorded, and the residual matrix is generated by subtracting the current texture matrix from the difference value matrix using the matrix residual iteration method. The cumulative matrix is generated by adding the last round residual to the current residual item by item through the iterative update method, and the new texture matrix is updated. The matrix is smoothed using a 3x3 convolution kernel. Finally, the residual iteration texture matrix is output.

[0056] The specific steps for generating the residual iteration texture matrix are as follows:

[0057] Based on the timing alignment matrix, three frames of gray sequence are captured and the difference is calculated. The monotony or fluctuation amplitude of the gray change is judged, the current gray or average value is selected as the replacement value, and the gray replacement matrix is generated.

[0058] Based on the gray replacement matrix, the Fourier transform is used to calculate the residual field formed by the replacement value and the aligned frame difference. The residual field is added frame by frame and the true and false judgment is performed. The abnormal blocks are filtered out to generate the residual judgment matrix.

[0059] Based on the residual judgment matrix, the texture data is reconstructed according to the index and the residual information is accumulated. The residual distribution is recorded and the texture matrix is output to obtain the residual iteration texture matrix.

[0060] Based on the timing alignment matrix, the gray sequence of three consecutive frames is captured, and the pixel-by-pixel difference operation is used to calculate the corresponding pixels of the three frames. The input is the gray value corresponding to the position index, and the output is two groups of difference results. The sign judgment operation is performed on the difference value result. If the difference value signs are consistent for two consecutive times, it is judged as monotonic change. If the absolute value of the difference value is greater than the set threshold of 20, it is judged as too large fluctuation amplitude. The current frame gray value is retained as the replacement value for the pixels judged as monotonic change. The arithmetic mean value is calculated for the pixels judged as fluctuation, the input is the gray value of three frames, and the output is the average value. Finally, the matrix is written point by point to generate the gray replacement matrix.

[0061] Based on the gray scale substitution matrix, using fast Fourier transform FFT algorithm for matrix frame processing, input is two-dimensional gray scale substitution matrix, set the parameter for the number of sampling points 1024, transform direction for positive transform, output frequency domain matrix, point by point restore to time domain residual value and difference with the corresponding pixel value of the aligned frame to generate residual field, residual field according to the frame index order item by item superposition and form the cumulative matrix, set the threshold value of the cumulative matrix 50 to execute true false judgment step, more than threshold value mark false, below threshold value mark true, mark false pixel block is directly screened out and zero, output the remaining data, generate residual judgment matrix;

[0062] Based on the residual judgment matrix, reconstruct the texture data according to the index order, call the matrix reconstruction step to find the effective index in the judgment matrix point by point and extract the corresponding gray scale from the original frame matrix to the reconstruction matrix, perform addition operation point by point between the reconstruction matrix and the residual judgment matrix and get the residual cumulative matrix, use the distribution record method to count the residual sum of each row and each column in the matrix and output the distribution table, write the content of the distribution table into the independent storage unit and form the residual distribution matrix, finally combine the residual cumulative matrix and the residual distribution matrix to output, get the residual iteration texture matrix.

[0063] Fourier transform, first receive the pixel data in the gray scale substitution matrix as input, convert the discrete signal expressed in time domain of each pixel gray value into frequency domain expression, get the amplitude and phase of each frequency component, and execute discrete Fourier transform on the input pixel gray sequence, decompose into a finite number of sine and cosine basis function superposition form, then map the difference signal between the substitution value and the aligned frame in frequency domain, calculate the residual intensity under frequency distribution, form the residual field, and after frame by frame superposition, use the amplitude threshold value and phase consistency condition to judge true and false, remove the abnormal blocks in the frequency domain characteristics, generate the residual judgment matrix;

[0064] Fourier transform, according to the formula:

[0065] ;

[0066] Among them: Residual frequency domain cumulative value, Frame number, Total number of superposition frames, Time domain weight coefficient of the first Frame, Horizontal space index, Vertical space index, Horizontal direction size of matrix pixel number, Vertical direction size of matrix pixel number, Space window function weight, Residual field of the first Frame, represents a gray scale substitution matrix pixel value, represents a first represents an aligned frame difference matrix pixel value, represents a horizontal direction sub-pixel offset correction amount, represents a vertical direction sub-pixel offset correction amount, represents a horizontal direction frequency index, represents a vertical direction frequency index, represents an imaginary unit, represents an abnormal block mask value of 0 or 1, represents performing rejection on the abnormal block position in the calculation process;

[0067] The execution process is as follows: first, select the t-th frame image in the input video sequence and calculate the residual field , wherein A(x, y) is a gray scale substitution matrix pixel value, is an aligned frame difference matrix pixel value, then perform sub-pixel offset correction on the residual field in the spatial domain to obtain so as to compensate for the inter-frame displacement error, then apply a window function in the spatial position to reduce the boundary effect and spectral leakage, and then perform rejection on the detected abnormal area through an abnormal block mask to ensure that the effective pixels participate in the calculation, then perform discrete Fourier transform on the corrected residual field in the frequency domain and obtain the frequency domain components corresponding to the frequency indexes u and v, and further utilize the weight coefficients to perform weighted accumulation on the multi-frame residual frequency domain results to suppress high-noise frames and enhance the contribution of stable frames, finally obtain the residual frequency domain accumulation and generate a residual judgment matrix to realize video definition improvement.

[0068] The specific steps for generating the texture compensation matrix are as follows:

[0069] Based on the residual iteration texture matrix, extract the pixel neighborhood block and calculate the row difference and column difference, accumulate the absolute difference to obtain the energy value, compare the energy value with the center difference and generate a flag combined with the threshold, record the index and establish a frequency domain marking matrix;

[0070] Based on the frequency domain marking matrix, select the neighborhood block corresponding to the marked area, call the center difference and energy value according to the index, distinguish the high-frequency or low-frequency position and write the mark, and generate the texture compensation matrix;

[0071] Based on the residual iterative texture matrix, a neighborhood block with a size of 3 rows and 3 columns is extracted for each pixel point in the matrix, a row difference algorithm is used to subtract each point in the neighborhood block in the row direction and output a row difference array, a column difference algorithm is used to subtract each point in the neighborhood block in the column direction and output a column difference array, the absolute value of each item in the row difference array and the column difference array is taken and accumulated to obtain the energy value corresponding to the neighborhood block, and a comparison operation is performed on the energy value and the center pixel difference value. When the result of the energy value minus the center difference value is greater than a set threshold 30, a flag 1 is generated, and when the result is less than or equal to the threshold 30, a flag 0 is generated. The generated flag is stored in the index table in one-to-one correspondence with the index of the neighborhood block, and a two-dimensional matrix is established in the frequency coordinate system. The flags in the index table are written point by point, and a frequency domain marker matrix is output.

[0072] Based on the frequency domain marker matrix, the matrix position is scanned point by point and the region with a marker value of 1 is selected, the corresponding neighborhood block is located and the original index position is extracted, the center difference value recorded in the index table is called and compared with the energy value. If the energy value is greater than the center difference value, it is determined to be a high-frequency position and a marker value 2 is written, and if the energy value is less than or equal to the center difference value, it is determined to be a low-frequency position and a marker value 1 is written. The written markers are filled into a new matrix point by point and correspond to the original neighborhood block index. Finally, a combined matrix is output as a texture compensation matrix.

[0073] The specific steps for generating the texture compensation matrix are as follows:

[0074] Based on the frequency domain marker matrix, the mean value of the low-frequency region is updated according to the weight proportion, and the correction result is written into the matrix unit to generate a low-frequency brightness matrix.

[0075] Based on the low-frequency brightness matrix, a convolutional neural network is used to calculate the difference value of the high-frequency region and combine the directional difference to correct the edge. The edge correction value is combined with the original high-frequency region to generate a high-frequency correction matrix.

[0076] Based on the high-frequency correction matrix, the residual iterative texture matrix is called and added point by point, and the low-frequency and high-frequency matrices are combined and written into the output frame to obtain the texture compensation matrix.

[0077] Based on the frequency domain marker matrix, the gray value of each point in the low-frequency region is extracted, and a weighted average algorithm is used for brightness update. The weight proportion matrix is set to 3 rows and 3 columns, the center position weight value is 0.4, the four edge position weight values are 0.1, and the four corner position weight values are 0.05. The gray values of the pixels in the neighborhood are multiplied by the corresponding weights point by point and accumulated to obtain the updated brightness value of the low-frequency region. The updated brightness value is written into the original matrix unit position to replace the old value. The low-frequency brightness matrix is finally output by traversing all low-frequency regions of the matrix and completing the writing point by point.

[0078] Based on the low-frequency brightness matrix, the high-frequency region is processed by using the convolutional neural network (CNN) method. The input is the high-frequency region pixel block, the convolution kernel size is set to 3 rows and 3 columns, the step is 1, the padding method is same, the activation function is ReLU, and the loss function is mean square error. The difference between the current gray value and the neighborhood average is calculated point by point for the high-frequency region to form a difference matrix. The direction gradient calculation module is called to perform difference accumulation in the horizontal and vertical directions respectively and generate a direction difference matrix. The difference matrix and the direction difference matrix are weighted point by point, and the weight value is set to 0.6 for the difference and 0.4 for the direction difference. The edge correction value is obtained. The edge correction value is added point by point to the original high-frequency region matrix and written into a new matrix. Finally, the high-frequency correction matrix is output.

[0079] Based on the high-frequency correction matrix, the residual iterative texture matrix is called. The two matrices are added point by point, the input is the value of the corresponding position of the two matrices, and the output is the point-by-point cumulative value to form an intermediate merging matrix. The intermediate merging matrix and the low-frequency brightness matrix are added point by point to obtain a complete texture matrix. The texture matrix is written point by point into the output frame buffer to record the cumulative value of each row and each column and update the output frame unit. Finally, the texture compensation matrix is generated.

[0080] The convolutional neural network receives the high-frequency region pixel block marked by the low-frequency brightness matrix as input. The convolution operation is performed through multiple convolution units in turn. Each convolution unit uses a preset convolution kernel to extract the direction gradient and texture boundary feature. After convolution operation, the pooling unit is used to downsample the feature mapping, reducing the data dimension and preserving the main edge feature distribution. After convolution and pooling processing, the full connection unit is used to calculate the difference amplitude and direction difference of the high-frequency region, and the edge correction value is generated. The edge correction value is merged with the original high-frequency region point by point to form a high-frequency correction matrix.

[0081] Fourier transform, according to the formula:

[0082] ;

[0083] Wherein: represents the high-frequency correction matrix, represents the high-frequency region matrix, represents the horizontal direction pixel index, represents the vertical direction pixel index, represents the global enhancement weight coefficient, represents the low-frequency brightness matrix, represents the low-pass convolution kernel, represents the soft threshold function controlled by the threshold , represents the edge confidence map output by the convolutional neural network, represents the edge confidence threshold parameter, represents the direction angle an edge convolution kernel is constructed, a direction parameter representing edge detection, a magnitude operation is represented, a nonlinear enhancement index is represented;

[0084] The execution process is as follows: first, the high-frequency region matrix is read , then the low-frequency brightness matrix is used , and the low-pass convolution kernel is , a two-dimensional convolution operation is performed to obtain , and the difference term is calculated , then the edge confidence map output by the convolutional neural network is obtained , and the soft threshold function is used for nonlinear compression , then the edge convolution kernel is constructed with the direction parameter , and the convolution result is calculated , the result is taken as the magnitude , and the nonlinear enhancement index is applied , the global enhancement weight coefficient is further applied to the product of the above difference term, confidence correction term and direction enhancement factor, and finally the weighted correction result is added point by point with to obtain the high-frequency correction matrix . The specific steps for generating a high-definition sequence frame result are as follows:

[0085] Based on the texture compensation matrix, the gray sequence is established on the time axis, the adjacent difference is calculated, the difference and the threshold value are judged, the weighted average or the center frame is selected for pixel synthesis, and the pixel synthesis buffer is generated;

[0086] Based on the pixel synthesis buffer, the gray data is written frame by frame and the time sequence is maintained, the synthesis frame is updated according to the index and the output is established, and the high-definition sequence frame result is obtained;

[0087] Based on the texture compensation matrix, the gray sequence of each pixel point is extracted on the time axis, the difference operation method is used to perform subtraction operation point by point on the adjacent frame gray values and output the difference sequence, the difference sequence is compared point by point with the set threshold value 50, when the difference is greater than the threshold value, the weighted average method is used, the weight is set to 0.3 for the previous frame, 0.4 for the current frame and 0.3 for the next frame, the three frame gray values are multiplied point by point and accumulated to obtain the synthesis value, when the difference is less than or equal to the threshold value, the center frame gray value is directly selected as the synthesis value, the synthesis result of each pixel point is written point by point into the cache matrix, and finally the pixel synthesis buffer is generated;

[0088] Based on the texture compensation matrix, the gray sequence of each pixel point is extracted on the time axis, the difference operation method is used to perform subtraction operation point by point on the adjacent frame gray values and output the difference sequence, the difference sequence is compared point by point with the set threshold value 50, when the difference is greater than the threshold value, the weighted average method is used, the weight is set to 0.3 for the previous frame, 0.4 for the current frame and 0.3 for the next frame, the three frame gray values are multiplied point by point and accumulated to obtain the synthesis value, when the difference is less than or equal to the threshold value, the center frame gray value is directly selected as the synthesis value, the synthesis result of each pixel point is written point by point into the cache matrix, and finally the pixel synthesis buffer is generated;

[0089] ​Based on the pixel synthesis buffer, the cache matrix is traversed frame by frame and the gray data is extracted, a sequential writing method is used to write the gray matrix of each frame into the output buffer in sequence, the writing order is kept consistent with the time index, the index updating step is performed on the synthesis frame, the index updating method is that the frame number is sorted in ascending order according to the timestamp, then the old index is covered point by point and the synthesis frame is rewritten, after the updating operation of all frames is completed, the data in the output buffer is written into the output sequence in time sequence frame by frame, and finally the high-definition sequence frame result is obtained.

[0090] A video definition enhancement system is used to execute the above-mentioned video definition enhancement method, and the system comprises:

[0091] A time sequence alignment module: based on the index sorting of the low-resolution video frame sequence, the pixel gray sequence is extracted, the gray difference and the coordinate difference of adjacent frames are calculated, the minimum displacement is selected by window accumulation, the mapping is updated and the boundary is weighted, and the time sequence alignment matrix is obtained.

[0092] A residual construction module: based on the time sequence alignment matrix, the difference value is calculated by grabbing three gray sequences, the monotony or fluctuation amplitude is judged and written into the replacement matrix, the Fourier transform is performed on the replacement value and the alignment frame difference to generate the residual field, the abnormal blocks are determined by frame superposition and amplitude and phase, and the residual iterative texture matrix is obtained.

[0093] A frequency domain marking module: based on the residual iterative texture matrix, the neighborhood block is extracted, the row difference and column difference are calculated and the absolute value is accumulated, the energy value and the center difference are compared combined with the threshold value to output the mark, and the frequency domain marking matrix is obtained.

[0094] A texture compensation module: based on the frequency domain marking matrix, the brightness is updated according to the weight mean in the low-frequency area, the difference value and the directional difference are called in the high-frequency area and the edge is corrected by the convolutional neural network, the result is written into the matrix, and the texture compensation matrix is obtained.

[0095] A sequence synthesis module: based on the texture compensation matrix, the time axis gray sequence is calculated, the adjacent difference value is compared with the threshold value to select the weighted average or the center frame to write into the buffer, the index is updated frame by frame and output, and the high-definition sequence frame result is obtained.

[0096] The above is only a preferred embodiment of the present application, and does not limit the present application in other forms. Any skilled person in the art can modify or change the above disclosed technical content to equivalent embodiments applied to other fields, but any simple modification, equivalent change and modification made according to the technical essence of the present application to the above embodiments still belongs to the protection scope of the technical solution of the present application.

Claims

1. A method of video sharpness enhancement, characterized by, The method comprises the following steps: S1: based on the low-resolution video frame sequence, establish frame index and order, extract pixel gray sequence, calculate adjacent frame gray difference and coordinate difference, window cumulative difference and select minimum displacement, update pixel mapping and weight boundary neighborhood, generate time sequence alignment matrix; S2: based on the time sequence alignment matrix, capture three frame gray sequences and calculate the difference, judge the monotony or fluctuation amplitude, select the current gray or mean value as the replacement, use Fourier transform, calculate the residual field formed by the replacement value and the alignment frame difference, superimpose the residual field and perform true and false judgment to exclude abnormal blocks, obtain the residual iterative texture matrix; S3: based on the residual iterative texture matrix, extract pixel neighborhood block and calculate row difference and column difference, accumulate absolute difference to obtain energy value, compare energy value and center difference and combine threshold value to output flag, flag is high frequency or low frequency and record index, generate frequency domain marker matrix; S4: based on the frequency domain marker matrix, update brightness by weighted mean value in low frequency area, use convolutional neural network, correct edge in high frequency area according to maximum difference and direction difference, combine low frequency and high frequency and add point by point with the residual iterative texture matrix, obtain texture compensation matrix; S5: based on the texture compensation matrix, establish gray sequence on time axis and calculate difference, judge difference and threshold value relationship to select weighted average or center frame, perform pixel synthesis and write into buffer, establish high-definition sequence frame result; The time sequence alignment matrix includes pixel position index, gray difference value distribution and neighborhood weighting parameter, the residual iterative texture matrix includes residual value field, judgment marker set and texture enhancement block, the frequency domain marker matrix includes high frequency block index, low frequency block index and energy difference value record, the texture compensation matrix includes brightness update value, edge correction value and synthesis pixel value, and the high-definition sequence frame result includes frame index sequence, pixel gray set and time sequence smoothing output set.

2. The video resolution enhancement method of claim 1, wherein, The specific steps for generating the residual iterative texture matrix are: based on the low-resolution video frame sequence, establish frame index and order, extract pixel gray sequence, calculate adjacent frame gray difference and coordinate difference, window cumulative difference and select minimum displacement, update pixel mapping and weight boundary neighborhood, generate time sequence alignment matrix; based on the time sequence alignment matrix, call the mapping relationship of frame index, compare pixel position point by point, filter and record mapping index according to difference value size, generate residual iterative texture matrix.

3. The video resolution enhancement method of claim 1, wherein, The specific steps for generating the residual iterative texture matrix are: based on the time sequence alignment matrix, capture three frame gray sequences and calculate the difference, judge the monotony or fluctuation amplitude of gray change, select the current gray or mean value as the replacement value, generate gray replacement matrix; based on the gray replacement matrix, use Fourier transform to calculate the residual field formed by the replacement value and the alignment frame difference, superimpose the residual field frame by frame and perform true and false judgment to exclude abnormal blocks, generate residual judgment matrix; based on the residual judgment matrix, reconstruct texture data according to index and accumulate residual information, record residual distribution and output texture matrix, obtain residual iterative texture matrix.

4. The video resolution enhancement method of claim 3, wherein, The Fourier transform first receives pixel data in the gray scale replacement matrix as input, converts the discrete signal represented by the gray scale value of each pixel point in the time domain into a frequency domain representation, obtains the amplitude and phase of each frequency component, and performs a discrete Fourier transform on the input pixel gray scale sequence to decompose it into a superposition form of a limited number of sine and cosine basis functions. Then, the difference signal between the replacement value and the aligned frame is mapped in the frequency domain, the residual intensity under the frequency distribution is calculated, the residual field is formed, and after frame-by-frame superposition, true and false are determined using the amplitude threshold value and the phase consistency condition, abnormal blocks in the frequency domain features are removed, and the residual judgment matrix is generated.

5. The video resolution enhancement method of claim 1, wherein, The specific steps for generating the texture compensation matrix are: Based on the residual iterative texture matrix, the pixel neighborhood block is extracted and the row difference and column difference are calculated, the energy value is accumulated by the absolute difference, the energy value is compared with the center difference and combined with the threshold to generate the flag, the index is recorded and the frequency domain marking matrix is established; Based on the frequency domain marking matrix, the neighborhood block corresponding to the marked area is selected, the center difference and energy value are called according to the index, the high frequency or low frequency position is distinguished and the mark is written, and the texture compensation matrix is generated.

6. The video resolution enhancement method of claim 1, wherein, The specific steps for generating the texture compensation matrix are: Based on the frequency domain marking matrix, the mean value of the low frequency area is updated according to the weight ratio, the corrected result is written into the matrix unit, and the low frequency brightness matrix is generated; Based on the low frequency brightness matrix, a convolutional neural network is used to calculate the difference value of the high frequency area and combine the directional difference to correct the edge, and the edge correction value and the original high frequency area are merged to generate a high frequency correction matrix; Based on the high frequency correction matrix, the residual iterative texture matrix is called and added point by point, the low frequency and high frequency matrices are merged and written into the output frame to obtain the texture compensation matrix.

7. The video resolution enhancement method of claim 6, wherein, The convolutional neural network receives the high frequency region pixel block marked by the low frequency brightness matrix as input, and performs convolution operation through multiple convolution units in turn. Each convolution unit uses a preset convolution kernel to extract directional gradient and texture boundary features. After convolution operation, a pooling unit is used to downsample the feature mapping, reduce the data dimension, and preserve the main edge feature distribution. After convolution and pooling processing, the difference amplitude and directional difference of the high frequency region are calculated using a fully connected unit, and an edge correction value is generated. The edge correction value and the original high frequency region are merged point by point to form a high frequency correction matrix.

8. The video resolution enhancement method of claim 1, wherein, The specific steps for generating the high-definition sequence frame result are: Based on the texture compensation matrix, a gray scale sequence is established on the time axis and the adjacent difference is calculated, the difference value and the threshold value are judged, the weighted average or the center frame is selected for pixel synthesis, and a pixel synthesis buffer is generated; Based on the pixel synthesis buffer, the gray scale data is written frame by frame and the time sequence is maintained, the synthesis frame is updated according to the index and the output is established to obtain the high-definition sequence frame result.

9. A video resolution enhancement system characterized by, The video definition enhancement method according to any one of claims 1-8, the system comprising: a time sequence alignment module: based on the low-resolution video frame sequence, an index sequence is established, a pixel gray scale sequence is extracted, the gray scale difference and the coordinate difference between adjacent frames are calculated, the minimum displacement is selected by window accumulation, the mapping is updated and the boundary is weighted, and a time sequence alignment matrix is obtained; Residual construction module: based on the time alignment matrix, the difference value of three frames of gray sequence is calculated, the monotony or fluctuation amplitude is judged, the substitute matrix is written, the Fourier transform is performed on the substitute value and the difference of the alignment frame to generate the residual field, the abnormal block is determined frame by frame, and the residual iterative texture matrix is obtained; Frequency domain marking module: based on the residual iterative texture matrix, the neighborhood block is extracted, the row difference and column difference are calculated and the absolute value is accumulated, the energy value and the center difference are compared combined with the threshold value to output the mark, and the frequency domain marking matrix is obtained; Texture compensation module: based on the frequency domain marking matrix, the low frequency area is updated according to the weight mean brightness, the difference value and the direction difference are called in the high frequency area, and the edge is corrected through the convolutional neural network, the result is written into the matrix, and the texture compensation matrix is obtained; Sequence synthesis module: based on the texture compensation matrix, the time axis gray sequence is calculated, the adjacent difference value is compared with the threshold value, the weighted average or the center frame is selected and written into the buffer area, the index is updated frame by frame, and the high definition sequence frame result is obtained.

Images