Video processing method and device, terminal equipment and computer readable storage medium

Abstract

The invention provides a video processing method and apparatus, a terminal device and a computer readable storage medium. The method comprises the steps of obtaining to-be-processed video information; determining a target video frame based on the ambiguity of the to-be-processed video information; and performing alignment fusion processing on the target video frame to obtain target video information. According to the video processing method provided by the invention, the video frame with relatively high ambiguity in the video is taken as the reference of the video frame with relatively poor ambiguity, and the blurred object in the video frame with relatively poor ambiguity is replaced, fused and the like through the same object in the video frame with relatively high ambiguity, so that the blurring in the current video frame is eliminated; and the clarity of the current video frame is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video processing method, apparatus, terminal device and computer-readable storage medium. Background Art

[0002] During video shooting, there is relative motion between the scene and the camera during camera exposure. This image blur caused by relative motion is called motion blur. Video motion blur is mainly caused by long exposure time of a single frame in poor light, fast-moving objects in the scene, and rapid shaking or rotation of the video shooting device. The most direct way to remove motion blur is to shorten the exposure time or increase the frame rate.

[0003] However, whether it is blind deblurring and non-blind deblurring based on traditional methods or AI-based methods, it is difficult to accurately eliminate motion blur from a moving terminal camera. Summary of the invention

[0004] The present application provides a video processing method, which uses a clear video frame as a reference to adjust the blurred objects in the video, thereby improving the blurriness of the current video frame.

[0005] In a first aspect, the present application provides a video processing method, the method comprising:

[0006] Get the video information to be processed;

[0007] Determining a target video frame based on the fuzziness of the video information to be processed;

[0008] The target video frames are aligned and fused to obtain target video information.

[0009] In a second aspect, the present application further provides a video processing device, the device comprising:

[0010] An acquisition module, used to acquire video information to be processed;

[0011] A determination module, configured to determine a target video frame based on the blurriness of the video information to be processed;

[0012] The adjustment module is used to perform alignment and fusion processing on the target video frame to obtain target video information.

[0013] In a third aspect, the present application also provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in any one of the video processing methods described.

[0014] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the steps in any one of the video processing methods.

[0015] The video processing method provided by the present application eliminates the blur in the current video frame by using a video frame with higher blur as a reference for a video frame with lower blur, and replaces or merges the blurred object in the video frame with lower blur with the same object in a video frame with higher blur, thereby improving the clarity of the current video frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 is a schematic diagram of a scenario of a video processing system provided in an embodiment of the present application;

[0018] Figure 2 This is a flow chart of an embodiment of a video processing method in an embodiment of the present application;

[0019] Figure 3 This is a flow chart of an embodiment of a video processing method in an embodiment of the present application;

[0020] Figure 4 This is a schematic diagram of the effect of an embodiment of the video processing method in the embodiment of the present application;

[0021] Figure 5 This is a schematic diagram of the effect of an embodiment of the video processing method in the embodiment of the present application;

[0022] Figure 6 This is a schematic diagram of the effect of an embodiment of the video processing method in the embodiment of the present application;

[0023] Figure 7 This is a schematic diagram of the effect of an embodiment of the video processing method in the embodiment of the present application;

[0024] Figure 8 is a schematic diagram of a functional module of a video processing device in an embodiment of the present application;

[0025] Fig. 9 It is a schematic diagram of the structure of the terminal device in the embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0027] In the description of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the feature. In the description of the present application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0028] In this application, the word "exemplary" is used to mean "used as an example, illustration or description". Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. At the same time, it is to be understood that in the specific implementation of this application, when user information, user data and other related data are involved, when the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data shall comply with relevant laws, regulations and standards of relevant countries and regions.

[0029] In order to enable any person skilled in the art to implement and use the present application, the following description is provided. In the following description, details are listed for the purpose of explanation. It should be understood that those of ordinary skill in the art will recognize that the present application can be implemented without using these specific details. In other examples, known structures and processes will not be elaborated in detail to avoid unnecessary details that make the description of the present application obscure. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in the present application.

[0030] The present application provides a video processing method, apparatus, device and storage medium, which are described in detail below.

[0031] See also Figure 1 , Figure 1 Schematic diagram of a video processing system provided in an embodiment of the present application. The video processing system may include a terminal device 100 and a storage device 200. The storage device 200 may transmit data to the terminal device 100. Figure 1 The terminal device 100 in the embodiment can obtain the video stored in the storage device 200 to execute the video processing method in the present application.

[0032] In the embodiment of the present application, the terminal device 100 includes but is not limited to a desktop computer, a portable computer, a network server, a PDA (Personal Digital Assistant), a tablet computer, a wireless terminal device, an embedded device, a mobile phone, etc.

[0033] In an embodiment of the present application, communication between the terminal device 100 and the storage device 200 can be achieved through any communication method, including but not limited to mobile communications based on the 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), and Worldwide Interoperability for Microwave Access (WiMAX), or computer network communications based on the TCP / IP Protocol Suite (TCP / IP) and User Datagram Protocol (UDP).

[0034] It should be noted that Figure 1 The scene diagram of the video processing system shown is only an example. The video processing system and scene described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the video processing system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0035] like Figure 2 As shown, Figure 2 This is a flow chart of an embodiment of a video processing method in an embodiment of the present application. The video processing method may include the following steps 201 to 203:

[0036] 201. Obtain video information to be processed.

[0037] In the embodiment of the present application, the video information to be processed can be a video file of any format type. Of course, the video information to be processed can also be a video being shot, or several consecutive video frames in a video. The method of obtaining the video information to be processed can be obtained through a storage medium, or it can be video information sent in real time by an image sensor, which is not limited by the specific embodiment of the present application. Therefore, the current video frame can be the latest video frame of the video being shot, and the historical video frame can be all the historical video frames before the current video frame in the video currently being shot.

[0038] 202. Determine a target video frame based on the blurriness of the video information to be processed.

[0039] In the embodiment of the present application, in order to replace or merge the blurred object in the video frame with poor blurriness with the same object in the video frame with high blurriness, the blurriness in the current video frame is eliminated, thereby improving the clarity of the current video frame. Therefore, it is necessary to determine the target video frame with the lowest blurriness in the video information to be processed.

[0040] In the embodiment of the present application, the blurriness of each video frame in the video information to be processed can be determined in the following manner:

[0041] Tenengrad gradient method, the Tenengrad gradient method uses the Sobel operator to calculate the gradient in the horizontal and vertical directions respectively. In the same scene, the higher the gradient value, the clearer the image. The measurement indicator is the average gray value of the image processed by the Sobel operator. The larger the value, the clearer the image.

[0042] Variance method, variance is a measurement method used in the probability wheel to examine the degree of discreteness (deviation) between a set of discrete data and its expectation (i.e., the mean of the data). The larger the variance, the greater the deviation between the data in this set, and some of the data in the group are larger and some are smaller, and the distribution is uneven; the smaller the variance, the smaller the deviation between the data in this set, and the data in the group are evenly distributed and similar in size. The grayscale difference between the data of a clearly focused image should be greater than that of a blurred image, that is, its variance should be larger. The variance of the image grayscale data can be used to measure the blurriness of the image. The larger the variance, the greater the blurriness.

[0043] Of course, any other existing technology may be adopted according to actual conditions to determine the blurriness of the current video frame and each historical video frame.

[0044] Furthermore, according to the above-listed methods, after determining the blurriness of each video frame in the video information to be processed, the video frame with the lowest blurriness can be used as the target video frame. It should be noted that the lower the blurriness, the higher the picture clarity.

[0045] 203. Perform alignment and fusion processing on the target video frame to obtain target video information.

[0046] According to the above embodiment, after determining the target video frame with the lowest blur, the blurriness of the video information to be processed can be processed to improve the blurriness of the video information to be processed. Specifically, the processing method is as follows:

[0047] First, the similarity between each video frame except the target video frame in the video information to be processed and the target video frame can be determined. When there are similar video frames whose similarity exceeds a certain threshold, the same object in the target video frame can be replaced with the corresponding object in the similar video frame. For example, the picture repetition between two completely adjacent video frames is almost the same. Assuming that both the target video frame and the current video frame include an object A, and object A is blurred in the current video frame, then object A in the target video frame can be replaced with object A in the current video frame. For example, a similarity threshold can be set, and then the pixel colors at the same pixel positions in the two video frames can be calculated. Assuming that more than 95% of the pixels at the same position have the same color, the two video frames can be considered to have a similarity of 95%. If the similarity of 95% exceeds the similarity threshold, object A in the current video frame can be removed, and then object A with lower blur in the target video frame can be merged to the position where object A is removed in the current video frame, and some pixel alignment operations can be performed to obtain a current video frame with smaller blur after processing. At this time, the video including the current video with smaller blur is the target video information.

[0048] The video processing method provided by the present application eliminates the blur in the current video frame by using a video frame with higher blur as a reference for a video frame with lower blur, and replaces or merges the blurred object in the video frame with lower blur with the same object in a video frame with higher blur, thereby improving the clarity of the current video frame.

[0049] In order to better implement the embodiment of the present application, in one embodiment of the present application, based on the blurriness of the video information to be processed, determining the target video frame includes:

[0050] The blurriness of each video frame in the video information to be processed is determined to obtain the video frame with the lowest blurriness; based on the matching relationship between the video frame with the lowest blurriness and the current video frame being played, the target video frame is determined.

[0051] However, in actual situations, especially when it comes to video information transmitted in real time through an image sensor, it is divided into a current video frame and a historical video frame. The current video frame is the video frame being played, and the historical video frame is the video frame before the current video frame. If the current video frame is not clear, it is only necessary to perform blur processing on the current video frame, and there is no need to perform corresponding processing on the historical video frame, thereby reducing the number of video frames to be processed, thereby reducing the processing burden of the device. Therefore, if and only if the video frame with the lowest blur is not the current video frame being played, a mismatch relationship is obtained, and only then is the video frame with the lowest blur determined to be the target video frame. At this time, the blur of the current video frame is processed based on the clearest target video frame, and the video frame viewed by the user can be the clearest video frame. Otherwise, if the video frame with the lowest blur is the current video frame, since the historical video frame has been played, there is no need to process the historical video frame.

[0052] It should be noted that the implementation method of determining the blurriness of each video frame in the video information to be processed and obtaining the video frame with the lowest blurriness is the same as the processing method in the above embodiment, and will not be described in detail here. Among them, when there is a historical video frame with the lowest blurriness among the historical video frames, the target video frame of the historical video frame with the lowest blurriness can be used as a reference historical video frame for the current video frame.

[0053] In addition, the embodiment of the present application also provides an implementation method. According to the above description, the target video frame with the lowest blur can be obtained. If the target video frame is not the current video frame, the target video frame can be determined to be one of the historical video frames. In order to provide a better processing effect in the subsequent process of performing blur removal processing on the current video frame to improve the clarity of the current video frame, in the embodiment of the present application, the frame number difference between the target video frame and the current video frame can be determined. If the frame number difference between the target video frame and the current video frame is less than the frame number threshold in the above steps, it can be determined that the picture repetition between the target video frame and the current video frame is high, and the two are two completely adjacent video frames. Since the picture repetition between the two completely adjacent video frames is almost the same, assuming that the target video frame and the current video frame both include an object A, and object A is in a blurred state in the current video frame, then the object A in the target video frame can be replaced with the object A in the current video frame.

[0054] For example: In the embodiment of the present application, a frame number threshold can be set to obtain the first few frames of video frames adjacent to the current video frame. For example: The frame number threshold can be any natural number from 4 to 6. When the frame number threshold is 6, only the first 6 frames of historical video frames adjacent to the current video frame are obtained in the acquisition of historical video frames. In the embodiment of the present application, the purpose of not acquiring all historical video frames is that the adjacent reference historical video frame with the lowest blur in the subsequent process has a high degree of picture repetition with the current video frame, and the picture repetition between the two can basically reach more than 95%. Therefore, when the clear object in the historical video frame is replaced or merged with the blurred same object in the current video frame, the terminal has a smaller data processing amount, and the naturalness of the current video frame after processing is also higher, and there will be no sense of disharmony in the picture. It should be noted that in the embodiment of the present application, the value of the frame number threshold is preferably a positive integer less than or equal to 10. Of course, the specific value can also be determined according to the actual situation, and the embodiment of the present application is not limited.

[0055] In the specific implementation process, the blurred area in the current video frame can be determined. At this time, since the number of video frames of the target video frame and the current video frame is less than a certain frame number threshold, the picture repetition is highly consistent, and the picture resolution and picture size are also consistent. At this time, the specific pixel coordinates of the blurred area in the current video frame can be determined, and then based on the pixel coordinates, a target area at the same position in the target video frame is located, and finally the target area replaces the blurred area in the current video frame, and the processed current video frame can be obtained.

[0056] In order to better implement the embodiment of the present application, in one embodiment of the present application, determining the blurriness of each video frame in the video information to be processed to obtain the video frame with the lowest blurriness includes:

[0057] Taking any video frame in the video information to be processed as the video frame to be tested, the video frame to be tested is reduced to obtain a reduced video frame to be tested; determining the blurriness of each video frame to be tested; and determining the video frame to be tested with the lowest blurriness as the target video frame.

[0058] In the above embodiment, a solution is listed for determining the blurriness of the current video frame and each historical video frame directly based on the size of the video frame. In order to improve the timeliness of the calculation process, that is, to reduce the amount of data processing, the data processing process can be shortened. The blurriness of the video frame can be calculated on a zoom graph. In actual operation, each video frame in the video information to be processed can be scaled to 1 / 8 of its original size. At this time, the amount of data of the video frame is compressed, and the speed of determining the blurriness of each video frame is faster, and the time for processing the unclear current video frame is also faster, which can improve the timeliness of the processing. It should be noted that the specific ratio of the video frame reduction can also be determined according to the actual situation, and the embodiment of the present application does not limit it.

[0059] In addition, in the embodiment of the present application, the blurriness of each video frame to be tested can be calculated by the following formula ①:

[0060]

[0061] Among them, the represents the Laplacian operator, * represents the convolution operation, b t is the blur of the video frame to be tested, F t The discrete pixel coordinates are represented as (x, y), where x∈[0, w), y∈[0, h), w and h represent the width and height of the image, respectively. This blur measurement does not give an absolute assessment of the image blur, but rather a relative blur when compared with the blur of the historical frame. Similarly, the blur of the historical frame is denoted by b t-i , i∈[1,N].

[0062] After obtaining the blurriness of the video frame to be tested, it is necessary to determine the target video frame with the lowest blurriness. The embodiment of the present application also provides a solution for obtaining the target video frame with the lowest blurriness through formula ②, which is specifically as follows:

[0063] w b =b t / b t-i ......②

[0064] Among them, if the formula ②w b Greater than 1, indicating that the historical video frame F t-i The blurriness of the current video frame is higher. The purpose of this step is to find the historical video frame N relative to the current frame F t The clearest frame, the clearest frame can be recorded as F′ t .

[0065] In order to better implement the embodiment of the present application, in one embodiment of the present application, the target video frame is aligned and fused to obtain the target video information, including:

[0066] If the target video frame is not the current video frame being played; determining a fusion coefficient between the target video frame and the current video frame; and based on the fusion coefficient, fusing the target video frame with the current video frame to adjust the blurriness of the current video frame.

[0067] The above embodiment provides a solution to directly replace the blurred area at the corresponding position in the current video frame with the clear area in the target video frame with the lowest clarity. In order to further improve the natural appearance of the processed current video frame, the embodiment of the present application also provides a solution to fuse two video frames based on two kinds of optical flow information.

[0068] In the above embodiment, a solution is provided to determine which video frame is the clearest video frame among the video frames by using a ratio, and the current frame F is calculated using formula ② t The clearest frame F′ t Blur ratio. The larger the blur ratio between the current frame and the clearest frame in history, the blurrier the current video frame is relative to the historical video frame, and a larger fusion coefficient needs to be considered. Therefore, a linear function can be set, that is, w b The larger the value, the larger the fusion coefficient is, and a linear function is formed. The slope of the linear function can be set according to actual conditions and is not limited in the embodiments of the present application.

[0069] After the fusion coefficient is obtained according to the above linear function, the target video frame is fused with the current video frame based on the fusion coefficient to adjust the blurriness of the current video frame. The blurriness of the current video frame can also be adjusted according to formula ③. The specific formula ③ is as follows:

[0070]

[0071] Among them, w(x, y) is the fusion coefficient obtained according to the above linear function; F is the current video frame after adjustment; t (x, y) is the current video frame; F′ t (x, y) is the target video frame with the lowest blur.

[0072] In order to better implement the embodiment of the present application, in one embodiment of the present application, determining the fusion coefficient between the target video frame and the current video frame includes:

[0073] Determine the gyroscope optical flow information between the target video frame and the current video frame; determine the inter-frame optical flow information between the target video frame and the current video frame based on the gyroscope optical flow information; obtain the initial fusion coefficient based on the inter-frame optical flow information; determine the average gradient within the block between the target video frame and the current video frame; if the average gradient within the block is less than a preset first gradient threshold, adjust the initial fusion coefficient according to a preset first adjustment coefficient to obtain the fusion coefficient between the target video frame and the current video frame; if the average gradient within the block is greater than the first gradient threshold and less than a preset second gradient threshold, adjust the initial fusion coefficient according to a preset second adjustment coefficient to obtain the fusion coefficient between the target video frame and the current video frame.

[0074] In the embodiment of the present application, since the current terminals such as cameras or mobile phones that can shoot videos are equipped with gyroscopes, the gyroscope optical flow information between the target video frame and the current video frame can be determined by the gyroscope carried in the camera or mobile phone. However, the gyroscope can only measure rotation, and for actual frame alignment, it is not enough to refer to the gyroscope optical flow information alone, and the optical flow method is also needed for synchronization reference.

[0075] In the embodiment of the present application, based on the gyroscope optical flow information, the inter-frame optical flow information between the target video frame and the current video frame is determined. The optical flow method in the prior art, for example, the DIS optical flow method (dense optical flow algorithm) can be used to determine the inter-frame optical flow information between the two video frames, as shown in the following formula ③:

[0076]

[0077] Among them, F′ in formula ④ t is the clearest target video frame; F t is the current video frame; flow DIS Indicates that the DIS optical flow algorithm is used to calculate F t and F′ t The inter-frame optical flow information between gyro is the gyroscope optical flow information, which can be used as the initial value of DIS optical flow; Represents a function for calculating inter-frame optical flow information based on gyroscope optical flow information.

[0078] In the embodiment of the present application, flow DIS To determine the initial fusion coefficient. DIS The greater the amplitude of the pixel movement represented by the inter-frame optical flow information, the more blurred the current video frame is, and the larger the fusion coefficient is required. DIS A linear function between the initial fusion coefficient and the flow DISThe slope of the linear function between the initial fusion coefficient can also be set according to the specific situation, and is not limited in the embodiment of the present application.

[0079] After the initial fusion coefficient is determined according to the above embodiment, in the embodiment of the present application, the purpose of considering the average gradient within the block is that the video frame degradation caused by motion blur is more obvious in the high-frequency area. After the fusion coefficient is calculated by the above method, the fusion coefficient is further adjusted based on the local block average gradient information, with the purpose of weakening the fusion coefficient of the smooth area. Therefore, to determine the average gradient within the block between the target video frame and the current video frame, refer to formula ⑤, formula ⑥ and formula ⑦:

[0080]

[0081]

[0082] Wherein, K and L in formula ⑤ and formula ⑥ are constants, and the specific value can be 8; F t is the current video frame; g x (x, y) is the horizontal gradient, g y (x, y) is the gradient in the vertical direction.

[0083] After obtaining the horizontal and vertical gradients, the average gradient within the block can be obtained according to formula ⑦ and formula ⑧, as shown below:

[0084] g(x,y)=sqrt(g x (x, y)*g x (x,y)+g y (x, y)*g y (x, y))......⑦

[0085] Among them, g(x, y) is the average gradient within the block.

[0086] After obtaining the average gradient within the block, if the average gradient within the block is less than the preset first gradient threshold, the initial fusion coefficient is adjusted according to the preset first adjustment coefficient to obtain the fusion coefficient between the target video frame and the current video frame; if the average gradient within the block is greater than the first gradient threshold and less than the preset second gradient threshold, the initial fusion coefficient is adjusted according to the preset second adjustment coefficient to obtain the fusion coefficient between the target video frame and the current video frame. The initial fusion coefficient can also be adjusted according to formula ⑧ to obtain the final fusion coefficient. The specific formula ⑧ is as follows:

[0087]

[0088] Wherein, in formula ⑧, w(x, y)′ is the initial fusion coefficient obtained according to the linear function; g_th1 is the first gradient threshold, g_th2 is the second gradient threshold, the first gradient threshold value can be 4, and the second gradient threshold value can be 8; w(x, y) is the adjusted fusion coefficient. However, the values ​​of the first gradient threshold and the second gradient threshold can be set according to actual conditions, and the embodiments of the present application are not limited thereto.

[0089] After the final fusion coefficient w(x, y) is obtained, the current video frame can be processed according to the above formula ③.

[0090] In order to better implement the embodiment of the present application, in one embodiment of the present application, an initial fusion coefficient is obtained based on the inter-frame optical flow information, including:

[0091] Based on the inter-frame optical flow information, the pixel distance between the target video frame and the current video frame is determined; based on the inter-frame optical flow information and the gyroscope optical flow information, the similarity coefficient between the target video frame and the current video frame is determined; the pixel block alignment error between the target video frame and the current video frame is determined; according to the pixel distance, the similarity coefficient and the pixel block alignment error, the initial fusion coefficient is obtained.

[0092] According to the above embodiments, the present application provides a solution for determining the initial fusion coefficient. However, fast-moving objects in the scene or optical flow calculation errors will cause pixel alignment errors. If the pixel alignment error is not considered, the clear frame obtained by the final fusion may have ghosting or artifact problems. Therefore, in order to eliminate ghosting or artifacts, the embodiment of the present application also needs to take into account the pixel alignment error when calculating the initial fusion coefficient.

[0093] Specifically, based on the inter-frame optical flow information, the pixel distance between the target video frame and the current video frame is determined, which can be obtained by referring to the following formula 9:

[0094]

[0095] Among them, σ d It is a constant and is actually set to 80. Current video frame F t The pixel point is relative to the target video frame F′ with the lowest blur t The smaller the displacement of the corresponding pixel point, the larger the pixel distance coefficient.

[0096] However, it should be noted that DIS optical flow is more dependent on the feature points of the image itself, and optical flow calculation errors may occur in areas with weak textures or repeated textures. The gyroscope can measure the rotation of the terminal device itself and is not affected by the content of the picture. Therefore, it is necessary to consider the difference between DIS optical flow and gyroscope calculated optical flow. At the same time, the optical flow difference considered should not only consider the difference in optical flow direction, but also the difference in size. Therefore, based on the inter-frame optical flow information and the gyroscope optical flow information, the similarity coefficient between the target video frame and the current video frame can be determined using the following formula ⑩ and formula The calculation results are:

[0097]

[0098] Among them, the flow in formula ⑩ DIS Represents the optical flow information between frames; flow gyro is the gyroscope optical flow information; w f (x, y) is the Tanimoto similarity coefficient; "." represents dot product. The smaller the difference between the optical flow calculated by DIS and the optical flow calculated by the gyroscope, the larger the fusion coefficient, and vice versa.

[0099]

[0100] Among them, the Tanimoto similarity coefficient obtained according to formula ⑩ may get a negative value. If the obtained Tanimoto similarity coefficient is a negative value, the current video frame cannot be effectively processed. Therefore, it is necessary to rely on formula The Tanimoto similarity coefficient is processed. As shown in the formula As shown, if the obtained Tanimoto similarity coefficient is a negative value, it is taken as 0; if the Tanimoto similarity coefficient obtained according to formula ⑩ is not a negative value, the result obtained according to formula ⑩ is directly determined to determine the Tanimoto similarity coefficient.

[0101] Specifically, the pixel block alignment error between the target video frame and the current video frame can be determined according to the formula get:

[0102]

[0103] Among them, the formula K and L in it are constants, and the specific value can be 8; E(x, y) is the pixel block alignment error.

[0104] After determining the pixel block alignment error, similarity coefficient, and pixel distance according to the above formula, the initial fusion coefficient can be obtained. Specifically, the initial fusion coefficient can be obtained according to the formula Get, formula As shown below:

[0105]

[0106] Among them, w(x, y)′ is the initial fusion coefficient; w luck (x, y) is the coefficient determined by the pixel distance; w f (x, y) is the similarity coefficient; E(x, y) is the coefficient determined by the pixel block alignment error; w′ b is a global coefficient, which can be an adjustable constant.

[0107] In order to better implement the embodiment of the present application, in one embodiment of the present application, determining a pixel block alignment error between a target video frame and a current video frame includes:

[0108] According to the preset block pixel alignment parameters, an initial pixel block alignment error between the target video frame and the current video frame is determined; according to the preset alignment error adjustment parameters, the initial pixel block alignment error is adjusted to obtain the pixel block alignment error between the target video frame and the current video frame.

[0109] In the above embodiment, a method is provided according to the formula A scheme for calculating the initial fusion coefficient. In order to make the calculated initial fusion coefficient more convenient for the fusion of two video frames. The embodiment of the present application also provides a scheme for calculating the pixel block alignment error. Specifically, according to the formula As shown:

[0110]

[0111] Among them, the formula E(x, y) in the above formula is The pixel block alignment error obtained in , and in the embodiment of the present application, E(x, y) can be regarded as the initial pixel block alignment error. e is the alignment error adjustment parameter, which can be set to 20 in practice; and w e (x, y) is the pixel block alignment error adjusted according to the alignment error adjustment parameter. Therefore, the above formula It can be changed to w(x, y) = w′ b *w luck (x, y)*w f (x, y)*w e (x, y).

[0112] In order to better implement the embodiment of the present application, in one embodiment of the present application, after determining that the video frame with the lowest blurriness among the current video frame and the historical video frames is the target video frame, the method further includes:

[0113] If the target video frame is the current video frame, image enhancement is performed on the current video frame.

[0114] According to the above embodiment, in actual situations, the target video frame may be the current video frame. Therefore, if the target video frame is the current video frame, the current video frame may be subjected to Laplace image enhancement to make the video effect of the current video frame better. Among them, the essence of Laplace enhancement is to use the second differential of the image to degenerate the image (the role of image sharpening is to enhance the grayscale contrast, thereby making the blurred image clearer). In the image field, differentiation is sharpening, integration is blurring, and using the second differential to degenerate the image is to use the neighborhood pixels to improve the contrast. The specific enhancement process can refer to the prior art, and will not be described in detail here.

[0115] Each of the above embodiments provides a video processing method, and the overall process can be as follows: Figure 3 The specific processing effect diagram can be shown as follows Figure 4 as well as Figure 5 As shown, Figure 4 is the blurred current video frame, Figure 5 is the current video frame obtained after processing. Figure 6 as well as Figure 7 As shown, Figure 6 is the gyroscope optical flow information calculated based on the gyroscope, Figure 7 It is the inter-frame optical flow information optimized by DIS.

[0116] It should be noted that x and y in any formula in the embodiments of the present application represent the pixel coordinates of the pixel point in the video frame.

[0117] In order to better implement the video processing method in the embodiment of the present application, in addition to the video processing method, the embodiment of the present application also provides a video processing device, such as Figure 8 As shown, the device 300 includes:

[0118] An acquisition module 301 is used to acquire video information to be processed;

[0119] A determination module 302, configured to determine a target video frame based on the blurriness of the video information to be processed;

[0120] The adjustment module 303 is used to perform alignment and fusion processing on the target video frame to obtain target video information.

[0121] The video processing device provided in the present application can first obtain the video information to be processed through the acquisition module 301, and then use the video frame with higher blur in a video as a reference for the video frame with lower blur through the determination module 302. After that, the blurred object in the video frame with lower blur is replaced, merged, etc. with the same object in the video frame with higher blur through the adjustment module 303, thereby eliminating the blur in the current video frame and improving the clarity of the current video frame.

[0122] In some embodiments of the present application, the determination module 302 is specifically used to:

[0123] Determine the blurriness of each video frame in the video information to be processed, and obtain the video frame with the lowest blurriness;

[0124] Based on the matching relationship between the video frame with the lowest blur and the current video frame being played, the target video frame is determined.

[0125] In some embodiments of the present application, the determination module 302 is further configured to:

[0126] Taking any video frame in the video information to be processed as the video frame to be tested, reducing the image of the video frame to be tested to obtain a reduced video frame to be tested;

[0127] Determining the blurriness of each video frame to be tested;

[0128] The video frame to be tested with the lowest blur is determined as the target video frame.

[0129] In some embodiments of the present application, the adjustment module 303 is specifically used to:

[0130] If the target video frame is not the current video frame being played;

[0131] Determine a fusion coefficient between a target video frame and a current video frame;

[0132] Based on the fusion coefficient, the target video frame is fused with the current video frame to adjust the blurriness of the current video frame.

[0133] In some embodiments of the present application, the adjustment module 303 is further used to:

[0134] Determine gyroscope optical flow information between the target video frame and the current video frame;

[0135] Based on the gyroscope optical flow information, determine the inter-frame optical flow information between the target video frame and the current video frame;

[0136] Based on the inter-frame optical flow information, the initial fusion coefficient is obtained;

[0137] determining an intra-block average gradient between a target video frame and a current video frame;

[0138] If the average gradient within the block is less than a preset first gradient threshold, adjusting the initial fusion coefficient according to a preset first adjustment coefficient to obtain a fusion coefficient between the target video frame and the current video frame;

[0139] If the average gradient within the block is greater than the first gradient threshold and less than a preset second gradient threshold, the initial fusion coefficient is adjusted according to a preset second adjustment coefficient to obtain a fusion coefficient between the target video frame and the current video frame.

[0140] In some embodiments of the present application, the adjustment module 303 is further used to:

[0141] Based on the inter-frame optical flow information, determine the pixel distance between the target video frame and the current video frame;

[0142] Determine the similarity coefficient between the target video frame and the current video frame based on the inter-frame optical flow information and the gyroscope optical flow information;

[0143] determining a pixel block alignment error between a target video frame and a current video frame;

[0144] The initial fusion coefficient is obtained according to the pixel distance, similarity coefficient and pixel block alignment error.

[0145] In some embodiments of the present application, the adjustment module 303 is further used to:

[0146] Determining an initial pixel block alignment error between a target video frame and a current video frame according to a preset block pixel alignment parameter;

[0147] The initial pixel block alignment error is adjusted according to a preset alignment error adjustment parameter to obtain a pixel block alignment error between a target video frame and a current video frame.

[0148] In some embodiments of the present application, the video processing device further includes an enhancement module 304, and the enhancement module 304 is specifically used to:

[0149] If the target video frame is the current video frame, image enhancement is performed on the current video frame.

[0150] The present application also provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of any video processing method in the present application. The terminal device integrates any video processing method provided in the present application, such as Fig. 9 As shown, it shows a schematic diagram of the structure of the terminal device involved in the embodiment of the present application, specifically:

[0151] The terminal device may include one or more processing core processors 401, one or more computer-readable storage media memories 402, a power supply 403, an input unit 404 and other components. Those skilled in the art will appreciate that Fig. 9 The terminal device structure shown in the figure does not constitute a limitation on the terminal device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0152] The processor 401 is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 executes various functions of the terminal device and processes data, thereby monitoring the terminal device as a whole. Optionally, the processor 401 may include one or more processing cores; the processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface and the application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0153] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0154] The terminal device also includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, power consumption management and other functions through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0155] The terminal device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0156] Although not shown, the terminal device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the terminal device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, such as:

[0157] Get the video information to be processed;

[0158] Determine a target video frame based on the fuzziness of the video information to be processed;

[0159] The target video frames are aligned and fused to obtain the target video information.

[0160] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0161] To this end, the embodiment of the present application provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any video processing method provided in the embodiment of the present application. For example, the computer program loaded by the processor can execute the following steps:

[0162] Get the video information to be processed;

[0163] Determine a target video frame based on the fuzziness of the video information to be processed;

[0164] The target video frames are aligned and fused to obtain the target video information.

[0165] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.

[0166] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.

[0167] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0168] The above is a detailed introduction to a video processing method and device provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video processing method, characterized in that: The method comprises: Get the video information to be processed; Determining a target video frame based on the fuzziness of the video information to be processed; The target video frames are aligned and fused to obtain target video information.

2. The video processing method according to claim 1, characterized in that: The step of determining a target video frame based on the blurriness of the video information to be processed includes: Determine the blurriness of each video frame in the video information to be processed, and obtain the video frame with the lowest blurriness; Based on the matching relationship between the video frame with the lowest blur and the current video frame being played, a target video frame is determined.

3. The video processing method according to claim 2, characterized in that: Determining the blurriness of each video frame in the video information to be processed to obtain the video frame with the lowest blurriness includes: Taking any one video frame in the video information to be processed as a video frame to be tested, reducing the image of the video frame to be tested to obtain a reduced video frame to be tested; Determining the blurriness of each of the video frames to be tested; The video frame to be tested with the lowest blur is determined as the target video frame.

4. The video processing method according to claim 1, characterized in that: The performing alignment and fusion processing on the target video frame to obtain target video information includes: If the target video frame is not the current video frame being played; Determining a fusion coefficient between the target video frame and the current video frame being played; Based on the fusion coefficient, the target video frame is fused with the current video frame to adjust the blurriness of the current video frame.

5. The video processing method according to claim 4, characterized in that: The determining a fusion coefficient between the target video frame and the current video frame includes: Determining gyroscope optical flow information between the target video frame and the current video frame; Determining inter-frame optical flow information between the target video frame and the current video frame based on the gyroscope optical flow information; Based on the inter-frame optical flow information, an initial fusion coefficient is obtained; Determining an intra-block average gradient between the target video frame and the current video frame; If the average gradient within the block is less than a preset first gradient threshold, adjusting the initial fusion coefficient according to a preset first adjustment coefficient to obtain a fusion coefficient between the target video frame and the current video frame; If the intra-block average gradient is greater than the first gradient threshold and less than a preset second gradient threshold, the initial fusion coefficient is adjusted according to a preset second adjustment coefficient to obtain a fusion coefficient between the target video frame and the current video frame.

6. The video processing method according to claim 5, characterized in that: The obtaining of the initial fusion coefficient based on the inter-frame optical flow information includes: Determine a pixel distance between the target video frame and the current video frame based on the inter-frame optical flow information; Determine a similarity coefficient between the target video frame and the current video frame based on the inter-frame optical flow information and the gyroscope optical flow information; Determining a pixel block alignment error between the target video frame and the current video frame; An initial fusion coefficient is obtained according to the pixel distance, the similarity coefficient and the pixel block alignment error.

7. The video processing method according to claim 6, characterized in that: The determining a pixel block alignment error between the target video frame and the current video frame comprises: Determining an initial pixel block alignment error between the target video frame and the current video frame according to a preset block pixel alignment parameter; The initial pixel block alignment error is adjusted according to a preset alignment error adjustment parameter to obtain a pixel block alignment error between the target video frame and the current video frame.

8. A video processing device, characterized in that: The device comprises: An acquisition module, used to acquire video information to be processed; A determination module, configured to determine a target video frame based on the blurriness of the video information to be processed; The adjustment module is used to perform alignment and fusion processing on the target video frame to obtain target video information.

9. A terminal device, characterized in that: The terminal device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of the video processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in the video processing method according to any one of claims 1 to 7.

Images