Video processing method and related equipment

Abstract

This application provides a video processing method and related equipment, relating to the field of video processing. The video processing method includes: obtaining a video stream, the video stream including multiple frames of original images; determining whether the nth frame of the original image in the video stream is the clearest frame within a preset frame number range, the preset frame number range including the n-kth frame of the original image to the n+kth frame of the original image; if not, using the clearest frame within the preset frame number range as a reference frame corresponding to the nth frame of the original image; and fusing the reference frame with the nth frame of the original image to obtain a target image corresponding to the nth frame of the original image. This method can repair images with motion blur in the video, restore their details, and improve video quality.

Description

Technical Field

[0001] The present application relates to the field of video processing, and in particular to a video processing method and related equipment. Background Art

[0002] With the widespread use of electronic devices, recording videos with them has become a daily routine. For example, when people use mobile phones to record videos, hand tremors, movement, or the subject's movements can cause inconsistent clarity between frames in the recorded video, a phenomenon known as motion blur.

[0003] To improve the clarity of recorded videos and reduce the effects of motion blur, various technologies have emerged to enhance video quality. For example, when recording videos in low light and requiring long exposure times, exposure times can be reduced, or optical image stabilization (OIS) devices can be added to electronic devices to reduce the effects of motion blur. However, the reduction in exposure time can introduce noise and other issues, and the range of OIS devices is also limited. Therefore, the aforementioned existing technologies cannot completely eliminate the motion blur that occurs during video recording.

[0004] Therefore, how to completely and efficiently remove motion blur becomes an urgent problem that needs to be solved. Summary of the Invention

[0005] The present application provides a video processing method and related equipment, which can repair images with motion blur in the video, restore their details, and improve the video quality.

[0006] To achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, a video processing method is provided, comprising:

[0008] Acquire a video stream, wherein the video stream includes multiple frames of original images;

[0009] Determine whether the nth original image frame in the video stream is the clearest frame within a preset frame number range, where n≥1, k≥1, n and k are positive integers, and the preset frame number range includes the nkth original image frame to the n+kth original image frame;

[0010] If not, the clearest frame within the preset frame number range is used as the reference frame corresponding to the nth frame of the original image;

[0011] The reference frame and the n-th original image are fused to obtain a target image corresponding to the n-th original image.

[0012] Among them, the multiple frames of original images are all images in the YUV domain.

[0013] In an embodiment of the present application, a preliminary screening is performed by determining whether the nth frame original image in the video is the clearest frame within a preset frame number range. If it is not the clearest frame, the clearest frame within the preset frame number range is determined as a reference frame. After using this as a reference to repair and improve the nth frame original image, the target image corresponding to the nth frame original image is determined. Similarly, each frame of the original image can be repaired and improved, thereby improving the clarity of the entire video and enhancing the user experience.

[0014] In one possible implementation, determining whether the nth original image frame in the video stream is the clearest frame within a preset frame number range includes:

[0015] Determine the blurriness corresponding to the nkth frame original image to the n+kth frame original image respectively;

[0016] Determining whether the blurriness corresponding to the nth frame of the original image is a minimum value according to the blurriness corresponding to the nkth frame of the original image to the n+kth frame of the original image;

[0017] If so, the nth frame of the original image is the clearest frame within the preset frame number range.

[0018] In this implementation, the nth original image is compared with the blurriness of multiple surrounding original images to determine whether to deblur the nth original image. If the nth original image is the clearest frame within the comparison range, no other frames are needed for deblurring. If the nth original image is not the clearest frame within the comparison range, a clearer frame is used for deblurring.

[0019] In a possible implementation, determining blurriness corresponding to the original image from the nkth frame to the n+kth frame includes:

[0020] The blurriness corresponding to the nkth frame original image to the n+kth frame original image is determined according to the angle information collected by the gyroscope sensor in the electronic device and / or the acceleration information collected by the acceleration sensor.

[0021] In this implementation, the fuzziness corresponding to each frame of the original image is determined using angle information and / or acceleration information, which is convenient and accurate.

[0022] In one possible implementation, determining whether the nth original image frame in the video stream is the clearest frame within a preset frame number range includes:

[0023] Determining first information corresponding to the original image of the nkth frame to the original image of the n+kth frame, respectively, wherein the first information is a gradient or contrast of image content;

[0024] Determine whether the first information corresponding to the nth frame of original image is a maximum value based on the first information corresponding to the nkth frame of original image to the n+kth frame of original image; or

[0025] determining whether a minimum value of ratios of the first information corresponding to the nth original image frame and the first information corresponding to the original images within the preset frame number range is 1;

[0026] If the first information corresponding to the n-th frame original image is a maximum value, or if the minimum value of the ratio is 1, the n-th frame original image is the clearest frame within the preset frame number range.

[0027] In this implementation, the gradient or contrast of the image content is used to directly compare whether the nth frame of the original image is the clearest frame. If the nth frame of the original image is the clearest frame within the preset frame number range, no deblurring processing is required, which reduces the amount of calculation and is convenient and efficient.

[0028] In a possible implementation, the method further includes:

[0029] If the first information corresponding to the n-th frame of the original image is not a maximum value, or if the minimum value of the ratio is not 1, then the n-th frame of the original image is not the clearest frame within the preset frame number range;

[0030] When the nth original image frame is not the clearest frame within the preset frame number range, if the first information corresponding to the nth original image frame is less than or equal to the preset first information threshold; or

[0031] If the minimum value of the ratio of the first information corresponding to the nth frame original image to the first information corresponding to the original images within the preset frame number range is less than or equal to the preset ratio threshold, the clearest frame within the preset frame number range is used as the reference frame corresponding to the nth frame original image.

[0032] In this implementation, if the nth frame original image is not the clearest frame, it is necessary to compare the clearest frame within a preset range so that the clearest frame can be used as a reference frame to deblur the nth frame original image.

[0033] In a possible implementation, the method further includes:

[0034] It is determined whether the n-th original image frame is a clear frame or a blurred frame.

[0035] If the nth original image frame is a clear frame, no processing is performed; if the nth original image frame is a blurred frame, it is determined whether the nth original image frame in the video stream is the clearest frame within a preset frame number range.

[0036] In this implementation, the original images in the video stream are preliminarily screened by using blurriness, and the clear frames screened out are not processed, and only the blurry frames are subsequently processed. This can save some computational effort and improve processing efficiency.

[0037] In a possible implementation, determining whether the nth original image frame is a clear frame or a blurred frame includes:

[0038] Determining the blurriness corresponding to the n-th frame of the original image;

[0039] If the blurriness corresponding to the n-th frame original image is greater than a preset blurriness threshold, the n-th frame original image is a blurry frame; otherwise, the n-th frame original image is a clear frame.

[0040] In this implementation, the blurriness corresponding to each frame of the original image is compared with a preset blurriness threshold to achieve preliminary screening of clear frames and blurry frames.

[0041] In a possible implementation, fusing the reference frame with the n-th original image to obtain a target image corresponding to the n-th original image includes:

[0042] Determining, based on the nth frame of the original image, a corresponding global motion mask image and a corresponding highlight mask image, wherein the global motion mask image is used to indicate masking of a near scene in the nth frame of the original image, and the highlight mask image is used to indicate masking of an area in the nth frame of the original image having a grayscale value greater than a preset grayscale value;

[0043] Determine a local motion mask image according to the reference frame, the n-th original image, the global motion mask image, and the highlight mask image;

[0044] The reference frame and the n-th original image are fused according to the global motion mask image and the local motion mask image to obtain the target image corresponding to the n-th original image.

[0045] In this implementation, by determining the global motion mask image, the highlight mask image and the local motion mask image, the areas that need to be fused and the areas that do not need to be fused in the reference frame and the nth frame original image can be divided, which facilitates subsequent more delicate fusion and improves the quality of the fused target image.

[0046] In a possible implementation, determining a local motion mask image according to the reference frame, the n-th original image, the global mask image, and the highlight mask image includes:

[0047] Registering the reference frame and the n-th original image;

[0048] Determining the difference image according to the registered reference frame and the n-th original image;

[0049] The local motion mask map is determined according to the difference map, the global motion mask map and the highlight mask map.

[0050] In this implementation, by determining the difference map, the areas with differences between the aligned reference frame and the nth frame original image can be determined. For these areas with differences, secondary screening can be performed to filter out areas that do not need to be fused except for the close-up and the highlight areas.

[0051] In a possible implementation, determining the difference image according to the registered reference frame and the n-th original image frame includes:

[0052] Determining an initial difference image using an inter-frame difference method according to the registered reference frame and the n-th original image;

[0053] Thresholding processing is performed on the initial difference map to generate the difference map.

[0054] In this implementation, the difference between the registered reference frame and the n-th original image frame can be determined by inter-frame difference distribution, and then the degree of difference can be made larger after thresholding.

[0055] In a possible implementation, before determining the local motion mask map based on the difference map, the global motion mask map, and the highlight mask map, the method further includes:

[0056] The highlight mask image is processed using a dilation method.

[0057] In this implementation, the dilation method can be used to expand the highlight area shown in the highlight mask image so that the range of subsequent highlight area fusion is slightly larger.

[0058] In a possible implementation, fusing the reference frame and the n-th original image according to the global motion mask image and the local motion mask image to obtain the target image corresponding to the n-th original image includes:

[0059] determining a radial weight according to the global motion mask;

[0060] Determining a brightness weight according to the n-th frame original image;

[0061] The registered reference frame and the n-th original image are fused using the formula Pr=(1-a×b×Pj)×Pm+(a×b×Pj)×Pc to obtain the target image corresponding to the n-th original image;

[0062] Among them, a is the radial weight, b is the brightness weight, Pj is the value in the local motion mask image, Pm is the pixel value of the nth frame original image, Pc is the pixel value of the aligned reference frame, and Pr is the pixel value of the target image.

[0063] This implementation combines radial weights, luminance weights, and values ​​in the local motion mask to achieve fusion in the blended areas and non-fusion in the non-blended areas. By adjusting the magnitude of the weight changes and the magnitude of the values ​​in the local motion mask, the transition between the blended and non-blended areas can be adjusted, resulting in a more detailed, natural, and visually enhanced target image.

[0064] In a second aspect, an electronic device is provided, comprising a module / unit for executing the first aspect or any one of the methods in the first aspect.

[0065] In a third aspect, an electronic device is provided, including a camera module, a processor, and a memory;

[0066] The camera module is used to collect a video stream, wherein the video stream includes multiple frames of original images;

[0067] The memory is used to store a computer program that can be executed on the processor;

[0068] The processor is used to execute the processing steps in the first aspect or any one of the methods in the first aspect.

[0069] In a fourth aspect, a chip is provided, characterized in that it includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip executes the steps of processing in the first aspect or any one of the methods in the first aspect.

[0070] In a fifth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the steps of processing in the first aspect or any one of the methods in the first aspect.

[0071] In a sixth aspect, a computer program product is provided, comprising: a computer program code, which, when executed by an electronic device, enables the electronic device to execute the steps of processing in the first aspect or any one of the methods in the first aspect.

[0072] In an embodiment of the present application, a preliminary screening is performed by determining whether the nth frame original image in the video is the clearest frame within a preset frame number range. If it is not the clearest frame, the clearest frame within the preset frame number range is determined as a reference frame. After using this as a reference to repair and improve the nth frame original image, the target image corresponding to the nth frame original image is determined. Similarly, each frame of the original image can be repaired and improved, thereby improving the clarity of the entire video and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is two frames of images in a video recorded using existing technology;

[0074] Figure 2 This is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0075] Figure 3 Schematic diagram of the video processing method provided in the embodiment of the present application;

[0076] Figure 4 is a schematic diagram of a global motion mask provided in an embodiment of the present application;

[0077] Figure 5 is a schematic diagram of a highlight mask provided in an embodiment of the present application;

[0078] Figure 6 is a schematic diagram of determining a difference map provided by an embodiment of the present application;

[0079] Figure 7 is a schematic diagram of determining a local motion mask provided by an embodiment of the present application;

[0080] Figure 8 is a schematic diagram of a highlight mask image after expansion provided by an embodiment of the present application;

[0081] Figure 9 is a schematic diagram of a local motion mask provided in an embodiment of the present application;

[0082] Figure 10 is a schematic diagram of determining a target image provided by an embodiment of the present application;

[0083] Figure 11 is a schematic diagram of a radial weight provided in an embodiment of the present application;

[0084] Figure 12 Schematic diagram of the effect of the video processing method provided by the embodiment of the present application;

[0085] Figure 13 is a schematic diagram of a hardware system of an apparatus suitable for the present application;

[0086] Figure 14 A schematic diagram of the structure of a video processing device provided in an embodiment of the present application;

[0087] Figure 15 A schematic diagram of the structure of a chip provided for an application embodiment. DETAILED DESCRIPTION

[0088] The technical solution in this application will be described below with reference to the accompanying drawings.

[0089] In the description of the embodiments of this application, unless otherwise specified, " / " represents or. For example, A / B can represent A or B. "And / or" in this article is simply a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of this application, "plurality" means two or more than two.

[0090] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0091] First, some of the terms used in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0092] 1. The RGB (red, green, blue) color space, or RGB domain, refers to a color model related to the structure of the human visual system. According to the structure of the human eye, all colors are considered to be different combinations of red, green, and blue.

[0093] 2. Pixel value refers to the set of color components corresponding to each pixel in a color image in the RGB color space. For example, each pixel corresponds to a set of three primary color components, where the three primary color components are red (R), green (G), and blue (B).

[0094] 3. YUV color space, or YUV domain, refers to a color encoding method where Y represents brightness, and U and V represent chrominance. While the RGB color space focuses on the human eye's perception of color, the YUV color space focuses on visual sensitivity to brightness. RGB and YUV color spaces are convertible.

[0095] 4. Grayscale image: A grayscale image is a single-channel image used to represent varying degrees of brightness, with the brightest being completely white and the darkest being completely black. In other words, each pixel in a grayscale image corresponds to a varying degree of brightness between black and white. To describe the brightness variation between the brightest and darkest levels, the image is typically divided, for example, into 256 parts, representing 256 levels of brightness, referred to as 256 grayscales (grayscale 0 to grayscale 255). Each pixel corresponds to a grayscale value, also known as a grayscale value.

[0096] 5. A binary image is one in which each pixel has only two possible values ​​or grayscale states. For example, the grayscale values ​​corresponding to the pixels in the image can only be 0 or 255, with 0 and 255 representing white and black, respectively. Alternatively, the values ​​corresponding to the pixels in the image can only be 0 or 1, with 0 and 1 representing white and black, respectively.

[0097] The above is a brief introduction to the nouns involved in the embodiments of this application, and no further details will be given below.

[0098] With the widespread use of electronic devices, recording videos with them has become a daily routine. For example, when people use mobile phones to record videos, hand tremors, movement, or the subject's movements can cause inconsistent clarity between frames in the recorded video, a phenomenon known as motion blur.

[0099] For example, Figure 1 Two frames of images in a video recorded using the prior art are shown.

[0100] like Figure 1 As shown in (a), it is an image frame with motion blur in the video. Figure 1 (c) in the figure shows Figure 1 The local content in the center of (a). Figure 1 (b) in the figure shows a relatively clear image frame in the video. Figure 1 (d) in the figure shows Figure 1 The local content located at the center in (b).

[0101] It should be understood that in addition to the pedestrians being photographed moving, the photographer himself is also moving. Therefore, the still objects such as buildings and trees photographed by the photographer are also moving relative to the photographer, which leads to inconsistent clarity between frames in the recorded video.

[0102] To improve the clarity of recorded videos and reduce the effects of motion blur, various technologies have emerged to enhance video quality. For example, when recording video in low ambient light, electronic devices typically increase the exposure time to improve imaging quality. However, to reduce motion blur caused by inconsistent clarity between frames, the exposure time can be reduced. Furthermore, optical image stabilization (OIS) devices can be added to electronic devices to reduce motion blur through anti-shake technology.

[0103] However, due to the limitation of the reduction in exposure time, noise and other problems may be introduced into the recorded video, and the range of the introduced OIS device is also limited. Therefore, neither of the above two existing technologies can completely remove motion blur.

[0104] In view of this, an embodiment of the present application provides a video processing method, which determines the clear frames and blurred frames in the video, and then uses the inter-frame differences between the clear frames and the blurred frames to repair the blurred frames, thereby improving the clarity of the video and enhancing the user experience.

[0105] First, the application scenarios of the embodiments of the present application are briefly described.

[0106] Figure 2 FIG2 is a schematic diagram of an application scenario provided by an embodiment of the present application. The video processing method provided by the present application can be applied to remove motion blur on an image in a video.

[0107] In one example, the electronic device is a mobile phone. Figure 2 As shown in (a) of FIG, the graphical user interface (GUI) of the electronic device. When the electronic device detects that the user clicks the icon of the camera application on the interface, the camera application can be started and the display can be as follows: Figure 2 Another GUI shown in (b) in FIG. 8 may be referred to as a preview interface.

[0108] The preview interface may include a viewfinder 21. In the preview state, the viewfinder 21 may display a preview image in real time. The preview interface may also include multiple shooting mode options and a first control, namely, a capture button 11. The multiple shooting mode options include, for example, photo mode and video mode. The capture button 11 is used to indicate whether the current shooting mode is photo mode, video mode, or another mode. The camera application is generally in photo mode by default when it is opened.

[0109] For example, Figure 2 As shown in (b), after the electronic device starts the camera application, the electronic device runs the program corresponding to the video processing method, and obtains and stores the video in response to the user clicking the shooting key 11.

[0110] It should be understood that during the filming process, if there are pedestrians walking in the scene to be filmed, and the photographer is also walking, then the static objects such as buildings and trees in the scene to be filmed are also moving relative to the photographer, resulting in inconsistent clarity between frames in the recorded video, which cannot be effectively avoided using related technologies. However, the video processing method of the present application can detect blurred frames and clear frames with different clarity in the video, and use the inter-frame differences between the clear frames and the blurred frames to restore the details in the blurred frames and obtain high-quality video. Among them, the blurred frames are used to indicate images with relatively low clarity, and the clear frames are used to indicate images with relatively high clarity.

[0111] It should be understood that the above Figure 2 The scenarios shown are examples of application scenarios and do not limit the application scenarios of this application. The video processing method provided in the embodiments of this application can be applied to but not limited to the following scenarios:

[0112] Video calls, video conferencing applications, long and short video applications, live video applications, online video courses, smart camera applications, system camera recording function video, video surveillance, smart cat-eye and other shooting scenarios, etc.

[0113] The video processing method provided in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0114] The video processing method provided in the embodiment of the present application can be used in a video mode, wherein the video mode can instruct the electronic device to perform video recording; or, the video mode can also instruct the electronic device to perform live video broadcasting.

[0115] For example, the video processing method provided in the embodiment of the present application can be applicable to video modes at night or in dimly lit environments.

[0116] Figure 3 This is a flow chart of the video processing method provided in the embodiment of the present application. Figure 3 As shown, the video processing method 30 includes the following S11 to S20.

[0117] S11. Obtain a video stream, which includes multiple frames of original images.

[0118] It should be understood that the video stream may be collected by the electronic device using its own camera or obtained from other devices, and the embodiments of the present application do not impose any restrictions on this.

[0119] For example, when an electronic device collects data using a camera set by itself, the process may include: the electronic device starts the camera, displays Figure 2 The preview interface shown in (b) includes a first control, which may be a capture button 11. When the electronic device detects a first operation of the capture button 11 by the user, the camera captures multiple frames of original images in response to the first operation to generate a video stream.

[0120] Among them, the multiple frames of original images are all images in the YUV domain.

[0121] It should be understood that the multiple frames of original images included in the video stream may be original images directly generated by an image sensor in a camera, or may be images obtained by performing one or more processing operations on the original images.

[0122] S12. Determine whether the nth original image frame is a clear frame or a blurred frame, where n≥1, and n is a positive integer.

[0123] For example, the blurriness corresponding to the n-th frame of the original image may be determined, and then a blurriness threshold may be set to screen the n-th frame of the original image as a clear frame or a blurry frame.

[0124] It should be understood that blurriness and sharpness are two relative but interrelated concepts that describe the degree of image clarity (blurriness). A sharper image is associated with higher quality, corresponding to greater sharpness and less blurriness. A less clear image, or more blurry, is associated with lower quality, corresponding to less sharpness and greater blurriness.

[0125] If the electronic device is equipped with a gyroscope sensor and / or an accelerometer, the blurriness corresponding to the n-th frame of the original image is determined based on the angle information collected by the gyroscope sensor and / or the acceleration information collected by the accelerometer. Alternatively, the blurriness corresponding to the n-th frame of the original image may be determined using other algorithms, which are not limited in this embodiment of the present application. The blurriness threshold may also be set and adjusted as needed, which is not limited in this embodiment of the present application.

[0126] Based on this, if the blurriness corresponding to the nth frame original image is greater than the blurriness threshold, the nth frame original image is a blurry frame. If the blurriness corresponding to the nth frame original image is less than or equal to the blurriness threshold, the nth frame original image is a clear frame.

[0127] Based on the above determination, if the nth original image is initially determined to be a clear frame, no processing is performed. The n+1th original image can then be determined to be a clear frame or a blurred frame, and so on. However, if the nth original image is a blurred frame, subsequent processing of the nth original image is required.

[0128] It should be understood that the above process is mainly for preliminary screening of the original images in the video stream. The clear frames screened out are not processed, and only the blurred frames are subsequently processed. This can save some computational effort and improve processing efficiency.

[0129] S13. Determine whether the nth original image frame in the video stream is the clearest frame within a preset frame number range, k≥1, k is a positive integer, and the preset frame number range is the nkth original image frame to the n+kth original image frame.

[0130] It should be understood that when n takes different values, the original images included in the preset frame number range corresponding to the original image frame are different. When k takes different values, for the same original image, the original images included in the preset frame number range corresponding to the original image are also different.

[0131] It should be understood that the clearest frame referred to herein is a relatively clear original image frame within the corresponding range. As n and k change, the preset frame range corresponding to the nth original image includes different original images. Accordingly, the clearest frame determined within the preset frame range may not necessarily be the same.

[0132] S14: If the nth original image frame is the clearest frame among the multiple original image frames within the preset frame number range, no processing is performed.

[0133] It should be understood that when the nth frame of the original image is the clearest frame among multiple frames of original images within the preset frame number range, it means that it is already the clearest image relative to other frames and cannot be improved based on other frames, so it does not need to be processed.

[0134] S15. If the nth frame original image is not the clearest frame among the multiple frames of original images within the preset frame number range, a frame of the clearest original image can be determined from the multiple frames of original images other than the nth frame original image within the preset frame number range. This frame of original image is the clearest frame. Therefore, this frame of original image can be used as a reference frame corresponding to the nth frame original image.

[0135] It should be understood that when the nth frame original image is not the clearest frame among the multiple frames of original images within the preset frame number range, it means that the quality of the nth frame original image is poor compared with other frame images and there is relatively serious blur. Therefore, a frame of the clearest frame can be screened out from the multiple frames of original images other than the nth frame original image within the preset frame number range, and used as a reference frame for processing the nth frame original image. Then, subsequent processing is performed based on the reference frame and the nth frame original image to achieve the restoration of the nth frame original image.

[0136] In the above S13, the following two methods can be used to determine whether the nth original image in the video stream is the clearest frame within the preset frame number range. Of course, other methods can also be used for determination, and the embodiment of the present application does not impose any limitation on this.

[0137] Optionally, as an example, the above S13 may include the following S131 to S134.

[0138] S131. If the electronic device is equipped with a gyroscope sensor and / or an accelerometer, the blurriness corresponding to each original image within a preset frame number range of the nth original image is determined based on the angle information collected by the gyroscope sensor and / or the acceleration information collected by the accelerometer. In other words, the blurriness corresponding to each of the nth original image frame through the n+kth original image frame is determined.

[0139] S132 , determining whether the blurriness corresponding to the nth frame original image is a minimum value based on the blurriness corresponding to the nkth frame original image to the n+kth frame original image.

[0140] S133: If yes, the nth frame of the original image is the clearest frame within the preset frame number range and does not need to be processed.

[0141] S134. If not, the nth frame original image is not the clearest frame within the preset frame number range. Therefore, it is necessary to determine the original image corresponding to the minimum blur within the preset frame number range, and use the original image as the clearest frame within the preset frame number range, and then use the clearest frame within the preset frame number range as the reference frame corresponding to the nth frame original image.

[0142] For example, assuming n=5, k=1, and determining that the blurriness corresponding to the fifth frame of the original image is greater than a preset blurriness threshold, the fifth frame of the original image is a blurred frame. Therefore, the next step is to determine whether the fifth frame of the original image is the clearest frame within the range of the fourth frame of the original image to the sixth frame of the original image.

[0143] If the fifth frame is judged as blurry, but it is still the clearest frame among its two adjacent frames, then no further processing is required on the fifth frame, because the other two frames are blurrier than the fifth frame.

[0144] If the fifth original image is determined to be blurry, and the blurriness of the fourth original image is determined to be less than that of both the fifth and sixth original images, then the fourth original image is the clearest frame within the range of the fourth to sixth original images. Therefore, the fourth original image can be used as a reference frame for the fifth original image.

[0145] Optionally, as another example, S13 may include the following S135 to S138.

[0146] S135 . Determine first information corresponding to the nk-th original image to the n+k-th original image, respectively, where the first information is a gradient or contrast of image content.

[0147] It should be understood that the image gradient refers to the trend of change in the image's grayscale values ​​along a certain direction within any frame from the original image frame nk to the original image frame n+k. A larger gradient indicates a faster change in the image's grayscale values, resulting in a clearer image; a smaller gradient indicates a slower change in the image's grayscale values, resulting in a blurrier image.

[0148] It should be understood that image contrast refers to the measurement of the brightness levels between the brightest white and darkest black in the light and dark areas of an image. A larger range of difference indicates greater contrast, while a smaller range indicates less contrast. Generally, a higher contrast indicates a clearer image, while a lower contrast indicates a blurrier image.

[0149] S136. Determine whether the first information corresponding to the original image of the nth frame is a maximum value based on the first information corresponding to the original image of the nkth frame to the original image of the n+kth frame; or determine whether the minimum value of the ratio of the first information corresponding to the original image of the nth frame to the first information corresponding to the original images within a preset frame number range is 1.

[0150] Among them, the preset frame number range includes the nth frame original image itself. Therefore, when determining the ratio of the first information corresponding to the nth frame original image and the first information corresponding to the original images within the preset frame number range, the ratio of the first information corresponding to the nth frame original image to itself is 1.

[0151] S137: If the first information corresponding to the nth frame original image is the maximum value, or if the minimum value in the ratio is 1, the nth frame original image is the clearest frame within the preset frame number range and does not need to be processed.

[0152] S138. If the first information corresponding to the n-th frame original image is not the maximum value, or if the minimum value in the ratio is not 1, then the n-th frame original image is not the clearest frame within the preset frame number range. Therefore, the clearest frame within the preset frame number range needs to be used as the reference frame corresponding to the n-th frame original image.

[0153] It should be understood that if the first information corresponding to the nth frame original image is not the maximum value, it can be determined which of the remaining original images has the maximum value in the first information corresponding to the nkth original image to the n+kth frame original image. The original image corresponding to the maximum value is the clearest frame. Therefore, the original image corresponding to the maximum value is used as the reference frame of the nth frame original image.

[0154] Alternatively, if the minimum value among the ratios is not 1, it indicates that there is an original image with a larger first information than the first information of the nth original image. The minimum value among the ratios of the first information corresponding to the nth original image and the first information corresponding to the n-th through n+kth original images can be determined. The original image corresponding to this minimum value is the clearest frame, and thus the original image corresponding to this minimum value is used as the reference frame for the nth original image.

[0155] For example, if n is currently 6, k is 2, and the first information is contrast, then the above S23 includes:

[0156] Step 1: Determine the contrasts corresponding to the original images of the 4th frame to the 8th frame respectively.

[0157] Step 2: Determine the ratios of the contrast corresponding to the 6th frame original image to the contrast corresponding to the 4th frame original image to the 8th frame original image, which are B1, B2, 1, B3 and B4 respectively.

[0158] Step 3: Based on the ratios B1, B2, 1, B3, and B4, if the minimum value among the ratios is 1, it means that the contrasts of the other frames of original images are all smaller than the contrast corresponding to the sixth frame of original image. Only then can the corresponding ratios be greater than 1. Therefore, the sixth frame of original image is the clearest frame.

[0159] Step 4: If the minimum value among the ratios is not 1, it means that there is an original image with a greater contrast than the sixth frame, and the clarity of the original image is higher. Therefore, the minimum value among the ratios B1, B2, B3 and B4 can be determined.

[0160] Step 5: If B4 is the minimum of the ratios B1, B2, B3, and B4, then the contrast of the 8th frame is the highest, and the 8th frame is the clearest. This means that the 8th frame can be used as the reference frame for subsequent processing of the 6th frame.

[0161] In addition, in combination with the second example above, in the video processing method provided in the embodiment of the present application, in order to reduce the amount of calculation, another screening condition can be added.

[0162] For example, the first information corresponding to the nkth original image through the n+kth original image has been determined, where the first information is the gradient or contrast of the image content. Furthermore, based on the first information corresponding to the nkth original image through the n+kth original image, it is determined that the first information corresponding to the nth original image is not a maximum value.

[0163] If the first information corresponding to the nth original image frame is not the maximum value within the range of neighboring frames, but the first information is greater than a preset first information threshold, the nth original image frame may not be processed. Alternatively, if the minimum value of the ratio of the first information corresponding to the nth original image frame to the first information corresponding to the original images within the preset frame range is not 1, but the minimum value is greater than a preset ratio threshold, the nth original image frame may not be processed.

[0164] That is to say, even if the nth frame original image is not the clearest frame within the range of neighboring frames, the overall video may be clear and meet the visual requirements, so the nth frame original image may not be subsequently processed.

[0165] If the first information corresponding to the nth frame original image is less than or equal to the preset first information threshold, or if the minimum value of the ratio of the first information corresponding to the nth frame original image to the first information corresponding to the original images within the preset frame number range is less than or equal to the preset ratio threshold, then the nth frame original image needs to be subsequently processed.

[0166] S16 : Determine the corresponding global motion mask image and highlight mask image according to the n-th frame original image.

[0167] The global motion here refers to the foreground within the background content of the nth original image frame, and the global motion mask is an image that masks the foreground within the nth original image frame. It should be understood that the background content can also be considered as a still object within the original image. The global motion mask is a binary image.

[0168] For example, Figure 4 A schematic diagram illustrating a global motion mask.

[0169] like Figure 4 As shown in (a) in the figure, in this image, the foreground can be considered as pedestrians, and the rest of the image content is considered as the background. In the background content, the building on the left, the venue, trees and street lights on the right, and the road below can be considered as the foreground of the background content. Since the foreground of the background content is generally distributed at the edge of the image, the following can be generated: Figure 4 The global motion mask shown in (b).

[0170] like Figure 4 As shown in (b) in the figure, in the global motion mask image, the foreground content regarded as background content can be divided into a first sub-area, and the pixel values ​​of the first sub-area are all set to 0, corresponding to the black area; for other content, it can be divided into a second sub-area, and the pixel values ​​of the second sub-area are all set to 1, corresponding to the white area.

[0171] Optionally, as an implementable approach, the sizes of the first sub-region and the second sub-region can be divided according to a preset pixel size. For example, with the center of the global mask image as the center, the pixel area within 200 pixels to the left and right of the center and 100 pixels above and below the center is divided into the second sub-region, and the rest is divided into the first sub-region.

[0172] Alternatively, as another achievable method, the sizes of the first sub-region and the second sub-region may be divided according to a preset ratio. For example, with the center of the global mask image as the center, the first sub-region and the second sub-region may be divided in the global mask image according to a preset side length ratio of 2:3.

[0173] It should be understood that the close-up of the background content is usually static and has obvious scaling, so it is difficult to align the close-up after global transformation and is prone to ghosting. In addition, the user's attention is generally focused on the center of the picture when watching a video, so no further processing is done on the close-up distributed on the edge. The image content located in the center of the original image of the nth frame corresponding to the second sub-area is the image range that needs to be repaired and improved subsequently.

[0174] In addition, it should be understood that the highlight mask image is used to indicate that the area in the nth frame of the original image whose grayscale value is greater than the preset grayscale value is masked. The grayscale value corresponding to each pixel can be calculated from the three primary color pixel values ​​corresponding to each pixel in the nth frame of the original image.

[0175] It should be understood that when recording videos at night or in a dimly lit environment, the blurring of highlight areas is more obvious. Therefore, the highlight areas with more obvious blurring can be screened out in combination with the original image of the nth frame, and then a highlight mask map can be generated to facilitate subsequent processing of the highlight content in the original image of the nth frame.

[0176] For example, Figure 5 A schematic diagram illustrating a highlight mask image is shown.

[0177] By comparing the grayscale value of each pixel in the nth frame of the original image with a preset grayscale value, the highlight mask image is divided into highlighted and non-highlighted areas. The highlight mask image is a binary image, where pixels in the highlighted area of ​​the highlight mask image are assigned a value of 1, and pixels in the non-highlighted area are assigned a value of 0.

[0178] For example, the original image of the nth frame is Figure 5 As shown in (a) in FIG, if the preset grayscale value is 200, the region with a grayscale value greater than 200 determined in the n-th frame original image is divided into the highlight region in the highlight mask image, as shown in FIG. Figure 5 Q1, Q2 and Q3 shown in (b); the area with a grayscale value less than 200 is divided into the non-highlight area in the highlight mask image, such as Figure 5 The region other than Q1, Q2 and Q3 shown in (b).

[0179] S17: Register the reference frame with the nth original image (image registration).

[0180] It should be understood that registration refers to the matching of geographic coordinates of different images obtained by different imaging methods in the same area, which may include three aspects: geometric correction, projection transformation, and scale unification.

[0181] For example, a homography transformation can be used to describe the position mapping relationship between the world coordinate system and the pixel coordinate system of an object in the reference frame and the nth frame original image. The transformation matrix corresponding to the reference frame and the nth frame original image is called a homography matrix. Then, by calculating the homography matrix corresponding to the reference frame and the nth frame original image, the reference frame and the nth frame original image can be aligned. The homography matrix information can be provided by an electronic image stabilization (EIS), for example.

[0182] In addition, registration can also be performed using feature points, optical flow algorithms, etc. Of course, other methods can also be used for registration, and the embodiments of the present application do not impose any restrictions on this.

[0183] S18. Determine a difference image based on the registered reference frame and the n-th original image frame.

[0184] Optionally, an inter-frame difference method may be used to determine an initial difference map corresponding to the registered reference frame and the nth frame of the original image. Then, a thresholding process is performed on the initial difference map to generate a difference map.

[0185] The thresholding process is used to indicate that pixels in the initial difference image whose values ​​are greater than a preset grayscale difference threshold are updated to a first value, and pixels whose values ​​are less than the preset grayscale difference threshold are updated to a second value.

[0186] The preset grayscale difference threshold can be set and changed as needed, and the magnitudes of the first grayscale value and the second grayscale value can also be set as needed, and the present embodiment does not impose any restrictions on this. For example, the preset grayscale difference threshold can be set to 20, the first value can be 1, and the second value can be 0.

[0187] It should be understood that the inter-frame difference method involves subtracting the grayscale values ​​of each pixel in the registered reference frame and the nth original image to generate an initial difference map. This initial difference map reflects areas with significant differences between the registered reference frame and the nth original image. The initial difference map is a grayscale image.

[0188] On this basis, after thresholding the initial difference image after subtraction, the area with large differences between the reference frame after registration and the n-th frame original image can be enhanced. The difference image is a binary image.

[0189] For example, the values ​​corresponding to pixels in the initial difference image whose grayscale values ​​are less than a preset grayscale difference threshold are updated to 0, and the values ​​corresponding to pixels in the initial difference image whose grayscale values ​​are greater than the preset grayscale difference threshold are updated to 1. In this way, the generated difference image is a binary image.

[0190] For example, Figure 6 A schematic diagram of determining a difference map is shown. Figure 6 As shown, the reference frame and the n-th original image are registered, and then the grayscale value of each pixel in the registered reference frame and the n-th original image is subtracted to generate an initial difference image as indicated by P1.

[0191] Among them, the grayscale value range corresponding to the pixel in the initial difference image is, for example, 0 to 255. The larger the grayscale value, the greater the difference between the reference frame after alignment and the n-th frame original image at the pixel position; the smaller the grayscale value, the smaller the difference between the reference frame after alignment and the n-th frame original image at the pixel position, and the more similar they are.

[0192] Combine Figure 6 P1 shown in FIG can be an initial difference image determined based on the reference frame and the nth frame original image, in which the difference area represented is very subtle visually. Based on this, a thresholding process can be performed on the initial difference image, and the values ​​corresponding to the pixels below the preset grayscale difference threshold of 20 are updated to 0, and the values ​​corresponding to the pixels above the preset grayscale threshold of 20 are updated to 1. For example, the image after thresholding is as follows: Figure 6In this way, the regions with large differences between the registered reference frame and the n-th original image can be enhanced, so as to facilitate further processing of the difference regions.

[0193] S19: Determine a local motion mask map according to the difference map, the global motion mask map, and the highlight mask map.

[0194] Optionally, the highlight mask image may be processed using a dilation method.

[0195] It should be understood that the dilation method refers to adding pixels to the content boundary in the image, making the white area (grayscale value 1) "fatter", and the width of this fat circle can be specified by a parameter, which can be set and changed as needed. The embodiment of the present application does not impose any restrictions on this.

[0196] For example, Figure 7 As shown in Figure 1, the highlight mask image is processed using the dilation method. That is, the white areas in the highlight mask image are dilated, resulting in the image shown in P3. P3 is the highlight mask image after dilation. At this point, each white area shown in P3 is larger than the initial highlight mask image.

[0197] Then, the local motion mask image is determined based on the difference image, the global motion mask image, and the highlight mask image. Based on the difference image, the global motion mask image and the highlight mask image are removed from the difference image. That is, the pixels in the difference image corresponding to the close-up in the global motion mask image, or in the first sub-area, with a value of 1, are updated to 0, and the pixels corresponding to the highlight area in the highlight mask image with a value of 1 are updated to 0. Thus, the local motion mask image can be obtained, as shown in FIG. Figure 7 P4 shown in .

[0198] It should be understood that the foreground shown in the original image of the nth frame is greatly scaled, the global registration error is large, the global registration problem is difficult to solve, and the local registration is time-consuming and labor-intensive. Therefore, after comprehensive consideration, the foreground content shown in the global motion mask image, or the content of the first sub-area, is not considered for fusion. Based on this, the white area in the differential image corresponding to the first sub-area of ​​the global motion mask image is removed.

[0199] It should also be understood that the highlight area in the original image of the nth frame is relatively bright and has a large difference from the surrounding pixels. Therefore, the blur in the highlight area is more obvious when observed by the human eye, and is the area that needs to be fused later. However, in order to facilitate the determination of the area that does not need to be fused in the difference image, the white area in the difference image corresponding to the highlight area belonging to the highlight mask image can be removed.

[0200] In this way, the white area shown in the determined local motion mask is the content that is completely different between the n-th frame original image and the reference frame after registration. Since the content indicated by the local motion mask is the area with large changes, no subsequent fusion is required.

[0201] For example, Figure 8 As shown in (a), a part of the white area in the expanded highlight mask image is equivalent to the first sub-area in the global motion mask image, and the other part is equivalent to the second sub-area in the global motion mask image. Therefore, combined with Figure 8 As shown in (b) of Figure 1, when determining the local motion mask, the highlight content in the first subregion of the expanded highlight mask is also removed. In other words, the global motion mask takes precedence over the highlight mask. If the content of a certain area belongs to both the foreground and the highlight region, fusion is not required even if the content belongs to the highlight region if the foreground is not fused.

[0202] For example, Figure 9 As shown, after removing the global motion mask and the expanded highlight mask from the difference image, the white area in the local motion mask is equivalent to the second sub-area located only in the global motion mask, which can represent the non-highlighted content other than the foreground that does not need to be fused.

[0203] S20 , according to the global motion mask image, the highlight mask image and the local motion mask image, the registered reference frame and the n-th original image are fused to obtain a target image corresponding to the n-th original image.

[0204] Alternatively, as Figure 10 As shown, the above S20 may include the following S201 to S203.

[0205] S201: Determine a corresponding radial weight according to a global motion mask image.

[0206] S202: Determine a brightness weight according to the n-th frame of the original image.

[0207] S203 , according to the registered reference frame and the n-th original image, the target image corresponding to the n-th original image is determined using the formula Pr=(1-a×b×Pj)×Pm+(a×b×Pj)×Pc.

[0208] Among them, a is the radial weight, b is the brightness weight, Pj is the value in the local motion mask image, Pm is the pixel value of the nth frame original image, Pc is the pixel value of the reference frame after alignment, and Pr is the pixel value of the target image.

[0209] Figure 11A schematic diagram showing a method of determining radial weights based on a global motion mask image is shown.

[0210] like Figure 11 As shown in the figure, taking the global motion mask as a rectangle as an example, the radial weight corresponding to the center of the global motion mask can be set to 1, the radial weight corresponding to the edge of the second sub-area of ​​the global motion mask can be set to an intermediate number between 0 and 1 for transition, and the radial weights corresponding to the first sub-area can all be set to 0. Therefore, when performing fusion, the foreground of the original image in the nth frame can be omitted, and only the content within the first sub-area can be fused, with the fusion strength increasing as the image moves toward the center.

[0211] It should be understood that the brightness weight is determined based on the grayscale value of the nth frame of the original image. A corresponding brightness weight can be determined for each pixel in the entire image. Moreover, the higher the brightness, the greater the corresponding brightness weight, and the lower the brightness, the smaller the corresponding brightness weight.

[0212] For example, for the nth frame of the original image, the brightness weight ranges from 0 to 1. When the grayscale value of a pixel in the nth frame of the original image is 0, the corresponding brightness weight can be 0; when the grayscale value of a pixel is 255, the corresponding brightness weight can be 1; when the grayscale value of a pixel is 128, the corresponding brightness weight can be 0.5.

[0213] It should be understood that the radial weight and brightness weight can be set and changed as needed, and the embodiments of the present application do not impose any restrictions on this.

[0214] It should be understood that the local motion mask indicates the area where the difference between the registered reference frame and the original image of the nth frame is relatively large, and this application does not fuse this area. The values ​​of the pixels in the local motion mask are usually 0 or 1. Here, in order to smoothly transition at the boundary between the fused area and the unfused area, the local motion mask can be filtered to smooth the boundary between the fused area and the unfused area, that is, the value is changed to an intermediate value between 0 and 1.

[0215] It should be understood that the above is for the same pixel position, the pixel value of the reference frame after registration and the pixel value of the nth frame original image, to determine the pixel value of the target image, and each pixel corresponds to the radial weight, brightness weight and value in the local motion mask map.

[0216] Figure 12 A schematic diagram showing the effect of the video processing method provided in an embodiment of the present application is shown.

[0217] like Figure 12 As shown in (a), there is a motion blurred image frame in the video. Figure 12 As shown in (b) in Figure 12The image shown in (a) is restored by the video processing method provided in the embodiment of the present application, and the corresponding image after the motion blur is removed. Compared with the two, the clarity of the image is greatly improved after the motion blur is removed.

[0218] This embodiment of the application provides a video processing method that performs preliminary screening by determining whether the nth original image frame in a video is the clearest frame within a preset frame range. If it is not the clearest frame, the clearest frame within the preset frame range is determined as a reference frame. This reference is used to repair and enhance the nth original image, and then the target image corresponding to the nth original image is determined. Similarly, each original image frame can be repaired and enhanced, thereby improving the clarity of the entire video and enhancing the user experience.

[0219] It should be understood that the above examples are intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to the specific numerical values ​​or specific scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or variations based on the above examples, and such modifications or variations also fall within the scope of the embodiments of the present application.

[0220] Combined with the above Figures 2 to 12 , the video processing method of the embodiment of the present application is described in detail, and the following will be combined with Figures 13 to 15 , describing in detail the hardware systems, devices, and chips of the electronic devices to which this application applies. It should be understood that the hardware systems, devices, and chips in the embodiments of this application can execute the various video processing methods in the aforementioned embodiments of this application. That is, the specific working processes of the various products below can refer to the corresponding processes in the aforementioned method embodiments.

[0221] The video processing method provided in the embodiment of the present application can be applicable to various electronic devices. Correspondingly, the video processing device provided in the embodiment of the present application can be electronic devices in various forms.

[0222] In some embodiments of the present application, the electronic device may be various camera devices such as SLR cameras and compact cameras, mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc., or may be other devices or apparatuses capable of performing image processing. The embodiments of the present application do not impose any restrictions on the specific type of electronic device.

[0223] The following text takes the mobile phone as an example. Figure 13 A structural diagram of an electronic device 100 provided in an embodiment of the present application is shown.

[0224] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0225] It should be noted that Figure 13 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include Figure 13 More or fewer components than those shown, or the electronic device 100 may include Figure 13 Combinations of some of the components shown, or alternatively, the electronic device 100 may include Figure 13 Subassemblies of some of the components shown. Figure 13 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0226] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0227] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0228] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0229] In an embodiment of the present application, the processor 110 can run the software code of the video processing method provided in the embodiment of the present application, so as to capture a video with higher definition.

[0230] Figure 13 The connection relationship between the modules shown is only for illustrative purposes and does not limit the connection relationship between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of the multiple connection modes in the above embodiments.

[0231] The wireless communication function of the electronic device 100 can be implemented through components such as the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor.

[0232] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0233] Electronic device 100 can implement display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0234] Display screen 194 may be used to display images or videos.

[0235] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0236] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can perform algorithmic optimization on image noise, brightness, and color. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0237] The camera 193 is used to capture images or videos. It can be triggered to turn on through application instructions to realize the shooting function, such as shooting and obtaining a video stream of any scene. The camera may include components such as an imaging lens, a filter, and an image sensor. The light emitted or reflected by the object enters the imaging lens, passes through the filter, and finally converges on the image sensor. The image sensor is mainly used to focus the light emitted or reflected by all objects in the shooting angle (also known as the scene to be shot, the target scene, or the scene image that the user expects to shoot) to form an image; the filter is mainly used to filter out unnecessary light waves in the light (for example, light waves other than visible light, such as infrared); the image sensor is mainly used to perform photoelectric conversion on the received light signal, convert it into an electrical signal, and input it into the processor 130 for subsequent processing. Among them, the camera 193 can be located on the front or back of the electronic device 100. The specific number and arrangement of the cameras can be set according to needs, and this application does not impose any restrictions.

[0238] In an embodiment of the present application, the camera 193 can acquire a video stream including multiple frames of original images.

[0239] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0240] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0241] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, the y-axis, and the z-axis) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used in scenarios such as navigation and somatosensory games.

[0242] For example, in an embodiment of the present application, the gyro sensor 180B may be used to collect angle information, and the angle information may be used to determine the blurriness corresponding to the original image.

[0243] Accelerometer 180E can detect the magnitude of the acceleration of electronic device 100 in various directions (generally the x-axis, y-axis, and z-axis). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. Accelerometer 180E can also be used to identify the posture of electronic device 100, which can serve as an input parameter for applications such as landscape / portrait switching and pedometers.

[0244] For example, in the embodiment of the present application, the acceleration sensor 180E can be used to collect the magnitude of acceleration, that is, acceleration information, and the acceleration information can be used to determine the blurriness corresponding to the original image.

[0245] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0246] Figure 14 This is a schematic diagram of the structure of a video processing device provided in an embodiment of the present application. Figure 14 As shown, the video processing device 200 includes an acquisition module 210 and a processing module 220 .

[0247] The video processing device 200 can perform the following schemes:

[0248] The acquisition module 210 is configured to acquire a video stream including multiple frames of original images.

[0249] Processing module 220 is used to determine whether the nth frame original image in the video stream is the clearest frame within a preset frame number range, where n≥1, k≥1, n and k are positive integers, and the preset frame number range is from the nkth frame original image to the n+kth frame original image; if not, the clearest frame within the preset frame number range is used as the reference frame corresponding to the nth frame original image.

[0250] The processing module 220 is further configured to fuse the reference frame with the n-th original image to obtain a target image corresponding to the n-th original image.

[0251] It should be noted that the video processing device 200 is implemented in the form of a functional module. The term "module" here can be implemented in the form of software and / or hardware, and is not specifically limited to this.

[0252] For example, a "module" may be a software program, a hardware circuit, or a combination of the two that implements the aforementioned functionality. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group processor) and memory for executing one or more software or firmware programs, combined logic circuits, and / or other suitable components that support the described functionality.

[0253] Therefore, the modules of each example described in the embodiments of this application can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0254] An embodiment of the present application also provides another electronic device, including a camera module, a processor and a memory.

[0255] The camera module is used to obtain a video stream, which includes multiple frames of original images.

[0256] Memory is used to store computer programs that can be executed on the processor.

[0257] The processor is used to execute the processing steps in the video processing method as described above.

[0258] Optionally, the camera module may include at least one of a wide-angle camera, a main camera, and a telephoto camera.

[0259] The embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions; when the computer-readable storage medium is run on a video processing device, the video processing device is caused to execute the following Figure 3 The method shown. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more servers that can be integrated with the medium. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0260] The present application also provides a computer program product including computer instructions, which, when executed on a video processing device, enables the video processing device to execute Figure 3 The technical solution shown.

[0261] Figure 15 A schematic diagram of the structure of a chip provided in an embodiment of the present application. Figure 15 The chip shown can be a general purpose processor or a dedicated processor. The chip includes a processor 301. The processor 301 is used to support the video processing device to execute Figure 3 The technical solution shown.

[0262] Optionally, the chip further includes a transceiver 302, which is configured to accept control of the processor 301 and to support the communication device in executing Figure 3 The technical solution shown.

[0263] Optional, Figure 15 The chip shown may further include: a storage medium 303 .

[0264] It should be noted that Figure 15The chip shown can be implemented using the following circuits or devices: one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits that can perform the various functions described throughout this application.

[0265] The electronic device, video processing device, computer storage medium, computer program product, and chip provided in the above-mentioned embodiments of the present application are all used to execute the methods provided above. Therefore, the beneficial effects that can be achieved can refer to the corresponding beneficial effects of the methods provided above, and will not be repeated here.

[0266] It should be understood that the above is only to help those skilled in the art better understand the embodiments of the present application, and is not intended to limit the scope of the embodiments of the present application. Based on the above examples given, those skilled in the art can obviously make various equivalent modifications or changes. For example, certain steps in each embodiment of the above detection method may be unnecessary, or certain new steps may be added. Or a combination of any two or any multiple embodiments described above. Such modifications, changes, or combined solutions also fall within the scope of the embodiments of the present application.

[0267] It should also be understood that the above description of the embodiments of the present application focuses on emphasizing the differences between the various embodiments. The same or similar points that are not mentioned can be referenced with each other. For the sake of brevity, they will not be repeated here.

[0268] It should also be understood that the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0269] It should also be understood that in the embodiments of the present application, "pre-setting" and "pre-definition" can be achieved by pre-saving corresponding codes, tables or other methods that can be used to indicate relevant information in a device (for example, including an electronic device), and the present application does not limit its specific implementation method.

[0270] It should also be understood that the division of the modes, situations, categories and embodiments in the embodiments of the present application is only for the convenience of description and should not constitute a special limitation. The features of various modes, categories, situations and embodiments can be combined without contradiction.

[0271] It should also be understood that in the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0272] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A video processing method, characterized in that: Applied to electronic equipment, the method includes: Acquire a video stream, wherein the video stream includes multiple frames of original images; Determine whether the nth original image frame in the video stream is the clearest frame within a preset frame number range, where n≥1, k≥1, n and k are positive integers, and the preset frame number range includes the nkth original image frame to the n+kth original image frame; If not, the clearest frame within the preset frame number range is used as the reference frame corresponding to the nth frame of the original image; According to the n-th frame original image, a corresponding global motion mask map and a highlight mask map are determined, the global motion mask map is used to indicate that the foreground in the n-th frame original image is to be masked, and the highlight mask map is used to indicate that the area in the n-th frame original image whose grayscale value is greater than a preset grayscale value is to be masked; according to the reference frame, the n-th frame original image, the global motion mask map and the highlight mask map, a local motion mask map is determined; according to the global motion mask map and the local motion mask map, the reference frame and the n-th frame original image are fused to obtain a target image corresponding to the n-th frame original image.

2. The video processing method according to claim 1, wherein: Determining whether the nth original image frame in the video stream is the clearest frame within a preset frame number range includes: Determine the blurriness corresponding to the nkth frame original image to the n+kth frame original image respectively; Determining whether the blurriness corresponding to the nth frame of the original image is a minimum value according to the blurriness corresponding to the nkth frame of the original image to the n+kth frame of the original image; If so, the nth frame of the original image is the clearest frame within the preset frame number range.

3. The video processing method according to claim 2, wherein: Determining the blurriness corresponding to the nkth frame original image to the n+kth frame original image respectively includes: The blurriness corresponding to the nkth frame original image to the n+kth frame original image is determined according to the angle information collected by the gyroscope sensor in the electronic device and / or the acceleration information collected by the acceleration sensor.

4. The video processing method according to claim 1, wherein: Determining whether the nth original image frame in the video stream is the clearest frame within a preset frame number range includes: Determining first information corresponding to the original image of the nkth frame to the original image of the n+kth frame, respectively, wherein the first information is a gradient or contrast of image content; Determine whether the first information corresponding to the nth frame of original image is a maximum value based on the first information corresponding to the nkth frame of original image to the n+kth frame of original image; or determining whether a minimum value of ratios of the first information corresponding to the nth original image frame and the first information corresponding to the original images within the preset frame number range is 1; If the first information corresponding to the n-th frame original image is a maximum value, or if the minimum value of the ratio is 1, the n-th frame original image is the clearest frame within the preset frame number range.

5. The video processing method according to claim 4, wherein: The method further comprises: If the first information corresponding to the n-th frame of the original image is not a maximum value, or if the minimum value of the ratio is not 1, then the n-th frame of the original image is not the clearest frame within the preset frame number range; When the nth original image frame is not the clearest frame within the preset frame number range, if the first information corresponding to the nth original image frame is less than or equal to the preset first information threshold; or If the minimum value of the ratio of the first information corresponding to the nth frame original image to the first information corresponding to the original images within the preset frame number range is less than or equal to the preset ratio threshold, the clearest frame within the preset frame number range is used as the reference frame corresponding to the nth frame original image.

6. The video processing method according to any one of claims 1 to 5, characterized in that: The method further comprises: Determining whether the nth original image frame is a clear frame or a blurred frame; If the nth original image frame is a clear frame, no processing is performed; if the nth original image frame is a blurred frame, it is determined whether the nth original image frame in the video stream is the clearest frame within a preset frame number range.

7. The video processing method according to claim 6, characterized in that: Determining whether the nth original image frame is a clear frame or a blurred frame includes: Determining the blurriness corresponding to the n-th frame of the original image; If the blurriness corresponding to the n-th frame original image is greater than a preset blurriness threshold, the n-th frame original image is a blurry frame; otherwise, the n-th frame original image is a clear frame.

8. The video processing method according to claim 1, wherein: Determining a local motion mask image according to the reference frame, the n-th original image, the global motion mask image, and the highlight mask image, comprising: Registering the reference frame and the n-th original image; Determine a difference image according to the registered reference frame and the n-th original image; The local motion mask map is determined according to the difference map, the global motion mask map and the highlight mask map.

9. The video processing method according to claim 8, characterized in that: Determining the difference image according to the registered reference frame and the n-th original image includes: Determining an initial difference image using an inter-frame difference method according to the registered reference frame and the n-th original image; Thresholding processing is performed on the initial difference map to generate the difference map.

10. The video processing method according to claim 8 or 9, characterized in that: Before determining the local motion mask map based on the difference map, the global motion mask map, and the highlight mask map, the method further includes: The highlight mask image is processed using a dilation method.

11. The video processing method according to claim 8, wherein: According to the global motion mask image and the local motion mask image, the reference frame and the n-th original image are fused to obtain the target image corresponding to the n-th original image, including: determining a radial weight according to the global motion mask; Determining a brightness weight according to the n-th frame original image; The registered reference frame and the n-th original image are fused using the formula Pr=(1-a×b×Pj)×Pm+(a×b×Pj)×Pc to obtain the target image corresponding to the n-th original image; Among them, a is the radial weight, b is the brightness weight, Pj is the value in the local motion mask image, Pm is the pixel value of the nth frame original image, Pc is the pixel value of the aligned reference frame, and Pr is the pixel value of the target image.

12. An electronic device, characterized in that: Including camera module, processor and memory; The camera module is used to collect a video stream, wherein the video stream includes multiple frames of original images; The memory is used to store a computer program that can be executed on the processor; The processor is configured to execute the processing steps in the video processing method according to any one of claims 1 to 11.

13. A chip, characterized in that: include: The processor is configured to call and run a computer program from a memory, so that a device equipped with the chip executes the video processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to perform the video processing method according to any one of claims 1 to 11.

Images