Video processing method, device, equipment and medium

Abstract

An embodiment of the present disclosure relates to a video processing method, apparatus, device, and medium, the method including: obtaining a plurality of image groups based on a sequence of video frames of an initial video; performing motion blur processing based on each frame image in the target image group, performing fusion on the images obtained by performing the motion blur processing on each frame image, to obtain a motion blur image corresponding to the target image group; determining a subject object region and a background region corresponding to the target image group based on a designated frame image in the target image group, where each image group in the plurality of image groups is a target image group; performing fusion on the motion blur image and the designated frame image according to the subject object region and the background region, to obtain a target fusion image; and generating a target video based on the target fusion image corresponding to each of the plurality of image groups.

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application is based on and claims priority from a Chinese application having application number 202210705983.3 and filing date June 21, 2022, the entire disclosure of which is incorporated herein by reference.

[0002] [Technical field] The present disclosure relates to the field of video processing technology, and more particularly to a video processing method, device, apparatus and medium. [Background technology]

[0003] In the field of video creation, creators generally shoot videos according to their needs. Different shooting methods can produce different video effects. In some cases, creators need to shoot a video effect in which the main object is clear and the background is blurred, creating a frame-drop effect. Such video effects often require professional photography equipment to be used for slow shutter shooting and / or moving shots, and also require the video creator to have solid photography skills and a suitable shooting scene. Summary of the Invention

[0004] An embodiment of the present disclosure includes obtaining a plurality of image groups based on a video frame sequence of an initial video, performing motion blur processing based on each frame image in a target image group, and fusing the images obtained by performing the motion blur processing on each frame image to obtain a motion blur image corresponding to the target image group, wherein each image group in the plurality of image groups is the target image group, determining a main object region and a background region corresponding to the target image group based on a designated frame image in the target image group, and fusing the motion blur image and the designated frame image according to the main object region and the background region, thereby obtaining a target image. A video processing method is provided, which includes: obtaining a get fusion image, wherein an image portion in the subject object region of the target fusion image is an image portion in the subject object region of the specified frame image, and an image portion in the background region of the target fusion image is an image portion in the background region of the motion blur image; and generating a target video based on the target fusion images corresponding to each of the plurality of image groups, wherein a playback order in the target video of the target fusion images corresponding to each of the plurality of image groups is the same as a playback order in the initial video of the plurality of image groups.

[0005] In some embodiments, the step of performing motion blur processing based on each frame image in the target image group and fusing the images obtained by motion blur processing each frame image includes adopting an optical flow interpolation algorithm to insert a specified number of intermediate frame images between adjacent frame images in the target image group, and treating all frame images in the target image group with inserted frames as images obtained by motion blur processing each frame image in the target image group, and performing average fusion on the images obtained by motion blur processing each frame image.

[0006] In some embodiments, the step of inserting a specified number of intermediate frame images between adjacent frame images in the target image group using an optical flow interpolation algorithm includes obtaining bidirectional motion vectors of pixel blocks between adjacent frame images in the target image group, and inserting the specified number of intermediate frame images between the adjacent frame images using the bidirectional motion vectors of the pixel blocks and a block motion compensation algorithm.

[0007] In some embodiments, the step of obtaining bidirectional motion vectors of pixel blocks between adjacent frame images in the target image group includes obtaining bidirectional motion vectors of pixel blocks between adjacent frame images in the target image group using an improved DIS optical flow algorithm, wherein the resolution of the bottom layer image of the image pyramid used in the improved DIS optical flow algorithm is smaller than the resolution of the bottom layer image of the image pyramid used in the original DIS optical flow algorithm, and / or the number of iterations used in the improved DIS optical flow algorithm is smaller than the number of iterations used in the original DIS optical flow algorithm.

[0008] In some embodiments, the step of determining a main object region and a background region based on a designated frame image in the target image group includes: selecting an image located at an intermediate position in the target image group as a designated frame image; employing an object instance segmentation algorithm to process the designated frame image; and obtaining a main object region and a background region corresponding to the target image group based on the processing result.

[0009] In some embodiments, the step of performing image fusion on the motion-blurred image and the designated frame image according to the subject object region and the background region includes: obtaining a subject object mask image according to the subject object region and the background region; acquiring a weighting coefficient corresponding to the subject object mask image; adjusting pixel values ​​of the subject object mask image based on the weighting coefficient to obtain the adjusted subject object mask image; and performing image fusion on the motion-blurred image and the designated frame image based on the adjusted subject object mask image.

[0010] In some embodiments, the step of obtaining a weighting factor corresponding to the subject object mask image includes obtaining a global motion range corresponding to each frame image in the target image group by an optical flow method, and determining a weighting factor corresponding to the subject object mask image according to the global motion range.

[0011] In some embodiments, performing image fusion on the motion-blurred image and the designated frame image based on the adjusted subject object mask image includes performing image fusion on the motion-blurred image and the designated frame image by employing the following formula:

number

[0012] In some embodiments, the step of obtaining a plurality of image groups based on the sequence of video frames of the initial video includes dividing the sequence of video frames of the initial video at specified intervals to obtain a plurality of image groups, with a predetermined number of overlapping frame images between two adjacent image groups.

[0013] An embodiment of the present disclosure includes an image group acquisition module for obtaining a plurality of image groups based on a video frame sequence of an initial video; a blur processing module for performing motion blur processing based on each frame image in a target image group, fusing the images obtained by performing the motion blur processing on each frame image, and obtaining a motion blurred image corresponding to the target image group, wherein each image group in the plurality of image groups is the target image group; a region determination module for determining a main object region and a background region corresponding to the target image group based on a designated frame image in the target image group; and fusing the motion blurred image and the designated frame image according to the main object region and the background region, and obtaining a motion blurred image corresponding to the target image group. the video processing device further includes: a fusion module for obtaining a target fusion image, wherein an image portion in the subject object region of the target fusion image is an image portion in the subject object region of the designated frame image, and an image portion in the background region of the target fusion image is an image portion in the background region of the motion blur image; and a video generation module for generating a target video based on the target fusion images corresponding to each of the plurality of image groups, wherein a playback order of the target fusion images corresponding to each of the plurality of image groups in the target video is the same as a playback order of the target fusion images in the initial video of the plurality of image groups.

[0014] An embodiment of the present disclosure further provides an electronic device including a processor and a memory for storing instructions executable by the processor, the processor being configured to read the executable instructions from the memory and execute the instructions to implement a video processing method according to an embodiment of the present disclosure.

[0015] An embodiment of the present disclosure further provides a computer-readable storage medium having stored thereon a computer program that, when run by a processor, causes the processor to perform a video processing method according to an embodiment of the present disclosure.

[0016] An embodiment of the present disclosure further provides a computer program comprising instructions that, when executed by a processor, cause the processor to perform a video processing method according to an embodiment of the present disclosure.

[0017] It should be understood that the contents described in this section are not intended to identify key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily apparent from the following specification. [Brief explanation of the drawings]

[0018] The drawings herein, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

[0019] In order to more clearly describe the technical solutions in the embodiments of the present disclosure or the prior art, the following briefly describes the drawings that need to be used in the description of the embodiments or the prior art. Obviously, those skilled in the art can obtain other drawings based on these drawings without any creative efforts. [Figure 1] 1 is a schematic diagram illustrating the flow of a video processing method according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a schematic diagram of frame interleaving between adjacent frame images according to an embodiment of the present disclosure. [Figure 3]1 is a schematic diagram illustrating the configuration of a video processing device according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram illustrating the configuration of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0020] In order to make the above-mentioned objects, features and advantages of the present disclosure more clearly understandable, the following further describes the aspects of the present disclosure. It should be noted that, unless contradictory, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0021] In the following description, many specific details are set forth in order to fully understand the present disclosure; however, the present disclosure may be embodied in other forms different from those described herein. Obviously, the embodiments in the specification are only some of the embodiments of the present disclosure, but not all of the embodiments.

[0022] As mentioned above, in order to achieve a video effect in which the main object is clear and the background is blurred, giving the impression of frame dropping, it is often necessary to use professional filming equipment to perform slow shutter shooting and / or film using a moving shot, and it is also necessary for the video creator to have solid filming skills and to film an appropriate scene. For many video creators, it is difficult to meet the above filming conditions, and it is therefore difficult to achieve the above video effect through filming.

[0023] To achieve a video effect in which the main object is clear and the background is blurred, creating a sense of frame skipping, generally requires professional photography equipment, solid photography skills, and an appropriate shooting scene. For example, a professional photography stabilizer and tripod must be used in combination to perform slow shutter photography, which creates a blurred background and motion smear to create a sense of blurred screen. Furthermore, the slow shutter must be professionally adjusted during shooting, and proper exposure must be used to achieve the desired effect. Furthermore, to achieve the above video effect through photography, high requirements must be placed on the shooting scene, such as a nighttime or dark scene; otherwise, sufficient light is likely to result in overexposure.

[0024] Related technologies often generate motion smear by controlling the shooting frame rate and exposure time, but this method is limited to shooting scenes and can only capture dark scenes, so it cannot be applied to all scenes. Furthermore, it cannot protect the main object in the video, and only generates smear throughout the image, making it difficult to ensure the main object is clear. Furthermore, professional video stabilizers are often lacking for personal video recording, and the main object is often blurred due to the user's hand shake.

[0025] To address the above-mentioned problems, the embodiments of the present disclosure provide a video processing method, device, equipment, and medium that can process a normally shot video into a video with a clear main subject image and a blurred background with a dropped frame effect through software processing, as described in detail below.

[0026] 1 is a schematic diagram illustrating a flow of a video processing method according to an embodiment of the present disclosure. The method may be performed by a video processing device, which may be implemented in software and / or hardware and generally integrated into electronic equipment. As shown in FIG. 1, the method mainly includes the following steps S102 to S110.

[0027] Step S102: Obtain a group of images based on a sequence of video frames of the initial video.

[0028] The initial video may be a video obtained by shooting without restrictions on shooting tools, shooting skills, and shooting scenes, for example, a video obtained by a user simply shooting an arbitrary scene with a mobile phone. The initial video may be a video obtained by a user shooting in real time, or a pre-recorded video uploaded by a user.

[0029] In some embodiments, a video frame sequence of an initial video may be divided at specified intervals to obtain multiple image groups. The embodiments of the present disclosure are not limited to a specific division method. The division method may be, for example, average division (i.e., division at equal intervals), non-average division, or cross division (where overlapping frame images exist between adjacent image groups obtained by cross division). The specified interval may be a certain number. Therefore, the number of frame images in each image group may be the same, and each image group may contain N frame images. The number N may be flexibly set as needed and may be determined, for example, with reference to the frame rate of the initial video and the actual frame rate of the desired video. For example, the value of N may be the ratio between the frame rate of the initial video and the actual frame rate of the desired video. For example, if the ratio is not an integer, the value closest to the ratio may be an integer. In some embodiments, the frame images between two adjacent image groups are completely different. In other embodiments, some frame images between two adjacent image groups are the same, that is, some frame images are overlapped. In other words, there is a predetermined number of overlapping frame images between two adjacent image groups. This ensures a reasonable number of image groups (i.e., a reasonable frame rate for the video generated later), while also ensuring the image fusion effect during subsequent processing of each image group. For ease of understanding, the following is an example explanation.

[0030] Assuming the original frame rate of the initial video is X fps, N frames of images may be processed as a set to generate a continuous frame-skipped video. This allows for subsequent fusion of N frames into one frame. For example, if the actual frame rate of the desired video is 10 fps to 15 fps, N may be selected as X / 10, i.e., X / 10 original frames are fused into one frame. Assuming the original frame rate is 30 fps, three original frames are fused into one frame. Assuming the original frame rate is 60 fps, six original frames are fused into one frame. The above is merely an example of selecting the value of N and should not be considered limiting. Let Pi be the image of the ith frame of the video frame sequence to be processed. In some embodiments, P1 to P6 are one set of images, P7 to P12 are one set of images, P12 to P17 are one set of images, and so on. The number of image groups obtained in this way is generally small, resulting in a low frame rate for the final video, resulting in noticeable frame drops. On the other hand, if the number of frame images in the image groups is reduced—for example, P1 to P3 as one image group and P4 to P6 as another—and only three frames are fused each time, the degree of motion smear is small, and noticeable flow effects are not easily observed. To achieve a better fusion effect, embodiments of the present disclosure may multiplex image frames. While six frames are still selected and processed as a set, there are overlapping frames between adjacent image groups, i.e., P1 to P6 as one image group, P4 to P9 as another image group, P7 to P12 as another image group, P10 to P15 as another image group, and so on. In other words, between any two adjacent image groups, three frames of images are superimposed, and by multiplexing the frame images in this way, it is possible to double the number of image groups while ensuring that each image group contains six frames of images.Therefore, the rational number of image groups can be ensured while ensuring the fusion effect during subsequent processing of multiple frames in each image group, which means that the sense of flow and blurring of the entire screen can be improved under the premise of ensuring the generated video frame rate.

[0031] Each image group is set as a target image group, that is, the following steps S104 to S108 are executed for each image group.

[0032] Step S104: Motion blur processing is performed based on each frame image in the target image group, and the images obtained by performing the motion blur processing on each frame image are fused to obtain a motion blurred image corresponding to the target image group.

[0033] Motion Blur is a post-processing method for capturing the motion effect of an object (such as an object, animal, or person), mainly simulating the imaging technique of exposing the object while it is moving. For example, it simulates the indirect exposure function of capturing a moving object during imaging, creating a dynamic effect in the image, such as the effect of an object blurring or moving. For example, the motion blur is along a specified direction.

[0034] In an embodiment of the present disclosure, motion blur processing is performed based on each frame image in the target image group, and all images obtained by the motion blur processing are then fused. For example, the images obtained by the motion blur processing may not only include the processed original frame images in the target image group, but may also include frame images added based on the original frame images during the motion blur processing. Finally, after fusion is performed on all images, a motion blur image corresponding to the target image group can be obtained. The motion blur image has a blurred and shaky screen effect.

[0035] Step S106: Based on the designated frame image in the target image group, a main object region and a background region corresponding to the target image group are determined.

[0036] The embodiments of the present disclosure do not impose any limitations on the type of the subject object, and the subject object may be, for example, a person, an animal, or an object such as a vehicle.

[0037] To ensure that the subject object portion of a video appears relatively clearly on the screen, an embodiment of the present disclosure proposes an object protection policy. For example, a designated frame image may be selected from a target image group, and the designated frame image may be an intermediate frame in the target image group. Object segmentation is performed on the designated frame image, and a subject object region and a background region corresponding to the target image group can be finally obtained based on the segmentation results. The subject object region and background region can then be used to realize protection for the subject object. For example, object segmentation is performed on the designated frame image (e.g., human image segmentation is performed when the subject object is a person), and the subject object region and background region in the designated frame image can be obtained. The subject object region and background region in the designated frame image can then be used as the subject object region and background region corresponding to the target image group, with the background region being the region other than the subject object region.

[0038] Note that the above steps S104 and S106 may be executed in parallel, regardless of the order of precedence.

[0039] Step S108: Fusion is performed on the motion blurred image and the designated frame image according to the main object region and the background region to obtain a target fusion image.

[0040] The image portion in the subject object region of the target fusion image is the image portion in the subject object region of the designated frame image, and the image portion in the background region of the target fusion image is the image portion in the background region of the motion-blurred image. That is, the subject object region in the target fusion image is composed of pixels in the subject object region of the designated frame image, and the background region in the target fusion image is composed of pixels in the background region of the motion-blurred image. With the above method, the target fusion image has a blurred and shaky background, but a relatively clear subject object.

[0041] For example, after dividing a designated frame image in a target image group into a main object region and a background region, the main object region and the background region may be distinguished using a specific method. For example, a main object mask image may be generated based on the main object region and the background region. The main object mask image may be labeled by adopting different pixel values ​​for different regions. For example, all pixel values ​​of the background region in the main object mask image are 0, and all pixel values ​​of the main object region are 1. Then, based on the main object mask image, the motion blur image and the designated frame image are fused to obtain a target fusion image that combines the clear main object in the designated frame image with the blurred and blurred background in the motion blur image.

[0042] Step S110: Generate a target video based on target fusion images corresponding to each of the plurality of image groups, and the playback order of the target fusion images corresponding to each of the plurality of image groups in the target video is the same as the playback order of the target fusion images in the initial video of the plurality of image groups.

[0043] Each image group is treated as a target image group and the above steps S104 to S108 are performed on it. Therefore, each image group has one target fusion image. All target fusion images are arranged in a corresponding front-to-back positional relationship in the video frame sequence of the initial video of the multiple image groups. Each target fusion image is considered to be one frame that constitutes the target video. After being arranged in order, multiple target fusion images can constitute the video frame sequence of the target video. In other words, the video frame sequence composed of target fusion images is the target video. The number of video frames included in the target video is fewer than the number of video frames in the initial video, and each frame image in the target video is obtained by fusing multiple frames in the initial video after performing processing such as motion blur and subject object protection. Therefore, the target video can have a certain frame skipping effect, and while the background of the image screen is blurred and shaken, the subject person remains clear.

[0044] By using the above method and software algorithm, a normally shot video can be processed into a video with a clear main subject and a blurred background with a frame-drop effect, allowing users to easily and quickly achieve the above video shooting effect without being limited by the shooting tools, shooting skills, or shooting scene.

[0045] The above technical solution according to the embodiments of the present disclosure obtains a plurality of image groups based on a sequence of video frames of an initial video, and each image group is used as a target image group to perform the following operations: perform motion blur processing based on each frame image in the target image group, and fuse the images obtained by motion blur processing each frame image to obtain motion blur images corresponding to the target image group; determine a main object region and a background region corresponding to the target image group based on a designated frame image in the target image group; then fuse the motion blur images with the designated frame image according to the main object region and background region to obtain a target fusion image; and finally generate a target video based on the target fusion images corresponding to each of the plurality of image groups. Using the above method and a software algorithm, a normally shot video can be processed into a video with a clear main subject and a blurred background with a frame-drop effect. Users can easily and quickly achieve the above video shooting effect regardless of the limitations of their shooting tools, shooting skills, and shooting scenes.

[0046] In some embodiments, the step of performing motion blur processing based on each frame image in the target image group and fusing the images obtained by performing the motion blur processing on each frame image may be performed with reference to the following steps A to B.

[0047] Step A: An optical flow interpolation algorithm is employed to insert a specified number of intermediate frame images between adjacent frame images in the target image group, and all frame images in the target image group into which frames have been inserted are images obtained by applying motion blur processing to each frame image in the target image group.

[0048] Optical flow is the "instantaneous velocity" of pixel movement on the observed image plane of an object moving in space. Optical flow studies utilize the time-domain changes and correlations of pixel intensity data in an image sequence to determine the "motion" of each pixel location. In other words, optical flow algorithms match pixels in one image with pixels in another image, and through matching, determine how the pixels "move" or "flow" from one image to the other. After matching for each pixel, local pixel movement can be used to interpolate intermediate views between the two images. In some embodiments, to save computational power and improve processing efficiency, frame interpolation may be performed using sparse optical flow interpolation. For example, a frame image may be divided into pixel blocks of a specified size (e.g., 16*16), and pixel block matching and motion vector calculation are performed on a pixel block-by-pixel basis. The motion vectors corresponding to all pixels in the same pixel block are the same, while the motion vectors between different pixel blocks may be the same or different. This method can significantly reduce computational power. Regardless of whether it is the service side or the mobile terminal, the video processing can be performed directly by the above method. Based on this, in some embodiments, the above step A may be performed by referring to the following steps A1 to A2.

[0049] Step A1: Obtain bidirectional motion vectors of pixel blocks between adjacent frame images in a target image group.

[0050] For example, bidirectional motion vectors include forward motion vectors and backward motion vectors. For example, adjacent frame images are a previous frame image Fa and a next frame image Fb, respectively. Using Fa as a reference, pixel blocks in Fa are matched with pixel blocks in Fb, and a forward motion vector is calculated in the direction from Fa to Fb. Using Fb as a reference, pixel blocks in Fb are matched with pixel blocks in Fa, and a backward motion vector is calculated in the direction from Fb to Fa. Bidirectional motion vectors can accurately and reliably represent the tendency of optical flow movement between pixel block images.

[0051] In some embodiments, a modified DIS optical flow algorithm may obtain bidirectional motion vectors of pixel blocks between adjacent frame images in a target image set.

[0052] For example, the resolution of the bottom layer image of the image pyramid employed in the improved DIS optical flow algorithm is smaller than the resolution of the bottom layer image of the image pyramid employed in the original DIS optical flow algorithm.

[0053] For example, the number of iterations used in the improved DIS optical flow algorithm is smaller than the number of iterations used in the original DIS optical flow algorithm. For example, the resolution of the bottom image of the image pyramid used in the original DIS optical flow algorithm is the resolution of the original image, and the resolution of the bottom image of the image pyramid used in the improved DIS optical flow algorithm is 1 / 4 of the resolution of the original image. The number of iterations used in the original DIS optical flow algorithm is 12, and the number of iterations used in the improved DIS optical flow algorithm is 5.

[0054] The DIS optical flow algorithm is an abbreviation for the dense inverse search-based method. The original DIS optical flow algorithm belongs to the dense optical flow algorithm. In the embodiment of the present disclosure, an improvement is made based on the original DIS optical flow algorithm to save computational power. For example, the DIS algorithm zooms images to different sizes to build an image pyramid, and then estimates optical flow layer by layer, starting from the layer with the smallest resolution and working downward. The optical flow estimated at each layer is used as the initialization for the estimation at the next layer, thereby achieving the goal of accurately estimating motion at different resolutions. In the embodiment of the present disclosure, only a sparse optical flow is required (i.e., instead of calculating a corresponding optical flow for each pixel, all pixels in each pixel block share a single optical flow, which can represent a motion vector). Therefore, the DIS optical flow algorithm is improved to reduce the resolution of the bottom layer image of the image pyramid (i.e., the highest resolution). For example, the highest resolution is set to 1 / 4 of the original image. Furthermore, even at the highest resolution, a sparse optical flow can be obtained without a densification step. Furthermore, since the embodiment of the present disclosure only requires a sparse optical flow and does not require high accuracy, a small number of iterations can be used to find a solution using gradient descent. Therefore, the 12 iterations of the original DIS optical flow algorithm are reduced to 5 iterations. After improving the DIS optical flow algorithm, the improved DIS optical flow algorithm can be used to quickly obtain bidirectional motion vectors of pixel blocks between adjacent frame images.

[0055] Step A2: Insert a specified number of intermediate frame images between adjacent frame images according to the bidirectional motion vectors of the pixel blocks and the block motion compensation algorithm. The intermediate frame images are images inserted between adjacent frame images.

[0056] Motion compensation is a method for describing the differences between adjacent frames, such as how each pixel block in a previous frame image gradually moves to a certain position in a subsequent frame image. In a block motion compensation algorithm (also known as blocking motion compensation), each frame image is divided into a number of pixel blocks. Based on the pixel blocks in the original frame image and their corresponding motion vectors, their positions in intermediate frame images can be predicted. For example, if the bidirectional motion vectors between pixel blocks in adjacent frame images are known, the pixel blocks in the adjacent frame images can be sampled M times at equal distances along their motion paths, with one frame inserted for each sample. The value M of the sampling number can represent the fineness of the image fusion. The larger the M value, the more natural the image fusion. The smaller the M value, the coarser the image fusion and the more noticeable overlapping marks are likely to appear. Frame insertion is performed using block motion compensation to obtain a blur effect diagram between adjacent frames. For ease of understanding, refer to the schematic diagram of frame insertion between adjacent frame images shown in Figure 2. Fa and Fb are adjacent frames. For any pixel block block_i in frame Fa, the corresponding block_i0 and block_iM are found from the previous and next frames. Based on the bidirectional motion vector (forward motion vector F_ab and backward motion vector F_ba) of the pixel block, M equal-distance samplings are performed along the corresponding motion path, with one frame inserted for each sampling. For example, the pixel block positions used in the jth and kth samplings are as shown in Figure 2, where block_ij is the pixel block corresponding to the jth sampling and block_ik is the pixel block corresponding to the kth sampling. As shown in Figure 2, each pixel is duplicated and superimposed along the motion path of its corresponding pixel block, creating a realistic and smooth motion blur effect. Using the above method, multiple intermediate frame images can be inserted between adjacent frame images through multiple samplings, and each intermediate frame image is a blurred image.

[0057] Step B: Average fusion is performed on the images obtained by motion blurring each frame image.

[0058] By averaging the pixel values ​​of all images obtained through motion blur processing (the original adjacent image frames and the inserted intermediate frame images), a motion blurred image corresponding to the target image group can be obtained. In this way, the final motion blurred image can simulate the indirect exposure function of capturing a moving object during shooting, creating a dynamic motion blur effect in the image. In addition, the pixel block processing method reduces the required computing power while ensuring the image fusion effect, effectively improving the overall performance of the algorithm and ensuring feasibility for mobile devices.

[0059] The above method can generate a corresponding motion-blurred image based on the frame images in each image group. The degree of blur in the motion-blurred image is generally proportional to the degree of motion; the faster the motion, the longer the smear. The implementation principle and achievable effect of adopting the above algorithm are consistent with the actual slow shutter principle and the degree of blur in captured images. Therefore, the following problems exist: When a user desires a moving, blurred background but a relatively clear main object, neither the above blur processing algorithm nor the actual shooting effect can avoid the main object being blurred due to the subject's movement or the shaking of the shooting device. In other words, the main object in the motion-blurred image obtained by the above motion blur processing method according to the embodiment of the present disclosure is also blurred, making it difficult to present it clearly to the user. To address this issue, the embodiment of the present disclosure proposes an object protection policy to perform object segmentation based on specified frame images in the target image group, obtain the main object region and background region corresponding to the target image group, and perform object protection according to the main object region and background region. For example, an image located at the middle of the target image group may be selected as the designated frame image, which contributes to a more natural fusion process later.

[0060] In some embodiments, an image located at the middle position of the target image group is set as a designated frame image, and an object instance segmentation algorithm is employed to process the designated frame image, and a subject object region and a background region corresponding to the target image group are obtained based on the processing result. For example, a subject object mask image may be obtained according to the subject object region and the background region. In some embodiments, the designated frame image may contain at least one object. Therefore, a subject object mask may be determined from at least one object mask. The subject object mask is the object mask closest to the image center, thereby obtaining a subject object mask image.

[0061] In some embodiments of determining the subject object mask image, reference may be made to steps 1 to 4 below.

[0062] Step 1: Image erosion is performed on the object segmentation results (alpha segmentation diagram) of the specified frame image to reduce connections between multiple objects.

[0063] Step 2: The eroded image is binarized, and then connected region detection is performed to find the largest connected region closest to the center of the image and set it as the main object.

[0064] Step 3: The selected connected regions are dilated and mapped to the original Alpha partition map to obtain the main object mask.

[0065] Step 4: The subject object mask is optimized, for example, by performing box blur and edge smoothing processing, to obtain a subject object mask image.

[0066] Through the above method, a subject object mask image can be obtained, and the subject object mask image can be used to easily protect the subject object later.

[0067] In the embodiment of the present disclosure, the two processes of acquiring the motion blur image and the subject object mask image corresponding to the target image group may be executed in parallel, regardless of the order.

[0068] After obtaining the motion blur image and the subject object mask image through the above method, in some embodiments, a target fusion image corresponding to the target image group may be obtained based on the motion blur image, the subject object mask image and the designated frame image.

[0069] To make the obtained target video frame image more realistic, the embodiment of the present disclosure may control the degree of protection of the subject object. For example, to avoid a sense of incongruity, when the global motion range is large, the subject object is not particularly clear. Based on this, the step of performing image fusion on the motion blur image and the designated frame image based on the subject object mask image may refer to the following steps (1) to (3).

[0070] In step (1), a weighting factor corresponding to the subject object mask image is obtained. The weighting factor is related to the degree of protection of the subject object, and the larger the weighting factor, the higher the degree of protection of the subject object and the clearer the subject object.

[0071] In some embodiments, an optical flow method may be used to obtain a global motion range corresponding to each frame image in the target image group, and a weighting factor corresponding to the subject object mask image may be determined based on the global motion range. The embodiments of the present disclosure are not limited to the optical flow method. For example, a sparse optical flow method may be employed to determine pixel block motion information. This allows the global motion range corresponding to each frame image in the target image group to be obtained. The global motion range is negatively correlated with the weighting factor. The larger the global motion range, i.e., the faster the motion, the smaller the weighting factor, resulting in a relatively lower sharpness of the subject object (however, the sharpness is still higher than the sharpness of the blurred background, and the subject object simply becomes less sharp). As described above, the embodiments of the present disclosure can adjust the degree of object protection based on the global motion range due to lens shift.

[0072] In step (2), pixel values ​​of the subject object mask image are adjusted based on the weighting coefficients to obtain an adjusted subject object mask image. In some examples, the adjusted subject object mask image may be obtained by multiplying the pixel values ​​of the subject object mask image by the weighting coefficients.

[0073] In step (3), image fusion is performed on the motion blur image and the designated frame image based on the adjusted subject object mask image. For example, the following formula may be used to perform image fusion on the motion blur image and the designated frame image:

number

[0074] The target fusion image obtained by performing image fusion based on the above formula has a blurred and shaky background, but the main object is relatively clear, and the clarity can be adjusted based on a weighting factor. The weighting factor may be determined based on the global motion range caused by the lens shift, so that the clarity of the main object is related to the global motion range, resulting in a more realistic and natural screen effect.

[0075] The video frame sequence of the initial video is segmented (using methods such as average segmentation, non-average segmentation, and cross segmentation, without limitation) to obtain corresponding target fusion images for each of the resulting images. Then, all target fusion images are sequentially arranged to form the desired target video. Furthermore, multiple frames of the initial video are fused into one frame of the target video, thereby reducing the frame rate and providing the user with a sense of frame lag. As described above, a user can easily and quickly convert a normally shot video into a target video with a clear main object and a blurred background, without being limited by the shooting tools, shooting skills, or shooting scene, simply by using the above video processing method according to the embodiment of the present disclosure and employing a software algorithm. The target video has a unique style and can present the user with a video screen with a sense of movement and frame lag, while the main object in the video screen remains clear, making it stand out well. Taking a person as the subject object as an example, the above video effect can reflect the subject person's inner thoughts to a certain extent, leaving a deep impression. In addition, during the processing, for example, a sparse optical flow algorithm is adopted to perform motion blur, which can effectively reduce the computational power, improve the overall performance of the algorithm, and ensure the feasibility of mobile terminals. Therefore, it can be realized on both the service side and the mobile terminal side, and has a wider range of application.

[0076] Corresponding to the video processing method, an embodiment of the present disclosure provides a video processing device. Figure 3 is a schematic diagram of the configuration of a video processing device according to an embodiment of the present disclosure. The device may be realized by software and / or hardware, and may be integrated into a general electronic device, as shown in Figure 4.

[0077] The video processing device includes an image set acquisition module 302 for obtaining a plurality of image sets based on a sequence of video frames of an initial video; a blur processing module 304 for performing motion blur processing based on each frame image in a target image group, fusing the images obtained by performing the motion blur processing on each frame image, and obtaining a motion blur image corresponding to the target image group, wherein each image group in the plurality of image groups is the target image group; a region determination module 306 for determining a subject object region and a background region corresponding to the target image group based on a designated frame image in the target image group; a fusion module 308 for fusing the motion-blurred image and the designated frame image according to the main object region and the background region to obtain a target fusion image, wherein an image portion in the main object region of the target fusion image is an image portion in the main object region of the designated frame image, and an image portion in the background region of the target fusion image is an image portion in the background region of the motion-blurred image; and a video generation module 310 for generating a target video based on target fusion images corresponding to each of the plurality of image groups, wherein the playback order of the target fusion images corresponding to each of the plurality of image groups in the target video is the same as the playback order in the initial video of the plurality of image groups.

[0078] By using the above device and adopting a software algorithm, a video obtained by normal shooting can be processed into a video in which the main subject is clear and the background is blurred, giving the effect of frame dropping. Users can easily and quickly achieve the above video shooting effect without being limited by shooting tools, shooting skills, or shooting scenes.

[0079] In some embodiments, the blur processing module 304 is used to employ an optical flow interpolation algorithm to insert a specified number of intermediate frame images between adjacent frame images in the target image group, converting all frame images in the target image group into images obtained by motion blur processing each frame image in the target image group, and performing average fusion on the images obtained by motion blur processing each frame image.

[0080] In some embodiments, the blur processing module 304 is used to obtain bidirectional motion vectors of pixel blocks between adjacent frame images in the target image group, and to insert a specified number of intermediate frame images between the adjacent frame images using the bidirectional motion vectors of the pixel blocks and a block motion compensation algorithm.

[0081] In some embodiments, the blur processing module 304 is used to obtain bidirectional motion vectors of pixel blocks between adjacent frame images in the target image group using an improved DIS optical flow algorithm, and the resolution of the bottom image of the image pyramid used in the improved DIS optical flow algorithm is smaller than the resolution of the bottom image of the image pyramid used in the original DIS optical flow algorithm, and / or the number of iterations used in the improved DIS optical flow algorithm is smaller than the number of iterations used in the original DIS optical flow algorithm.

[0082] In some embodiments, the region determination module 306 is used to select an image located at the middle position of the target image group as a designated frame image, employ an object instance segmentation algorithm to process the designated frame image, and obtain a main object region and a background region corresponding to the target image group based on the processing result.

[0083] In some embodiments, the fusion module 308 is used to obtain a subject object mask image according to the subject object region and the background region, obtain a weighting factor corresponding to the subject object mask image, adjust pixel values ​​of the subject object mask image based on the weighting factor to obtain the adjusted subject object mask image, and perform image fusion on the motion blur image and the designated frame image based on the adjusted subject object mask image.

[0084] In some embodiments, the fusion module 308 is used to obtain a global motion range corresponding to each frame image in the target image group by an optical flow method, and determine a weighting coefficient corresponding to the subject object mask image according to the global motion range.

[0085] In some embodiments, the fusion module 308 is used to: perform image fusion on the motion blurred image and the designated frame image based on the adjusted subject object mask image; performing image fusion on the motion-blurred image and the designated frame image by employing the following formula:

number

[0086] In some embodiments, the image group acquisition module 302 is used to divide the video frame sequence of the initial video at specified intervals to obtain multiple image groups, with a predetermined number of overlapping frame images between two adjacent image groups.

[0087] A video processing device according to an embodiment of the present disclosure can execute a video processing method according to any embodiment of the present disclosure, and has functional modules and beneficial effects corresponding to the executed method.

[0088] For those skilled in the art to clearly understand, for convenience and simplicity of description, the operation processes of the above-described apparatus embodiments can be referred to the corresponding processes in the method embodiments, and will not be further described here.

[0089] An embodiment of the present disclosure further provides an electronic device including a processor and a memory for storing instructions executable by the processor, the processor reading the executable instructions from the memory and executing the instructions to realize the above video processing method. Figure 4 is a schematic diagram of the configuration of an electronic device according to an embodiment of the present disclosure. As shown in Figure 4, an electronic device 400 includes one or more processors 401 and a memory 402.

[0090] The processor 401 may be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities, and may control other components in the electronic device 400 to perform desired functions.

[0091] The memory 402 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored in the computer-readable storage medium, and the processor 401 may execute the program instructions to implement the video processing method of the embodiments of the present disclosure described above and / or other desired functions. The computer-readable storage medium may also store various contents, such as an input signal, a signal component, and a noise component.

[0092] In some examples, the electronic device 400 may further include input devices 403 and output devices 404, these components being interconnected by a bus system and / or other form of connection (not shown).

[0093] The input device 403 may further include, for example, a keyboard, a mouse, and the like.

[0094] The output device 404 can output various information to the outside, including determined distance information, direction information, etc. The output device 404 may include, for example, a display, a speaker, a printer, a communication network, and a remote output device connected thereto.

[0095] Of course, for simplicity, Fig. 4 shows only some of the components related to the present disclosure in the electronic device 400, and omits components such as a bus, an input / output interface, etc. Furthermore, depending on a specific application situation, the electronic device 400 may further include any other appropriate components.

[0096] In addition to the methods and apparatus described above, embodiments of the present disclosure may also be a computer program product including computer program instructions that, when run by a processor, cause the processor to perform a video processing method according to embodiments of the present disclosure.

[0097] The computer program product may include program code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as general procedural programming languages ​​such as "C" or similar. The program code may execute entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0098] An embodiment of the present disclosure may also be a computer-readable storage medium having stored thereon computer program instructions that, when run by a processor, cause the processor to perform a video processing method according to an embodiment of the present disclosure.

[0099] The computer-readable storage medium may employ any combination of one or more computer-readable media. The computer-readable medium may be a readable signal medium or a computer-readable storage medium. The computer-readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. Examples (non-exhaustive list) of computer-readable storage media include an electrical connection having one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), fiber optics, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0100] An embodiment of the present disclosure further provides a computer program product including computer programs / instructions that, when executed by a processor, implement the video processing method in an embodiment of the present disclosure.

[0101] An embodiment of the present disclosure further provides a computer program comprising instructions that, when executed by a processor, cause the processor to perform a video processing method according to an embodiment of the present disclosure.

[0102] It should be noted that, in this specification, relational terms such as "first" and "second" are used solely to distinguish one entity or operation from another and do not require or imply the existence of any actual relationship or order between those entities or operations. Furthermore, the terms "comprise," "include," or any other variant thereof, indicate a non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements but also other elements not expressly specified or elements inherent in such process, method, article, or device. Absent further limitations, an element qualified by "comprises a..." does not exclude the process, method, article, or device that includes said element from also including other identical elements.

[0103] The foregoing are merely specific embodiments of the present disclosure, and are used to enable those skilled in the art to understand or realize the present disclosure. Various modifications to these examples will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other examples without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to these examples herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. obtaining a plurality of images based on a sequence of video frames of an initial video; performing motion blur processing based on each frame image in a target image group, performing fusion on the images obtained by performing the motion blur processing on each frame image, and acquiring motion blur images corresponding to the target image group, and each image group in the plurality of image groups is the target image group; determining a main object region and a background region corresponding to the target image group based on a designated frame image in the target image group; performing fusion on the motion-blurred image and the designated frame image according to the main object region and the background region to obtain a target fusion image, an image portion in the main object region of the target fusion image being an image portion in the main object region of the designated frame image, and an image portion in the background region of the target fusion image being an image portion in the background region of the motion-blurred image; generating a target video based on target fusion images corresponding to each of the plurality of groups of images, wherein the playback order of the target fusion images corresponding to each of the plurality of groups of images in the target video is the same as the playback order of the target fusion images in the initial video of the plurality of groups of images.

2. The step of performing motion blur processing based on each frame image in a target image group, and performing fusion on the images obtained by performing motion blur processing on each of the frame images, comprises: An optical flow interpolation algorithm is employed to insert a specified number of intermediate frame images between adjacent frame images in the target image group, and all of the frame images in the target image group into which frames have been inserted are images obtained by performing motion blur processing on each frame image in the target image group; 2. The video processing method according to claim 1, further comprising: performing average fusion on images obtained by performing motion blur processing on each of the frame images.

3. The step of inserting a specified number of intermediate frame images between adjacent frame images in the target image group by employing an optical flow interpolation algorithm comprises: Obtaining bidirectional motion vectors of pixel blocks between adjacent frame images in the target image group; 3. The video processing method of claim 2, further comprising inserting a specified number of intermediate frame images between the adjacent frame images according to the bidirectional motion vectors of the pixel blocks and a block motion compensation algorithm.

4. The step of obtaining bidirectional motion vectors of pixel blocks between adjacent frame images in the target image group comprises: obtaining bidirectional motion vectors of pixel blocks between adjacent frame images in the target image group by an improved DIS optical flow algorithm based on dense reverse search; 4. The video processing method of claim 3, wherein the resolution of the bottom image of the image pyramid used in the improved DIS optical flow algorithm is smaller than the resolution of the bottom image of the image pyramid used in the original DIS optical flow algorithm, and / or the number of iterations used in the improved DIS optical flow algorithm is smaller than the number of iterations used in the original DIS optical flow algorithm.

5. The step of determining a main object region and a background region based on a designated frame image in the target image group comprises: an image located at an intermediate position of the group of target images is designated as a designated frame image; employing an object instance segmentation algorithm to process the designated frame image; The video processing method of claim 1 , further comprising: obtaining a subject object region and a background region corresponding to the target images based on the processing result.

6. The step of performing image fusion on the motion blur image and the designated frame image according to the main object region and the background region includes: obtaining a subject object mask image according to the subject object region and the background region; obtaining a weighting factor corresponding to the subject object mask image; adjusting pixel values ​​of the subject object mask image based on the weighting coefficients to obtain an adjusted subject object mask image; The video processing method according to claim 1 , further comprising: performing image fusion on the motion-blurred image and the designated frame image based on the adjusted subject object mask image.

7. The step of obtaining a weighting factor corresponding to the subject object mask image comprises: Obtaining a global motion width corresponding to each frame image in the target image group by an optical flow method; 7. The video processing method of claim 6, further comprising: determining a weighting factor corresponding to the subject object mask image in response to the global motion range.

8. The video processing method of claim 7 , wherein the global motion range is negatively correlated with the weighting factor.

9. The step of performing image fusion on the motion blur image and the designated frame image based on the adjusted subject object mask image, performing image fusion on the motion-blurred image and the designated frame image by employing the following formula: [Equation 1] 7. The video processing method of claim 6, wherein β is the weighting coefficient, mask_main is the subject object mask image, β*mask_main is the adjusted subject object mask image, Pn is the designated frame image, Merge_N is the motion blur image, and Merge_N′ is the target fusion image.

10. The step of obtaining a plurality of image groups based on a sequence of video frames of an initial video comprises: The video processing method of claim 1, further comprising: dividing the video frame sequence of the initial video at specified intervals to obtain the plurality of image groups, with a predetermined number of overlapping frame images between two adjacent image groups.

11. an image set acquisition module for obtaining a plurality of image sets based on a sequence of video frames of the initial video; a blur processing module that performs motion blur processing based on each frame image in a target image group, and performs fusion on the images obtained by performing the motion blur processing on each frame image to obtain motion blur images corresponding to the target image group, and each image group in the plurality of image groups is the target image group; a region determination module for determining a main object region and a background region corresponding to the target image group based on a designated frame image in the target image group; a fusion module for fusing the motion-blurred image and the designated frame image according to the main object region and the background region to obtain a target fusion image, wherein an image portion in the main object region of the target fusion image is an image portion in the main object region of the designated frame image, and an image portion in the background region of the target fusion image is an image portion in the background region of the motion-blurred image; a video generation module that generates a target video based on target fusion images corresponding to each of the plurality of groups of images, wherein a playback order of the target fusion images corresponding to each of the plurality of groups of images in the target video is the same as a playback order of the target fusion images corresponding to each of the plurality of groups of images in the initial video.

12. a processor; a memory for storing instructions executable by the processor; The electronic device, wherein the processor is configured to read the executable instructions from the memory and execute the instructions to implement the video processing method of any one of claims 1 to 10.

13. A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, causes the processor to perform the video processing method of any one of claims 1 to 10.

14. A computer program comprising instructions which, when executed by a processor, cause the processor to perform the video processing method of any one of claims 1 to 10.

Images