Video processing method, device, electronic device and storage medium

The video processing method addresses the lack of user interaction methods in current video applications by allowing the target object to disappear into the video, enhancing user experience through personalized visual effects.

JP7583165B2Active Publication Date: 2024-11-13DOUYIN VISION CO LTD

Patent Information

Application Number
JP2023528594
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-11-16
Filing Date
2021-11-15
Publication Date
2024-11-13
Estimated Expiration
2041-11-15

AI Technical Summary

Technical Problem

Current video applications fail to meet the demand for various user interaction methods during the video shooting or playback process, lacking the ability to provide personalized visual effects.

Method used

A video processing method that fixes a target frame image, removes a target object, and performs a complementary operation on the target area to generate a complementary frame image, allowing the target object to disappear into the video, enhancing user interaction and visual experience.

Benefits of technology

Provides users with a more personalized and enhanced visual experience by creating the effect of the target object disappearing into the video, improving user interaction and visual effects during shooting and playback.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007583165000001
    Figure 0007583165000001
  • Figure 0007583165000002
    Figure 0007583165000002
  • Figure 0007583165000003
    Figure 0007583165000003
Patent Text Reader

Abstract

The present disclosure provides a video processing method, device, electronic device, and storage medium. The video processing method provided by the present disclosure responds to a trigger command by fixing a target frame image of a target video, removing a target object in the target frame image, and performing an interpolation operation on a target region in the target frame image to generate and display an interpolated frame image, thereby realizing the special effect of the target object disappearing in the target video, thereby providing users with a more personalized and superior visual experience in video applications.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to the technical field of video processing, and in particular to a video processing method, device, electronic device and storage medium. [Background technology]

[0002] With the development of communication technologies and terminal devices, various terminal devices such as mobile phones and tablet computers have become an indispensable part of people's work and life. As terminal devices become more and more popular, video applications have become a major means of communication and entertainment.

[0003] However, in current video applications, the video shooting or playback by smart terminals can no longer meet the demand for diversity of user interaction means during the video shooting or playback process. Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides a video processing method, device, electronic device, storage medium, computer program product, and computer program to solve the technical problem that current video interaction applications only record the subject of shooting as an image and are unable to meet the demand for diversity of user interaction means during the video shooting or playback process. [Means for solving the problem]

[0005] In a first aspect, an embodiment of the present disclosure provides a video processing method, the method comprising: freeze a target frame image of the target video in response to a trigger command; In the target frame image EyesThe method includes a step of removing the target object and performing a complementation operation on a target area in the target frame image to generate and display a complemented frame image, the target area including a vacant area remaining after the target object is removed from the target frame image.

[0006] In a second aspect, an embodiment of the present disclosure provides a video processing device, the device comprising: Responds to trigger commands ,eye an image capture module used to fix a target frame image of a target video; In the target frame image Eyes The target object is removed, and the target frame image is Eyes and an image processing module used to perform a complementation operation on a target area and generate and display a complemented frame image, the target area including a vacant area remaining after the target object is removed from the target frame image.

[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device, the electronic device comprising: A processor; A memory used to store a computer program for the processor; a display adapted to display video processed by said processor; The processor is configured to execute the computer program to realize the video processing method as described in the first aspect above and various possible designs of the first aspect.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon, the computer-readable storage medium being operable, when executed by a processor, to realize a video processing method as described in the first aspect above and various possible designs of the first aspect.

[0009] In a fifth aspect, an embodiment of the present disclosure provides a computer program product including a computer program, which, when executed by a processing device, causes the steps of the method set forth in the first aspect of the present disclosure to be realized. Effect of the Invention

[0010] The video processing method, apparatus, electronic device, and storage medium provided by the embodiments of the present disclosure respond to a trigger command, fix a target frame image of a target video, remove a target object in the target frame image, and perform an interpolation operation on a target area in the target frame image to generate a interpolated frame image, thereby achieving a special effect in which the target object disappears in the target video during the process of recording or playing back the video, thereby providing users with a more personalized and better visual experience in video applications, and improving the user experience. [Brief description of the drawings]

[0011] In order to more clearly describe the embodiments of the present disclosure or the solutions in the prior art, the drawings that need to be used in the description of the embodiments or the prior art will be briefly described below. Of course, the drawings described below are some embodiments of the present disclosure, and those skilled in the art can think of other drawings based on these drawings without creative efforts. [Figure 1] FIG. 2 is a diagram of an application scene of a video processing method shown according to one exemplary embodiment of the present disclosure. [Diagram 2] 4 is a flowchart of a video processing method according to one exemplary embodiment of the present disclosure. [Diagram 3] FIG. 2 is a schematic diagram of one interface of a video processing process shown according to one exemplary embodiment of the present disclosure. [Figure 4] FIG. 11 is a diagram of another application scene of the video processing method shown according to one exemplary embodiment of the present disclosure. [Diagram 5] FIG. 2 is a schematic diagram of an intermediate process of a video processing process shown according to one exemplary embodiment of the present disclosure. [Figure 6] FIG. 13 is a schematic diagram of another interface of a video processing process shown according to one exemplary embodiment of the present disclosure. [Figure 7] 4 is a flowchart of a video processing method according to another exemplary embodiment of the present disclosure. [Figure 8] FIG. 1 is a structural schematic diagram of a video processing device shown according to one exemplary embodiment of the present disclosure. [Figure 9] FIG. 1 is a structural schematic diagram of an electronic device according to one exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments of the present disclosure will be described in more detail with reference to the drawings. Although the drawings show some embodiments of the present disclosure, it should be understood that the present disclosure can be realized in various forms and should not be interpreted as being limited to the embodiments described herein, but on the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure serve merely as examples and are not intended to limit the scope of protection of the present disclosure.

[0013] It should be understood that the steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. Additionally, method embodiments may include additional steps and / or omit performance of steps that are shown. The scope of the present disclosure is not limited in this respect.

[0014] As used herein, the term "comprises" and variations thereof mean open inclusive, i.e., "including, but not limited to." The term "based on" means "based at least in part on." The term "in one embodiment" means "at least one embodiment." The term "in another embodiment" means "at least one other embodiment." The term "in some embodiments" means "at least some embodiments." Definitions of other terms are provided below in the description.

[0015] It should be noted that the modifiers "a" and "a plurality" referred to in this disclosure are exemplary rather than limiting, and that one of ordinary skill in the art would understand "one or more" unless the context clearly indicates otherwise.

[0016] With the increasing popularity of terminal devices, video applications have become a major means of communication and entertainment. In the current video applications, the shooting and playing of videos by smart terminals can no longer meet the demand for a variety of interaction methods of users during the process of shooting and playing videos. Meanwhile, in the video applications, there is also a growing interest in providing users with a more personalized visual experience and better visual effects when shooting and playing videos.

[0017] The present disclosure aims to provide a video processing method, which responds to a trigger command, fixes a target frame image of a target video, removes a target object in the target frame image, and performs a complement operation on a target area in the target frame image, and generates and displays a complement frame image, thereby realizing that during the process of recording or playing a video, a user can achieve a special effect in which a target object disappears in the target video by inputting a specific command, thereby providing the user with a more personalized visual experience and better visual effects in video interaction applications.

[0018] Fig. 1 is a diagram of an application scene of a video processing method according to one exemplary embodiment of the present disclosure. As shown in Fig. 1, the video processing method provided by this embodiment can be executed by a terminal device equipped with a camera and a display screen. Specifically, a camera (e.g., a front camera, a rear camera, an exterior camera, etc.) in the terminal device can record a video of a target object (e.g., a person, an animal, a building, etc.), and the target video can be played back by the display screen of the terminal device.

[0019] Taking the case where the target object is a person as an example, in one possible scenario, when using a terminal device to shoot a video of a target person, generally, the camera on the terminal device is pointed at the target person to shoot. As can be understood, when taking a picture, the camera captures the target background at the same time as the target person. At this time, the user can input a trigger command (e.g., a target gesture command, a target voice command, a target facial expression command, a target character command, a target body command, etc.) into the terminal device to trigger the target effect, thereby fixing the target frame image of the target video currently being shot.

[0020] In another possible scenario, when a target video is played using a terminal device, the target video contains not only a target person but also a target background. At this time, the user can input a trigger command (e.g., a target gesture command, a target voice command, a target facial expression command, a target character command, a target body command, etc.) into the terminal device to trigger a target effect, thereby fixing a target frame image of the target video currently being shot.

[0021] The trigger command may be a trigger command input by a user, may be a trigger command issued from a target object in a video, or may include a trigger command issued by a terminal under a preset condition (such as a condition that the video is played up to a preset specific time point). The target frame image may be a first frame image after the trigger command is acquired, or may be a frame image after a preset time has elapsed since the trigger command is acquired. After the target frame image is fixed, the target object in the target frame image may be removed, and a complement operation, for example, a background complement operation, may be performed on the target area in the target frame image to generate and display a complement frame image, and the target area includes an empty area after the target object is removed from the target frame image. As can be seen from the above description, in the process of shooting or playing the target object, the trigger command may be used to trigger a target special effect, and the target object may be removed from the target frame image and a complement operation may be performed, thereby achieving the visual effect of the target object disappearing from the video.

[0022] 2 is a flowchart of a video processing method according to an exemplary embodiment of the present disclosure. As shown in FIG. 2, the video processing method provided by this embodiment includes the following steps:

[0023] In step 101, in response to a trigger command, a target frame image of a target video is frozen.

[0024] In one possible scenario, when using a terminal device to videotape a target object, typically the camera on the terminal device is aimed at the target object to shoot, and it can be understood that in the process of shooting the target object, the camera simultaneously captures not only the target object but also the target background.

[0025] In another possible scenario, when a target video is played using a terminal device, the target video generally includes a target object and a target background, and it is understood that when the target video is shot and generated, the target object is shot based on the target background. Here, for ease of understanding, a case where a target person is taken as a target object can be taken as an example. Still referring to FIG. 1, when a video of a target person is shot using a terminal device, the camera in the terminal device is generally aimed at the target person to shoot, and it is understood that when taking a composition, the camera simultaneously captures the target background in addition to the target person.

[0026] When a terminal device is used to play a target video, since the target video includes a target object and a target background, during playback, a related scene of the target object based on the target background is played.

[0027] 3 is a schematic diagram of an interface of a video processing process shown according to an exemplary embodiment of the present disclosure. As shown in FIG. 3, when a terminal device is used to videotape a target person or play a target video, a target background and a target person are simultaneously displayed on a display screen of the terminal device.

[0028] The terminal device responds to the trigger command and fixes the target frame image of the target video. Figure 4 is a diagram of another application scene of the video processing method shown according to one exemplary embodiment of the present disclosure. As shown in Figure 4, the scene of Figure 4 may be a scene in which a user inputs a target gesture command (e.g., a stretch command) in the process of recording a target video for a target person to trigger the effect that the person disappears from the image, and when the terminal device recognizes the target gesture command, it can fix the target frame image of the currently captured target video, where the current frame of the target video is fixed. In one embodiment, when the terminal device plays the target video, the user can use a target gesture command (e.g., a stretch command) to trigger the effect that the person disappears from the image, and when the terminal device recognizes the target gesture command, it can fix the target frame image of the currently captured target video, where the current frame of the target video is fixed.

[0029] In step 102, the target object in the target frame image is removed, and an interpolation operation is performed on the target area in the target frame image to generate and display an interpolated frame image.

[0030] After the target frame image is fixed, the target frame image can be processed to achieve the effect that the target object disappears in the target frame image, specifically, the target object in the target frame image can be removed, and a complement operation can be performed on the target area in the target frame image to generate and display a complement frame image, where the target area is the empty area after the target object is removed from the target frame image.

[0031] Here, the case where the target person is still taken as the target object can be described as an example. Figure 5 is a schematic diagram of an intermediate process of a video processing process shown according to an exemplary embodiment of the present disclosure. As shown in Figure 5, after the target frame image is fixed, the target person in the target frame image can be removed. Figure 6 is a schematic diagram of another interface of a video processing process shown according to an exemplary embodiment of the present disclosure. As shown in Figure 6, after the target person in the target frame image is removed, a background complement operation can be performed on the target area, i.e., the resulting empty area.

[0032] As shown in Figures 3 to 6, in the process of recording a target video for a target person, a user inputs a hand reaching command to trigger, so that during the video recording, the target person responds to the user's gesture and achieves a special effect of disappearing in the target video. In addition, although not shown, those skilled in the art should understand that in the process of playing the target video, a user can also use a trigger command to achieve a special effect of disappearing in the target video.

[0033] It should be noted that the above description of the target person as the target object is merely an example given for ease of understanding, and in this embodiment, the target object may have the form of a target person, a target animal, a target building, etc., and is not specifically limited here. In addition, the above trigger command may have the form of a target gesture command, a target voice command, a target facial expression command, a target character command, a target body command, etc., and is also not specifically limited here.

[0034] In this embodiment, by responding to a trigger command, fixing a target frame image of a target video, removing a target object in the target frame image, and performing a complement operation on a target area in a target frame image, and generating and displaying a complement frame image, it is realized that a user can achieve a special effect in which a target object disappears in a target video by inputting a specific command, and further provides the user with a more personalized visual experience in video applications and better visual effects.

[0035] 7 is a flowchart of a video processing method according to another exemplary embodiment of the present disclosure. As shown in FIG. 7, the video processing method provided by this embodiment includes the following steps:

[0036] In step 201, in response to a trigger command, a target frame image of a target video is frozen.

[0037] When using a terminal device to shoot a video of a target object, the camera on the terminal device is generally aimed at the target object to shoot, and it can be understood that in the process of shooting the target object, the camera simultaneously captures the target background in addition to the target object.

[0038] In another possible scene, when a target video is played using a terminal device, the target video generally includes a target object and a target background, and it can be understood that when the target video is shot and generated, the target object is shot based on the target background.

[0039] The terminal device can fix a target frame image of the target video by responding to a trigger command, and the target frame image may be the first frame image after the trigger command is obtained, or the corresponding frame image after a preset time has elapsed after the trigger command is obtained.

[0040] Here, for ease of understanding, a case where a target person is the target object can be taken as an example. Still referring to Fig. 1, when using a terminal device to take a video of a target person, the camera in the terminal device is generally aimed at the target person to take the video, but as can be understood, when taking a picture, the camera is configured to simultaneously capture the target background in addition to the target person.

[0041] When a terminal device is used to play a target video, the target video includes a target object and a target background, so that during playback, a related scene of the target object based on the target background is played.

[0042] As shown in FIG. 3, when a terminal device is used to shoot a video of a target person or play a target video, the target background and the target person are simultaneously displayed on the display screen of the terminal device.

[0043] The terminal device responds to the trigger command and fixes the target frame image of the target video. FIG. 4 is a diagram of another application scene of the video processing method shown according to one exemplary embodiment of the present disclosure. As shown in FIG. 4, the scene of FIG. 4 is a scene in which, in the process of recording a target video for a target person, the target person inputs a target gesture command (e.g., a stretch command) to trigger an effect that the person disappears from the image. When the terminal device recognizes the target gesture command, it can fix the target frame image of the currently captured target video, where the current frame of the target video is fixed. In one embodiment, when the terminal device plays the target video, the user can use a target gesture command (e.g., a stretch command) to trigger an effect that the person disappears from the image, and when the terminal device recognizes the target gesture command, it can fix the target frame image of the currently captured target video, where the current frame of the target video is fixed.

[0044] In addition, before triggering a special effect, it may be determined whether the currently captured video contains a target-type object, for example, it may be determined in advance that the currently captured video contains a person object, and then the subsequent special effect may proceed in response to a corresponding trigger command.

[0045] In step 202, a preset second special effect is displayed, which is used to present a visual blur effect in the target frame image.

[0046] After the target frame image of the target video is fixed, a preset second special effect can be displayed, which is used to make the target frame image show a visual blur effect. By adding the visual blur effect to the fixed target frame image, the problem of unnatural transition of the boundary between the complemented part and the original background caused by background complementation can be avoided.

[0047] In step 203, a preset third special effect is displayed, which is used to make the target frame image show a visual shaking effect.

[0048] In this step, after displaying the preset second special effect, a preset third special effect can also be displayed, which is used to make the target frame image show a visual shaking effect.By making the target frame image show a visual shaking effect, the disappearance process of the target object can have a stronger dynamic effect.

[0049] In this embodiment, the order of step 202 and step 203 is not limited, and step 202 may be executed before step 203 or after step 203, step 202 and step 203 may be executed separately, or only either step 202 or step 203 may be executed.

[0050] In step 204, the target object in the target frame image is removed, and an interpolation operation is performed on the target area in the target frame image to generate and display an interpolated frame image.

[0051] After the target frame image is fixed, the target frame image can be processed to achieve the effect that the target object disappears in the target frame image, specifically, the target object in the target frame image can be removed, and a background complement operation can be performed on the target area in the target frame image, so as to generate and display a complement frame image, where the target area is the empty area after the target object is removed from the target frame image.

[0052] Here, the case where the target person is the target object can be taken as an example for explanation. As shown in Fig. 5, after the target frame image is fixed, the target person in the target frame image can be removed. As shown in Fig. 6, after the target person in the target frame image is removed, a complement operation can be performed on the target area, i.e., the resulting empty area.

[0053] Specifically, the target binary image and the target frame image can be fused to obtain a model input image, where the target region in the target binary image includes each pixel point whose pixel value is the target value. Continuing to take the target person as an example, the target human body region in the image is obtained using a conventional human body segmentation model, and a binary image with the same size as the original image is obtained, where 255 represents that this pixel point does not belong to the target human body region, and 0 represents that this pixel point belongs to the target human body region. Then, the target frame image and the target binary image are reduced or enlarged by a certain ratio (for example, 256 / long side of the image), and then fused in an image processing channel and the pixel value is normalized to (-1, 1), thereby obtaining a model input image.

[0054] After obtaining the model input image, the model input image can be subsequently input into the image restoration model to perform a background complementation operation to generate a processed frame image, and then the target area in the processed frame image can be used to replace the target area in the target frame image to generate a complemented frame image.

[0055] The image restoration model may be one model or a combination of multiple models, and the image restoration model may be located on a cloud server or on a terminal device.

[0056] When the image restoration model may be deployed on a cloud server, the terminal device uploads the target frame image to the cloud server, and after completion using the deep learning method, returns the completed result image to the terminal device. In this way, uploading the image to the cloud server increases the interaction time and the network delay is also uncertain, so it cannot support scenes with high real-time requirements.

[0057] When using traditional completion algorithms (such as patchmatch series algorithms) on terminal devices, when the area that needs to be completed is large (such as the scene of background completion of the target human body in the above example), the semantic continuity of the completion result is poor. Therefore, an image inpainting model (such as an image inpainting model including a first image inpainting model and a second image inpainting model) can be arranged on the terminal device, and the model input image can be input into the first image inpainting model to generate a first completion image. The model input image can be input into the first image inpainting model with low accuracy to obtain a rough completion result image, i.e., the first completion image.

[0058] Then, pixel truncation is performed on the first complementary image using a preset pixel threshold value to generate a second complementary image, and the second complementary image is input to a second image restoration model to generate a third complementary image, and the complementation accuracy of the second image restoration model is higher than that of the first image restoration model. Next, pixel truncation is performed on the third complementary image using a preset pixel threshold value to generate a fourth complementary image, and finally, the target area in the fourth complementary image is used to replace the target area in the target frame image to generate the complementation frame image. Here, both the first image restoration model and the second image restoration model may be based on a UNet network structure. Specifically, the above first complementary image can be truncated pixel by pixel, that is, pixel points with pixel values ​​less than -1 are set to -1, pixel points with pixel values ​​greater than 1 are set to 1, and pixel values ​​of pixel points between -1 and 1 are kept to their original values, and the resultant image after truncation, i.e., the second complementary image, is input to a second image restoration model with high accuracy to obtain a corrected complementation resultant image, i.e., the third complementary image. Then, the third complementary image can be further subjected to pixel-by-pixel truncation, i.e., pixel points with pixel values ​​less than -1 are set to -1, pixel points with pixel values ​​greater than 1 are set to 1, and pixel values ​​of pixel points between -1 and 1 are kept at their original values, while the resulting image is reduced or enlarged to the size of the original image, and the part of this image corresponding to the human body area is used to replace the human body area of ​​the original image, to obtain the final resulting image, i.e., the required complementary frame image.

[0059] In addition, as shown in Figures 3 to 6, in the process of recording a target video for a target person, the user can input a trigger command to trigger the video, so that during the video recording, the target person responds to the user's trigger command and achieves a special effect of disappearing into the target video.

[0060] Similarly, when a target video is played using a terminal device, the target video contains not only a target person but also a target background at the same time. At this time, the user inputs a trigger command (e.g., a target gesture command, a target voice command, a target facial expression command, a target character command, a target body command, etc.) into the terminal device to trigger a target effect, thereby fixing a target frame image of the target video currently being shot.

[0061] The trigger command may be a trigger command input by a user, may be a trigger command issued from a target object in a video, or may include a trigger command issued by a terminal under a preset condition (such as a condition that the video has been played up to a preset specific point). The above description of a target person as a target object is merely an example given for ease of understanding, and in this embodiment, the target object may take the form of a target person, target animal, target building, etc., and is not specifically limited here. The above trigger command may be a target gesture command, a target voice command, a target facial expression command, Target text command and The form may be a target body command, etc., and is not specifically limited here.

[0062] In step 205, the first special effect sequence frame is played with the complementary frame image as a background, and the first special effect sequence frame is used to dynamically display the special effect particles according to a preset path.

[0063] After generating the complementary frame image, the first special effect sequence frame can be played with the complementary frame image as the background, and the first special effect sequence frame is used to dynamically display the special effect particles according to a preset path. By making the target frame image show the special effect particle effect, the atmosphere can be enriched, the effect of the target object can be shown more naturally, and the dynamic effect can be stronger.

[0064] In addition, based on the above embodiment, after generating the complementary frame image, the background complement operation can be continuously performed on the target area in the subsequent frame images of the target video, and the subsequent frame images are located after the complementary frame image in the target video. By continuously performing the complement operation on the target area in the subsequent frame images of the target video, the effect of the target object continuously disappearing in the target video can be achieved. An example can be taken of the target person being the target object. After removing the target person in the target frame image, the complement operation can be performed on the target area, i.e., the resulting empty area. Then, the user continues to shoot the target person, and continuously removes the target person in the subsequent frame images of the target video while continuously performing background complement, so that the effect of the target person continuously disappearing in the target video, i.e., the visual effect of the target person being hidden in the target video can be achieved.

[0065] 8 is a structural schematic diagram of a video processing device shown according to an exemplary embodiment of the present disclosure. As shown in FIG. 8, the video processing device 300 provided by this embodiment includes: an image capture module 301 used to capture a target frame image of a target video in response to a trigger command; In the target frame image Eyes and an image processing module 302 used to remove a target object and perform a complementing operation on a target area in the target frame image to generate and display a complemented frame image, the target area including a vacant area remaining after the target object is removed from the target frame image.

[0066] According to one or more embodiments of the present disclosure, the image processing module 302 is further used for playing a first special effect sequence frame with the complementary frame image as a background, and the first special effect sequence frame is used for dynamically displaying special effect particles according to a preset path.

[0067] According to one or more embodiments of the present disclosure, the image processing module 302 is further used to display a second pre-set special effect used to make the target frame image show a visual blur effect.

[0068] According to one or more embodiments of the present disclosure, the image processing module 302 is further used to display a third pre-set special effect used to make the target frame image show a visual shaking effect.

[0069] According to one or more embodiments of the present disclosure, the image processing module 302 is further used to continuously perform an interpolation operation on the target region in a subsequent frame image of the target video, the subsequent frame image being located after the interpolated frame image in the target video.

[0070] According to one or more embodiments of the present disclosure, the image processing module 302 is further used to determine the target object as a target type object.

[0071] According to one or more embodiments of the present disclosure, the trigger command includes at least one of a target gesture command, a target voice command, a target facial expression command, a target body command, and a target text command.

[0072] According to one or more embodiments of the present disclosure, the image processing module 302 includes: Recognizing each pixel point in the target frame image using a preset object segmentation model, and generating a target binary image having the same size as the target frame image; and A target region of the target object in the target frame image is determined based on the target binary image, the target region in the target binary image including pixel points whose pixel values ​​are target values.

[0073] According to one or more embodiments of the present disclosure, the image processing module 302 includes: Fusing the target binary image and the target frame image to obtain a model input image; and Inputting the model input image into an image inpainting model; generating a processed frame image; and replacing the target area in the target frame image with the target area in the processed frame image; The complementary frame image is generated.

[0074] According to one or more embodiments of the present disclosure, the image restoration model is located on a terminal device, the terminal device comprising: Process the target video based on the image inpainting model It is used for this purpose.

[0075] According to one or more embodiments of the present disclosure, the image processing module 302 specifically includes: inputting the model input image into a first image inpainting model to generate a first complement image; performing pixel truncation on the first complement image using a preset pixel threshold to generate a second complement image; The second complemented image is input to a second image restoration model to generate a third complemented image, and the complementation accuracy of the second image restoration model is higher than that of the first image restoration model; and performing pixel truncation on the third complementary image using the preset pixel threshold to generate a fourth complementary image, and the processed frame image includes the fourth complementary image.

[0076] It should be noted that the video processing device provided by the embodiment shown in FIG. 8 can be used to perform the method steps provided by any of the above method embodiments, and the specific implementation forms and technical effects are similar, so they will not be repeatedly described here.

[0077] FIG. 9 is a structural schematic diagram of an electronic device shown according to one exemplary embodiment of the present disclosure. As shown in FIG. 9, FIG. 9 shows a structural schematic diagram of an electronic device 400 suitable for implementing an embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include, but is not limited to, a mobile terminal with an image acquisition function, such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Portable Android Device, tablet computer), a PMP (Portable Media Player, portable multimedia player), an in-vehicle terminal (e.g., an in-vehicle navigation terminal), and a fixed terminal with an external image acquisition device, such as a digital TV (Television), a desktop computer, etc. The electronic device shown in FIG. 9 is merely an example and should not impose any limitations on the function and scope of use of the embodiment of the present disclosure.

[0078] 9, the electronic device 400 may include a processor (such as a central processing unit or a graphic processor) 401, which may perform various appropriate operations and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a memory 408 into a random access memory (RAM) 403. The RAM 403 also stores various programs and data necessary for the operation of the electronic device 400. The processor 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404. Memory is used to store a program for executing the video processing method described in each of the above method embodiments, and the processor is configured to execute the program stored in the memory.

[0079] Typically, input devices 406 including a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc., output devices 407 including a liquid crystal display (LCD), speaker, vibrator, etc., memory 408 including a magnetic tape, hard disk, etc., and communication devices 409 can be connected to the I / O interface 405. The communication devices 409 can enable the electronic device 400 to communicate and exchange data with other devices wirelessly or via wires. Although FIG. 9 shows the electronic device 400 with various devices, it should be understood that not all of the devices shown need be implemented or deployed. Alternatively, more or fewer devices can be implemented or deployed.

[0080] In particular, according to an embodiment of the present disclosure, the above processes described with reference to the flowcharts can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer readable storage medium including a computer program embodied in a non-transitory computer readable medium, the computer program including program code for performing the video processing method shown in the flowchart of the embodiment of the present disclosure. In such an embodiment, the computer program can be downloaded and installed from a network via the communication device 409, or installed from the memory 408, or installed from the ROM 402. When the computer program is executed by the processor 401, the above video processing functions defined in the method according to the embodiment of the present disclosure are performed.

[0081] It should be noted that the computer readable medium described in this disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of computer readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic memory component, or any suitable combination of the above. In this disclosure, a computer readable storage medium may be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with computer-readable program code embedded therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may transmit, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained in the computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, fiber optic cable, RF (Radio Frequency), etc., or any suitable combination of the above.

[0082] The computer-readable medium may be included in the electronic device, or may exist independently without being assembled in the electronic device.

[0083] The above-mentioned computer-readable storage medium is equipped with one or more programs, and when the one or more programs are executed by the electronic device, the electronic device responds to a trigger command, fixes a target frame image of a target video, removes a target object in the target frame image, and performs a complement operation on a target area in the target frame image, and generates and displays a complement frame image, where the target area is the vacant area after the target object is removed from the target frame image.

[0084] Computer program code for carrying out the operations of the present disclosure can be written in one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages, or a combination thereof. The program code can run entirely on the user's computer, partly on the user's computer, as a standalone software package, partly on the user's computer, partly on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any kind of network, such as a Local Area Network (LAN) or a Wide Area Network (WAN), or can be connected to an external computer (e.g., connecting via the Internet using an Internet Service Provider).

[0085] In some embodiments, the client terminals and servers may communicate using any conventional or future developed network protocol, such as HTTP (HyperText TrAnsfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), an internetwork (e.g., the Internet), an end-to-end network (e.g., an Ad hoc end-to-end network), and any conventional or future developed network.

[0086] The flowcharts and block diagrams in the drawings illustrate the architecture, functions, and operations that can be realized by the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, program segment, or part of code that includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions shown in the blocks may be executed in a different order than that shown in the figures. For example, two blocks shown as connected may actually be executed in parallel, or the blocks may be executed in the reverse order depending on the related functionality. It should be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or may be implemented using a combination of dedicated hardware and computer instructions.

[0087] The modules described in the embodiments of the present disclosure can be implemented in software or hardware. The names of the modules may differ depending on the situation. Module may not be intended to limit the R .

[0088] The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that may be used may include, but are not limited to, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), System-on-a-chips (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

[0089] In the context of this disclosure, a machine-readable medium may be a tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of machine-readable storage media may include electrical connections based on one or more wires, portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic memory components, or any suitable combination of the above.

[0090] In a first aspect, according to one or more embodiments of the present disclosure, there is provided a video processing method, the method comprising: freeze a target frame image of the target video in response to a trigger command; In the target frame image Eyes The method includes a step of removing the target object and performing a complementation operation on a target area in the target frame image to generate and display a complemented frame image, the target area including a vacant area remaining after the target object is removed from the target frame image.

[0091] According to one or more embodiments of the present disclosure, after the step of generating and displaying a complementary frame image, the method further comprises: The method further includes playing a first special effect sequence frame with the complementary frame image as a background, the first special effect sequence frame being used to dynamically display special effect particles according to a preset path.

[0092] According to one or more embodiments of the present disclosure, after the step of fixing a target frame image of the target video, the method includes: The method further includes the step of displaying a preset second special effect used to present a visual blur effect on the target frame image.

[0093] According to one or more embodiments of the present disclosure, after the step of fixing a target frame image of the target video, the method includes: The method further includes the step of displaying a third preset special effect used for making the target frame image show a visual shaking effect.

[0094] According to one or more embodiments of the present disclosure, after the step of generating and displaying a complementary frame image, the method further comprises: The method further includes the step of continuously performing an interpolation operation on the target region in a subsequent frame image of the target video, the subsequent frame image being located after the interpolated frame image in the target video.

[0095] According to one or more embodiments of the present disclosure, prior to the step of obtaining the trigger command, the method further comprises: The method further includes determining the goal object as a goal type object.

[0096] According to one or more embodiments of the present disclosure, the trigger command includes at least one of a target gesture command, a target voice command, a target facial expression command, a target body command, and a target written command.

[0097] According to one or more embodiments of the present disclosure, removing the target object in the target frame image and performing an interpolation operation on the target area in the target frame image. teeth , Recognizing each pixel point in the target frame image using a preset object segmentation model, and generating a target binary image having the same size as the target frame image; determining a target region of the target object in the target frame image based on the target binary image, the target region in the target binary image including pixel points whose pixel values ​​are target values; Including nothing.

[0098] According to one or more embodiments of the present disclosure, for the target region in the target frame image, Complementary Operations The step of: fusing the target binary image and the target frame image to obtain a model input image; inputting the model input image into an image inpainting model to generate a processed frame image; and replacing the target area in the target frame image with the target area in the processed frame image to generate the complementary frame image.

[0099] According to one or more embodiments of the present disclosure, the image restoration model is located on a terminal device, the terminal device comprising: Process the target video based on the image inpainting model It is used for this purpose.

[0100] According to one or more embodiments of the present disclosure, the step of inputting the model input image into an image inpainting model to generate a processed frame image includes: inputting the model input image into a first image inpainting model to generate a first complement image; performing pixel truncation on the first complement image using a preset pixel threshold to generate a second complement image; inputting the second complemented image into a second image restoration model to generate a third complemented image, where the complementation accuracy of the second image restoration model is higher than that of the first image restoration model; and performing pixel truncation on the third complementary image using the preset pixel threshold to generate a fourth complementary image, the processed frame image including the fourth complementary image.

[0101] In a second aspect, according to one or more embodiments of the present disclosure, there is provided a video processing device, comprising: Responds to trigger commands ,eye an image capture module used to fix a target frame image of a target video; In the target frame image Eyes The target object is removed, and the target frame image is Eyes and an image processing module used to perform a complementation operation on a target area and generate and display a complemented frame image, the target area including a vacant area remaining after the target object is removed from the target frame image.

[0102] According to one or more embodiments of the present disclosure, the image processing module is further used for playing a first special effect sequence frame with the complementary frame image as a background, and the first special effect sequence frame is used for dynamically displaying special effect particles according to a preset path.

[0103] According to one or more embodiments of the present disclosure, the image processing module is further used to display a second pre-set special effect used to make the target frame image show a visual blur effect.

[0104] According to one or more embodiments of the present disclosure, the image processing module is further used to display a third pre-set special effect used to make the target frame image show a visual shaking effect.

[0105] According to one or more embodiments of the present disclosure, the image processing module is further used to continuously perform a complementation operation on the target region in a subsequent frame image of the target video, the subsequent frame image being located after the complemented frame image in the target video.

[0106] According to one or more embodiments of the present disclosure, the image processing module is further adapted to determine the target object as a target type object.

[0107] According to one or more embodiments of the present disclosure, the trigger command includes at least one of a target gesture command, a target voice command, a target facial expression command, a target body command, and a target written command.

[0108] According to one or more embodiments of the present disclosure, the image processing module specifically includes: Recognizing each pixel point in the target frame image using a preset object segmentation model, and generating a target binary image having the same size as the target frame image; and The method is used to determine the target region of the target object in the target frame image based on the target binary image, the target region in the target binary image including each pixel point whose pixel value is a target value.

[0109] According to one or more embodiments of the present disclosure, the image processing module specifically includes: fusing the target binary image and the target frame image to obtain a model input image; and Inputting the model input image into an image inpainting model; generating a processed frame image; and replacing the target area in the target frame image with the target area in the processed frame image; The complementary frame image is generated.

[0110] According to one or more embodiments of the present disclosure, the image restoration model is located on a terminal device, the terminal device comprising: Process the target video based on the image inpainting model It is used for this purpose.

[0111] According to one or more embodiments of the present disclosure, the image processing module specifically includes: inputting the model input image into a first image inpainting model to generate a first complement image; performing pixel truncation on the first complement image using a preset pixel threshold to generate a second complement image; The second complemented image is input to a second image restoration model to generate a third complemented image, and the complementation accuracy of the second image restoration model is higher than that of the first image restoration model; and performing pixel truncation on the third complementary image using the preset pixel threshold to generate a fourth complementary image, and the processed frame image includes the fourth complementary image.

[0112] In a third aspect, an embodiment of the present disclosure provides an electronic device, the electronic device comprising: A processor; A memory used to store a computer program for the processor; a display adapted to display video processed by said processor; The processor is configured to execute the computer program to realize the video processing method as described in the first aspect above and various possible designs of the first aspect.

[0113] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon, the computer-readable storage medium being operable, when executed by a processor, to realize a video processing method as described in the first aspect above and various possible designs of the first aspect.

[0114] In a fifth aspect, an embodiment of the present disclosure provides a computer program product including a computer program, which, when executed by a processing device, performs the steps of the method described in any embodiment of the present disclosure.

[0115] The above description is merely a description of some preferred embodiments of the present disclosure and a description of the applied technical principles.Those skilled in the art should understand that the scope of the invention according to the embodiments of the present disclosure is not limited to the solution formed by a specific combination of the above technical features, but should also cover other solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept, for example, by replacing the above features with technical features having similar functions (but not limited to) disclosed in the present disclosure.

[0116] It should be noted that although operations are described in a particular order, this should not be understood as requiring that these operations be performed in the particular order or sequence shown. Multitasking or parallel processing may be advantageous in certain circumstances. Similarly, although the above description includes some specific implementation details, these should not be construed as limiting the scope of the disclosure. Certain features that are described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments, either alone or in any suitable subcombination.

[0117] Although the present subject matter has been described using language specific to structural features and / or methodological logical operations, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or operations described above. Rather, the specific features and operations described above are merely example forms for implementing the claims.

[0118] This application claims priority to a Chinese patent application filed with the China Intellectual Property Office on November 16, 2020, bearing application number 202011280804.3 and entitled "Video Processing Method, Apparatus, Electronic Device and Storage Medium," the entire contents of which are incorporated herein by reference.

Claims

1. 1. A video processing method comprising the steps of: freeze a target frame image of the target video in response to a trigger command; removing the target object from the target frame image and performing a complementing operation on a target area in the target frame image to generate and display a complemented frame image, the target area including a vacant area remaining after the target object is removed from the target frame image; The step of removing the target object in the target frame image and performing an interpolation operation on the target area in the target frame image includes: Recognizing each pixel point in the target frame image using a preset object segmentation model, and generating a target binary image having the same size as the target frame image; determining a target region of the target object in the target frame image based on the target binary image, the target region in the target binary image including pixel points whose pixel values ​​are target values; fusing the target binary image and the target frame image to obtain a model input image; inputting the model input image into a first image restoration model, and completing the model input image based on a completion accuracy of the first image restoration model to generate a first completion image; performing pixel truncation on the first complement image using a preset pixel threshold to generate a second complement image; inputting the second complemented image into a second image restoration model, and complementing the second complemented image based on a complementation accuracy of the second image restoration model to generate a third complemented image, wherein the complementation accuracy of the second image restoration model is higher than that of the first image restoration model; performing pixel truncation on the third complementary image using the preset pixel threshold to generate a fourth complementary image, the processed frame image including the fourth complementary image; and replacing the target area in the target frame image with the target area in the processed frame image to generate the complementary frame image.

13. A video processing method comprising:

2. After the step of generating and displaying the complementary frame image, 2. The video processing method of claim 1, further comprising the step of playing a first special effect sequence frame with the complementary frame image as a background, wherein the first special effect sequence frame is used to dynamically display special effect particles according to a preset path.

3. After the step of fixing the target frame image of the target video, 2. The video processing method according to claim 1, further comprising the step of displaying a second preset special effect used to present a visual blur effect in the target frame image.

4. After the step of fixing the target frame image of the target video, 2. The video processing method according to claim 1, further comprising the step of displaying a third preset special effect used to make the target frame image show a visual shaking effect.

5. After the step of generating and displaying the complementary frame image, 2. The video processing method of claim 1, further comprising the step of continuously performing an interpolation operation on the target region in a subsequent frame image of the target video, the subsequent frame image being located after the interpolated frame image in the target video.

6. Prior to the step of obtaining the trigger command, Video processing method according to any one of claims 1 to 5, further comprising the step of determining said goal object as a goal type object.

7. The video processing method according to any one of claims 1 to 5, characterized in that the trigger command comprises at least one of a target gesture command, a target voice command, a target facial expression command, a target body command, and a target text command.

8. The video processing method of claim 1 , wherein the image inpainting model is located in a terminal device, and the terminal device processes the target video based on the image inpainting model.

9. 1. A video processing device comprising: an image capture module adapted to capture a target frame image of a target video in response to a trigger command; an image processing module used to remove a target object in the target frame image and perform a complementing operation on a target area in the target frame image to generate and display a complemented frame image, the target area including a vacant area remaining after the target object is removed from the target frame image; The image processing module includes: Recognizing each pixel point in the target frame image using a preset object segmentation model, and generating a target binary image having the same size as the target frame image; determining a target region of the target object in the target frame image based on the target binary image, the target region in the target binary image including pixel points whose pixel values ​​are target values; fusing the target binary image and the target frame image to obtain a model input image; inputting the model input image into a first image restoration model to complement the model input image based on a complementation accuracy of the first image restoration model to generate a first complemented image; performing pixel truncation on the first complemented image using a preset pixel threshold to generate a second complemented image; inputting the second complemented image into a second image restoration model, and complementing the second complemented image based on a complementation accuracy of the second image restoration model to generate a third complemented image, wherein the complementation accuracy of the second image restoration model is higher than that of the first image restoration model; performing pixel truncation on the third complementary image using the preset pixel threshold to generate a fourth complementary image, the processed frame image including the fourth complementary image; replacing the target area in the target frame image with the target area in the processed frame image to generate the complementary frame image.

13. A video processing device comprising:

10. An electronic device, A processor; A memory used to store computer programs; a display adapted to display video processed by said processor; 9. An electronic device, characterized in that the processor is configured to implement the video processing method according to any one of claims 1 to 8 by executing the computer program.

11. A computer-readable storage medium having computer-executable instructions stored thereon, the computer-readable storage medium causing a video processing method according to any one of claims 1 to 8 to be realized when a processor executes the computer-executable instructions.

12. A computer program, characterized in that, when said computer program is executed by a processing device, the video processing method according to any one of claims 1 to 8 is implemented.

Citation Information

Patent Citations

  • Image processor, image processing method, imaging apparatus, and imaging method

    JP2008005084A

  • Program, information storage medium, terminal, and network system

    JP2011096018A

  • Image processing apparatus, image processing method, and program

    JP2013077873A

  • Method for realtime diminishing of moving object in moving image during photographing of moving image, moving image photographing apparatus for the same, and program for mentioned moving image photographing apparatus

    JP2014096661A

  • Program, information processing method, and information processing terminal

    JP2020129356A

Cited By

  • Video processing method and apparatus for triggering special effect, electronic device and storage medium

    US12425529B2