Creating videos using virtual characters

Abstract

This disclosure describes techniques for creating a video using a virtual character. Creation of the video may be initiated by a user. A camera input may be received, the camera input including a human body of a user. The camera input may be split into a first stream for removing a human body and a second stream for making an animation of a virtual character in the video. A repair filter may be applied to remove the human body from the camera input in real time. The repair filter may be configured to accelerate texture sampling. The output of the repair filter may be mixed with an image included in the camera input to generate a camera input background.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Application No. 17 / 751,222, filed on May 23, 2022, and entitled “CREATION OF VIDEOS USING VIRTUAL CHARACTERS,” the disclosure of which is incorporated herein by reference in its entirety. Background Art

[0003] Internet-based tools are increasingly used for communication. Internet-based tools can be any software or platform. Existing social media platforms enable users to communicate with each other by sharing images, videos and other information via static applications or web pages. As communication devices such as mobile phones become more and more advanced, people then expect new entertainment, social networking and communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The following detailed description may be better understood when read in conjunction with the accompanying drawings.For illustrative purposes, example embodiments of various aspects of the disclosure are shown in the drawings; however, the invention is not limited to the specific methods and instrumentalities disclosed.

[0005] Figure 1 An example system for distributing videos using virtual characters according to the present disclosure is shown.

[0006] Figure 2 An example diagram illustrating an inpainting filter applied to an image according to the present disclosure is shown.

[0007] Figure 3 An example diagram illustrating a ping-pong operation in accordance with the present disclosure is shown.

[0008] Figure 4 An example method for creating a video using a virtual character is shown, which may be performed by a user device according to the present disclosure.

[0009] Figure 5 Another example method for creating a video using a virtual character is shown, which may be performed by a user device according to the present disclosure.

[0010] Figure 6 Another example method for creating a video using a virtual character is shown, which may be performed by a user device according to the present disclosure.

[0011] Figure 7Another example method for creating a video using a virtual character is shown, which may be performed by a user device according to the present disclosure.

[0012] Figure 8 Another example method for creating a video using a virtual character is shown, which may be performed by a user device according to the present disclosure.

[0013] Fig. 9 Another example method for creating a video using a virtual character is shown, which may be performed by a user device according to the present disclosure.

[0014] Fig.10 An example computing device is shown that can be used to perform any of the techniques disclosed herein. DETAILED DESCRIPTION

[0015] A virtual character (e.g., a digital person, an avatar, a simulated character, etc.) is a simulation of a person on a computer. Virtual characters are becoming increasingly popular in today's digital ecosystem. For example, many movies, television shows, simulations, games, virtual worlds, and industries utilize virtual characters in some way. A virtual character can simulate the movement of a real person. For example, a person can shoot a video of himself swinging at a camera. If a video is created using a virtual character, the virtual character can replace one or more body parts of the person in the final video. For example, the final video may feature a person with a virtual character whose hand swings at the camera. The person's hand may no longer be visible. As another example, the final video may feature a person with a virtual character whose head swings at the camera. The person's head may no longer be visible. However, it may be difficult to smoothly remove (multiple) human body parts from a frame of a video in real time so that (multiple) human body parts are no longer visible. This may be particularly difficult when the virtual character is smaller than or does not completely occlude the human body part it replaces. Therefore, improved techniques for video creation using virtual characters are desirable.

[0016] Improved techniques for video creation using virtual characters are described herein. The improved techniques for video creation using virtual characters described herein can be used by, for example, Figure 1 Systems such as the system 100 shown utilize. Figure 1 An example system 100 for distributing content is illustrated. The system 100 may include a server 102 and a plurality of client devices 104a-104d. The server 102 and the plurality of client devices 104a-104d may communicate with each other via one or more networks 132.

[0017] The servers 102 may be located at a data center, such as a single site, or distributed in different geographic locations (e.g., at several sites). The servers 102 may provide services via one or more networks 132. The networks 132 include various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, etc. The networks 132 may include physical links, such as coaxial cable links, twisted pair cable links, optical fiber links, combinations thereof, etc. The networks 132 may include wireless links, such as cellular links, satellite links, Wi-Fi links, etc.

[0018] The server 102 may include multiple computing nodes that host various services. In an embodiment, the node hosts a video service 112. The video service 112 may include a content streaming service, such as an Internet protocol video streaming service. The video service 112 may be configured to distribute content 132 via various transmission technologies. The video service 112 is configured to provide content 132, such as video, audio, text data, combinations thereof, etc. The content 132 may include content streams (e.g., video streams, audio streams, information streams), content files (e.g., video files, audio files, text files), and / or other data. The content 132 may be stored in a database 122. For example, the video service 112 may include a video sharing service, a video hosting platform, a content distribution platform, a collaborative gaming platform, etc.

[0019] In an embodiment, the content 132 distributed or provided by the video service 112 includes a video. For example, the video can be a short video. The short video can have a duration less than or equal to a predetermined time limit, such as one minute, five minutes, or other predetermined minutes. By way of example and not limitation, the short video can include at least one but no more than four 15-second segments strung together. The short duration of the video can provide a quick burst of entertainment for the audience, allowing the user to watch a large amount of video in a short time frame. This quick burst of entertainment may become popular on social media platforms.

[0020] The video may include a pre-recorded audio overlay, such as a pre-recorded song clip or audio from a television program or movie. If the video includes a pre-recorded audio overlay, the video may be characterized by one or more individuals lip-syncing, dancing, or otherwise moving their bodies with the pre-recorded audio. For example, the video may be characterized by an individual completing a "dance challenge" of a popular song, or the video may be characterized by two individuals participating in lip-syncing or a couple dance. As another example, the video may be characterized by an individual completing a challenge that requires them to move his or her body in a manner corresponding to the pre-recorded audio overlay, such as in a manner corresponding to the beat or rhythm of a pre-recorded song characterized by the pre-recorded audio overlay. Other videos may not include a pre-recorded audio overlay. For example, these videos may be characterized by an individual doing sports, playing a prank, or giving advice, such as beauty and fashion advice, cooking tips, or home improvement tips.

[0021] In an embodiment, content 132 may be output to different client devices 104 via network 132. Content 132 may be streamed to client devices 104. The content stream may be a video stream received from video service 112. Multiple client devices 104 may be configured to access content 132 from video service 112. In an embodiment, client device 104 may include content application 106. Content application 106 outputs (e.g., displays, renders, presents) content 132 to a user associated with client device 104. Content may include video, audio, commentary, text data, etc.

[0022] The plurality of client devices 104 may include any type of computing device, such as a mobile device, a tablet device, a laptop computer, a desktop computer, a smart TV or other smart device (e.g., a smart watch, a smart speaker, smart glasses, a smart helmet), a gaming device, a set-top box, a digital streaming device, a robot, etc. The plurality of client devices 104 may be associated with one or more users. A single user may access the server 102 using one or more of the plurality of client devices 104. The plurality of client devices 104 may be moved to various locations and use different networks to access the server 102.

[0023] The video service 112 may be configured to receive input from a user. A user may be registered as a user of the video service 112 and may be a user of the content application 106 operating on the client device 104. The user input may include a video created by the user, a user comment associated with the video, or a "like" associated with the video. The user input may include a connection request and user input data, such as text data, digital image data, or user content. The connection request may include a request from a client device 104a to 104d to connect to the video service 112. The user input data may include information that the user connected to the video service 112 wants to share with other connected users of the video service 112, such as videos and / or user comments.

[0024] The video service 112 can receive different types of input from users using different types of client devices 104. For example, a user using a content application 106 on a first user device (such as a mobile phone or tablet computer) can use the content application 106 to create and upload videos. Users using the content application 106 on different mobile phones or tablet computers can also view, comment on, or "like" videos or comments written by other users. In another example, a user using the content application 106 on a smart TV, laptop computer, desktop computer, or gaming device cannot create and upload videos or comment on videos using the content application 106. Instead, a user using the content application 106 on a smart TV, laptop computer, desktop computer, or gaming device may only be able to use the content application 106 to view videos, view comments left by other users, and "like" videos.

[0025] In an embodiment, a user may use a content application 106 on a client device 104 to create a video and upload the video to a server 102. The client device 104 may access interfaces 108a to 108n of the content application 106. The interface 108 may include an input element. For example, the input element may be configured to allow a user to create a video. To create a video, a user may allow the content application 106 to access an image capture device, such as a camera or microphone of the client device 104. Using the content application 106, the user may select the duration of the video or set the speed of the video, such as "slow motion" or "speed up".

[0026] The user can edit the video using the content application 106. The user can add one or more text, filters, sounds or effects, such as beauty effects, to the video. To add a pre-recorded audio overlay to the video, the user can select a song or sound clip from the sound library of the content application 106. The sound library can include different songs, sound effects or audio clips from movies, albums, and television shows. In addition to or in lieu of adding a pre-recorded audio overlay to the video, the user can add a voice-over to the video using the content application 106. The voice-over can be a sound recorded by the user using the microphone of the client device 104. The user can add a text overlay to the video, and can use the content application 106 to specify when they want the text overlay to appear in the video. The user can assign subtitles, location annotations, and one or more tags to the video to indicate the subject of the video. The content application 106 can prompt the user to select a frame of the video to be used as a "cover image" for the video.

[0027] After the user has created the video, the user may upload the video to the server 102 using the content application 106 and / or save the video locally to the user device 104. When the user uploads the video to the server 102, they may choose whether they want the video to be viewable by all other users of the content application 106 or only by a subset of users of the content application 106. The video service 112 may store the uploaded video and any metadata associated with the video in one or more databases 122.

[0028] In an embodiment, a user may provide input on a video using a content application 106 on a client device 104. The client device 104 may access an interface 108 of the content application 106 that allows the user to provide input associated with the video. The interface 106 may include an input element. For example, the input element may be configured to receive input from the user, such as a comment or "like" associated with a particular video. If the input is a comment, the content application 106 may allow the user to set an emoticon associated with his or her input. The content application 106 may determine timing information for the input, such as when the user writes a comment. The content application 106 may send the input and associated metadata to the server 102. For example, the content application 106 may send a comment, an identifier of the user who wrote the comment, and timing information for the comment to the server 102. The video service 112 may store the input and associated metadata in a database 122.

[0029] Video service 112 can be configured to output uploaded videos and user inputs to other users. A user can be registered as a user of video service 112 to view videos created by other users. A user can be a user of content application 106 operated on client device 104. Content application 106 can output (display, render, present) videos and user comments to users associated with client device 104. Client device 104 can access interface 108 of content application 106. Interface 108 can include output elements. Output elements can be configured to display information about different videos so that users can select videos to be viewed. For example, output elements can be configured to display multiple cover images, subtitles or tags associated with a video. Output elements can also be configured to arrange videos according to categories associated with each video.

[0030] In an embodiment, user comments associated with a video may be output to other users who are watching the same video. For example, all users who access a video may view comments associated with the video. Video service 112 may output video and associated comments simultaneously. Comments may be output by video service 112 in real time or near real time. Content application 106 may display video and comments in various ways on client device 104. For example, comments may be displayed in an overlay above the content or in an overlay next to the content. As another example, a user who wants to view comments of other users associated with a video may need to select a button to view comments. Comments may be animated when displayed. For example, comments may be shown as scrolling on the video or scrolling on the overlay.

[0031] Multiple computing nodes can process tasks associated with video service 112. Multiple computing nodes can be implemented as one or more computing devices, one or more processors, one or more virtual computing instances, combinations thereof, and the like. Multiple computing nodes can be implemented by one or more computing devices. One or more computing devices may include virtualized computing instances. Virtualized computing instances may include virtual machines, such as simulations of computer systems, operating systems, servers, and the like. Virtual machines can be loaded by computing devices based on virtual images and / or other data defining specific software (e.g., operating systems, dedicated applications, servers) for simulation. As the demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on one or more computing devices. A hypervisor can be implemented to manage the use of different virtual machines on the same computing device.

[0032] As discussed above, a user may create a video using a content application 106 on a client device 104 and upload the video to a server 102. In an embodiment, a video created by a user via a content application 106 on a client device 104 may be a video featuring a virtual character. A virtual character may replace one or more body parts of a user. For example, a video created by a user may feature a virtual head replacing the user's head. As another example, a video created by a user may feature virtual hands, legs, arms, etc. replacing the user's hands, feet, arms, etc. Virtual body parts may be similar to body parts of animals, monsters, cartoon characters, etc. In some embodiments, a virtual character may replace an entire person in a video. For example, if the virtual character is a cartoon cat, the cartoon cat may replace an entire person in a video, so that the video is characterized by the cartoon cat, rather than any human body part.

[0033] In an embodiment, at least one of the video service 112 or the client device 104 includes a virtual character model 117. The virtual character model 117 can be used to remove one or more human body parts from the camera input so that the human body part(s) are not shown in the video. For example, the virtual character model 117 can be used to remove one or more human body parts from the camera input and replace the one or more human body parts with a virtual character so that only the virtual character is shown in the created video without showing the human body part(s).

[0034] In an embodiment, the virtual character model 117 may be configured to receive a camera input including a user's human body. The user may be a user of the video service 112. For example, the user may be a user associated with the client device 104. The user may initiate the creation of a video, such as via the content application 106. The user may indicate that he or she wants the video to feature a virtual character. For example, the user may select a filter or effect associated with the virtual character. If the user indicates that he or she wants the video to feature a virtual character, the virtual character model 117 may be configured to split the received camera input into a first stream for removing one or more human body parts in the video and a second stream for animating the virtual character in the video.

[0035] The virtual character model 117 can be configured to apply a repair filter to the first stream. Applying the repair filter to the first camera input stream can cause one or more human body parts to be removed, such as in real time based on the camera input (i.e., while the video is being captured). The repair filter is configured to accelerate texture sampling. For example, the repair filter is configured to accelerate texture sampling by saving downsampling and using a pyramid ping-pong operation to accelerate upsampling, such as described above with respect to Figure 3The pyramid ping-pong operation described in more detail. The inpainting filter is configured to perform multiple levels of sampling by doubling the resolution of the sampling of the previous level and performing a ping-pong operation on the sampling of each level of the multiple levels of sampling. Implementing the inpainting filter may also include generating a lookup table that includes a pre-computed Hammersley point set.

[0036] Figure 2 An example diagram 200 illustrating an inpainting filter applied to an image (e.g., an image included in a camera input) according to the present disclosure is shown. Applying the inpainting filter to a first camera input stream may, for example, include using an input mask to mask a human body part in an image 204 included in the first camera input stream. For example, to mask the human body part in the image 204, an input mask 202 may be applied to the image 204. The mask 202 may be configured to mask features of a user's head in the image 204 to create a masked image 206. However, in some embodiments, it should be understood that the mask may additionally or alternatively mask other human body parts or the entire human body.

[0037] Applying the inpainting filter to the masked image 206 can naturally remove the masked head from the image 204. For example, as shown in image 208, after applying the inpainting filter, the user's masked head disappears. The inpainting filter is configured to accelerate texture sampling, thereby enabling real-time removal of human body parts from the camera input stream. The inpainting filter according to the present disclosure does not perform downsampling of textures. The inpainting filter uses a pyramid ping-pong operation to accelerate upsampling.

[0038] The inpainting filter may perform multiple levels of sampling based on the mask image 206. Performing multiple levels of sampling based on the mask image 206 may include performing an initial level of sampling at a low resolution. For example, performing the initial level of sampling may include performing an initial sampling of the original texture of the mask image 206 at a resolution of 16x16. Performing the initial level of sampling may also include performing multiple iterations by performing a ping-pong operation.

[0039] Performing multiple levels of sampling may also include upsampling the texture generated by the sampling of the previous level. For example, the higher resolution may be twice the resolution of the sampling of the previous level. In an embodiment, performing multiple levels of sampling may include performing multiple iterations with the sampling of the current level by performing a ping-pong operation. The ping-pong operation minimizes the read / write of the same texture. Figure 3 The ping-pong operation is discussed in more detail.

[0040] Figure 3An example diagram 300 illustrating a pyramid ping-pong operation utilized by an inpainting filter according to the present disclosure is shown. The inpainting filter according to the present disclosure accelerates texture sampling by saving downsampling. The inpainting filter also uses a pyramid ping-pong operation to accelerate upsampling. For example, the pyramid ping-pong operation includes performing multiple levels of sampling by doubling the resolution of the previous level of sampling and performing a ping-pong operation with each level of sampling in the multiple levels of sampling. Utilizing the ping-pong operation can prevent a large amount of textures from being copied in a graphics processing unit (GPU), and can reduce the cost of a rendering pipeline.

[0041] The pyramid ping-pong operation can include multiple levels of texture sampling. In the example, the graph 300 includes four levels of texture sampling: level 0 (level zero), level 1 (level one), level 2 (level two), and level 3 (level three). Each level can include multiple iterations by performing a ping-pong operation. The ping-pong operation can include a ping portion 302 and a pong portion 304. Each level can be associated with a different resolution, and the resolution doubles when each level increases. For example, level 0 can be associated with a 16x16 resolution, level 1 can be associated with a 32x32 resolution, level 2 can be associated with a 64x64 resolution, level 3 can be associated with a 128x128 resolution, and so on.

[0042] The resolution associated with the last level can be the power of 2 closest to the resolution of the input masking texture (e.g., 512x512 for 720p and 540p, 256x256 for 360p, etc.). The resolution associated with the first level (e.g., level 0) can be half the resolution of the next level (e.g., level 1). The smallest resolution associated with level 0 can be 16x16 to limit the impact of the average value of the entire image.

[0043] In an embodiment, the number of levels may be determined by the input resolution. For example, if the input resolution is 720p, then the levels of the pyramid (from level 0 upwards) may be: 16x16, 32x32, 64x64, 128x128, 256x256, 512x512. In another example, if the input resolution is changed to 360p, then the levels of the pyramid (from level 0 upwards) may be: 16x16, 32x32, 64x64, 128x128, 256x256.

[0044] Each level can perform an iteration of the Laplacian kernel to converge the masked region. The number of iterations can be determined, for example, by the quality setting. A low quality setting has fewer iterations to converge the initial level and copy the result to the next level using interpolation. A high quality setting has as many iterations as required to converge the initial level and perform smoothing at the next level to obtain better results near the region boundaries. For example, the number of iterations for a low quality setting can be 5, while the number of iterations for a high quality setting can be 11. When executed, the initial level can be completely converged in the masked region. Each current level can then sample the previous level (e.g., level 1 will sample from level 0, and so on) as a base fill in the first run, and then each level can perform multiple Laplacian iterations according to the given quality.

[0045] In one example, if Figure 3 As illustrated, level 0 may include initial sampling of masked original texture 306 at a resolution of 16x16. Level 0 sampling may also include multiple iterations at a resolution of 16x16 by performing a ping-pong operation. The ping-pong operation of level 0 includes a ping portion 302a and a pong portion 304a. Level 1 may upsample the texture generated by the last iteration of level 0 at a higher resolution of 32x32. Level 1 may also perform iterations with a resolution of 32x32, including a ping portion 302b and a pong portion 304b. Level 2 may upsample the texture generated by the last iteration of level 1 at a resolution of 64x64. Level 2 may also perform iterations with a resolution of 64x64, including a ping portion 302c and a pong portion 304c. Level 3 may upsample the texture generated by the last iteration of level 2 at a resolution of 128x128. Level 2 may also perform iterations with a resolution of 128x128, including a ping portion 302d and a pong portion 304d. During the pyramid ping-pong process, copying textures only occurs for the initial sampling of the masked original texture, and there are no copied textures from previous levels. The pyramid ping-pong operation speeds up texture sampling by performing only upsampling. The pyramid ping-pong operation also speeds up texture sampling by saving copying textures in the GPU and reducing the cost of the rendering pipeline.

[0046] In other embodiments, applying the repair filter may also include generating a lookup table by pre-calculating a Hammersley point set to further speed up the sampling process. Pre-calculating the Hammersley sampling point set can optimize the GPU code to allow it to run in low-end devices with performance comparable to C++. The sampling points can be jittered by adding additional random numbers. Initial sampling can be performed on the masked original texture. For the remainder of iterations 2-N, a ping-pong operation can be performed at the same level on the pyramid to ensure that it is blurred smoothly. For the first iteration of level 1, the result of the previous level (level 0) (i.e., a 16x16 texture) can be upsampled, and then a ping-pong operation can be performed at level 1 until it is blurred smoothly. This process can be iteratively repeated for all levels.

[0047] Return to reference Figure 1 After applying the inpainting filter to the first camera input stream to remove the human body from the camera input in real time, the output from the inpainting filter can be used to generate the camera input background. For example, the output of the inpainting filter can be mixed with the original image included in the camera input stream to generate the camera input background.

[0048] In an embodiment, the virtual character model 117 can additionally or alternatively utilize human body input to drive the 3D virtual character. For example, the virtual character model 117 can render the animated 3D virtual character onto one or more camera input backgrounds, where real human body parts are erased. Rendering the virtual character on the camera input background may include rendering the virtual character at the location where (multiple) human body parts are removed on the camera input background. The virtual character model 117 can additionally or alternatively implement the animation of the virtual character based on real-time detection of the user's facial landmarks and movements. The virtual character model 117 can additionally or alternatively generate sounds for the virtual character in real time based on source sounds received from the user. The virtual character model 117 can additionally or alternatively generate and / or output a final video. As described above, one or more body parts or the entire human body of the user may not be visible in the final video. In the final video, the user can be naturally replaced by an animated virtual character with sound.

[0049] Figure 4 An example process 400 performed by a video service 112 and / or a client device 104 is illustrated. The video service 112 and / or the client device 104 may perform the process 400 to create a video using a virtual character, such as a video for uploading to the video service 112. Once uploaded to the video service 112, the video may be viewed by users of the video service. Figure 4 2 is depicted as a sequence of operations, but one of ordinary skill in the art will appreciate that various embodiments may add, remove, reorder, or modify the operations depicted.

[0050] In 402, creation of a video may be initiated. Creation of a video may be triggered by a user. The user may be a user of a video service (i.e., video service 112). For example, the user may be a user associated with a client device (i.e., client device 104). The user may initiate creation of a video, such as via a content application (i.e., content application 106). The user may indicate that he or she wishes the video to feature a virtual character. For example, the user may select a filter or effect associated with a virtual character.

[0051] At 404, a camera input may be received that includes a human body of a user. For example, a camera input may be received from a camera of a client device. For example, a human body of a user may include any number of body parts of a user. For example, a human body of a user may include the entire body of a user or one or more body parts of a user. If a user indicates that he or she wishes the video to feature a virtual character, the received camera input may be split into a first stream for removing one or more human body parts in the video and a second stream for animating the virtual character in the video. At 406, the camera input may be split into a first stream for removing a human body and a second stream for animating the virtual character in the video.

[0052] At 408, a repair filter may be applied to remove human bodies from the camera input in real time. The repair filter may be configured to accelerate texture sampling. For example, applying the repair filter may use a pyramid ping-pong operation to accelerate texture sampling, such as described above with respect to Figure 3 The output of the inpainting filter may be blended with the image included in the camera input to generate a camera input background at 410. For example, the result from the last iteration of the last level of the pyramid ping-pong operation may be blended with the image included in the camera input to generate a camera input background for creating a video.

[0053] Figure 5 An example process 500 performed by a video service 112 and / or a client device 104 is illustrated. The video service 112 and / or the client device 104 may perform the process 500 to create a video using a virtual character, such as a video for uploading to the video service 112. Once uploaded to the video service 112, the video may be viewed by users of the video service. Figure 5 2 is depicted as a sequence of operations, but one of ordinary skill in the art will appreciate that various embodiments may add, remove, reorder, or modify the operations depicted.

[0054] In 502, creation of a video may be initiated. Creation of a video may be triggered by a user. The user may be a user of a video service (i.e., video service 112). For example, the user may be a user associated with a client device (i.e., client device 104). The user may initiate creation of a video, such as via a content application (i.e., content application 106). The user may indicate that he or she wishes the video to feature a virtual character. For example, the user may select a filter or effect associated with a virtual character.

[0055] At 504, a camera input may be received that includes a human body of a user. For example, a camera input may be received from a camera of a client device. For example, a human body of a user may include any number of body parts of a user. For example, a human body of a user may include the entire body of a user or one or more body parts of a user. If a user indicates that he or she wishes the video to feature a virtual character, the received camera input may be split into a first stream for removing one or more human body parts in the video and a second stream for animating the virtual character in the video. At 506, the camera input may be split into a first stream for removing a human body and a second stream for animating the virtual character in the video.

[0056] The inpainting filter may be applied to remove a human body from the camera input in real time. To apply the inpainting filter, an input mask may be used to mask an image (e.g., an image included in the camera input). In 508, the input mask may be used to mask a human body in the image included in the camera input. Masking the image may include creating a copy of the destination texture. On the copy of the destination texture, the missing area may be set to zero. The input mask may be resampled to the resolution of the destination texture and may be stored in an alpha channel of the copy of the destination texture.

[0057] For example, applying the mask to the image may create a masked image. A human body may be masked in the masked image. In 510, sampling at multiple levels may be performed based on the masked image. Sampling at multiple levels based on the masked image may include performing sampling at an initial level at a resolution of 16x16. Each level may be associated with a different image resolution, with the resolution doubling as each level increases. For example, level 0 of each pyramid in the pyramid may be associated with a 16x16 resolution, level 1 of each pyramid in the pyramid may be associated with a 32x32 resolution, level 2 of each pyramid in the pyramid may be associated with a 64x64 resolution, level 3 of each pyramid in the pyramid may be associated with a 128x128 resolution, and so on.

[0058] In 512, a ping-pong operation may be performed at each level of sampling. Using the ping-pong operation may prevent a large number of textures from being copied in the GPU and reduce the cost of the rendering pipeline. In one example, Figure 3 As shown, level 0 performs initial sampling of the masked original texture at a resolution of 16x16. Then, level 0 can perform multiple iterations at a resolution of 16x16 by performing a ping-pong operation, and the ping-pong operation of level 0 includes a ping portion 302a and a pong portion 304a. The next level can upsample the texture generated by the last iteration of the previous level at a higher resolution (e.g., twice the resolution of the previous level). Then, the next level can perform multiple iterations at the same resolution by performing a ping-pong operation. The ping-pong operation minimizes the read / write of the same texture. By utilizing the ping-pong operation, the copy texture only occurs for the initial sampling of the masked original texture, and there is no copy texture from the previous level. The ping-pong operation speeds up texture sampling by saving copy textures in the GPU and reducing the cost of the rendering pipeline.

[0059] The output of the inpainting filter may be blended with the image included in the camera input to generate a camera input background at 514. For example, the result from the last iteration of the last level of the pyramid ping-pong operation may be blended with the image included in the camera input to generate a camera input background for creating a video.

[0060] Figure 6 An example process 600 performed by a video service 112 and / or a client device 104 is illustrated. The video service 112 and / or the client device 104 may perform the process 600 to create a video using a virtual character, such as a video for uploading to the video service 112. Once uploaded to the video service 112, the video may be viewed by users of the video service. Figure 6 2 is depicted as a sequence of operations, but one of ordinary skill in the art will appreciate that various embodiments may add, remove, reorder, or modify the operations depicted.

[0061] As described above, the inpainting filter can be applied to remove a human body from a camera input in real time. To apply the inpainting filter, an input mask can be used to mask an image (e.g., an image included in the camera input). In 602, the input mask can be used to mask a human body in the image included in the camera input. Masking the image can include creating a copy of the destination texture. On the copy of the destination texture, the missing area can be set to zero. The input mask can be resampled to the resolution of the destination texture and can be stored in the alpha channel of the copy of the destination texture.

[0062] For example, applying a mask to an image may create a masked image. A human body may be masked or blurred in the masked image. Applying a repair filter may include performing multiple levels of sampling based on the masked image. Performing multiple levels of sampling based on the masked image may include performing an initial level of sampling at a resolution of 16x16. In 604, the initial level of sampling may be performed at a resolution of 16x16. In 606, a texture generated by a previous level of sampling may be upsampled at a higher resolution. For example, the higher resolution may be twice the resolution of the previous level of sampling.

[0063] In 608, a ping-pong operation may be performed with each level of sampling in the plurality of levels of sampling. The plurality of levels of sampling may include sampling of the initial level. Utilizing the ping-pong operation may prevent a large number of textures from being copied in the GPU and reduce the cost of the rendering pipeline. The ping-pong operation minimizes reads / writes to the same texture. By utilizing the ping-pong operation, copying the texture only occurs for the initial sampling of the masked original texture, and there will be no copied textures from the previous level. The ping-pong operation speeds up texture sampling by saving copied textures in the GPU and reducing the cost of the rendering pipeline.

[0064] Figure 7 An example process 700 performed by the video service 112 and / or the client device 104 is illustrated. The video service 112 and / or the client device 104 may perform the process 700 to create a video using a virtual character, such as a video for uploading to the video service 112. Once uploaded to the video service 112, the video may be viewed by users of the video service. Figure 7 2 is depicted as a sequence of operations, but one of ordinary skill in the art will appreciate that various embodiments may add, remove, reorder, or modify the operations depicted.

[0065] As described above, the inpainting filter can be applied to remove a human body from a camera input in real time. To apply the inpainting filter, an input mask can be used to mask an image (e.g., an image included in the camera input). In 702, the input mask can be used to mask a human body in an image included in the camera input. Masking the image can include creating a copy of a destination texture. On the copy of the destination texture, the missing area can be set to zero. The input mask can be resampled to the resolution of the destination texture and can be stored in the alpha channel of the copy of the destination texture.

[0066] For example, applying the mask to the image may create a masked image. A human body may be masked in the masked image. Applying the inpainting filter may include performing multiple levels of sampling based on the masked image. Performing multiple levels of sampling based on the masked image may include performing an initial level of sampling at a resolution of 16x16. At 704, initial sampling may be performed on the original texture of the masked image at a resolution of 16x16.

[0067] In 706, by performing a ping-pong operation, multiple iterations can be performed with the same resolution of 16x16 with the initial level of sampling among the multiple levels of sampling. The initial level of sampling includes the initial sampling and multiple iterations with the same resolution of 16x16. The ping-pong operation is used to perform multiple iterations with the initial level of sampling. The ping-pong operation can speed up texture sampling by saving duplicate textures in the GPU and reducing the cost of the rendering pipeline.

[0068] In 708, sampling of other levels among the multiple levels of sampling may be performed. The number of levels may be determined by the input resolution. For example, if the input resolution is 720p, then the levels (from level 0 upwards) may be: 16x16, 32x32, 64x64, 128x128, 256x256, 512x512. In another example, if the input resolution is changed to 360p, then the levels of the pyramid (from level 0 upwards) may be: 16x16, 32x32, 64x64, 128x128, 256x256. The sampling of each level includes upsampling the texture generated by the last iteration of the previous level at a resolution higher than the resolution of the previous level (e.g., twice the resolution of the previous level), and then performing multiple iterations at the same resolution by utilizing a ping-pong operation. The ping-pong operation minimizes reads / writes to the same texture. The ping-pong operation speeds up texture sampling by saving duplicate textures in the GPU and reducing the cost of the rendering pipeline.

[0069] Figure 8 An example process 800 performed by a video service 112 and / or a client device 104 is illustrated. The video service 112 and / or the client device 104 may perform the process 800 to create a video using a virtual character, such as a video for uploading to the video service 112. Once uploaded to the video service 112, the video may be viewed by users of the video service. Figure 8 2 is depicted as a sequence of operations, but one of ordinary skill in the art will appreciate that various embodiments may add, remove, reorder, or modify the operations depicted.

[0070] At 802, creation of a video may be initiated. Creation of a video may be triggered by a user. The user may be a user of a video service (i.e., video service 112). For example, the user may be a user associated with a client device (i.e., client device 104). The user may initiate creation of a video, such as via a content application (i.e., content application 106). The user may indicate that he or she wishes the video to feature a virtual character. For example, the user may select a filter or effect associated with a virtual character.

[0071] At 804, a camera input may be received that includes a human body of a user. For example, a camera input may be received from a camera of a client device. For example, a human body of a user may include any number of body parts of a user. For example, a human body of a user may include the entire body of a user or one or more body parts of a user. If a user indicates that he or she wishes the video to feature a virtual character, the received camera input may be split into a first stream for removing one or more human body parts in the video and a second stream for animating the virtual character in the video. At 806, the camera input may be split into a first stream for removing a human body and a second stream for animating the virtual character in the video.

[0072] At 808, a repair filter may be applied to remove human bodies from the camera input in real time. The repair filter may be configured to accelerate texture sampling. For example, applying the repair filter may accelerate texture sampling by saving downsampling, and using a pyramid ping-pong operation to accelerate upsampling, such as described above with respect to Figure 3 The pyramid ping-pong operation described herein. Applying the inpainting filter may include performing multiple levels of sampling by doubling the resolution of the previous level of sampling and performing a ping-pong operation with the sampling at each level of the multiple levels of sampling. Implementing the inpainting filter may include generating a lookup table including the precomputed Hammersley point set.

[0073] The output of the inpainting filter may be blended with an image included in the camera input to generate a camera input background at 810. For example, the result from the last iteration of the last level of the pyramid ping-pong operation may be blended with the corresponding original image included in the camera input to generate a camera input background for creating a video.

[0074] Fig. 9 An example process 900 performed by a video service 112 and / or a client device 104 is illustrated. The video service 112 and / or the client device 104 may perform the process 900 to create a video using a virtual character, such as a video for uploading to the video service 112. Once uploaded to the video service 112, the video may be viewed by users of the video service. Fig. 92 is depicted as a sequence of operations, but one of ordinary skill in the art will appreciate that various embodiments may add, remove, reorder, or modify the operations depicted.

[0075] As described above, human input can be used to drive 3D virtual characters. For example, an animated 3D virtual character can be rendered onto one or more camera input backgrounds generated using the above-mentioned techniques. In 902, the virtual character can be rendered on the camera input background. The virtual character can be rendered at a location where the human body is removed or erased. In 904, the animation of the virtual character can be implemented based on real-time detection of the user's facial landmarks and movements. In 906, the sound of the virtual character can be operatively generated in real time based on the source sound received from the user who created the video. In 908, a final video can be generated and / or output. In the final video, one or more body parts or the entire human body of the user can be naturally replaced by the animated virtual character with the generated sound.

[0076] Fig.10 The diagram shows a computing device that can be used in various aspects, such as Figure 1 The services, networks, modules and / or devices depicted in. Figure 1 The example architecture can be represented by Fig.10 One or more instances of computing device 1000 implement video service 112 , virtual character model 117 , and / or client device 104 . Fig.10 The computer architecture shown illustrates a conventional server computer, workstation, desktop computer, laptop computer, tablet computer, network appliance, PDA, e-reader, digital cellular telephone, or other computing node and can be used to perform any aspects of the computer described herein, such as implementing the methods described herein.

[0077] The computing device 1000 may include a baseboard or "motherboard," which is a printed circuit board to which multiple components or devices may be connected via a system bus or other electrical communication paths. One or more central processing units (CPUs) 1004 may operate in conjunction with a chipset 1006. The CPU(s) 1004 may be a standard programmable processor that performs the arithmetic and logic operations necessary for the operation of the computing device 1000.

[0078] (Multiple) CPUs 1004 can perform necessary operations by manipulating switching elements to transition from one discrete physical state to the next discrete physical state, which distinguish and change these states. Switching elements can generally include electronic circuits that maintain one of two binary states (such as flip-flops) and electronic circuits that provide output states based on logical combinations of the states of one or more other switching elements (such as logic gates). These basic switching elements can be combined to create more complex logic circuits, including registers, adder-subtractors, arithmetic logic units, floating point units, etc.

[0079] CPU(s) 1004 may be enhanced or replaced with other processing units, such as GPU(s) 1005. GPU(s) 1005 may include processing units specialized for, but not necessarily limited to, highly parallel computations, such as graphics and other visualization-related processing.

[0080] Chipset 1006 may provide an interface between CPU(s) 1004 and the rest of the components and devices on the baseboard. Chipset 1006 may provide an interface to random access memory (RAM) 1008 used as main memory in computing device 1000. Chipset 1006 may also provide an interface to a computer-readable storage medium such as read-only memory (ROM) 1020 or non-volatile RAM (NVRAM) (not shown) for storing basic routines that may help boot computing device 1000 and transfer information between various components and devices. ROM 1020 or NVRAM may also store other software components necessary for the operation of computing device 1000 according to various aspects described herein.

[0081] The computing device 1000 can operate in a network environment using logical connections to remote computing nodes and computer systems through a local area network (LAN). Chipset 1006 may include functionality for providing network connectivity through a network interface controller (NIC) 1022 (such as a Gigabit Ethernet adapter). NIC 1022 can connect computing device 1000 to other computing nodes through network 1016. It should be understood that multiple NICs 1022 may be present in computing device 1000, thereby connecting the computing device to other types of networks and remote computer systems.

[0082] The computing device 1000 may be connected to a mass storage device 1028 that provides non-volatile storage for the computer. The mass storage device 1028 may store system programs, application programs, other program modules, and data, which have been described in more detail herein. The mass storage device 1028 may be connected to the computing device 1000 via a storage controller 1024 connected to the chipset 1006. The mass storage device 1028 may be composed of one or more physical storage units. The mass storage device 1028 may include a management component 1010. The storage controller 1024 may be connected to the physical storage unit interface via a serial attached SCSI (SAS) interface, a serial advanced technology attachment (SATA) interface, a fiber channel (FC) interface, or other types of interfaces for physically connecting and transferring data between a computer and a physical storage unit.

[0083] The computing device 1000 can store data on the mass storage device 1028 by transforming the physical state of the physical storage unit to reflect the information being stored. The specific transformation of the physical state may depend on various factors and different implementations of this description. Examples of such factors may include, but are not limited to, the technology used to implement the physical storage unit and whether the mass storage device 1028 is characterized as a primary storage device or a secondary storage device, etc.

[0084] For example, the computing device 1000 may store information to the mass storage device 1028 by issuing instructions through the storage controller 1024 to change the magnetic characteristics of a specific location within a disk drive unit, the reflective or refractive characteristics of a specific location in an optical storage unit, or the electrical characteristics of a specific capacitor, transistor, or other discrete component in a solid-state storage unit. Other transformations of the physical media are possible without departing from the scope and spirit of the present description, and the foregoing examples are provided only to facilitate the description. The computing device 1000 may also read information from the mass storage device 1028 by detecting the physical state or characteristics of one or more specific locations within the physical storage unit.

[0085] In addition to the mass storage device 1028 described above, the computing device 1000 may access other computer-readable storage media to store and retrieve information, such as program modules, data structures, or other data. It should be understood by those skilled in the art that computer-readable storage media may be any available media that provides storage of non-transitory data and can be accessed by the computing device 1000.

[0086] By way of example and not limitation, computer-readable storage media may include volatile and nonvolatile, transient computer-readable storage media and non-transient computer-readable storage media, and removable and non-removable media implemented in any method or technology. Computer-readable storage media include, but are not limited to, RAM, ROM, erasable programmable ROM ("EPROM"), electrically erasable programmable ROM ("EEPROM"), flash memory or other solid-state memory technology, compact disk ROM ("CD-ROM"), digital versatile disk ("DVD"), high-definition DVD ("HD-DVD"), Blu-ray disc or other optical storage, cassettes, magnetic tape, magnetic disk storage, other magnetic storage devices, or any other medium that can be used to store desired information in a non-transient manner.

[0087] Mass storage devices (such as Fig.10 The mass storage device 1028 depicted in the figure can store an operating system for controlling the operation of the computing device 1000. The operating system can include a version of the LINUX operating system. The operating system can include a version of the WINDOWS server operating system from Microsoft Corporation. According to other aspects, the operating system can include a version of the UNIX operating system. Various mobile phone operating systems, such as IOS and Android, can also be utilized. It should be understood that other operating systems can also be utilized. The mass storage device 1028 can store other systems or applications and data utilized by the computing device 1000.

[0088] The mass storage device 1028 or other computer-readable storage medium may also be encoded with computer-executable instructions that, when loaded into the computing device 1000, transform the computing device from a general-purpose computing system into a special-purpose computer capable of implementing the various aspects described herein. As described above, these computer-executable instructions transform the computing device 1000 by specifying how the CPU(s) 1004 transition between states. The computing device 1000 may access a computer-readable storage medium storing computer-executable instructions that, when executed by the computing device 1000, may perform the methods described herein.

[0089] Computing devices (such as Fig.10 The computing device 1000 depicted in FIG. 1000 may also include an input / output controller 1032 for receiving and processing input from a plurality of input devices such as a keyboard, a mouse, a touch pad, a touch screen, an electronic stylus, or other types of input devices. Similarly, the input / output controller 1032 may provide output to a display such as a computer monitor, a flat panel display, a digital projector, a printer, a plotter, or other types of output devices. It is to be understood that the computing device 1000 may not include Fig.10All components shown may include Fig.10 Other components not explicitly shown in the drawings or may be used in conjunction with Fig.10 The architecture shown is a completely different architecture.

[0090] As described herein, a computing device may be a physical computing device such as Fig.10 The computing device 1000 of the present invention may also include a virtual machine host process and one or more virtual machine instances. The computer executable instructions may be indirectly executed by the physical hardware of the computing device through interpretation and / or execution of instructions stored and executed in the context of the virtual machine.

[0091] It is to be understood that the methods and systems are not limited to specific methods, specific components or specific implementations. It is also to be understood that the terminology used herein is for describing particular embodiments only and is not intended to be limiting.

[0092] As used in this specification and the appended claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from "about" one particular value and / or to "about" another particular value. When such a range is expressed, another embodiment includes from the one particular value and / or to the other particular value. Similarly, when a value is expressed as an approximation by using the antecedent "about", it is understood that the particular value forms another embodiment. It is also understood that the endpoints of each range in the range are significant with respect to the other endpoints, and independent of the other endpoints.

[0093] "Optional" or "optionally" means that the subsequently described event or circumstance may or may not occur, and that the description includes instances where said event or circumstance occurs and instances where it does not.

[0094] In the description and claims of this specification, the word "include" and variations of the word (such as, "comprising" and "having") mean "including but not limited to", and are not intended to exclude, for example, other components, integers or steps. "Exemplary" means "an example of..." and is not intended to convey an indication of a preferred or ideal embodiment. "Such as" is not used in a limiting sense, but for explanatory purposes.

[0095] Components that can be used to perform the described methods and systems are described. When describing combinations, subsets, interactions, groups, etc. of these components, it is to be understood that, although specific references to each of the various individual and collective combinations and arrangements of these components may not be explicitly described, each is specifically contemplated and described herein for all methods and systems. This applies to all aspects of the application, including but not limited to the operations in the described methods. Therefore, if there are various additional operations that can be performed, it is to be understood that each of these additional operations can be performed with any specific embodiment or combination of embodiments of the described methods.

[0096] The present method and system may be understood more readily by reference to the following detailed description of the preferred embodiments and examples included therein and to the accompanying drawings and their description.

[0097] As will be appreciated by those skilled in the art, the method and system may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. In addition, the method and system may take the form of a computer program product on a computer-readable storage medium having computer-readable program instructions (e.g., computer software) implemented in the storage medium. More specifically, the method and system may take the form of computer software implemented by a web. Any suitable computer-readable storage medium may be utilized, including a hard disk, a CD-ROM, an optical storage device, or a magnetic storage device.

[0098] Embodiments of the method and system are described below with reference to block diagrams and flowchart illustrations of the method, system, apparatus, and computer program product. It is to be understood that each block illustrated in the block diagram and flowchart, and combinations of blocks in the block diagram and flowchart illustration, can be implemented by computer program instructions, respectively. These computer program instructions can be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed on the computer or other programmable data processing device create components for implementing the functions specified in one or more flowchart blocks.

[0099] These computer program instructions may also be stored in a computer-readable memory that may instruct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture that includes computer-readable instructions for implementing the functions specified in one or more flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flowchart blocks.

[0100] The various features and processes described above can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. In addition, in some implementations, certain methods or process blocks may be omitted. The methods and processes described herein are not limited to any particular sequence, and the blocks or states associated therewith may be performed in other appropriate sequences. For example, the blocks or states described may be performed in an order different from that specifically described, or a plurality of blocks or states may be combined in a single block or state. Example blocks or states may be performed serially, in parallel, or in some other manner. Blocks or states may be added to the described example embodiments, or removed from the described example embodiments. The example systems and components described herein may be configured differently from those described. For example, compared to the described example embodiments, elements may be added, removed, or rearranged.

[0101] It is also understood that various items are illustrated as being stored in memory or on storage devices when in use, and that these items or portions thereof may be transferred between memory and other storage devices for purposes of memory management and data integrity. Alternatively, in other embodiments, some or all of the software modules and / or systems may be executed in memory on another device and communicate with the illustrated computing system via inter-computer communication. In addition, in some embodiments, some or all of the systems and / or modules may be implemented or provided in other ways, such as at least partially implemented or provided in firmware and / or hardware, including but not limited to one or more application specific integrated circuits (“ASICs”), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field programmable gate arrays (“FPGAs”), complex programmable logic devices (“CPLDs”), etc. Some or all of the modules, systems, and data structures may also be stored (e.g., as software instructions or structured data) on a computer-readable medium, such as a hard disk, memory, network, or portable media article, so as to be read by an appropriate device or via an appropriate connection. Systems, modules, and data structures may also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media, including wireless and wire / cable-based media, and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, such computer program products may also take other forms. Therefore, the present invention may be practiced with other computer system configurations.

[0102] Although the method and system have been described in conjunction with preferred embodiments and specific examples, the scope is not intended to be limited to the particular embodiments set forth, as the embodiments herein are intended in all respects to be illustrative rather than restrictive.

[0103] Unless otherwise expressly stated, any method described herein is in no way intended to be construed as requiring that its operations be performed in a specific order. Therefore, in the event that a method claim does not actually enumerate the order in which its operations are to be followed or that the operations are not otherwise specifically provided in the claim or description to be limited to a specific order, the order is in no way intended to be inferred in any respect. This applies to any possible non-express basis for interpretation, including: matters of logic with respect to the arrangement of steps or operational flows; ordinary meaning derived from grammatical organization or punctuation; and the number or type of embodiments described in the specification.

[0104] It is obvious to those skilled in the art that various modifications and variations can be made without departing from the spirit or scope of the present disclosure. Other embodiments will be apparent to those skilled in the art by considering the specification and practice described herein. The specification and example drawings are intended to be considered as exemplary only, with the true scope and spirit being indicated by the following claims.

Claims

1. A method for creating a video using a virtual character, comprising: User-initiated video creation; receiving a camera input, the camera input including a human body of the user; splitting the camera input into a first stream for removing the human body and a second stream for animating a virtual character in the video; applying an inpainting filter to remove the human body from the camera input in real time, wherein the inpainting filter is configured to accelerate texture sampling; as well as An output of the inpainting filter is blended with an image included in the camera input to generate a camera input background.

2. The method of claim 1 , wherein applying the inpainting filter comprises: masking the human body in an image included in the camera input using an input mask; performing a plurality of levels of sampling based on the masked image, wherein performing a plurality of levels of sampling includes performing an initial level of sampling at a resolution of 16x16; as well as A ping-pong operation is performed with samples at each level of the plurality of levels of samples.

3. The method of claim 2, wherein performing an initial level of sampling further comprises: performing initial sampling of the original texture of the masked image at the resolution of 16x16; as well as By performing the ping-pong operation, a plurality of iterations are performed with the initial level of sampling.

4. The method of claim 2, wherein performing multiple levels of sampling further comprises: Upsampling a texture produced by a previous level of sampling at a higher resolution, wherein the higher resolution is twice the resolution of the previous level of sampling; as well as By performing the ping-pong operation, multiple iterations are performed with samples at the current level.

5. The method of claim 1, wherein implementing the inpainting filter comprises: A lookup table is generated, wherein the lookup table includes a precomputed set of Hamersley points.

6. The method according to claim 1, further comprising: The virtual character is rendered on the camera input background, wherein the virtual character is rendered at a location where the human body is removed.

7. The method according to claim 6, further comprising: Animation of the virtual character is achieved based on real-time detection of facial landmarks and movements of the user.

8. The method according to claim 6, further comprising: Sound is generated in real time for the virtual character based on source sound received from the user.

9. The method according to claim 8, further comprising: The video is generated and output, wherein the user is not visible in the video, and wherein the video includes the virtual character having the animation and having the sound.

10. A system comprising: at least one processor in communication with at least one memory, the at least one memory comprising computer readable instructions that, when executed by the at least one processor, configure the system to perform operations comprising: User-initiated video creation; receiving a camera input, the camera input including a human body of the user; splitting the camera input into a first stream for removing the human body and a second stream for animating a virtual character in the video; applying an inpainting filter to remove the human body from the camera input in real time, wherein the inpainting filter is configured to accelerate texture sampling; and An output of the inpainting filter is blended with an image included in the camera input to generate a camera input background.

11. The system of claim 10, wherein applying the inpainting filter comprises: masking the human body in an image included in the camera input using an input mask; performing a plurality of levels of sampling based on the masked image, wherein performing a plurality of levels of sampling includes performing an initial level of sampling at a resolution of 16x16; as well as A ping-pong operation is performed with samples at each level of the plurality of levels of samples.

12. The system of claim 11, wherein performing an initial level of sampling further comprises: performing initial sampling of the original texture of the masked image at the resolution of 16x16; as well as By performing the ping-pong operation, a plurality of iterations are performed with the initial level of sampling.

13. The system of claim 11, wherein performing multiple levels of sampling further comprises: Upsampling a texture produced by a previous level of sampling at a higher resolution, wherein the higher resolution is twice the resolution of the previous level of sampling; as well as By performing the ping-pong operation, multiple iterations are performed with samples at the current level.

14. The system of claim 10, wherein implementing the inpainting filter comprises: A lookup table is generated, wherein the lookup table includes a precomputed set of Hamersley points.

15. The system of claim 10, the operations further comprising: rendering the virtual character on the camera input background, wherein the virtual character is rendered at a location where the human body is removed; Animating the virtual character based on real-time detection of facial landmarks and movements of the user; generating sounds for the virtual character in real time based on source sounds received from the user; as well as The video is generated and output, wherein the user is not visible in the video, and wherein the video includes the virtual character having the animation and having the sound.

16. A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a processor, cause the processor to perform operations comprising: User-initiated video creation; receiving a camera input, the camera input including a human body of the user; splitting the camera input into a first stream for removing the human body and a second stream for animating a virtual character in the video; applying an inpainting filter to remove the human body from the camera input in real time, wherein the inpainting filter is configured to accelerate texture sampling; as well as An output of the inpainting filter is blended with an image included in the camera input to generate a camera input background.

17. The non-transitory computer-readable storage medium of claim 16, wherein applying the inpainting filter comprises: masking the human body in an image included in the camera input using an input mask; performing a plurality of levels of sampling based on the masked image, wherein performing a plurality of levels of sampling includes performing an initial level of sampling at a resolution of 16x16; as well as A ping-pong operation is performed with samples at each level of the plurality of levels of samples.

18. The non-transitory computer-readable storage medium of claim 17, wherein performing an initial level of sampling further comprises: performing initial sampling of the original texture of the masked image at the resolution of 16x16; as well as By performing the ping-pong operation, a plurality of iterations are performed with the initial level of sampling.

19. The non-transitory computer readable storage medium of claim 18, wherein performing multiple levels of sampling further comprises: Upsampling a texture produced by a previous level of sampling at a higher resolution, wherein the higher resolution is twice the resolution of the previous level of sampling; as well as By performing the ping-pong operation, multiple iterations are performed with samples at the current level.

20. The non-transitory computer readable storage medium of claim 16, the operations further comprising: rendering the virtual character on the camera input background, wherein the virtual character is rendered at a location where the human body is removed; Animating the virtual character based on real-time detection of facial landmarks and movements of the user; generating sounds for the virtual character in real time based on source sounds received from the user; as well as The video is generated and output, wherein the user is not visible in the video, and wherein the video includes the virtual character having the animation and having the sound.

Images