Creating real-time interactive videos

Abstract

This disclosure describes techniques for creating real-time interactive videos. A source image is generated by a first machine learning model based on capturing an image of a user. The image includes a face of the user. One or more facial images of the user are captured. The one or more facial images depict one or more facial expressions. The source image and information extracted from the one or more facial images are input into a second machine learning model. The second machine learning model is configured and trained to transfer the creator’s facial expressions to the machine-generated images in real-time. A real-time interactive video is created by dynamically driving the source image based on the one or more facial expressions.

Description

Cross-reference to related applications

[0001] This application claims priority to U.S. Application 18 / 388,785, filed November 10, 2023, entitled “Creating Real-Time Interactive Video,” the disclosure of which is incorporated herein by reference in its entirety. Background Technology

[0002] Machine learning models are increasingly being used across various industries to perform a wide range of tasks. These tasks can include content generation. Improvements in content generation techniques leveraging machine learning models are anticipated. Attached Figure Description

[0003] The following detailed description will be better understood when read in conjunction with the accompanying drawings. For illustrative purposes, exemplary embodiments of various aspects of this disclosure are shown in the drawings; however, the invention is not limited to the specific methods and means disclosed.

[0004] Figure 1 An example system for creating real-time interactive video according to this disclosure is shown.

[0005] Figure 2 An example user interface for creating real-time interactive video is shown according to this disclosure.

[0006] Figure 3 An example user interface for creating real-time interactive video is shown according to this disclosure.

[0007] Figure 4 An example user interface for creating real-time interactive video is shown according to this disclosure.

[0008] Figure 5 An example user interface for creating real-time interactive video is shown according to this disclosure.

[0009] Figure 6 An example user interface for creating real-time interactive video is shown according to this disclosure.

[0010] Figure 7 An example system for creating real-time interactive video according to this disclosure is shown.

[0011] Figure 8 An example system for creating real-time interactive video according to this disclosure is shown.

[0012] Figure 9 An example process for creating real-time interactive video according to this disclosure is shown.

[0013] Figure 10 An example process for creating real-time interactive video according to this disclosure is shown.

[0014] Figure 11 An example process for creating real-time interactive video according to this disclosure is shown.

[0015] Figure 12 An example process for creating real-time interactive video according to this disclosure is shown.

[0016] Figure 13 An example process for creating real-time interactive video according to this disclosure is shown.

[0017] Figure 14 An example computing device is shown that can be used to perform any of the techniques disclosed herein. Detailed Implementation

[0018] Machine learning models can be used for content generation. Specifically, they can be used to generate still images for users. However, existing technologies for generating content using machine learning models may lack user interactivity. Furthermore, existing technologies may not be able to generate high-quality video content, such as live video content. Therefore, improved technologies for content generation are needed.

[0019] This article describes an improved technique for generating real-time interactive video. An initial image or live footage of the user can be captured. First, face detection can be used to identify the user's facial features, which are preserved for generating one or more new images based on the initial image. AI can then be used to generate new images based on the initial image. The new images can dynamically display the user's current facial features and / or expressions. For example, if the user opens their mouth, the new image could display a machine-generated character resembling the user and also opening their mouth. The new images can continue to reflect changes in the user's facial expressions in real time. For example, the new images can form frames in a video. The new images can be displayed on the device's interface in real time.

[0020] Figure 1 An example system 100 according to this disclosure is illustrated. System 100 can be used to create real-time interactive video. For example, system 100 can utilize a user's (multiple) facial expressions to drive machine-generated images to create dynamic, real-time interactive video. System 100 may include a first machine learning model 104 and a second machine learning model 112. User equipment (e.g., client device, mobile computing device, etc.) may include system 100. For example, user equipment may include the first machine learning model 104 and the second machine learning model 112.

[0021] The user may be prompted (e.g., via an interface of a user device, such as a client device, mobile computing device, etc.) to capture an image of his or her face. For example, the user may be prompted or guided to position his or her face at a predetermined location on the interface. The user may use a camera device associated with the user device to capture user image 102 (e.g., an image including the user's face). The camera device may be a component of the user device or separate from the user device. User image 102 may depict the user's face. User image 102 may be used to generate source image 106. For example, user image 102 may be input into a first machine learning model 104. First machine learning model 104 may generate source image 106 based on user image 102. First machine learning model 104 may generate source image 106 by scanning the user's face positioned at a predetermined location. Source image 106 may depict a machine-generated character or avatar, such as a machine-generated character or avatar similar to the user in user image 102. Source image 106 may be input into a second machine learning model 112. Source image 106 may be output and displayed on the interface of a client device.

[0022] A user can capture one or more additional images (e.g., multiple user images 108, driving images, etc.). The one or more additional images can be captured sequentially. For example, the one or more additional images can be frames of real-time video depicting multiple facial expressions of the user. User images 108 can be captured in response to prompts to make various facial expressions or movements via an interface such as a user device (e.g., a client device, a mobile computing device, etc.). The user can utilize a camera device associated with the user device to capture multiple user images 108. Information can be extracted from each user image in user images 108 in real time. This information may include facial image information, such as facial landmark data. Information can be extracted from each user image in user images 108 using any suitable facial landmark identification technology. The extracted facial image information 110 can be input into a second machine learning model 112.

[0023] The second machine learning model 112 can be configured and trained to transfer a user's (e.g., a creator's) facial expressions to a machine-generated image, such as source image 106, in real time. The second machine learning model 112 can transfer facial expressions depicted in user image 108 to source image 106. For example, the second machine learning model 112 can use extracted facial image information 110 to transfer facial expressions depicted in user image 108 to source image 106. The second machine learning model 112 can display the facial expressions depicted in user image 108 on source image 106, such as in real time via an interface of a client device. For example, if extracted facial image information 110 corresponding to a first user image in user image 108 indicates that the user is smiling in the first user image, then the second machine learning model 112 can make the machine-generated character or avatar depicted in source image 106 smile. If extracted facial image information 110 corresponding to a next user image in user image 108 indicates that the user is frowning in the next user image, then the second machine learning model 112 can make the machine-generated character or avatar depicted in source image 106 stop smiling and start frowning.

[0024] In an embodiment, the second machine learning model 112 can create real-time interactive video by dynamically driving the source image 106 based on facial expressions depicted in the user image 108. For example, the second machine learning model 112 can utilize neural generative adversarial network (GAN) technology to dynamically drive the source image 106 based on facial movements in the user image(s) 108. Dynamically driving the source image 106 based on facial movements in the user image(s) 108 enhances user interactivity during the content creation process and creates a more dynamic effect for the final video output.

[0025] Figures 2 to 6 The user interface associated with creating real-time interactive video using System 100 is shown. (Example) Figure 2 As shown, a user interface (UI) 200 configured to guide a user to position their face at a predetermined location 202 can be displayed. The UI 200 can be displayed on a user device (e.g., a client device, mobile computing device, etc.). The user device can be associated with a camera device. Figure 3 As shown, UI 300 can be displayed. UI 300 can be configured to instruct the user to avoid moving his or her face out of the predetermined position 202 and / or to avoid changing his or her facial expression.

[0026] A source image can be generated based on a face at a predetermined location 202. For example, a source image can be generated based on a face scanned and positioned at the predetermined location 202. Figure 4As shown, a UI 400 can be displayed to indicate that the source image is being generated. The source image can be generated, for example, by a first machine learning model. The source image can depict a machine-generated character or avatar, such as a machine-generated character or avatar resembling a user with their face located at a predetermined position 202. Figure 5 As shown, UI 500 can be displayed based on (e.g., in response to) a source image. UI 500 can display information (e.g., prompts) 502 configured to prompt the user to show one or more facial expressions. The user can move his or her face and / or show one or more facial expressions, such as in response to viewing information 502. For example, the user can close her mouth. Figure 6 As shown, UI 600 can be displayed based on (e.g., in response to) a user moving his or her face. UI 600 can display one or more facial expressions on a source image. For example, UI 600 can display a source image with a closed mouth based on the user closing her mouth. Real-time interactive video can be created by dynamically driving the source image based on one or more facial expressions of the user (e.g., the creator).

[0027] Figure 7 An example system 700 according to this disclosure is illustrated. System 700 can be used to create real-time interactive video. For example, system 700 can utilize a user's facial expressions(s) to drive multiple machine-generated images(s) to create dynamic, real-time interactive video. System 700 may include a neuromotor model 712. The neuromotor model 712 can run on and / or be installed on a user device (e.g., a client device, a mobile computing device, etc.).

[0028] The user can be prompted (e.g., via an interface of a user device, such as a client device, mobile computing device, etc.) to capture an image of his or her face. For example, the user can be prompted or guided to position his or her face at a predetermined location on the interface. The user can use a camera device 701 associated with the user device to capture an initial user image (e.g., an image of the user's face at the predetermined location). The camera device 701 can be a component of the user device or separate from the user device. The initial user image can depict the user's face. The initial user image can be used to generate a source image 706. For example, the initial user image can be input into a first machine learning model. The first machine learning model can generate the source image 706 based on the initial user image. The first machine learning model can generate the source image 706 by scanning the user's face positioned at the predetermined location. The source image 706 can depict a machine-generated character or avatar, such as a machine-generated character or avatar similar to the user in the initial user image. The source image 706 can be input into a neuromotor model 712. The source image 706 can be output and displayed on the interface of the client device. For example, the output of source image 706 can be caused by neural motion model 712.

[0029] A user may capture one or more additional images (e.g., multiple driving frames 708, driving images, etc.). Multiple driving frames 708 may be captured sequentially. For example, multiple driving frames 708 may be frames depicting a real-time video image of the user's face. The user may capture multiple driving frames 708 in response to prompts to make various facial expressions or movements via an interface such as a user device (e.g., a client device, a mobile computing device, etc.). The user may also capture multiple driving frames 708 using a camera device 701 associated with the user device.

[0030] The sign detection model 707 can be configured to extract information from each of the driving frames(s) 708 in real time and / or determine information associated with each of the driving frames(s) 708. This information may include facial image information, such as facial sign data. The sign detection model 707 can use any suitable facial sign identification technique to extract and / or determine the information. The extracted facial image information can be input into the neuromotor model 712.

[0031] Information associated with source image 706 can be determined. This information may indicate one or more features associated with source image 706. These features may indicate, for example, the size (e.g., length, width) and / or shape of a machine-generated character's head, hair color and / or length, the location and / or shape of facial features (e.g., eyes, nose, mouth, ears, eyebrows, etc.), or any other feature of source image 706. Information associated with source image 706 may indicate the relationship between source image 706 and driving frames(s) 708. For example, information associated with source image 706 may indicate the size ratio between the machine-generated head depicted in source image 706 and the user's head in driving frames(s) 708. Information associated with source image 706 may be input into neuromotor model 712 along with source image 706 and driving frames(s) 708.

[0032] The neural motion model 712 can be configured and trained to transfer a creator's facial expressions to machine-generated images, such as source image 706, in real time. The neural motion model 712 can transfer facial expressions depicted in multiple driving frames 708 to source image 706. For example, the neural motion model 712 can use extracted facial image information and / or information associated with source image 706 to transfer facial expressions depicted in the neural motion model 712 to source image 706. The neural motion model 712 can display the facial expressions depicted in multiple driving frames 708 on source image 706, such as in real time via an interface of a client device.

[0033] In an embodiment, the neuromotor mobility model 712 can create real-time interactive video by dynamically driving source images 706 based on facial expressions depicted in multiple driving frames 708. For example, the neuromotor mobility model 712 can utilize neuromotor generative adversarial network (GAN) technology to dynamically drive source images 706 based on facial movements in multiple driving frames 708. Dynamically driving source images 706 based on facial movements in multiple driving frames 708 enhances user interactivity during content creation and creates a more dynamic effect for the final video output 714.

[0034] Figure 8An example system 800 according to this disclosure is illustrated. System 800 can be used to create real-time interactive video. For example, system 800 can utilize a user's facial expressions(s) to drive machine-generated images to create dynamic, real-time interactive video. System 800 may include a preprocessing component 801, a sign detector 803, a neural keypoint detector 805, a multilayer perceptron (MLP) neural keypoint mapping model 807, an MLP motion estimation model 809, a convolutional neural network (CNN) hourglass model 811, and an image inpainting generator 819. The neural keypoint detector 805, the MLP neural keypoint mapping model 807, the MLP motion estimation model 809, the convolutional neural network (CNN) hourglass model 811, and / or the image inpainting generator 819 may each be a component of a neuromotor model (e.g., neuromotor model 712) that runs on and / or is installed on a user device (e.g., a client device, a mobile computing device, etc.).

[0035] The user may be prompted (e.g., via the interface of a user device, such as a client device, mobile computing device, etc.) to capture an image of his or her face. For example, the user may be prompted or guided to position his or her face at a predetermined location on the interface. The user may use a camera device associated with the user device to capture an initial user image (e.g., an image of the user's face at the predetermined location). The camera device may be a component of the user device or separate from it. The initial user image may depict the user's face. The initial user image may be used to generate source image 806. For example, the initial user image may be input into a first machine learning model. The first machine learning model may generate source image 806 based on the initial user image. The first machine learning model may generate source image 806 by scanning the user's face positioned at the predetermined location. Source image 806 may depict a machine-generated character or avatar, such as a machine-generated character or avatar similar to the user in the initial user image.

[0036] A user may capture one or more additional images (e.g., multiple driving frames 808, driving images, etc.). Multiple driving frames 808 may be captured sequentially. For example, multiple driving frames 808 may be frames depicting a real-time video image of the user's face. A user may capture multiple driving frames 808 in response to prompts to make various facial expressions or movements via an interface such as a user device (e.g., a client device, a mobile computing device, etc.). A user may also capture multiple driving frames 808 using a camera device associated with the user device.

[0037] The marker detection model 803 can be configured to extract information from each of the driving frames(s) 808 in real time and / or determine information associated with each of the driving frames(s) 808. This information may include facial image information, such as facial marker data. The marker detection model 803 can use any suitable facial marker identification technique to extract and / or determine the information. The extracted facial image information can be fed into the neural keypoint detector 805.

[0038] Preprocessing model 801 can determine information associated with source image 806. This information can indicate one or more features associated with source image 806. These features can indicate, for example, the size (e.g., length, width) and / or shape of a machine-generated character's head, hair color and / or length, the location and / or shape of facial features (e.g., eyes, nose, mouth, ears, eyebrows, etc.), or any other feature of source image 806. The information associated with source image 806 can indicate the relationship between source image 806 and driving frames(s) 808. For example, the information associated with source image 806 can indicate the size ratio between the machine-generated head depicted in source image 806 and the user's head in driving frames(s) 808. The information associated with source image 806 can be input together with source image 806 and driving frames(s) 808 into neural keypoint detector 805.

[0039] The neural keypoint detector 805 can be configured to detect (e.g., determine) keypoints indicating one or more motion fields associated with one or more facial expressions depicted in the driving frames 808. The neural keypoint detector 805 can be configured to detect keypoints by identifying and locating specific points of interest in the driving frames 808. Keypoints can provide basic information about the position, pose, and structure of a user's face or facial expressions within the driving frames 808. The output of the neural keypoint detector 805 (e.g., keypoints) can represent one or more motion fields associated with the facial expressions. The motion fields can indicate multiple moving locations (e.g., the user's mouth, eyes, etc.) in the driving frames 808.

[0040] Keypoints can be input into an MLP neural keypoint mapping model 807. The MLP neural keypoint mapping model 807 can generate a deformable file (e.g., a deformable map) 813 based on the keypoints. The deformable file 813 can indicate how to distort the source image 806. The source image 806 can be distorted to generate a distorted source image 815. An MLP motion estimation model 809 and a CNN hourglass model 811 can optimize the motion field associated with (multiple) facial expressions and generate an occlusion map 817. The occlusion map 817 can indicate how to optimize, update, or improve the distorted source image 815 to generate (multiple) images that more closely resemble (multiple) facial expressions in (multiple) driving frames 808.

[0041] Image inpainting generator 819 can receive a distorted source image 815 and an occlusion map 817 as input. Image inpainting generator 819 can generate a result (e.g., a result image / video) 814 based on the distorted source image 815 and the occlusion map 817. Result 814 more closely resembles the facial expressions(s) in driving frame 808 than the distorted source image 815. Dynamically driving the source image 806 based on the facial expressions(s) and / or facial movements(s) in driving frame 808 enhances interactivity during the content creation process and creates a more dynamic effect for the final video output 814.

[0042] Figure 9 The illustration depicts a sample process 900 for creating live interactive video. Although in Figure 9 The operations are depicted as a sequence of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder, or modify the depicted operations.

[0043] At position 902, a source image can be generated. The source image can be generated by a first machine learning model. The source image can be generated based on an image captured of the user. The user's image may include the user's face. The user can capture an image using a camera device associated with the user's device. The camera device may be a component of the user's device or separate from the user's device. The first machine learning model can generate the source image by scanning the user's face in the user's image. The source image may depict a machine-generated character or avatar, such as a machine-generated character or avatar resembling the user in the user's image.

[0044] At position 904, one or more facial images of the user can be captured. These one or more facial images can depict one or more facial expressions. The one or more facial images can be captured sequentially. For example, the one or more facial images can be frames of a live video frame depicting the user's face. Facial images can be captured in response to prompts to make various facial expressions or facial movements, such as through an interface on a user device (e.g., a client device, mobile computing device, etc.). The user can utilize a camera device associated with the user device to capture facial images. Information can be extracted from each facial image in real time. This information can include facial image information, such as facial marker data. Any suitable facial marker identification technology can be used to extract information from each facial image.

[0045] At position 906, the source image can be fed into a second machine learning model. Information extracted from one or more facial images can be fed into the second machine learning model. The second machine learning model can be configured and trained to transfer the creator's facial expressions to the machine-generated images in real time. The second machine learning model can transfer facial expressions depicted in facial images to the source image. For example, the second machine learning model can use extracted facial image information to transfer facial expressions depicted in facial images to the source image.

[0046] The second machine learning model can display facial expressions depicted in a facial image on a source image, such as in real time via a client device's interface. At 908, one or more facial expressions can be displayed. For example, if extracted facial image information corresponding to a first facial image(s) indicates that the user is smiling in the first facial image, the second machine learning model can make the machine-generated character or avatar depicted in the source image smile. If extracted facial image information corresponding to a next facial image(s) indicates that the user is frowning in the next facial image, the second machine learning model can make the machine-generated character or avatar depicted in the source image stop smiling and begin frowning.

[0047] At point 910, real-time interactive video can be created. Real-time interactive video can be created by dynamically driving source images based on one or more facial expressions. For example, a second machine learning model can utilize a neuro-motor generative adversarial network (GAN) technique to dynamically drive source images based on one or more facial expressions. Dynamically driving source images based on one or more facial expressions enhances user interactivity during the content creation process and creates a more dynamic effect for the final video output.

[0048] Figure 10 The illustration shows a sample process 1000 for creating live interactive video. Although in Figure 10 The operations are depicted as a sequence of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder, or modify the depicted operations.

[0049] At point 1002, an interface can be displayed. The interface can be configured to guide the user to position their face at a predetermined location. The interface can be displayed on a user device (e.g., a client device, mobile computing device, etc.). The user device can be associated with a camera device. The interface can be further configured to instruct the user to avoid moving their face out of the predetermined location and / or to avoid changing their facial expressions. For example, an image of the user's face at the predetermined location can be captured by the camera device.

[0050] At point 1004, a source image can be generated. The source image can be generated by a first machine learning model. The source image can be generated based on a captured image of the user (e.g., the user's face). The user can utilize a camera device associated with their device to capture an image of themselves. The camera device can be a component of the user device or separate from it. The first machine learning model can generate the source image based on a scan of a face positioned at a predetermined location. The source image can depict a machine-generated character or avatar, such as a machine-generated character or avatar of the user in an image similar to the user.

[0051] At point 1006, the source image can be fed into the second machine learning model. Information extracted from one or more facial images of the user can be fed into the second machine learning model. The second machine learning model can be configured and trained to transfer the creator's facial expressions to the machine-generated images in real time. The second machine learning model can transfer facial expressions depicted in facial images to the source image. For example, the second machine learning model can use extracted facial image information to transfer facial expressions depicted in facial images to the source image.

[0052] Figure 11 The illustration shows an example process 1100 for creating live interactive video. Although in Figure 11 The operations are depicted as a sequence of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder, or modify the depicted operations.

[0053] At 1102, a source image can be generated. The source image can be generated by a first machine learning model. The source image can be generated based on an image captured of the user. The user's image may include the user's face. The user can capture an image of themselves using a camera device associated with the user's device. The camera device may be a component of the user's device or separate from the user's device. The first machine learning model can generate the source image by scanning the user's face in the user's image. The source image may depict a machine-generated character or avatar, such as a machine-generated character or avatar resembling the user in the user's image. At 1104, the source image generated by the first machine learning model can be displayed. For example, the source image can be displayed via the user's device interface.

[0054] At 1106, information configured to prompt a user to display facial expressions can be made to appear. This information can be displayed based on (e.g., in response to) the generation of a source image. The user can move his or her face and / or display one or more facial expressions, such as in response to viewing the information. For example, a user can close their mouth. One or more facial expressions can be displayed on the source image. For example, based on the user closing their mouth, a source image showing a closed mouth can be displayed. Real-time interactive video can be created by dynamically driving the source image based on one or more facial expressions.

[0055] Figure 12 The illustration shows a sample process 1200 for creating live interactive video. Although in Figure 12 The operations are depicted as a sequence of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder, or modify the depicted operations.

[0056] At point 1202, one or more facial images of the user can be captured. These facial images can depict one or more facial expressions. The one or more facial images can be captured continuously. For example, the one or more facial images can be frames of a live video image depicting the user's face. Facial images can be captured in response to prompts to make various facial expressions or facial movements, such as through an interface on a user device (e.g., a client device, mobile computing device, etc.). The user can also capture facial images using a camera device associated with the user device.

[0057] At point 1204, facial landmark data can be extracted. Facial landmark data can be extracted from one or more facial images. Facial landmark data can be extracted in real time. Facial landmark data can be extracted from each facial image in the facial images using any suitable facial landmark identification technique. At point 1206, the facial landmark data can be fed into a second machine learning model. The second machine learning model can be configured and trained to transfer the creator's facial expressions to machine-generated images, such as the source image, in real time. The second machine learning model can transfer facial expressions depicted in one or more facial images to the source image. For example, the second machine learning model can use facial landmark data to transfer facial expressions depicted in one or more facial images to the source image. The second machine learning model can display the facial expressions depicted in one or more facial images on the source image, such as in real time via a client device interface. For example, if the extracted facial landmark data corresponding to the first facial image indicates that the user is smiling in the first facial image, the second machine learning model can make the machine-generated character or avatar depicted in the source image smile. If the extracted facial landmark data corresponding to the next facial image indicates that the user is frowning in the next facial image, the second machine learning model can make the machine-generated character or avatar depicted in the source image stop smiling and start frowning.

[0058] Figure 13 The illustration shows a sample process 1300 for creating live interactive video. Although in Figure 13 The operations are depicted as a sequence of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder, or modify the depicted operations.

[0059] The second machine learning model may include a first sub-model (e.g., a neural keypoint detector), a second sub-model (e.g., an MLP neural keypoint mapping model, an MLP motion estimation model, and a CNN hourglass model), and a third sub-model (e.g., an image inpainting generator). The second machine learning model may run on and / or be installed on a user device (e.g., a client device, a mobile computing device, etc.).

[0060] The user can be prompted (e.g., via the interface of a user device, such as a client device, mobile computing device, etc.) to capture an image of his or her face. For example, the user can be prompted or guided to position his or her face at a predetermined location on the interface. The user can use a camera device associated with the user device to capture an initial user image (e.g., an image of the user's face at the predetermined location). The camera device can be a component of the user device or separate from it. The initial user image can depict the user's face. The initial user image can be used to generate a source image. For example, the initial user image can be input into a first machine learning model. The first machine learning model can generate the source image based on the initial user image. The first machine learning model can generate the source image by scanning the user's face positioned at the predetermined location. The source image can depict a machine-generated character or avatar, such as a machine-generated character or avatar similar to the user in the initial user image. The source image can be input into a second machine learning model.

[0061] A user can capture one or more additional images (e.g., multiple drive frames, drive images, etc.). Multiple drive frames can be captured sequentially. For example, multiple drive frames could be frames depicting a real-time video view of the user's face. A user can capture multiple drive frames in response to prompts to make one or more facial expressions or movements via an interface such as a user device (e.g., a client device, mobile computing device, etc.). A user can also capture multiple drive frames using a camera device associated with their user device.

[0062] The marker detection model can be configured to extract information from each of the multiple driving frames in real time and / or determine information associated with each of the multiple driving frames. This information may include facial image information, such as facial marker data. The marker detection model can use any suitable facial marker identification technique to extract and / or determine the information. The extracted facial image information can be fed into a first sub-model of a second machine learning model (e.g., a neural keypoint detector).

[0063] At position 1302, keypoints can be detected. Keypoints can indicate one or more motion fields. One or more motion fields can be associated with one or more facial expressions depicted in the driving frame. Keypoints can be detected by a first sub-model of a second machine learning model (e.g., a neural keypoint detector). Keypoints can be detected based on extracted facial image information. The neural keypoint detector can be configured to detect (e.g., determine) keypoints indicating one or more motion fields associated with one or more facial expressions depicted in the driving frame. The neural keypoint detector can be configured to detect keypoints by identifying and locating specific points of interest in the driving frame. Keypoints can provide basic information about the location, pose, and structure of a user's face or facial expressions within the driving frame.

[0064] Keypoints can be input into a second sub-model of the second machine learning model (e.g., an MLP neural keypoint mapping model). At position 1304, a deformable file can be generated. The deformable file can be generated by the second sub-model of the second machine learning model based on the keypoints. The MLP neural keypoint mapping model can generate deformable files (e.g., deformable mappings) based on the keypoints. The deformable file can indicate how to distort the source image.

[0065] At position 1306, one or more motion fields can be optimized by a second sub-model of the second machine learning model (e.g., an MLP motion estimation model and a CNN hourglass model). The occlusion map can be generated by the second sub-model of the second machine learning model. The occlusion map can indicate how to optimize, update, or improve the warped source image to more closely resemble the facial expressions and / or facial movements in the driving frames.

[0066] At point 1308, the source image can be deformed (e.g., distorted). The source image can be deformed based on a deformable file. The source image can be deformed by a second sub-model of a second machine learning model. The source image can be deformed based on a deformable file to generate a deformed / distorted source image.

[0067] At 1310, the occlusion map and the deformed source image can be input into the third sub-model of the second machine learning model (e.g., an image inpainting generator). The deformed source image can be input into the third sub-model of the second machine learning model. The third sub-model can receive the deformed source image and the occlusion map as input. At 1312, one or more result images (e.g., result image 814) can be generated. The one or more result images can depict one or more facial expressions and / or facial movements in the (multiple) driving frames (e.g., driving frame 808) on the source image (e.g., source image 806). The one or more result images can be generated by the third sub-model of the second machine learning model. The third sub-model can generate one or more result images by further optimizing the image of the deformed source image based on the occlusion map. The one or more result images generated by the third sub-model more closely resemble the (multiple) facial expressions and / or (multiple) facial movements in the (multiple) driving frames than the deformed source image. Dynamically driving source images based on multiple facial expressions and / or multiple facial movements in multiple driving frames enhances interactivity during the content creation process and creates more dynamic effects for the final video output.

[0068] Figure 14 The illustration shows that it can be used in any, such as Figures 1 to 8 The computing devices used in the various aspects such as services, networks, modules, and / or devices described herein. About Figures 1 to 8 Any or all of the components in a component can be individually manufactured by... Figure 14 This is achieved through one or more instances of the computing device 1400. Figure 14 The computer architecture shown illustrates conventional server computers, workstations, desktop computers, laptop computers, tablet computers, network devices, PDAs, e-readers, digital cellular phones, or other computing nodes, and can be used to perform any aspect of the computer described herein, such as to implement the methods described herein.

[0069] The computing device 1400 may include a substrate or “motherboard,” which is a printed circuit board on which a number of components or devices can be connected via a system bus or other electrical communication path. One or more central processing units (CPUs) 1404 may operate in conjunction with chipset 1406. The CPUs 1404 may be standard programmable processors that perform the arithmetic and logic operations necessary to perform the operation of the computing device 1400.

[0070] Multiple CPUs 1404 can perform necessary operations by manipulating switching elements that distinguish and change these states, transitioning from one discrete physical state to the next. Switching elements typically include electronic circuitry, such as flip-flops, that maintains one of two binary states, and electronic circuitry, such as logic gates, that provides an output state based on a logical combination of the states of one or more other switching elements. These basic switching elements can be combined to create more complex logic circuits, including registers, adder-subtractor units, arithmetic logic units, floating-point units, etc.

[0071] The (multiple) CPUs 1404 can be expanded or replaced by other processing units such as (multiple) GPUs 1405. The (multiple) GPUs 1405 may include processing units specifically designed for, but not limited to, highly parallel computing, such as graphics and other visualization-related processing.

[0072] Chipset 1406 can provide an interface between CPU(s) 1404 and the remaining components and devices on the substrate. Chipset 1406 can also provide an interface for random access memory (RAM) 1408, which serves as the main memory in computing device 1400. Chipset 1406 can also provide an interface for computer-readable storage media such as read-only memory (ROM) 1420 or non-volatile RAM (NVRAM) (not shown) for storing basic routines that can help boot computing device 1400 and transfer information between various components and devices. ROM 1420 or NVRAM can also store other software components necessary for the operation of computing device 1400 according to the aspects described herein.

[0073] Computing device 1400 can operate in a networked environment using logical connections to remote computing nodes and computer systems via a local area network (LAN). Chipset 1406 may include functionality for providing network connectivity via a network interface controller (NIC) 1422, such as a Gigabit Ethernet adapter. NIC 1422 may be able to connect computing device 1400 to other computing nodes via network 1416. It should be understood that multiple NICs 1422 may exist in computing device 1400, connecting the computing device to other types of networks and remote computer systems.

[0074] Computing device 1400 can be connected to mass storage device 1428, which provides non-volatile storage for the computer. Mass storage device 1428 can store system programs, application programs, other program modules, and data, as described in more detail herein. Mass storage device 1428 can be connected to computing device 1400 via storage controller 1424 connected via chipset 1406. Mass storage device 1428 may include one or more physical storage units. Mass storage device 1428 may include management component 1410. Storage controller 1424 can interface with physical storage units via Serial Attached SCSI (SAS) interface, Serial Advanced Technology Attached (SATA) interface, Fibre Channel (FC) interface, or other types of interfaces used for physically connecting and transferring data between the computer and physical storage units.

[0075] The computing device 1400 can store data on the mass storage device 1428 by changing the physical state of the physical storage units to reflect that information is being stored. The specific changes in physical state can depend on various factors and different implementations described herein. Examples of such factors may include, but are not limited to, the technology used to implement the physical storage units and whether the mass storage device 1428 is characterized as a primary storage device or a secondary storage device.

[0076] For example, computing device 1400 can store information in mass storage device 1428 by issuing instructions via storage controller 1424 to change the magnetic properties of a specific location within a disk drive unit, the reflection or refraction properties of a specific location in an optical storage unit, or the electrical properties of a specific capacitor, transistor, or other discrete component in a solid-state storage unit. Other transformations of the physical medium are possible without departing from the scope and spirit of this specification; the foregoing examples are provided merely for ease of description. Computing device 1400 can also read information from mass storage device 1428 by detecting the physical state or characteristics of one or more specific locations within a physical storage unit.

[0077] In addition to the aforementioned high-capacity storage device 1428, the computing device 1400 can access other computer-readable storage media to store and retrieve information, such as program modules, data structures, or other data. Those skilled in the art will understand that a computer-readable storage medium can be any available medium that provides storage for non-transitory data and can be accessed by the computing device 1400.

[0078] By way of example and not limitation, computer-readable storage media may include volatile and non-volatile, transient and non-transitory computer-readable storage media implemented in any method or technology, as well as removable and non-removable media. Computer-readable storage media include, but are not limited to, RAM, ROM, erasable programmable ROM (“EPROM”), electrically erasable programmable ROM (“EEPROM”), flash memory or other solid-state memory technologies, compact disc ROM (“CD-ROM”), digital versatile disc (“DVD”), high-definition DVD (“HD-DVD”), Blu-ray or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices, other magnetic storage devices, or any other medium that may be used to store desired information in a non-transitory manner.

[0079] Such as Figure 14 Mass storage devices such as the mass storage device 1428 shown can store the operating system used to control the operation of the computing device 1400. The operating system may include a version of the Linux operating system. The operating system may include a version of the Windows Server operating system from Microsoft Corporation. According to a further aspect, the operating system may include a version of the UNIX operating system. Various mobile phone operating systems, such as iOS and Android, may also be used. It should be understood that other operating systems may also be used. Mass storage device 1428 can store other systems, applications, and data used by the computing device 1400.

[0080] Mass storage device 1428 or other computer-readable storage medium may also be encoded with computer-executable instructions that, when loaded into computing device 1400, transform the computing device from a general-purpose computing system into a special-purpose computer capable of implementing the aspects described herein. As described above, these computer-executable instructions transform computing device 1400 by specifying how CPU(s)1404 transition between states. Computing device 1400 can access the computer-readable storage medium storing the computer-executable instructions, which, when executed by computing device 1400, can perform the methods described herein.

[0081] Such as Figure 14 The computing device 1400 shown may also include an input / output controller 1432 for receiving and processing input from multiple input devices such as a keyboard, mouse, touchpad, touchscreen, electronic pen, or other types of input devices. Similarly, the input / output controller 1432 may provide output to a display such as a computer monitor, flat panel display, digital projector, printer, plotter, or other types of output devices. It should be understood that the computing device 1400 may not include... Figure 14All components shown may include Figure 14 Other components not explicitly shown in the document, or those that can be utilized with Figure 14 The architecture shown is completely different.

[0082] As described in this article, a computing device can be a physical computing device, such as... Figure 14 The computing device 1400. A computing node may also include virtual machine host processes and one or more virtual machine instances. Computer-executable instructions can be indirectly executed by the physical hardware of the computing device by interpreting and / or executing instructions stored and executed in the context of the virtual machine.

[0083] It should be understood that the methods and systems are not limited to specific methods, specific components, or specific implementations. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.

[0084] As used in the specification and appended claims, the singular forms “a,” “an,” and “the” include plural indicators unless the context clearly indicates otherwise. A range may be expressed herein as from “about” a particular value and / or to “about” another particular value. When such a range is expressed, another embodiment includes from one particular value and / or to another particular value. Similarly, when a value is expressed as an approximation using the antecedent “about,” it should be understood that the particular value forms another embodiment. It should also be understood that the endpoints of each range are significant both relative to and independent of the other endpoint.

[0085] "Optional" or "optionally" means that the event or situation described below may or may not occur, and the description includes instances where the event or situation occurs and instances where it does not occur.

[0086] Throughout the description and claims of this specification, the word “comprising” and variations thereof, such as “comprising” and “including,” mean “including, but not limited to,” and are not intended to exclude, for example, other components, integers, or steps. “Exemplary” means “an example of…” and is not intended to convey indications of preferred or ideal embodiments. “Like” is not used in a limiting sense but for illustrative purposes.

[0087] Components that can be used to perform the described methods and systems are described. When describing combinations, subsets, interactions, groups, etc., of these components, it should be understood that although specific references to each of the various individual and collective combinations and arrangements of these components may not be explicitly described, each of the methods and systems is specifically considered and described herein. This applies to all aspects of this application, including but not limited to the operations in the described methods. Therefore, if various additional operations exist that can be performed, it should be understood that each of these additional operations can be performed using any particular embodiment or combination of embodiments of the described methods.

[0088] The methods and systems of the present invention can be more readily understood by referring to the following detailed description of preferred embodiments and examples included therein, as well as the accompanying drawings and their descriptions.

[0089] Those skilled in the art will understand that the methods and systems may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the methods and systems may take the form of a computer program product on a computer-readable storage medium having computer-readable program instructions (e.g., computer software) embodied within the storage medium. More specifically, the methods and systems may take the form of computer software implemented on the web. Any suitable computer-readable storage medium may be used, including hard disks, CD-ROMs, optical storage devices, or magnetic storage devices.

[0090] The following description of embodiments of methods and systems is based on block diagrams and flowcharts of methods, systems, apparatuses, and computer program products. It should be understood that each block in the block diagrams and flowcharts, as well as combinations of blocks in the block diagrams and flowcharts, can be implemented by computer program instructions. These computer program instructions can be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to create a machine, such that the instructions, which execute on the computer or other programmable data processing apparatus, create components for implementing the functions specified in one or more flowchart blocks.

[0091] These computer program instructions may also be stored in a computer-readable storage medium that can instruct a computer or other programmable data processing apparatus to operate in a particular manner, causing the instructions stored in the computer-readable storage medium to produce an article of manufacture including computer-readable instructions for implementing the functions specified in one or more flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus, thereby producing a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowchart blocks.

[0092] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Furthermore, certain methods or processing blocks may be omitted in some implementations. The methods and processes described herein are not limited to any particular order and can be executed in other suitable orders with respect to their associated blocks or states. For example, described blocks or states may be executed in an order different from that specifically described, or multiple blocks or states may be combined in a single block or state. Example blocks or states may be executed serially, in parallel, or in some other manner. Blocks or states may be added to or removed from the described example embodiments. The example systems and components described herein may be configured differently from those described. For example, elements may be added to, removed from, or rearranged from the described example embodiments compared to the described exemplary embodiments.

[0093] It should also be understood that the items are illustrated as being stored in memory or on a storage device when in use, and these items or portions thereof may be transferred between memory and other storage devices for memory management and data integrity purposes. Alternatively, in other embodiments, some or all of the software modules and / or systems may be executed in memory on another device and communicate with the illustrated computing system via inter-computer communication. Furthermore, in some embodiments, some or all of the systems and / or modules may be implemented or provided in other ways, such as at least in part as firmware and / or hardware, including, but not limited to, one or more application-specific integrated circuits (“ASICs”), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field-programmable gate arrays (“FPGAs”), complex programmable logic devices (“CPLDs”), etc. Some or all of the modules, systems, and data structures may also be stored (e.g., as software instructions or structured data) on computer-readable media, such as hard disks, memory, networks, or portable media articles, for retrieval by appropriate devices or via appropriate connections. The system, modules, and data structures can also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) over various computer-readable transmission media, including wireless and wired / cable-based media, and can take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, such computer program products can also take other forms. Therefore, the present invention can be implemented using other computer system configurations.

[0094] While methods and systems have been described in conjunction with preferred embodiments and specific examples, they are not intended to limit the scope to the particular embodiments illustrated, as the embodiments herein are intended in all respects to be illustrative rather than restrictive.

[0095] Unless otherwise expressly stated, it is not intended that any method described herein require its operations to be performed in a particular order. Therefore, no order is intended to be inferred in any way where the method claims do not actually describe the order of their operations or where the claims or description do not otherwise specify that the operations will be limited to a particular order. This applies to any possible non-expressive basis of interpretation, including: logical questions concerning the arrangement of steps or flow of operations; general meanings derived from grammatical organization or punctuation; and the number or type of embodiments described in the description.

[0096] It will be apparent to those skilled in the art that various modifications and variations can be made without departing from the scope or spirit of this disclosure. Other embodiments will be apparent to those skilled in the art in light of the description and practice described herein. The description and example figures are to be considered exemplary only, and their true scope and spirit are indicated by the appended claims.

Claims

1. A method for creating real-time interactive video, comprising: A first machine learning model generates a source image based on an image captured of the user, wherein the image includes the user's face; Capture one or more facial images of the user, wherein the one or more facial images depict one or more facial expressions; The source image and information extracted from the one or more facial images are input into a second machine learning model, wherein the second machine learning model is configured and trained to transfer the creator's facial expressions to the machine-generated image in real time; Display the one or more facial expressions on the source image; as well as The real-time interactive video is created by dynamically driving the source image based on one or more facial expressions.

2. The method according to claim 1, further comprising: The display interface is configured to guide the user to position the face at a predetermined location. as well as The source image is generated by the first machine learning model based on a scan of the face located at the predetermined position.

3. The method according to claim 1, further comprising: Display the source image generated by the first machine learning model; as well as The display is configured to prompt the user to show information about facial expressions.

4. The method according to claim 1, further comprising: Extract facial landmark data from the one or more facial images; as well as The facial landmark data is then input into the second machine learning model.

5. The method according to claim 4, further comprising: The first sub-model of the second machine learning model detects key points indicating one or more motion fields, which are associated with the one or more facial expressions.

6. The method of claim 5, further comprising: The second sub-model of the second machine learning model generates the deformed file based on the key points; as well as The second sub-model of the second machine learning model optimizes the one or more motion fields and generates an occlusion map.

7. The method according to claim 6, further comprising: The second sub-model of the second machine learning model deforms the source image based on the deformed file.

8. The method of claim 7, further comprising: The source image with the occlusion mapping and deformation is input into the third sub-model of the second machine learning model; as well as One or more images, including the one or more facial expressions on the source image, are generated by the third sub-model of the second machine learning model.

9. The method of claim 1, wherein the method is implemented by a mobile computing device.

10. A system for creating real-time interactive video, comprising: At least one processor; as well as At least one memory, communicatively coupled to the at least one processor, and including computer-readable instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, the operations including: A first machine learning model generates a source image based on an image captured of the user, wherein the image includes the user's face; Capture one or more facial images of the user, wherein the one or more facial images depict one or more facial expressions; The source image and information extracted from the one or more facial images are input into a second machine learning model, wherein the second machine learning model is configured and trained to transfer the creator's facial expressions to the machine-generated image in real time; Display the one or more facial expressions on the source image; and The real-time interactive video is created by dynamically driving the source image based on one or more facial expressions.

11. The system of claim 10, further comprising: The display interface is configured to guide the user to position the face at a predetermined location. as well as The source image is generated by the first machine learning model based on a scan of the face located at the predetermined position.

12. The system of claim 10, further comprising: Display the source image generated by the first machine learning model; as well as The display is configured to prompt the user to show information about facial expressions.

13. The system of claim 10, further comprising: Extract facial landmark data from the one or more facial images; as well as The facial landmark data is then input into the second machine learning model.

14. The system according to claim 13, wherein the operation further comprises: The first sub-model of the second machine learning model detects key points indicating one or more motion fields, which are associated with the one or more facial expressions.

15. The system of claim 14, further comprising: The second sub-model of the second machine learning model generates the deformed file based on the key points; The second sub-model of the second machine learning model deforms the source image based on the deformable file; as well as The second sub-model of the second machine learning model optimizes the one or more motion fields and generates an occlusion map.

16. The system of claim 15, further comprising: The source image with the occlusion mapping and deformation is input into the third sub-model of the second machine learning model; as well as One or more images, including the one or more facial expressions on the source image, are generated by the third sub-model of the second machine learning model.

17. A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a processor, cause the processor to perform operations, the operations including: A first machine learning model generates a source image based on an image captured of the user, wherein the image includes the user's face; Capture one or more facial images of the user, wherein the one or more facial images depict one or more facial expressions; The source image and information extracted from the one or more facial images are input into a second machine learning model, wherein the second machine learning model is configured and trained to transfer the creator's facial expressions to the machine-generated image in real time; Display the one or more facial expressions on the source image; as well as The real-time interactive video is created by dynamically driving the source image based on one or more facial expressions.

18. The non-transitory computer-readable storage medium of claim 17, further comprising: Extract facial landmark data from the one or more facial images; as well as The facial landmark data is then input into the second machine learning model.

19. The non-transitory computer-readable storage medium of claim 18, further comprising: Key points indicating one or more motion fields are detected by a first sub-model of the second machine learning model, the one or more motion fields being associated with the one or more facial expressions; The second sub-model of the second machine learning model generates the deformed file based on the key points; The second sub-model of the second machine learning model deforms the source image based on the deformable file; as well as The second sub-model of the second machine learning model optimizes the one or more motion fields and generates an occlusion map.

20. The non-transitory computer-readable storage medium of claim 19, further comprising: The source image with the occlusion mapping and deformation is input into the third sub-model of the second machine learning model; as well as One or more images, including the one or more facial expressions on the source image, are generated by the third sub-model of the second machine learning model.

Images