Method and apparatus for presenting avatar in video, and device and medium

Abstract

Provided are a method and apparatus for presenting an avatar in a video, and a device and a medium. The method comprises: acquiring a first video for specifying an action sequence; receiving a user interaction for a second video, wherein the second video comprises a physical scene, and the user interaction is used for specifying a target location, for placing an avatar, in the physical scene; and on the basis of a third video comprising the physical scene, and the action sequence, the avatar, and the target location, generating a fourth video comprising the physical scene and the avatar, wherein the avatar is located at the target location and executes the action sequence. By using the exemplary implementation mode of the present disclosure, a video comprising an avatar can be obtained in a simpler and more effective mode. In this way, a user can conveniently take a group photo with the avatar, and then a video comprising richer visual content is obtained.

Description

Method, device, apparatus and medium for presenting virtual image in video

[0001] This application claims priority to the Chinese invention patent application entitled “Methods, devices, equipment and media for presenting virtual images in videos” and application number 2023112668431, filed on September 27, 2023. The entire contents of that application are incorporated herein by reference. Technical Field

[0002] Exemplary implementations of the present disclosure generally relate to media data processing, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for presenting virtual avatars in videos. Background Art

[0003] Three-dimensional graphics processing technology has been widely used in various fields. For example, three-dimensional graphics processing technology can be used to generate images or videos of three-dimensional characters. Furthermore, three-dimensional graphics processing technology can be combined with video technology to provide augmented reality services. When users take photos and videos, they may want to take a photo with a virtual image (for example, a movie character, a comic character, etc.). Although the function of adding "stickers" to photos has been developed, the virtual image and its actions in the stickers are predetermined, and the actions of the virtual image cannot be specified. At this time, it is expected that media data can be processed in a more convenient and effective way, and the desired virtual image can be added to the media data.

[0004] Summary of the Invention

[0005] In a first aspect of the present disclosure, a method for presenting an avatar in a video is provided. In this method, a first video is obtained that specifies an action sequence. A user interaction is received with respect to a second video that includes a physical scene and specifies a target location for placing the avatar within the physical scene. Based on a third video that includes the physical scene, the action sequence, the avatar, and the target location, a fourth video is generated that includes the physical scene and the avatar, with the avatar positioned at the target location and performing the action sequence.

[0006] In a second aspect of the present disclosure, a device for presenting an avatar in a video is provided. The device includes: an acquisition module configured to acquire a first video for specifying an action sequence; a receiving module configured to receive a user interaction with a second video, the second video including a physical scene and the user interaction specifying a target location for placing the avatar in the physical scene; and a generation module configured to generate a fourth video including the physical scene and the avatar based on a third video including the physical scene, the action sequence, the avatar, and the target location, the avatar being located at the target location and performing the action sequence.

[0007] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform the method according to the first aspect of the present disclosure.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor implements the method according to the first aspect of the present disclosure.

[0009] It should be understood that the content described in this summary section is not intended to limit the key features or important features of the implementation of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of various implementations of the present disclosure will become more apparent hereinafter with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0011] FIG1 shows a block diagram of an application environment according to an exemplary implementation of the present disclosure;

[0012] FIG2 illustrates a block diagram for presenting an avatar in a video according to some implementations of the present disclosure;

[0013] FIG3 illustrates a block diagram for extracting an action sequence from a first video according to some implementations of the present disclosure;

[0014] FIG4 illustrates a block diagram of key point data of an avatar according to some implementations of the present disclosure;

[0015] FIG5 shows a block diagram for specifying a state for placing the virtual avatar in the physical scene according to some implementations of the present disclosure;

[0016] FIG6 illustrates a block diagram for generating new video frames based on frames in a third video and extracted actions according to some implementations of the present disclosure;

[0017] FIG7 illustrates a block diagram for generating video frames in a fourth video according to some implementations of the present disclosure;

[0018] FIG8 illustrates a block diagram of a fourth video according to some implementations of the present disclosure;

[0019] FIG9 illustrates a block diagram of generating video frames based on a field of view of an acquisition device according to some implementations of the present disclosure;

[0020] FIG10 illustrates a block diagram of generating a video frame based on a field of view of an acquisition device according to some implementations of the present disclosure;

[0021] FIG11 shows a flowchart of a method for presenting a virtual image in a video according to some implementations of the present disclosure;

[0022] FIG12 shows a block diagram of an apparatus for presenting a virtual image in a video according to some implementations of the present disclosure; and

[0023] FIG13 illustrates a block diagram of a device capable of implementing various implementations of the present disclosure. DETAILED DESCRIPTION

[0024] The following describes implementations of the present disclosure in more detail with reference to the accompanying drawings. Although certain implementations of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the implementations described herein. Rather, these implementations are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and implementations of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0025] In the description of the implementation of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "an implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". The following may also include other explicit and implicit definitions. As used herein, the term "model" can represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on a variety of technical solutions currently known and / or to be developed in the future.

[0026] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0027] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0028] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0029] As an optional but non-limiting implementation, in response to receiving a user's active request, a prompt message may be sent to the user, for example, in the form of a pop-up window, in which the prompt message may be presented in text form. Furthermore, the pop-up window may also include a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0030] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0031] As used herein, the term "in response to" refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of executing a subsequent action executed in response to the event or condition is not necessarily strongly correlated with the time when the event occurs or the condition is satisfied. For example, in some cases, a subsequent action may be executed immediately upon the occurrence of the event or the satisfaction of the condition; in other cases, the subsequent action may be executed some time after the occurrence of the event or the satisfaction of the condition.

[0032] Sample Environment

[0033] When taking photos or videos, users may wish to take a photo with a virtual avatar (e.g., a movie character, comic book character, etc.). In this case, the user can take a photo with a sculpture or puppet representing the virtual avatar. Figure 1 shows a block diagram 100 of an application environment according to an exemplary implementation of the present disclosure. As shown in Figure 1, a user can take a photo with a sculpture or puppet representing the virtual avatar. In this case, media data 110 may include a virtual avatar 120 and a user image 130.

[0034] With the advancement of image processing technology, the ability to add "stickers" to photos has been developed. Users can now select a desired "sticker" within an image processing tool and add it to a captured photo. However, the avatars and their actions within the stickers are predetermined, and the avatar's actions cannot be specified. There is a desire to process media data in a more convenient and efficient manner, allowing the addition of desired avatars to the media data.

[0035] Presenting an overview of the avatar

[0036] In order to at least partially address the deficiencies in the prior art, according to an exemplary implementation of the present disclosure, a method for presenting a virtual image in a video is proposed. In summary, a user can specify a desired virtual image. For example, a user can specify to take a photo with a virtual character "robot". At this time, a three-dimensional model of the robot can be generated based on artificial intelligence generated content (AIGC). The user can specify the sequence of actions performed by the "robot" and its position in the physical scene. Then, a video can be generated, which can include the physical scene of the virtual image of the "robot". For example, the user can be in the physical scene and appear in the video at the same time as the "robot".

[0037] An overview of an exemplary implementation of the present disclosure is provided with reference to FIG2 , which illustrates a block diagram 200 for presenting an avatar in a video according to some implementations of the present disclosure. As shown in FIG2 , a first video 210 may be acquired for specifying an action sequence. Here, the action sequence may specify the actions that the avatar is expected to perform. For example, the first video 210 may be acquired through live filming, reading pre-filmed video, and / or other methods.

[0038] For ease of description, the process of presenting an avatar will be described using a scenario involving two users as an example. For example, user A (e.g., a photographer) may hold a client device with capture functionality and film another user B performing an action sequence 212 (e.g., dancing) to record a first video 210. Furthermore, user A may continue to film a second video 220 within the physical scene, and may perform user interaction 224 with respect to the second video 220. For example, user A may specify a target location 222 within the physical scene where they wish to place the avatar by clicking, selecting, or otherwise. The second video 220 and the specified target location 222 may then be received.

[0039] User A can continue filming in physical scene 232. For ease of description, the resulting video will be referred to as third video 230. It should be understood that physical scene 232 includes the physical environment surrounding user A and may or may not include other people (as indicated by the dashed box). For example, physical scene 232 may or may not include user B. At this point, a fourth video 240 can be generated, including physical scene 232 and avatar 250, based on third video 230 including the physical scene, action sequence 212 extracted from first video 210, the action sequence 212, and the target location of the action sequence. At this point, avatar 250 is located at target location 222 and performing action sequence 212. In other words, avatar 250 in fourth video 240 can dance, and user B can perform any desired actions, such as dancing, moving around in physical scene 232, or interacting with avatar 250.

[0040] It should be understood that while the above description of the video generation process uses a scenario with two users, alternatively and / or additionally, there may be only one user. For example, a user may load and / or receive a pre-recorded first video 210, and may use a tripod to shoot a third video 230. For another example, the user may use a front-facing camera to shoot the third video 230. In this way, videos including avatars can be generated in a variety of application scenarios.

[0041] Using the exemplary implementations of this disclosure, videos featuring avatars can be generated in a simpler and more efficient manner. Specifically, the desired avatar can be directly generated using AIGC technology, without the need for pre-fabricated sculptures or dolls. Furthermore, a sequence of actions for the avatar can be specified, resulting in a video of the avatar performing those actions in a physical scene. In this way, users can conveniently take photos with the avatar, resulting in videos with richer visual content.

[0042] Detailed process of presenting virtual images

[0043] Having described an overview of presenting an avatar in a video, further details regarding presenting an avatar in a video are described below. According to an exemplary implementation of the present disclosure, a first video 210 may be filmed live. For example, user A may film a series of actions performed by another user. Alternatively and / or additionally, a pre-filmed video including an action sequence 212 may be received. For example, a video including a popular dance clip may be read from a client and / or downloaded from a network. In this manner, a user may be allowed to pre-specify the actions they wish the avatar to perform, thereby generating a video including rich content.

[0044] According to an exemplary implementation of the present disclosure, a user may be allowed to edit first video 210. For example, a video segment may be selected from first video 210, such as a video segment of a predetermined length or within a predetermined time period. Upon receiving an edit interaction with respect to first video 210, the first video may be updated based on the edit interaction. Furthermore, a corresponding action sequence may be extracted from the updated first video 210.

[0045] FIG3 illustrates a block diagram 300 for extracting an action sequence 212 from a first video 210, according to some implementations of the present disclosure. As shown in FIG3 , each video frame in the first video 210 can be processed one by one, and corresponding actions can be extracted from each video frame. Assuming that the first video 210 includes N video frames 310, ..., 320, ..., 330, corresponding N actions 312, ..., 322, ..., 332 can be extracted, respectively.

[0046] According to an example implementation of the present disclosure, various actions can be extracted based on key point detection technology. Figure 4 shows a block diagram 400 of key point data of a virtual image according to some implementations of the present disclosure. As shown in Figure 4, key points in various parts of the human body can be predefined. Specifically, referring to the key point data 410, the central position of the human torso (such as numbered 0) can be defined as the 0th key point; the positions of the hip joints (such as numbers 1 and 2) can be defined as the 1st and 2nd key points respectively; the waist position (such as numbered 3) can be defined as the 3rd key point, ..., and the right hand palm position is defined as the 23rd key point. For ease of illustration, Figure 4 only schematically shows a part of the 24 key points. It should be understood that more or fewer key points can be defined to describe human body movements. In this way, the positions of key points can be used to represent human body movements.

[0047] According to an example implementation of the present disclosure, each video frame in the first video 210 can be processed based on key point recognition technology. At this time, the processing result can be output, for example, using a 3+24*4 dimensional vector to represent the processing result. In the vector, the first 3 dimensions can represent the displacement of the virtual image (that is, the displacement relative to the position of the virtual image in the first frame). For example, the spatial coordinate system can be defined as the X-axis is in the right direction, the Y-axis is in the upward direction, and the Z-axis is in the right direction. Further, the subsequent 24*4 dimensions can represent the quaternion representation of the relevant rotation of the 24 key points, and the X-axis of the coordinate system where the quaternion is located is in the left direction, the Y-axis is in the upward direction, and the Z-axis is in the forward direction. In the example of Figure 3, N actions can be extracted, and at this time, N vectors can be obtained, and each vector includes 3+24*4 dimensions.

[0048] After successfully extracting the action from each frame, the extracted action can be used to drive the three-dimensional model of the virtual image. It should be understood that the three-dimensional model of the virtual image here can be a pre-built three-dimensional model. Alternatively and / or additionally, it can also be a three-dimensional model generated by AIGC based on the virtual image specified by the user. For example, the user can enter the name of a famous character and obtain the corresponding three-dimensional model. For example, AIGC can generate a corresponding three-dimensional model based on the user input of "robot". Furthermore, the user can make adjustments to the obtained three-dimensional model. For example, the user can specify: the robot has two blue antennas on the top of the head, etc. At this time, AIGC will adjust the generated model and add antennas on the top of the model's head.

[0049] Furthermore, the extracted action can be used to drive a three-dimensional model in a T-pose as shown in FIG4 . In the process of driving the three-dimensional model, the key points in the three-dimensional model can be bound according to the key point sequence as shown in FIG4 , and then the extracted N vectors can be used to respectively determine the positions of the key points of the three-dimensional model during the movement. According to an example implementation of the present disclosure, the texture of the surface of the three-dimensional model can be determined using a texture mapping algorithm that is currently available and / or to be developed in the future. For example, the above process can be performed based on a skinmesh algorithm. At this point, the three-dimensional model of the virtual image can be driven to perform the extracted action. The obtained data can be stored in the memory of the client device for subsequent use.

[0050] In the context of the present disclosure, the user can specify where to place the virtual image. According to an example implementation of the present disclosure, the user can be allowed to perform user interaction 224 in the second video 220 to specify a target position 222 for placing the virtual image. Figure 5 shows a block diagram 500 for specifying a state for placing the virtual image in the physical scene according to some implementations of the present disclosure. As shown in Figure 5, the user can specify a plane in the second video 220 by operations such as clicking and / or selecting. For example, it is possible to specify that the virtual image be placed on the ground in the physical scene. Further, a specific location on the ground can be specified. For example, a certain floor tile on the ground can be specified by operations such as box selection to place the virtual image.

[0051] According to an example implementation of the present disclosure, additional controls may be provided to the user for specifying further details regarding the avatar. For example, a speed parameter control 510 may be provided, which allows the speed of the action sequence to be adjusted. In this case, a speed parameter specifying the speed of the avatar's movement may be received, and a target action may be determined from the action sequence based on the speed parameter. Specifically, the speed may be accelerated to generate fast motion, decelerated to generate slow motion, and so on.

[0052] According to an example implementation of the present disclosure, a control 520 for a scaling parameter may be provided, and the dimensions (i.e., size) of the virtual image may be adjusted through the control 520. At this time, a scaling parameter for specifying the dimensions of the virtual image may be received, and the dimensions of the virtual image may be updated based on the scaling parameter. For example, the virtual image may be placed on the ground of the physical scene, and the dimensions of the virtual image may be set to normal dimensions. At this time, in the generated video, the virtual image may stand on the ground and dance. For another example, the virtual image may be placed on the desktop of the physical scene, and the dimensions of the virtual image may be set to smaller dimensions. At this time, in the generated video, the virtual image may stand on the desktop and dance, and so on. In this way, a richer visual effect may be generated.

[0053] According to an exemplary implementation of the present disclosure, a rotation parameter control 530 may be provided, through which the rotation of the avatar may be adjusted. In this case, a rotation parameter specifying the rotation of the avatar may be received, and the rotation angle of the avatar may be updated based on the rotation parameter. For example, the avatar may be placed on the ceiling, rotated 180 degrees, and so on. In this case, the generated video may show the avatar dancing upside down on the ceiling, and so on.

[0054] According to an exemplary implementation of the present disclosure, a mirroring parameter 540 may be provided, which allows adjustment of whether the avatar's left and right sides are mirrored. In this case, a mirroring setting specifying a flip of the avatar may be received, and the avatar representation frame may be updated based on the mirroring of the avatar representation frame. If mirroring parameter 540 is set to "on," the avatar will be displayed in a mirrored manner in the generated video.

[0055] According to an example implementation of the present disclosure, the first video can be parsed using the method described above to extract the action sequence, i.e., to obtain N vectors. Furthermore, an avatar representation of the avatar can be generated based on the action sequence. Specifically, the three-dimensional model can be projected into a two-dimensional space, and the avatar can be formed in each frame. Furthermore, the avatar representation can be combined with the third video to generate a fourth video.

[0056] According to an example implementation of the present disclosure, each video frame in the third video 230 can be processed one by one, and then the virtual image performing the target action can be placed at the corresponding target position in each video frame. Figure 6 shows a block diagram 600 for generating new video frames based on the frames in the third video and the extracted actions according to some implementations of the present disclosure. Referring to Figure 6, the third video 230 may include multiple video frames 610, ..., 620, ..., 630. The third video 230 may have a predetermined length, and alternatively and / or additionally, a fourth video may be generated in real time, in which case the length of the third video 230 is uncertain and continues to increase as the shooting progresses.

[0057] Each video frame in the third video 230 can be processed in a similar manner. For example, for a particular video frame (e.g., referred to as a target frame), a target action can be determined from the action sequence based on the target frame's offset in the third video. Subsequently, an avatar representation frame corresponding to the target frame can be generated in the avatar representation based on the keypoint matrix associated with the target action and the initial image of the avatar (e.g., the 3D model in the T-pose described above).

[0058] 6 , for the first frame (video frame 610) in the third video 230, the first action (action 312) in the action sequence 212 can be used to drive the 3D model of the avatar, thereby overlaying the 3D model onto the video frame 610. For the i-th frame (video frame 620) in the third video 230, the i-th action (action 322) in the action sequence 212 can be used to drive the 3D model of the avatar, thereby overlaying the 3D model onto the video frame 620.

[0059] It should be understood that the above example illustrates the case where the speed parameter 510 is set to 1. Alternatively and / or additionally, when the speed parameter 510 is set to other values, the actions used to drive the three-dimensional model can be adjusted accordingly. Assuming that the speed parameter is set to M, the (i mod M)th action can be used to drive the three-dimensional model (mod represents a modulo operation), which is then superimposed with the i-th video frame in the third video 230 to generate the i-th video frame in the fourth video 240.

[0060] According to an exemplary implementation of the present disclosure, if the duration of the third video 230 is greater than the duration of the action sequence, the fourth video 240 may be generated based on a loop of the action sequence. For example, assuming the length of the action sequence is N, then for the i-th video frame in the third video 230, the (i mod N)-th action may be used to drive the 3D model, thereby generating the i-th video frame in the fourth video 240.

[0061] FIG7 illustrates a block diagram 700 for generating video frames in a fourth video, according to some implementations of the present disclosure. As shown in FIG7 , when generating corresponding video frames 720 in the fourth video using a video frame (e.g., target frame 520) in the third video, the corresponding action 322 can be extracted from the action sequence 212, and an avatar representation frame 710 can be generated based on the vector of the action. Furthermore, the avatar representation 710 can be superimposed on the target frame 520, thereby generating corresponding video frames 720 in the fourth video.

[0062] According to an exemplary implementation of the present disclosure, during the generation of an avatar representation frame, the avatar's displacement can be determined based on the keypoint matrix and the avatar's initial image. Furthermore, the avatar's physical position in the physical scene can be determined based on the target position and the displacement, and an avatar representation frame corresponding to the physical position can be generated.

[0063] Specifically, information such as acceleration data and angular velocity data of the client device can be obtained to determine the posture of the client device. Specifically, a tracking module can be used to determine the position for placing the virtual image in each frame. For example, a tracking module can be constructed based on a Simultaneous Localization and Mapping (SLAM) algorithm to detect the position for placing the virtual image. For example, the target position 222 specified by the user can be input into the SLAM algorithm, and the corresponding position of the target position 222 in each frame of the third video 230 can be tracked. Furthermore, the generated three-dimensional model of the virtual image can be placed at the corresponding position in each frame.

[0064] At this time, the tracking module can output the corresponding Model View Projection (MVP) matrix for each video frame input. Here, the M matrix can map the vertex coordinates in the three-dimensional model from the model space to the world space of the physical scene, the V matrix can map the coordinates from the world space to the visual space (that is, the spatial coordinate system centered on the acquisition device), and the P matrix can map the coordinates from the visual space to the acquisition space (that is, if the vertex is not within the visible range, it does not need to be processed). Then, the model can be driven according to the MVP matrix and the key point data of the three-dimensional model obtained in the above manner. In this way, a virtual image can be presented in the video based on a simple mathematical operation process.

[0065] For example, the avatar's displacement can be determined based on the first three dimensions of the action vector, and then the positions of key points in the avatar can be determined based on the 24*4 dimensions. This allows the avatar's displacement relative to the initial action to be determined. Furthermore, the avatar can be mapped to the world space of the physical scene using the MVP matrix.

[0066] According to an example implementation of the present disclosure, each video frame in the fourth video can be generated based on the method described above. At this time, the virtual object in the fourth video can perform a pre-specified action sequence 212. It should be understood that the physical scene may include a variety of objects, and the objects here include at least any one of static objects and dynamic objects. For example, static objects may include environmental objects in the physical scene, such as buildings, tables, chairs, etc., and dynamic objects may include people, animals, and other moving objects such as vehicles. Assuming that the physical scene is a park, the generated fourth video 240 may include a virtual image, an image of the user taking a photo with the virtual image, and buildings and / or other tourists in the park, etc.

[0067] FIG8 illustrates a block diagram 800 of a fourth video according to some implementations of the present disclosure. As shown in FIG8 , the fourth video 240 may include a physical scene 232, an avatar 250 performing an action sequence 212, and may further include a user 820 performing any action sequence 810. It should be understood that the action sequence 810 may be the same as or different from the action sequence 212. For example, the avatar 250 may perform a dance move, and the user 820 may perform other actions, such as looking toward the location of the avatar 250 and clapping. In this case, the fourth video 240 will present a scene in which the user 820 watches the avatar 250 dance and applauds.

[0068] It should be understood that the position and orientation of the capture device of the third video may change during the capture process. According to an exemplary implementation of the present disclosure, when generating a virtual avatar representation frame corresponding to a physical location, the virtual avatar representation frame can be generated only when the physical location is determined to be within the field of view of the capture device of the third video. In this way, relevant mathematical operations are performed only when necessary, thereby reducing various resource overheads.

[0069] Figure 9 illustrates a block diagram 900 for generating video frames based on a capture device's field of view, according to some implementations of the present disclosure. Figure 9 shows a field of view 920 of a capture device 910 for a third video 230. Assuming that action sequence 212 represents dancing in place, the avatar will dance in place at target location 222. As shown in Figure 9, target location 222 is within field of view 920. Therefore, the avatar's corresponding physical location is also within field of view 920, and a avatar representation frame can be generated. Consequently, in the generated video frame 930, the avatar will dance in place at target location 222.

[0070] FIG10 illustrates a block diagram 1000 for generating video frames based on a capture device's field of view, according to some implementations of the present disclosure. As shown in FIG10 , assuming that the capture device 910's field of view 1010 moves to the position shown in FIG10 , where the target location is outside of field of view 1010, the corresponding physical location of the avatar is also outside of field of view 1010, and no avatar representation frame is generated. Consequently, in the generated video frame 1020, the avatar is not present, and only the physical scene 232 may be present. Alternatively and / or additionally, the video frame 1020 may include a person in the physical scene 232.

[0071] It should be understood that, for ease of description, Figures 9 and 10 only illustrate the case where action sequence 212 represents dancing in place. Alternatively and / or additionally, action sequence 212 may also represent a walking motion. In this case, target position 222 does not overlap with the physical position of the avatar, and subsequent steps should be performed based on whether the physical position of the avatar is within the field of view.

[0072] According to an example implementation of the present disclosure, the presentation position of the virtual object in the fourth video changes as the field of view of the capture device of the third video changes. For example, as the position and orientation of the capture device change, the field of view of the capture device will change. Compared to a virtual image that only presents fixed actions at a fixed position, the virtual object in the present disclosure can appear in different positions in the fourth video according to the definition of the action sequence and the changes in the field of view of the capture device. For example, users who wish to film together can interact with the virtual image, etc., to generate richer video content.

[0073] It should be understood that although the above description uses the example of placing the virtual image at a specified position on the ground without obstructions to describe the process of presenting the virtual image, in some cases, the virtual image may be blocked by certain objects in the physical scene. In this case, in the generated fourth video 240, there may be an occlusion relationship between the virtual image and other objects in the physical scene. For example, assuming that there is a table in front of the target position, a part of the virtual image behind the table (for example, the legs) may be blocked by the table. In this way, the generated virtual image can interact with other objects in the physical scene, thereby providing a richer visual expression.

[0074] According to an example implementation of the present disclosure, the spatial position of each object in a physical scene can be determined based on object positioning technology. For example, the world coordinate system of the space where the physical scene is located can be used to represent the spatial position. Furthermore, the three-dimensional model of the virtual image can be driven according to the method described above to determine the physical position of the three-dimensional model of the virtual image in the physical scene at a specific point in time. The spatial position of the object can be compared with the physical position of the three-dimensional model to determine whether there is an occlusion relationship between the virtual image and the object. If there is an occlusion relationship, a virtual image representation frame can be generated according to the occlusion relationship. At this time, only the unobstructed part of the virtual image can be presented.

[0075] It should be understood that although the above description uses a static ground surface as an example of the target location to describe the process of placing the avatar on the static ground surface, alternatively and / or additionally, the target location specified by the user can be located at a dynamic object in the physical scene. For example, the target location can be located on the roof of a slowly moving vehicle in the physical scene. In this case, the target location will slowly move with the movement of the vehicle. In this case, during the process of generating the fourth video, the tracking module will continuously track the position of the roof in each frame of the third video, and then place the avatar on the slowly moving roof. In this case, in the generated fourth video, the avatar will dance on the slowly moving roof.

[0076] For another example, if the user specifies a forward-facing acquisition device, they can set the target location to their shoulders and reduce the avatar's dimensions to 1 / 10 of its normal size. In this case, in the generated fourth video, the avatar can dance at the user's shoulders in response to the user's movements. This exemplary implementation of the present disclosure can generate richer visual effects.

[0077] Using the exemplary implementations of the present disclosure, videos featuring avatars can be obtained in a simpler and more efficient manner. Furthermore, a sequence of actions for the avatar can be specified, resulting in a video of the avatar performing those actions in a physical scene. In this way, users can conveniently take photos with the avatar, resulting in videos with richer visual content.

[0078] Example Process

[0079] FIG11 illustrates a flow chart of a method 1100 for presenting an avatar in a video, according to some implementations of the present disclosure. At block 1110, a first video specifying an action sequence is acquired. At block 1120, a user interaction is received with respect to a second video, the second video including a physical scene and the user interaction specifying a target location within the physical scene for placing the avatar. At block 1130, a fourth video including the physical scene and the avatar is generated based on a third video including the physical scene, the action sequence, the avatar, and the target location, with the avatar positioned at the target location and performing the action sequence.

[0080] According to an example implementation of the present disclosure, generating a fourth video includes: parsing the first video to extract an action sequence; generating an avatar representation of the avatar according to the action sequence; and combining the avatar representation and the third video to generate the fourth video.

[0081] According to an example implementation of the present disclosure, generating a virtual image representation includes: for a target frame in a third video, determining a target action from an action sequence based on an offset of the target frame in the third video; and generating a virtual image representation frame in the virtual image representation corresponding to the target frame based on a key point matrix associated with the target action and an initial image of the virtual image.

[0082] According to an example implementation of the present disclosure, generating a virtual image representation frame includes: determining the displacement of the virtual image based on a key point matrix and the initial image of the virtual image; determining the physical position of the virtual image in the physical scene based on the target position and the displacement; and generating a virtual image representation frame corresponding to the physical position.

[0083] According to an example implementation of the present disclosure, generating a virtual avatar representation frame corresponding to the physical location further includes: generating the virtual avatar representation frame in response to determining that the physical location is within the field of view of the capture device of the third video.

[0084] According to an example implementation of the present disclosure, there is an occlusion relationship between the virtual image and the objects in the physical scene.

[0085] According to an example implementation of the present disclosure, the method 1100 further includes: determining the spatial position of an object in a physical scene; determining an occlusion relationship between the virtual image and the object based on the spatial position and the physical position of the virtual image in the physical scene; and generating a virtual image representation frame according to the occlusion relationship.

[0086] According to an example implementation of the present disclosure, the method 1100 further includes: receiving a speed parameter for specifying a movement speed of the avatar; and determining a target action from the action sequence based on the speed parameter.

[0087] According to an example implementation of the present disclosure, the method 1100 further includes: receiving a scaling parameter for specifying dimensions of the avatar; and updating the dimensions of the avatar based on the scaling parameter.

[0088] According to an example implementation of the present disclosure, the method 1100 further includes: receiving a rotation parameter for specifying the rotation of the avatar; and updating the rotation angle of the avatar based on the rotation parameter.

[0089] According to an example implementation of the present disclosure, the method 1100 further includes: receiving a mirror setting for specifying a flip of the avatar; and updating the avatar representation frame based on the mirroring of the avatar representation frame.

[0090] According to an example implementation of the present disclosure, the method 1100 further includes: in response to receiving an editing interaction for the first video, updating the first video based on the editing interaction.

[0091] According to an example implementation of the present disclosure, the method 1100 further includes: in response to determining that the time length of the third video is greater than the time length of the action sequence, generating a fourth video based on a loop of the action sequence.

[0092] According to an example implementation of the present disclosure, in the method 1100 , the physical scene includes objects, the objects include at least any one of static objects and dynamic objects, and the dynamic objects include people, animals, and moving objects.

[0093] According to an example implementation of the present disclosure, the method 1100 further includes: in response to determining that the target position is located at the dynamic object, moving the target position along with the movement of the dynamic object.

[0094] According to an example implementation of the present disclosure, the presentation position of the virtual object in the fourth video changes as the field of view of the acquisition device of the third video changes.

[0095] Example devices and equipment

[0096] FIG12 shows a block diagram of an apparatus 1200 for presenting an avatar in a video according to some implementations of the present disclosure. The apparatus 1200 includes: an acquisition module 1210 configured to acquire a first video for specifying an action sequence; a receiving module 1220 configured to receive a user interaction with respect to a second video, the second video including a physical scene and the user interaction specifying a target location for placing the avatar in the physical scene; and a generation module 1230 configured to generate a fourth video including the physical scene and the avatar based on a third video including the physical scene, the action sequence, the avatar, and the target location, wherein the avatar is located at the target location and performs the action sequence.

[0097] According to an example implementation of the present disclosure, the generation module includes: a parsing module configured to parse a first video to extract an action sequence; an action-based generation module configured to generate a virtual image representation of a virtual image according to the action sequence; and a combination module configured to combine the virtual image representation and the third video to generate a fourth video.

[0098] According to an example implementation of the present disclosure, the action-based generation module includes: an action determination module, configured to determine, for a target frame in a third video, a target action from an action sequence based on an offset of the target frame in the third video; and a frame generation module, configured to generate a virtual image representation frame in the virtual image representation corresponding to the target frame based on a key point matrix associated with the target action and an initial image of the virtual image.

[0099] According to an example implementation of the present disclosure, the frame generation module includes: a displacement determination module, configured to determine the displacement of the virtual image based on the key point matrix and the initial image of the virtual image; a physical position determination module, configured to determine the physical position of the virtual image in the physical scene based on the target position and the displacement; and a position-based frame generation module, configured to generate a virtual image representation frame corresponding to the physical position.

[0100] According to an example implementation of the present disclosure, the position-based frame generation module includes: a field of view-based generation module configured to generate a virtual image representation frame in response to determining that the physical position is within the field of view of the capture device of the third video.

[0101] According to an example implementation of the present disclosure, there is an occlusion relationship between the virtual image and the objects in the physical scene.

[0102] According to an example implementation of the present disclosure, the device 1200 further includes: a spatial position determination module, configured to determine the spatial position of an object in a physical scene; an occlusion relationship determination module, configured to determine the occlusion relationship between the virtual image and the object based on the spatial position and the physical position of the virtual image in the physical scene; and a generation module based on the occlusion relationship, configured to generate a virtual image representation frame according to the occlusion relationship.

[0103] According to an example implementation of the present disclosure, the device 1200 further includes: a speed parameter receiving module, configured to receive speed parameters for specifying the movement speed of the virtual image; and a speed parameter-based determination module, configured to determine the target action from the action sequence based on the speed parameters.

[0104] According to an example implementation of the present disclosure, the apparatus 1200 further includes: a scaling parameter receiving module configured to receive scaling parameters for specifying dimensions of the virtual image; and an updating module based on the scaling parameters configured to update the dimensions of the virtual image based on the scaling parameters.

[0105] According to an example implementation of the present disclosure, the apparatus 1200 further includes: a rotation parameter receiving module configured to receive rotation parameters for specifying the rotation of the virtual image; and an updating module configured to update the rotation angle of the virtual image based on the rotation parameters.

[0106] According to an example implementation of the present disclosure, the device 1200 further includes: a mirror parameter receiving module, configured to receive a mirror setting for specifying the flipping of the virtual image; and an update module based on the mirror parameter, configured to update the virtual image representation frame based on the mirroring of the virtual image representation frame.

[0107] According to an exemplary implementation of the present disclosure, the apparatus 1200 further includes: an editing module configured to, in response to receiving an editing interaction for the first video, update the first video based on the editing interaction.

[0108] According to an example implementation of the present disclosure, the apparatus 1200 further includes: a loop module configured to generate a fourth video based on a loop of the action sequence in response to determining that the time length of the third video is greater than the time length of the action sequence.

[0109] According to an exemplary implementation of the present disclosure, a physical scene includes objects, and the objects include at least any one of static objects and dynamic objects, and the dynamic objects include people, animals, and moving objects.

[0110] According to an exemplary implementation of the present disclosure, the apparatus 1200 further includes: in response to determining that the target position is located at the dynamic object, the target position moves along with the movement of the dynamic object.

[0111] According to an example implementation of the present disclosure, the presentation position of the virtual object in the fourth video changes as the field of view of the acquisition device of the third video changes.

[0112] FIG13 shows a block diagram of a device 1300 capable of implementing various implementations of the present disclosure. It should be understood that the computing device 1300 shown in FIG13 is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described herein. The computing device 1300 shown in FIG13 can be used to implement the methods described above.

[0113] As shown in Figure 13, computing device 1300 is in the form of a general-purpose computing device. Components of computing device 1300 may include, but are not limited to, one or more processors or processing units 1310, memory 1320, storage device 1330, one or more communication units 1340, one or more input devices 1350, and one or more output devices 1360. Processing unit 1310 may be a real or virtual processor and is capable of performing various processes according to a program stored in memory 1320. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of computing device 1300.

[0114] The computing device 1300 typically includes a plurality of computer storage media. Such media can be any available media accessible to the computing device 1300, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 1320 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 1330 can be a removable or non-removable medium and can include a machine-readable medium such as a flash drive, a disk, or any other medium that can be used to store information and / or data (e.g., training data for training) and can be accessed within the computing device 1300.

[0115] The computing device 1300 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 13 , a disk drive for reading from or writing to a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 1320 may include a computer program product 1325 having one or more program modules configured to perform various methods or actions of various implementations of the present disclosure.

[0116] The communication unit 1340 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the computing device 1300 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the computing device 1300 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or other network nodes.

[0117] Input device 1350 may be one or more input devices, such as a mouse, keyboard, or trackball. Output device 1360 may be one or more output devices, such as a display, speaker, or printer. Computing device 1300 may also communicate with one or more external devices (not shown) via communication unit 1340, as needed, such as storage devices, display devices, or the like, with one or more devices that allow a user to interact with computing device 1300, or with any device that allows computing device 1300 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0118] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0119] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0120] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0121] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0122] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0123] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for presenting a virtual image in a video, comprising: Obtaining a first video for a specified action sequence; receiving a user interaction for a second video, the second video comprising a physical scene and the user interaction being used to specify a target location for placing the avatar in the physical scene; as well as Based on the third video including the physical scene, the action sequence and the virtual image and the target position, a fourth video including the physical scene and the virtual image is generated, the virtual image is located at the target position and performs the action sequence.

2. The method according to claim 1, wherein generating the fourth video comprises: parsing the first video to extract the action sequence; generating an avatar representation of the avatar in accordance with the sequence of actions; as well as The avatar representation and the third video are combined to generate the fourth video.

3. The method of claim 1 , wherein generating the avatar representation comprises: For the target frame in the third video, determining a target action from the action sequence based on an offset of the target frame in the third video; Based on the key point matrix associated with the target action and the initial image of the virtual image, an avatar representation frame corresponding to the target frame in the virtual image representation is generated.

4. The method of claim 3, wherein generating the avatar representation frame comprises: Determining a displacement of the avatar based on the key point matrix and an initial image of the avatar; determining a physical position of the avatar in the physical scene based on the target position and the displacement; The virtual avatar representation frame corresponding to the physical position is generated.

5. The method of claim 4, wherein generating the avatar representation frame corresponding to the physical location further comprises: In response to determining that the physical location is within the field of view of the capture device of the third video, the avatar representation frame is generated.

6. The method according to claim 4, wherein there is an occlusion relationship between the virtual image and the object in the physical scene.

7. The method according to claim 6, further comprising: determining a spatial position of the object in the physical scene; Determining the occlusion relationship between the virtual image and the object based on the spatial position and the physical position of the virtual image in the physical scene; as well as The virtual image representation frame is generated according to the occlusion relationship.

8. The method according to claim 3, further comprising: receiving a speed parameter for specifying a movement speed of the avatar; as well as The target action is determined from the action sequence based on the speed parameter.

9. The method according to claim 1, further comprising: receiving a scaling parameter for specifying dimensions of the avatar; as well as The dimensions of the avatar are updated based on the scaling parameters.

10. The method according to claim 1, further comprising: receiving a rotation parameter for specifying a rotation of the avatar; as well as The rotation angle of the avatar is updated based on the rotation parameter.

11. The method according to claim 3, further comprising: receiving a mirror image setting for specifying a flip of the avatar; as well as The avatar representation frame is updated based on a mirror image of the avatar representation frame.

12. The method according to claim 1, further comprising: In response to receiving an editing interaction for the first video, the first video is updated based on the editing interaction.

13. The method according to claim 1, further comprising: In response to determining that the time length of the third video is greater than the time length of the action sequence, the fourth video is generated based on a loop of the action sequence.

14. The method according to claim 1, wherein the physical scene includes objects, the objects include at least any one of static objects and dynamic objects, and the dynamic objects include people, animals, and moving objects.

15. The method according to claim 14, further comprising: In response to determining that the target position is located at the dynamic object, the target position moves as the dynamic object moves.

16. The method according to claim 1, wherein the presentation position of the virtual object in the fourth video changes as the field of view of a capture device of the third video changes.

17. A device for presenting a virtual image in a video, comprising: An acquisition module, configured to acquire a first video for specifying an action sequence; a receiving module configured to receive a user interaction for a second video, wherein the second video includes a physical scene and the user interaction is used to specify a target location for placing the virtual image in the physical scene; as well as A generation module is configured to generate a fourth video including the physical scene and the virtual image based on a third video including the physical scene, the action sequence and the virtual image and the target position, wherein the virtual image is located at the target position and performs the action sequence.

18. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 16 when executed by the at least one processing unit.

19. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 16.

Images