Video generation method and device, electronic equipment and storage medium
By extracting the bone data and partial contour data in the action reference video to generate intermediate video, and extracting the head contour data from the intermediate video to optimize the target video, the problem of virtual characters' image deviating from the expected style in the generative model is solved, and the image is realized consistency and smoothness of movement.
Patent Information
- Application Number
- CN202411637716.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-15
- Publication Date
- 2025-02-11
AI Technical Summary
When introducing action data into the generative model, it is difficult to avoid referring to the image information of the character to affect the generated virtual character image, causing the image to deviate from the expected style.
By extracting the bone data and first contour data of the action subject (excluding head contour data) from the action reference video and generating intermediate video based on these data, ensuring that the outline image of the synthetic subject is consistent with the action subject. Then, the head contour data of the synthetic subject is extracted from the intermediate video, and the video generation model is combined with the bone data to generate the target video again, optimizing head details and motion fluency.
It effectively avoids the influence of the head image of the reference character, ensures that the generated virtual character image meets the expected style, and improves the smoothness of the action.
Smart Images
Figure CN119421015A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision and artificial intelligence technology, and in particular to methods, devices, electronic devices and storage media for generating videos. Background Art
[0002] Generative models have been a hot topic in the field of AI (Artificial Intelligence) in recent years, and have achieved remarkable success in image and video generation. With the development of technology, the application prospects of generative models in virtual character animation design have gradually attracted attention. For example, the generative model can automatically generate animated videos of various expressions and actions of virtual characters to reduce the workload of designers in manually creating animations.
[0003] In order to accurately control the action performance of the virtual character in the video generated by the generative model, the related technology generally finds a suitable action reference video, and extracts the reference character's action data and other control information from the action reference video to control the generative model to optimize the action performance of the virtual character in the animation video.
[0004] However, when introducing motion data into the generative model, the image information of the reference character is inevitably introduced, which affects the image of the virtual character ultimately generated by the generative model and causes the generated virtual character image to deviate from the expected style. Summary of the invention
[0005] To overcome the problems existing in the related art, this specification provides a method, device, electronic device and storage medium for generating a video.
[0006] According to a first aspect of an embodiment of this specification, a method for generating a video is provided, the method comprising:
[0007] Extracting skeleton data and first contour data of the body of the action subject from the action reference video, wherein the first contour data includes contour data of the remaining parts of the body except the head; and determining first control information including at least the skeleton data and the first contour data, and controlling the video generation model to generate an intermediate video based on the first control information and the reference character data;
[0008] Extract second contour data of the body of the synthetic subject from the intermediate video, wherein the second contour data includes contour data of the head of the body; and determine second control information including at least the skeleton data and the second contour data, and control the video generation model to generate a target video based on the second control information.
[0009] According to a second aspect of an embodiment of this specification, there is provided a device for generating a video, including:
[0010] An intermediate video generation module is used to extract skeleton data and first contour data of the body of the action subject from the action reference video, wherein the first contour data includes contour data of the remaining parts of the body except the head; and determine first control information including at least the skeleton data and the first contour data, and control the video generation model to generate an intermediate video based on the first control information and the reference character data;
[0011] A target video generation module is used to extract second contour data of the body of the synthetic subject from the intermediate video, wherein the second contour data includes contour data of the head of the body; and to determine second control information including at least the skeleton data and the second contour data, and control the video generation model to generate a target video based on the second control information.
[0012] According to a third aspect of the embodiments of this specification, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method described in the first aspect are implemented.
[0013] According to a fourth aspect of the embodiments of this specification, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method described in the first aspect.
[0014] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:
[0015] In the embodiment of the specification, the process of generating a target video based on the video generation model includes two closely related stages: in the first stage, the skeleton data and the first contour data of the body of the action subject are extracted from the action reference video, and the video generation model is controlled to generate an intermediate video based on the first control information and the benchmark character data at least including the skeleton data and the first contour data, which can ensure that the synthetic subject in the intermediate video maintains the same contour image and action details as the action subject. In addition, the first contour data does not include the head contour data of the action subject, which can effectively avoid the head image of the action subject affecting the head details of the synthetic subject, so that the video generation model can generate the head details of the synthetic subject with reference to the head image of the benchmark character. In the second stage, the contour data of the head of the synthetic subject is extracted separately from the intermediate video to control the video generation model to further optimize the head details of the synthetic subject of the intermediate video, and the skeleton data generated in the first stage is combined again to control the video generation model to generate the target video, which can further improve the action fluency of the synthetic subject of the intermediate video.
[0016] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.
[0018] Figure 1 The flowchart of a method for generating a video is shown in the specification according to an exemplary embodiment.
[0019] Figure 2 It is a schematic diagram of a video generation process shown in this specification according to an exemplary embodiment.
[0020] Figure 3 It is a schematic diagram of repairing distorted elements in a target video according to an exemplary embodiment of the present specification.
[0021] Figure 4 It is a structural schematic diagram of an electronic device according to an exemplary embodiment of the present specification.
[0022] Figure 5 It is a block diagram of a device for generating a video according to an exemplary embodiment of the present specification. DETAILED DESCRIPTION
[0023] Generative models have been a hot topic in the field of AI (Artificial Intelligence) in recent years, especially in image and video generation. With the development of technology, the application prospects of generative models in virtual character animation design have gradually attracted attention. For example, the generative model can automatically generate animated videos of various expressions and actions of virtual characters. In the process of virtual character animation generation, the generative model can generate animation sequences with complex expressions, actions and environments frame by frame. The final generated virtual character animation video can be applied to different fields such as social media, live broadcast, and games.
[0024] In order to accurately control the action performance of the virtual character in the video generated by the generative model, the related technology generally finds a suitable action reference video, and extracts the reference character's action data and other control information from the action reference video to control the generative model to optimize the action performance of the virtual character in the animation video.
[0025] However, when introducing motion data into the generative model, the image information of the reference character is inevitably introduced, which affects the image of the virtual character ultimately generated by the generative model, causing the generated virtual character image to deviate from the expected style. For example, if a generative model is used to generate an animation video of a virtual character in a two-dimensional style, the motion reference video is a dance video of a real actor, and if the motion data extracted from the motion reference video includes the facial motion data of the real actor, since the expressions of the real actor usually have complex and subtle facial changes, such as subtle muscle movements, wrinkle changes, etc., the generative model introduces these complex details into the facial expressions of the virtual character, which may cause the generated facial expressions to be too complex and realistic, causing the character to lose its original cute and simple cartoon style, have a strong CG feel, and even produce an unnatural "sense of disharmony".
[0026] In response to the above technical problems, this solution proposes a method for generating videos to ensure that the virtual character in the action video generated by the generative model maintains consistent action performance with the reference character in the reference video, and the image of the generated virtual character will not be affected by the image of the reference character, ultimately ensuring that the image of the generated virtual character conforms to the expected style.
[0027] like Figure 1 As shown, Figure 1 1 is a flowchart of a method for generating a video according to an exemplary embodiment of the present specification, comprising the following steps 101-102:
[0028] Step 101: extract the skeleton data and first contour data of the body of the action subject from the action reference video, wherein the first contour data includes the contour data of the remaining parts of the body except the head; and determine the first control information including at least the skeleton data and the first contour data, and control the video generation model to generate an intermediate video based on the first control information and the benchmark character data.
[0029] like Figure 2 As shown, the skeleton data 23 and the first contour data 22 of the body of the action subject can be extracted from the action reference video 21, wherein the first contour data 22 includes the contour data of the rest of the body except the head.
[0030] Exemplarily, the action reference video 21 can be a video of real-life character movements such as a dance video, a sports video, or an animated video of an animated character. There can be more than one moving object in the action reference video 21, and the action subject can refer to an object that needs to extract the skeleton data 23 and the first contour data 22. For example, assuming that the action reference video contains two moving objects A and B, if the skeleton data 23 and the first contour data 22 of A are extracted, A can be regarded as the action subject. If the skeleton data 23 and the first contour data 22 of B are extracted, B can be regarded as the action subject. The role of the skeleton data 23 and the first contour data 22 is to provide specific action guidance and constraints to the video generation model 26, and to help the video generation model 26 generate video content that meets the expected action performance. Specifically, the skeleton data 23 can include but is not limited to the position data, angle data, and length data between different joints of the joints of each part of the body of the action subject in any frame image in the action reference video 21. The skeleton data 23 can control the skeleton movement of the synthetic subject (i.e., the virtual character appearing in the video) of the video generated by the video generation model 26, and can ensure the coherence of the joints and limbs of the synthetic subject. The first contour data 22 may include, but is not limited to, edge information of the body of the action subject in any frame image in the action reference video 21. The first contour data 22 defines the external form of the synthetic subject, ensuring that the appearance of the synthetic subject changes correctly when the subject moves. By combining the skeleton data 23 of the body of the action subject and the first contour data 22, the synthetic subject in the video generated by the video generation model 26 can be controlled to make smooth and realistic movements.
[0031] Because the slight expression changes of the head (especially the face) may introduce more contour changes, these contour changes will affect the facial image of the synthetic subject, causing the facial image of the generated synthetic subject to deviate from the expected style. Therefore, the first contour data 22 of this scheme can include the contour data of the rest of the body except the head of the action subject. This can ensure that the video generation model focuses on the action performance of the limbs and will not be affected by the head image of the action subject. For example, the first contour data 22 can include the contour data of the upper limbs except the head of the body, and can also include the contour data of the lower limbs except the head of the body, or the contour data of the remaining body parts except the head of the body.
[0032] After the skeleton data 23 and the first contour data 22 are extracted, the first control information 24 including at least the skeleton data 23 and the first contour data 22 is determined. Exemplarily, the first control information 24 may include, in addition to the skeleton data 23 and the first contour data 22, other data reflecting the motion state of the action subject, such as the speed and acceleration data of the motion subject. Of course, the first control information 24 may also include only the skeleton data 23 and the first contour data 22.
[0033] In order to control the image of the synthetic subject of the target video 30 finally generated by the video generation model 26 to conform to the expected style. The designer can design a reference character for the video generation model 26 to refer to according to the specific design task requirements, and use the reference character data 25 to control the process of generating videos by the video generation model 26, so that the synthetic subject finally generated is consistent with the style of the reference character. The reference character data 25 can be a number of video data or image data containing the reference character image, which is used to fine-tune the synthetic style of the video generation model 26.
[0034] After determining the first control information 24 and the benchmark character data 25, the video generation model 26 can be controlled based on the first control information 24 and the benchmark character data 25 to generate an intermediate video. Exemplarily, the Controlnet model can be combined with the video generation model 26, and the Controlnet model controls the generation process of the video generation model 26 according to the first control information 24. Exemplarily, the video generation model 26 can be composed of a Vincent graph model, an AnimateDiff model, and a Lora model, and the Lora model can be trained based on the benchmark character data 25. Specifically, the AnimateDiff model is used to give the Vincent graph model video generation capabilities, and the Lora model trained with the benchmark character data 25 fine-tunes the Vincent graph model so that it outputs a video that conforms to the style of the benchmark character. Exemplarily, the video generation model 26 can also be a generative model that independently has video generation capabilities, such as a generative model for generating videos such as MoCoGAN and VGAN, and the generative model can be fine-tuned through the benchmark character data 25 so that it can output a video that conforms to the style of the benchmark character.
[0035] In this solution, the video generation model 26 can be used to control the image of the synthetic subject in the video by referring to the image of the benchmark character in the reference benchmark character data 25, and the video generation model 26 can be used to control the motion of the synthetic subject in the video by referring to the motion data of the action subject in the action reference video 21. When the video generation model 26 is controlled to generate video content by combining the benchmark character data 25 and the motion data of the action subject, if the body proportions of the action subject and the body proportions of the benchmark character differ greatly, the motion of the synthetic subject generated in the end may appear distorted or unbalanced. For example, assuming that the action subject is a real person with a slender figure, and the benchmark character is a character with a cartoon style, the difference in body proportions will affect the range of motion of the joints of the synthetic subject and the relative positional relationship between the joints. For example, if the legs of the action subject are longer and the legs of the benchmark character are shorter, since the synthetic subject is the external image of the reference benchmark character and the motion style is the motion style of the reference action subject, the motion of the synthetic subject will not match its body shape, such as the leg bending is too large or too small, thus not conforming to the law of actual ergonomics.
[0036] In view of this technical problem, the benchmark character data 25 of the present solution may be an image containing the benchmark character, from which the proportional relationship between different parts of the body of the benchmark character can be determined. For example, the proportional relationship between the thigh, calf and upper body of the benchmark character is determined. After extracting the skeleton data 23 and the first contour data 22 of the body of the action subject from the action reference video 21, the proportional relationship of the body in the skeleton data 23 and the first contour data 22 can be adjusted to be the same as the proportional relationship of the body of the benchmark character, and the adjusted skeleton data 23 and the first contour data 22 are used as the first control information 24.
[0037] This solution adjusts the body proportion relationship in the skeleton data 23 and the first contour data 22 extracted from the action reference video 21 to be the same as the body proportion relationship of the reference character, which can ensure that the action of the synthesized subject matches its body shape, so that the generated action is more consistent with the physiological structure of the reference character. For example, the reference character is a cartoon-style character with short legs, while the action subject in the reference video is a real-life character with long legs. By adjusting the skeleton data 23 and the first contour data 22 of the action subject, the action generated by the video generation model 26 can be consistent with the unique body characteristics of the reference character.
[0038] Furthermore, the skeleton data 23 and the first contour data 22 may be position information about the character's joints, actions, and edges, and the skeleton data 23 and the first contour data 22 include the canvas on which they are located. If the size of the canvas carrying the skeleton data 23 and the first contour data 22 is inconsistent with the ratio of the image frame size output by the video generation model 26, the position of the skeleton data 23 in the image frame may be offset or misaligned, which will also cause the generated action to appear unnatural in the image. In response to this technical problem, the present solution adjusts the size of the canvas carrying the skeleton data 23 and the first contour data 22 to be consistent with the size of the output image frame set by the video generation model 26, and uses the adjusted skeleton data 23 and the first contour data 22 as the first control information 24, thereby ensuring the precise alignment of the skeleton data 23 and the first contour data 22 with the output image frame.
[0039] Step 102: extract second contour data of the body of the synthetic subject from the intermediate video, wherein the second contour data includes contour data of the head of the body; and determine second control information including at least the skeleton data and the second contour data, and control the video generation model to generate a target video based on the second control information.
[0040] The synthetic subject in the middle video 27 may refer to the main virtual character appearing in the middle video 27. If the middle video 27 contains multiple virtual characters, the virtual character whose actions or behaviors dominate is regarded as the synthetic subject. The synthetic subject may also be determined based on the image of each virtual character or the action of the virtual character. For example, a virtual character that is consistent with the image of the reference character and / or a virtual character that is consistent with the action of the action subject may be regarded as the synthetic subject. If the middle video contains only one virtual character, the virtual character may be regarded as the synthetic subject.
[0041] like Figure 2 As shown, after generating the intermediate video 27 according to step 102, the second contour data 28 of the body of the synthetic subject can be extracted from the intermediate video 27, and the second contour data 28 includes the contour data of the head of the body. Among them, the second contour data 28 can include only the contour data of the head of the body, or can include the contour data of the whole body of the synthetic subject. And determine the second control information 29 including at least the skeleton data 23 and the second contour data 28, and control the video generation model 26 to generate the target video 30 based on the second control information 29. Extracting the contour data of the head of the synthetic subject separately from the intermediate video can control the video generation model 26 to further optimize the head details of the synthetic subject of the intermediate video 27. And, in combination with the skeleton data 23 extracted in step 201, the video generation model 26 can be controlled again to generate the target video 30, further improving the action smoothness of the synthetic subject of the intermediate video 27. In addition, by using the skeleton data 23 extracted in step 201, it is possible to avoid extracting skeleton data from the intermediate video 27, thereby reducing the task workload. Among them, the skeleton data 23 can be the skeleton data extracted from the action reference video 21 without any processing. The skeleton data 23 may also be the skeleton data after the proportional relationship is adjusted as in the above-mentioned embodiment.
[0042] After the target video 30 is generated, this specification also proposes some solutions that can further optimize the content of the target video 30, see the following embodiments for details:
[0043] In an illustrated embodiment, the target video 30 can be input into the video generation model 26 again to output the optimized target video 30. The purpose of inputting the target video 30 into the video generation model 26 again is to further optimize the target video 30 generated for the first time, and to repair the defects of the target video 30 generated for the first time to improve the visual quality of the video. For example, the target video 30 generated for the first time may have blurred images or lost details. By inputting the target video 30 into the video generation model 26 again, the video generation model 26 can repair these problems by processing texture details frame by frame, thereby improving the video quality. Optionally, the target video 30 can be input into the video generation model 30 to obtain an output result, and then the output result is re-input into the video generation model 30 as input data, and the above steps are repeated until the optimization effect of the output result begins to decrease and stops, and the video with the highest output quality is selected from the multiple output results outputted previously.
[0044] In one illustrated embodiment, Figure 3 As shown, the distorted element 31 in the target video 30 can be identified, and the video generation model 26 or the optimization model 32 specifically used to optimize the distorted element can be called to repair the distorted element 31. Finally, the distorted element 31 in the target video 30 is replaced by the corresponding repaired element 33. The distorted element 31 can refer to various unnatural objects in the target video 30 or objects that do not meet the expected visual requirements. For example, there are objects with problems such as blur, noise, artifacts or color abnormalities in the target video 30. The object can be a background element in the target video 30, a hand, face, limbs, and other different objects of the synthetic subject. The model for repairing the distorted element can use the video generation model 26 or the optimization model 32 specifically used to optimize the distorted element. For example, some models are specifically used to optimize hand details or leg details. If the distorted element is a hand, the optimization model 32 specifically used to optimize the hand details can be used to optimize the hand. Similarly, if the distorted element is a leg, the optimization model 32 specifically used to optimize the leg details can be used to optimize the leg. For example, if it is necessary to repair the face of any frame image in the target video 30, the face in the frame image can be identified, and the image information of the face is input into the video generation model 26 for repair, and the face in the frame image is replaced with the corresponding repaired face. For example, if it is necessary to repair the hand of any frame image in the target video 30, the hand in the frame image can be identified, and the image information of the hand is input into the optimization model for optimizing the hand for repair, and the hand in the frame image is replaced with the corresponding repaired hand. In order to improve the optimization effect, the skeleton data and contour data of the hand of the action subject can be determined from the skeleton data 23 and the first contour data 22, and the optimization model 32 is controlled according to the skeleton data and contour data to repair the hand of the synthetic subject in the target video 30.
[0045] In an illustrated embodiment, in order to improve the quality of the target video 30, a new image frame may be inserted between any two frames in the target video 30 to make the motion transition smoother and more natural, thereby reducing visual freezes and discontinuities.
[0046] In an illustrated embodiment, super-resolution technology can be used to improve the quality of the target video 30, and by optimizing and enhancing the pixels of the image frame, the video picture is made clearer and more delicate, presenting a higher visual quality.
[0047] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of a device and a terminal to which it is applied.
[0048] like Figure 4 As shown, Figure 4 It is a structural diagram of an electronic device shown in this specification according to an exemplary embodiment. At the hardware level, the electronic device 400 includes a processor 402, an internal bus 404, a network interface 406, a memory 408 and a non-volatile memory 410, and of course may also include hardware required for other services. One or more embodiments of this specification can be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic module, but can also be hardware or logic devices.
[0049] like Figure 5 As shown, Figure 5 is a block diagram of a device for generating a video according to an exemplary embodiment of the present specification. The device can be applied to Figure 4 In the electronic device 500 shown in the figure, the technical solution of this specification is implemented. The device includes:
[0050] The intermediate video generation module 502 is used to extract the skeleton data and first contour data of the body of the action subject from the action reference video, wherein the first contour data includes contour data of the rest of the body except the head; and determine first control information including at least the skeleton data and the first contour data, and control the video generation model to generate the intermediate video based on the first control information and the reference character data;
[0051] The target video generation module 504 is used to extract second contour data of the body of the synthetic subject from the intermediate video, wherein the second contour data includes contour data of the head of the body; and to determine second control information including at least the skeleton data and the second contour data, and control the video generation model to generate a target video based on the second control information.
[0052] Optionally, the benchmark character data is an image containing a benchmark character, and the device also includes a proportion adjustment module 506, which is used to determine the proportional relationship between different parts of the body of the benchmark character from the image, adjust the proportional relationship of the body in the skeleton data and the first contour data according to the proportional relationship, and adjust the size ratio of the canvas carrying the skeleton data and the first contour data to be consistent with the size ratio of the output image frame set by the video generation model; the intermediate video generation module 502 is specifically used to determine the first control information including at least the adjusted skeleton data and the first contour data.
[0053] Optionally, the device also includes a distorted element repair module 508, which is used to identify the distorted elements in the target video, and call the video generation model or the optimization model used to optimize the distorted elements to repair the distorted elements; replace the distorted elements in the target video with corresponding repaired elements.
[0054] Optionally, the distorted elements include hands and / or face of the synthesized subject in the target video.
[0055] Optionally, the distorted element includes the hand of the synthetic subject in the target video, and the device also includes a hand control information determination module 510, which is used to determine the skeleton data and contour data of the hand from the skeleton data and the first contour data; the distorted element repair module 508 is specifically used to input the image information of the hand into an optimization model for optimizing the hand, and control the optimization model to repair the hand according to the skeleton data and contour data of the hand.
[0056] Optionally, the video generation model is composed of a Vincent graph model, an AnimateDiff model and a Lora model, and the Lora model is trained based on the benchmark character data.
[0057] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.
[0058] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Ordinary technicians in this field can understand and implement it without paying creative work.
[0059] The present specification also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the steps of any of the aforementioned methods for generating a video provided in the present application are implemented.
[0060] Specifically, computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (such as EPROM, EEPROM and flash memory devices), magnetic disks (such as internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks.
Claims
1. A method for generating a video, characterized in that: The method comprises: Extracting skeleton data and first contour data of the body of the action subject from the action reference video, wherein the first contour data includes contour data of the remaining parts of the body except the head; and determining first control information including at least the skeleton data and the first contour data, and controlling the video generation model to generate an intermediate video based on the first control information and the reference character data; Extract second contour data of the body of the synthetic subject from the intermediate video, wherein the second contour data includes contour data of the head of the body; and determine second control information including at least the skeleton data and the second contour data, and control the video generation model to generate a target video based on the second control information.
2. The method according to claim 1, characterized in that The reference character data is an image containing the reference character, and the method further comprises: Determine the proportional relationship between different parts of the body of the benchmark character from the image, adjust the proportional relationship of the body in the skeleton data and the first contour data according to the proportional relationship, and adjust the size ratio of the canvas carrying the skeleton data and the first contour data to be consistent with the size ratio of the output image frame set by the video generation model; The determining of the first control information including at least the skeleton data and the first contour data comprises: First control information including at least the adjusted skeleton data and the first contour data is determined.
3. The method according to claim 1, characterized in that The method further comprises: Identifying distorted elements in the target video, and calling the video generation model or an optimization model for optimizing the distorted elements to repair the distorted elements; The distorted elements in the target video are replaced with corresponding restored elements.
4. The method according to claim 3, characterized in that The distorted elements include hands and / or face of the synthesized subject in the target video.
5. The method according to claim 3, characterized in that: The distortion element includes a hand of a synthetic subject in the target video, The method further comprises: determining the skeleton data and the contour data of the hand from the skeleton data and the first contour data; The calling of the video generation model or the optimization model for optimizing the distorted element to repair the distorted element includes: inputting the image information of the hand into the optimization model for optimizing the hand, and controlling the optimization model to repair the hand according to the bone data and contour data of the hand.
6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: The target video is input into the video generation model to obtain an optimized target video.
7. The method according to any one of claims 1 to 5, characterized in that The video generation model is composed of a Vincent graph model, an AnimateDiff model and a Lora model, and the Lora model is trained based on the benchmark character data.
8. A device for generating a video, characterized in that: The device comprises: An intermediate video generation module is used to extract skeleton data and first contour data of the body of the action subject from the action reference video, wherein the first contour data includes contour data of the remaining parts of the body except the head; and determine first control information including at least the skeleton data and the first contour data, and control the video generation model to generate an intermediate video based on the first control information and the reference character data; A target video generation module is used to extract second contour data of the body of the synthetic subject from the intermediate video, wherein the second contour data includes contour data of the head of the body; and to determine second control information including at least the skeleton data and the second contour data, and control the video generation model to generate a target video based on the second control information.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.