Model training method, video generation method and related equipment
By dynamically changing training samples in digital human model training, first learning the reasoning of small pose poor poses, and then gradually learning the reasoning of large pose poor poses, the reasoning problem of difficult to effectively learn large pose changes in the existing technology is solved, and the model is stable and accurate inference under large pose changes is achieved.
Patent Information
- Application Number
- CN202411968104.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-26
- Publication Date
- 2025-04-08
AI Technical Summary
The existing digital human model training methods are difficult to effectively learn the reasoning of large pose changes, resulting in poor performance of models when dealing with large poses.
By dynamically changing the training samples during the model training process, first train with more samples with smaller pose differences, and then as the training rounds increase, the samples with larger pose differences are gradually increased, so that the network model can first learn the reasoning of small pose differences, and then gradually learn the reasoning of large pose differences.
The model that can handle large pose changes is achieved stably and accurately trained, which improves the model's reasoning ability when dealing with large pose changes.
Smart Images
Figure CN119785151A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular, to a model training method, a video generation method and related equipment. Background Art
[0002] "Digital human" refers to a virtual human image similar to a real person simulated by computer technology. With the continuous maturity of artificial intelligence technology, the technology of generating "digital human" has become closer and closer to the lives of ordinary people, and has been widely used in e-commerce sales, online live broadcasting, news broadcasting and other fields. Summary of the invention
[0003] The purpose of the present disclosure is to provide a model training method, a video generation method and related equipment for stably and accurately training a first model.
[0004] According to a first aspect of an embodiment of the present disclosure, a model training method is provided, the method comprising: Acquire a first sample image frame; the first sample image frame is an image frame obtained by performing mask processing on a first image frame in an image frame set, wherein the image frame set includes a plurality of image frames corresponding to a sample video; According to the first sample image frame, performing model training for a plurality of training rounds on the model to obtain a first model; Among them, for each training round of the multiple training rounds, a second sample image frame corresponding to the training round is determined from the second image frame of the multiple image frames, and the first image frame, the first sample image frame, the second sample image frame and the sample audio corresponding to the sample video are used as training samples; the number of third image frames in the second sample image frames is positively correlated with the training round, and the third image frame is an image frame whose posture difference value is greater than a preset difference value, and the posture difference value represents the difference between the posture of a specified part of the first object in the first image frame and the posture of the specified part in the second image frame.
[0005] According to a second aspect of an embodiment of the present disclosure, a method for generating a video is provided, the method comprising: Acquire a first object image frame, a second object image frame and a first audio, wherein the second object image frame is an image frame obtained by performing mask processing on the first object image frame; Using the first object image frame, the second object image frame and the first audio as inputs of a first model to obtain a plurality of third object image frames output by the first model, wherein the first model is trained according to the model training method described in the first aspect of the embodiment of the present disclosure; A first video is generated according to the plurality of third object image frames.
[0006] According to a third aspect of an embodiment of the present disclosure, a model training device is provided, the device comprising: an acquisition module configured to acquire a first sample image frame; the first sample image frame is an image frame obtained by performing mask processing on a first image frame in an image frame set, wherein the image frame set includes a plurality of image frames corresponding to a sample video; A training module is configured to perform a plurality of training rounds of model training on the model according to the first sample image frame to obtain a first model; Among them, for each training round of the multiple training rounds, a second sample image frame corresponding to the training round is determined from the second image frame of the multiple image frames, and the first image frame, the first sample image frame, the second sample image frame and the sample audio corresponding to the sample video are used as training samples; the number of third image frames in the second sample image frames is positively correlated with the training round, and the third image frame is an image frame whose posture difference value is greater than a preset difference value, and the posture difference value represents the difference between the posture of a specified part of the first object in the first image frame and the posture of the specified part in the second image frame.
[0007] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the steps of the method described in the first aspect of the embodiment of the present disclosure or the second aspect of the embodiment of the present disclosure are implemented.
[0008] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, including: a memory having a computer program stored thereon; A processor is used to execute the computer program in the memory to implement the steps of the method described in the first aspect of the embodiment of the present disclosure or the second aspect of the embodiment of the present disclosure.
[0009] According to a sixth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect of an embodiment of the present disclosure or the second aspect of an embodiment of the present disclosure.
[0010] Through the above technical solution, during the model training process, the present invention first uses more samples with smaller posture difference values for training, and then gradually increases the samples with larger posture difference values as the number of training rounds increases. The training samples in the model training process change dynamically, so that the network model can first learn the reasoning of small posture differences, and then gradually learn the reasoning of large posture differences. Therefore, the first model can be trained stably and accurately, and the trained first model can realize the reasoning of large posture changes.
[0011] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings: Figure 1 It is a flow chart of a training method of a digital human model in the related art.
[0013] Figure 2 A schematic diagram of an application environment of a video generation method provided by an embodiment of the present disclosure is shown.
[0014] Figure 3 The figure is a flowchart of a model training method according to an exemplary embodiment.
[0015] Figure 4 is a schematic diagram showing a posture difference value according to an exemplary embodiment.
[0016] Figure 5 The flowchart of a method for determining a second sample image frame is shown according to an exemplary embodiment.
[0017] Figure 6 The present invention is a flowchart of a method for determining a sample image frame set according to an exemplary embodiment.
[0018] Figure 7 The figure is a schematic diagram of a three-dimensional marking point according to an exemplary embodiment.
[0019] Figure 8 is a schematic diagram showing a head posture according to an exemplary embodiment.
[0020] Fig. 9 The figure is a schematic diagram showing a two-dimensional marking point according to an exemplary embodiment.
[0021] Fig.10 The figure is a schematic diagram showing a method of selecting training sample pairs according to an exemplary embodiment.
[0022] Fig.11 The present invention is a flowchart of a method for generating a video according to an exemplary embodiment.
[0023] Fig.12 It is a block diagram of a model training device according to an exemplary embodiment.
[0024] Fig.13 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION
[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0026] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0027] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0028] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0029] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0030] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0031] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0032] Reference Figure 1In the related art, 2D digital human generation technology usually uses two images as a pair of training samples for training, where one image provides the upper half of the face (including posture and blinking), and the lower half of the face image is fully masked. This image is called the inference frame, and the other image provides the full face (identity, lip shape), which is called the reference frame. The model is trained by the training sample pair composed of the inference frame and the reference frame and the audio data corresponding to the inference frame, and a model that can generate corresponding digital human videos based on any audio data is obtained.
[0033] During the model training process, since the training sample pairs of the inference frame and the reference frame are random, the posture changes of the digital human in the reference frame and the inference frame are random, which makes it difficult for the model to learn to reason about postures with large changes. Taking the posture of the digital human including the head posture as an example, the head posture changes randomly between the reference frame and the inference frame, and it is difficult for the model to learn to reason about head postures with large changes. Taking the posture of the digital human including the lip posture as an example, the lip posture changes randomly between the reference frame and the inference frame, and it is difficult for the model to learn to reason about lip postures with large changes.
[0034] In view of this, the embodiments of the present disclosure show a model training method, a video generation method, an apparatus, a medium, a device and a product. In the model training process, the present disclosure first uses more samples with smaller posture difference values for training, and then gradually increases the samples with larger posture difference values as the number of training rounds increases. The training samples in the model training process change dynamically, so that the network model can first learn the reasoning of small posture differences, and then gradually learn the reasoning of large posture differences. Therefore, the first model can be trained stably and accurately, and the trained first model can realize the reasoning of large posture changes.
[0035] In order to better understand the model training method, video generation method, device, electronic device, computer-readable storage medium, and computer program product provided by the embodiments of the present disclosure, the application environment applicable to the embodiments of the present disclosure is described below.
[0036] See also Figure 2 , Figure 2 A schematic diagram of an application environment of the model training method provided by an embodiment of the present disclosure is shown. As an implementation mode, the model training method provided by an embodiment of the present disclosure can be applied to an electronic device. The electronic device can be Figure 2 The server 110 shown in the figure can be connected to the terminal device 120 through a network. The network is used to provide a medium for a communication link between the server 110 and the terminal device 120. The network can include various connection types, such as a wired communication link, a wireless communication link, etc., which is not limited in the embodiment of the present disclosure.
[0037] It should be understood that Figure 2The server 110, network and terminal device 120 are only illustrative. According to the implementation requirements, there can be any number of servers 110, networks and terminal devices 120. For example, the server 110 can be a physical server 110, or a server 110 cluster composed of multiple servers 110, etc. The terminal device 120 can be a mobile phone, a tablet, a desktop computer, a laptop computer, a smart wearable device, etc.
[0038] As another implementation, the model training method and the video generation method provided in the embodiments of the present application can be applied to different electronic devices. For example, the video generation method is applied to electronic device A, and the model training method is applied to electronic device B. In addition, the video generation method and the model training method provided in the embodiments of the present application can also be applied to the same electronic device. The embodiments of the present application do not limit the electronic devices to which the above two methods are applied.
[0039] The embodiments of the present disclosure are further explained below with reference to the accompanying drawings.
[0040] Figure 3 is a flow chart of a model training method according to an exemplary embodiment. Figure 3 As shown, the method may include: Step S101, obtaining a first sample image frame. The first sample image frame is an image frame obtained by performing mask processing on a first image frame in an image frame set, and the image frame set includes multiple image frames corresponding to a sample video.
[0041] In the embodiment of the present disclosure, the first sample image frame may be an inference frame in the digital human model training, and the second sample image frame may be a reference frame in the digital human model training.
[0042] For example, for a sample video used for model training, multiple image frames corresponding to the sample video can be extracted to obtain an image frame set. The sample video can be one or more. For example, 20 image frames can be extracted from a 1s video. Correspondingly, if the total duration of the sample video is 1min, 1200 image frames can be extracted, that is, the final image frame set can include 1200 image frames.
[0043] In some embodiments, a first image frame may be selected from multiple image frames in an image frame set, and a mask process may be performed on the first image frame to obtain a first sample image frame, wherein the first sample image frame may be understood as an inference frame for model training, and the first image frame may be understood as an image frame corresponding to the first sample image frame before being masked. The first image frame may be randomly selected from multiple image frames, or the first image frame may be selected from the first image frame according to a preset rule. The present disclosure does not specifically limit the specific implementation method of selecting the first image frame.
[0044] Step S102: performing multiple rounds of model training on the model according to the first sample image frame to obtain a first model.
[0045] In other embodiments, reference Figure 4 , the vertical axis is the posture difference value, the horizontal axis is the frame number of the image frame, the points on the broken line represent multiple reference frames, the hash points represent multiple reasoning frames, and the length of the dotted line represents the posture difference value between the first image frame and the second image frame. It can be seen that the posture difference values between the reasoning frame and the reference frame are different. In order to ensure that the posture difference values between each pair of first sample image frames and second sample image frames in the model training process are different, there are both training sample pairs with large posture difference values and training sample pairs with small posture difference values, to avoid the problem that the posture difference values of the training sample pairs are too uniform and lack of training sample pairs with large posture difference values, a frame of image can be randomly selected from the image frame set as the first image frame, and the first image frame is masked to obtain the first sample image frame, and then multiple second sample image frames are selected from the second image frames other than the first image frame in the sample image frame set as training samples, wherein the second sample image frame can be understood as a reference frame for model training.
[0046] The posture difference value may represent the difference between the posture of the designated part of the first object in the first image frame and the posture of the designated part in the second image frame. For example, when the designated part is the mouth, the posture difference value may represent the difference between the mouth posture in the first image frame and the mouth posture in the second image frame. When the designated part is the head, the posture difference value may represent the difference between the head posture in the first image frame and the head posture in the second image frame.
[0047] For example, for each training round in multiple training rounds, a second sample image frame corresponding to the training round can be determined from the second image frame in the multiple image frames. The second sample image frame corresponding to each training round can be different, that is, as the training round changes, the second sample image frame also changes dynamically.
[0048] In some embodiments, the number of third image frames in the second sample image frames may be positively correlated with the training rounds, wherein the third image frames may be image frames whose posture difference values are greater than a preset difference value, that is, as the training rounds increase, the number of third image frames whose posture difference values in the second sample image frames are greater than the preset difference value changes dynamically. In this way, during the model training process, more samples with smaller posture difference values are first used for training, and then as the training rounds increase, the number of samples with larger posture difference values is gradually increased. The training samples during the model training process change dynamically, so that the network model can first learn the reasoning of small posture differences, and then gradually learn the reasoning of large posture differences, so that the first model can be stably and accurately trained.
[0049] In a possible implementation, the number of third image frames in the second sample image frames may be adjusted in each training round, that is, the number of third image frames greater than the preset difference value in the second sample image frames in each training round is different.
[0050] In another possible implementation, after each preset number of training rounds, the number of the third image frames in the second sample image frames is adjusted. Taking the preset number as 10 as an example, after 10 rounds of training, the number of the third image frames in the second sample image frames can be adjusted, that is, the number of the third image frames in every 10 training rounds is the same, and the number of the third image frames in the second sample image frames does not need to be adjusted within the 10 training rounds.
[0051] For example, the first image frame, the first sample image frame, the second sample image frame and the sample audio corresponding to the sample video in the second sample image frame can be used as training samples, and the model can be trained for multiple training rounds to obtain a first model.
[0052] In some embodiments, during the model training process, the first sample image frame, the second sample image frame and the sample audio can be used as the input of the preset network model, and the first image frame can be used as the output of the preset network model to train the preset network model to obtain the first model. The principle of model training is as follows: the preset network model can first determine the target lip shape of the first object based on the audio data corresponding to the first sample image frame in the sample audio, and then output the first sample image frame that completes the lower half of the face based on the image features and target lip shape of the lower half of the face of the first object in the second sample image frame. The preset network model is continuously corrected according to the difference between the first sample image frame that completes the lower half of the face and the first image frame, thereby obtaining the first model.
[0053] In other embodiments, the manner of masking the first image frame can be determined based on the designated part. For example, when the designated part is the mouth, the mouth or the lower half of the face of the first object can be masked. When the designated part is the eye, the eye or the upper half of the face of the first object can be masked.
[0054] In summary, in the model training process, the present invention first uses more samples with smaller posture difference values for training, and then gradually increases the samples with larger posture difference values as the number of training rounds increases. The training samples in the model training process change dynamically, so that the network model can first learn the reasoning of small posture differences, and then gradually learn the reasoning of large posture differences. Therefore, the first model can be trained stably and accurately, and the trained first model can realize the reasoning of large posture changes.
[0055] Figure 5 is a flowchart of a method for determining a second sample image frame according to an exemplary embodiment. Figure 5 As shown, the second sample image frame corresponding to each training round can be determined by the following steps: Step 201: For each training round, determine a sample selection probability set corresponding to the training round.
[0056] For example, each training round may correspond to a sample selection probability set, wherein the sample selection probability set includes a sample selection probability corresponding to each sample image frame set in a plurality of sample image frame sets, and the sample selection probability may characterize the probability of an image frame in the sample image frame set being selected as the second sample image frame.
[0057] In some embodiments, the second image frame may be divided according to the posture difference value to obtain multiple sample image frame sets, wherein the multiple sample image frame sets may be further divided into multiple first sample sets and multiple second sample sets according to preset thresholds corresponding to the posture difference values.
[0058] In some embodiments, the preset threshold may be 0, that is, the posture difference values with positive posture difference values may be divided into a plurality of first sample sets, and the posture difference values with negative posture difference values may be divided into a plurality of second sample sets.
[0059] Step 202: Determine a second sample image frame corresponding to the training round from a plurality of first sample sets or a plurality of second sample sets according to a sample selection probability set corresponding to the training round.
[0060] For example, the sample selection probability set corresponding to each training round may include a first probability set and a second probability set, wherein the first probability set may include the sample selection probability corresponding to each first sample set, and the second probability set may include the sample selection probability corresponding to each second sample set. The second sample image frame may be first selected from multiple first sample sets according to the first probability set, and the model may be trained for multiple rounds, and then the second sample image frame may be selected from multiple second sample sets according to the second probability set, and the model may be trained for multiple rounds, thereby obtaining the first model. The second sample image frame may also be first selected from multiple second sample sets according to the second probability set, and the model may be trained for multiple rounds, and then the second sample image frame may be selected from multiple first sample sets according to the first probability set, and the model may be trained for multiple rounds, thereby obtaining the first model.
[0061] In some other embodiments, an implementation of step S201 may be: The plurality of sample image frame sets are sorted according to a preset sequence of the posture difference values to obtain a first sequence number corresponding to each sample image frame set.
[0062] The preset order may be an order from small to large or from large to small. When the preset order is an order from small to large, the posture difference value corresponding to the sample image frame set may be positively correlated with the size of the first sequence number; when the preset order is an order from large to small, the posture difference value corresponding to the sample image frame set may be negatively correlated with the size of the first sequence number.
[0063] The sample selection probability set corresponding to the training round is determined according to the sample selection probability set of the previous training round and the first sequence number.
[0064] For example, the sample selection probability set can change with the change of training epochs. During the current round of training, the first selection probability in the sample selection probability set of the previous training round can be determined. The first selection probability can be the first sample selection probability greater than the first preset probability value determined in the sample selection probability set according to the preset arrangement order. Taking the sample selection probability set as {0.05, 0.45, 0.25, 0.15, 0.1}, the first preset probability value is 0.05, and the preset arrangement order is from front to back as an example, the first selection probability can be 0.45, and correspondingly, the second serial number can be 2. Then, the sample selection probability set corresponding to the training round can be determined based on the sample selection probability set of the previous training round and the second serial number of the sample image frame set corresponding to the first selection probability.
[0065] In some embodiments, in the early stage of model training, such as in 0-5 training rounds, the model currently does not have the ability to learn the posture changes between the first sample image frame and the second sample image frame, so it can. At this time, the larger the first serial number of the sample image frame set, the smaller the corresponding sample selection probability, and the smaller the first serial number of the sample image frame set, the larger the corresponding sample selection probability can be, so as to use more samples with small posture difference values to train the model. After the early model training, the model has learned small posture changes, so in the later stage of model training, such as in 5-10 training rounds, the larger the first serial number of the sample image frame set, the larger the corresponding sample selection probability, and the smaller the first serial number of the sample image frame set, the smaller the corresponding sample selection probability can be, so that the model gradually learns large posture changes.
[0066] For example, taking the sample image frame sets as 5 and the preset order from small to large as an example, in the early stage of model training, according to the order of the first serial number from small to large, the sample selection probabilities corresponding to the 5 sample image frame sets can be 0.45, 0.25, 0.15, 0.1, and 0.05 respectively. In the later stage of model training, the sample selection probabilities corresponding to the 5 sample image frame sets can be 0.05, 0.1, 0.15, 0.25, and 0.45 respectively. In this way, in the model training process, more second image frames with smaller posture difference values are first used as samples for training, and then as the number of training rounds increases, the number of second image frames with larger posture difference values as samples is gradually increased, so that the network model can first learn the reasoning of small posture differences, and then gradually learn the reasoning of large posture differences, so that the first model can be stably and accurately trained.
[0067] In a possible implementation, the sample selection probability corresponding to the sample image frame set may be adjusted in each training round, that is, the sample selection probability of the sample image frame set in each training round is different.
[0068] In another possible implementation, after each preset number of training rounds, the sample selection probability corresponding to the sample image frame set is adjusted. Taking the preset number as 10 as an example, after 10 rounds of training, the sample selection probability corresponding to each sample image frame set can be adjusted, that is, the sample selection probability of each 10 training rounds is the same, and there is no need to adjust the sample selection probability corresponding to each sample image frame set within 10 training rounds.
[0069] In other embodiments, the first probability value may be determined based on the number of sample image frame sets, the second sequence number, and the second preset probability value. When the preset sequence is in the order of posture difference values from small to large, the first selection probability may be reduced by the second preset probability value, and the sample selection probability corresponding to the sample image frame set having a sequence number greater than the second sequence number may be increased by the first probability value. When the preset sequence is in the order of posture difference values from large to small, the first selection probability may be reduced by the second preset probability value, and the sample selection probability corresponding to the sample image frame set having a sequence number less than the second sequence number may be increased by the first probability value.
[0070] For example, taking the preset order as the order from small to large posture difference value, the second preset probability value is 0.1, and the first probability value is 0.1 / (n-first serial number), n is the number of sample image frame sets, and the specific probability dynamic change rule can be: select the first selection probability minus 0.1, and each sample selection probability after the first selection probability plus 0.1 / (n-first serial number). When the first selection probability is less than 0.05, select the next group minus 0.1.
[0071] Taking the sample selection probability set corresponding to the first round of training as {0.45, 0.25, 0.15, 0.1, 0.05} as an example, the first selection probability is 0.45, the second sequence number of the second sample image frame set corresponding to 0.45 is 1, 0.45-0.1=0.35, 0.1 / (5-1)=0.025, and the probability of each group after adjustment is 0.45-0.1, 0.25+0.025, 0.15+0.025, 0.1+0.025, 0.05+0.025, that is, the sample selection probability set for the second round of training is {0.35, 0.275, 0.175, 0.125, 0.075}.
[0072] In the second round of training, the first selection probability in the sample selection probability set is 0.35, and the second serial number corresponding to 0.35 is 1, 0.35-0.1=0.25, 0.1 / (5-1)=0.025. After adjustment, the probability of each group is 0.35-0.1, 0.275+0.025, 0.175+0.025, 0.125+0.025, 0.075+0.025, that is, the sample selection probability set for the third round of training is {0.25, 0.3, 0.2, 0.15, 0.1}.
[0073] In the sample selection probability set for the third round of training, the first selection probability is 0.25, the second serial number corresponding to 0.25 is 1, 0.25-0.1=0.15, 0.1 / (5-1)=0.025. After adjustment, the probability of each group is 0.25-0.1, 0.3+0.025, 0.2+0.025, 0.15+0.025, 0.1+0.025, that is, the sample selection probability set for the fourth round of training is {0.15, 0.325, 0.225, 0.175, 0.125}.
[0074] In the fourth round of training, the first selection probability is 0.15, and the second serial number corresponding to 0.15 is 1, 0.15-0.1=0.05, 0.1 / (5-1)=0.025. After adjustment, the probability of each group is 0.15-0.1, 0.325+0.025, 0.225+0.025, 0.175+0.025, 0.125+0.025, that is, the sample selection probability set for the fifth round of training is {0.05, 0.35, 0.25, 0.2, 0.15}.
[0075] The first selection probability in the sample selection probability set of the fifth round of training is 0.35, the second serial number corresponding to 0.35 is 2, 0.35-0.1=0.25, 0.1 / (5-2)=0.033, and the probability of each group after adjustment is 0.05, 0.35-0.1, 0.25+0.04, 0.2+0.03, 0.15+0.03, that is, the sample selection probability set of the sixth round of training is {0.05, 0.25, 0.29, 0.23, 0.18}.
[0076] In the sample selection probability set of the sixth round of training, the first selection probability is 0.25, the second serial number corresponding to 0.25 is 2, 0.25-0.1=0.15, 0.1 / (5-2)=0.033, and the probability of each group after adjustment is 0.05, 0.25-0.1, 0.29+0.04, 0.23+0.03, 0.18+0.03, that is, the sample selection probability set of the seventh round of training is {0.05, 0.15, 0.33, 0.26, 0.21}.
[0077] In the sample selection probability set of the seventh round of training, the first selection probability is 0.15, the second serial number corresponding to 0.15 is 2, 0.15-0.1=0.15, 0.1 / (5-2)=0.033, and the probability of each group after adjustment is 0.05, 0.15-0.1, 0.33+0.04, 0.26+0.03, 0.21+0.03, that is, the sample selection probability set of the eighth round of training is {0.05, 0.05, 0.37, 0.29, 0.24}.
[0078] The first selection probability in the sample selection probability set of the eighth round of training is 0.37, the second serial number corresponding to 0.37 is 3, 0.37-0.1=0.27, 0.1 / (5-3)=0.05, and the probability of each group after adjustment is 0.05, 0.05, 0.37-0.1, 0.29+0.05, 0.24+0.05, that is, the sample selection probability set of the eighth round of training is {0.05, 0.05, 0.27, 0.34, 0.29}.
[0079] And so on, the sample selection probability in the sample selection probability set is dynamically adjusted.
[0080] Figure 6 is a flow chart showing a method for determining a sample image frame set according to an exemplary embodiment. Figure 6 As shown, a plurality of sample image frame sets are determined by the following steps: Step S301: Acquire a first marking point of a first image frame and a second marking point of each second image frame, wherein the marking point is used to mark a designated part in the image frame.
[0081] For example, for the first image frame and each second image frame, a first landmark point of the first image frame and multiple landmark points of each second image frame can be detected by a pre-trained landmark point detection model. The landmark point detection model can be a 3D landmark detection model, referring to Figure 7 The marker points may be 3D landmarks of each image, and the number of marker points in each image frame may be, for example, 68. By extracting the marker points of the image frame, the posture of the designated part of the first object in the image frame may be quickly and accurately acquired, so as to facilitate the subsequent determination of multiple sample image frame sets according to the posture difference value between the first image frame and the second image frame.
[0082] Step S302: determining a posture difference value according to the first marking point and the second marking point.
[0083] For example, a first posture value of a designated part in a first image frame can be determined according to the position of a first marking point, wherein the posture value can characterize the posture feature of the designated part. For each second image frame, a second posture value of a designated part in the second image frame can be determined according to the position of a second marking point, and then the difference between the first posture value and the second posture value can be used as a posture difference value.
[0084] In some embodiments, reference Figure 7 When the marking point is a three-dimensional marking point and the designated part is a facial organ, the three-dimensional marking point can be projected onto a two-dimensional plane to obtain a two-dimensional marking point, and the posture difference value can be calculated based on the two-dimensional marking point, so as to quickly and accurately obtain the posture value of the facial organ of the first object in the image frame.
[0085] In other embodiments, when the marker point is a three-dimensional marker point and the designated part is the head, the posture difference value can be directly calculated based on the three-dimensional marker point, so that the posture value of the head of the first object in the image frame can be quickly and accurately obtained. The posture value of the head may include at least one of the yaw angle (Yaw), pitch angle (Pitch) and roll angle (Roll) corresponding to the head of the first object in the preset coordinate system. Figure 8 The yaw angle describes the angle of rotation of the head left and right, and its rotation axis is perpendicular to the top of the head and the ground. In face gestures, the yaw angle determines the degree to which a person shakes his head left and right. The pitch angle describes the angle of the head tilting up and down, and its rotation axis is the horizontal axis and perpendicular to the forward direction of the human body. In face gestures, the pitch angle determines the degree to which a person raises or lowers his head. The roll angle describes the rotation of the head along the forward direction axis. In face gestures, the roll angle measures the degree of tilt of the head.
[0086] Taking the designated part as the mouth as an example, the first posture value or the second posture value can represent the degree of mouth opening. The first posture value can be determined according to the mark point of the mouth in the first mark point, and the second posture value can be determined according to the mark point of the mouth in the second mark point. Fig. 9 , the first posture value may be, for example, the distance between the highest mark point and the lowest mark point of the mouth in the first mark point, and the second posture value may be, for example, the distance between the highest mark point and the lowest mark point of the mouth in the second mark point. Accordingly, the first posture value and the second posture value may be calculated according to Formula 1.
[0087] Mouse_open_distance=abs(landmark(63,y) -landmark(67,y)) (Formula 1) Among them, Mouse_open_distance represents the size of the mouth, that is, the distance between point 63 and point 67, and landmark(63,y) represents Fig. 9 The coordinates of point 63, landmark(67,y) Fig. 9 The coordinates of point 67 in the middle.
[0088] The first posture value or the second posture value can also be expressed by the area of the mouth, referring to Fig. 9 The area of the mouth may be the area enclosed by point 61 to point 68, and the present disclosure does not make any specific limitation on this.
[0089] Reference Fig.10Taking the 3D landmarks with N second image frames and marker points in each image as an example, multiple second sample image frames can be selected from the N second image frames according to the sample selection probability, and a first sample image frame and multiple second sample image frames can be used as training sample pairs for model training.
[0090] Step S303: dividing the second image frame into a plurality of sample image frame sets according to the posture difference value.
[0091] For example, the second image frame can be divided into a third sample set and a fourth sample set according to the posture difference value and the preset threshold. In some embodiments, the preset threshold can be 0, and the second image frame with a positive posture difference value can be divided into the third sample set, and the second image frame with a negative posture difference value can be divided into the fourth sample set.
[0092] Taking the designated part as the mouth as an example, the posture difference value can represent the difference in the size of the mouth shape between the first image frame and the second image frame. A positive posture difference value can indicate that the mouth shape of the object in the second image frame is larger than that of the object in the first image frame. When the second image frame with a positive posture difference value is used for model training, an image of a small mouth shape is generated from an image of a large mouth shape. A negative posture difference value can indicate that the mouth shape of the object in the second image frame is smaller than that of the object in the first image frame. When the second image frame with a negative posture difference value is used for model training, an image of a large mouth shape is generated from a small image of the mouth shape. Among them, a large mouth shape can be understood as a mouth shape with a larger mouth opening, and a small mouth shape can be understood as a mouth shape with a smaller mouth opening.
[0093] In some other embodiments, the first image frame may be the first image frame, the second to nth image frames may be the second image frames, the first lip shape of the object in the first image frame may be represented as Infer_mod, and the second lip shape of the object in the second image frame may be represented as Ref_mod. n , the difference between the first and second mouth shapes, that is, the posture difference value can be expressed as Diff_mod n , where Diff_mod n =Ref_mod n -Infer_mod, can be based on Diff_mod n The symbol parameters are divided into the third sample set and the fourth sample set, where Diff_mod n The sign parameter can be used to characterize whether the attitude difference value is positive or negative.
[0094] In other embodiments, the second image frames in the third sample set and the fourth sample set can be sorted according to the preset order of the posture difference value, respectively, to obtain a sorted third sample set and a sorted fourth sample set. Then, the sorted third sample set is evenly divided according to the first preset set division number, and multiple adjacent second image frames are divided into the same set to obtain multiple first sample sets, and the sorted fourth sample set is evenly divided according to the second preset set division number, and multiple adjacent second image frames are divided into the same set to obtain multiple second sample sets. Among them, the first preset set division number and the second preset set division number can be set according to actual training needs, and the first preset set division number and the second preset set division number can be the same or different.
[0095] For example, the preset threshold is 0, the preset order is from small to large, the number of divisions in the first preset set and the number of divisions in the second preset set are both 3, the third sample set includes 6 second image frames A1, B1, C1, D1, E1, and F1, among which the posture difference values corresponding to A1, B1, C1, D1, E1, and F1 are 2, 4, 1, 8, 5, and 3, respectively; the fourth sample set includes A2, B2, C2, D2, E2, and F2, a total of 6 second image frames, among which the posture difference values corresponding to A2, B2, C2, D2, E2, and F2 are -5, -3, -4, -7, -1, and -2, respectively. Then, A1, B1, C1, D1, E1, F1, and A2, B2, C2, D2, E2, F2 can be sorted in ascending order according to the absolute value of the posture difference value, and the sorted third sample set is {C1, A1, F1, B1, E1, D1}, and the sorted fourth sample set is {E2, F2, B2, C2, A2, D2}. Then {C1, A1, F1, B1, E1, D1} is evenly divided into three first sample sets of {C1, A1}, {F1, B1}, and {E1, D1}, and {E2, F2, B2, C2, A2, D2} is evenly divided into three second sample sets of {E2, F2}, {B2, C2}, and {A2, D2}.
[0096] In this way, the multiple second image frames are first divided into a third sample set with positive posture difference values and a fourth sample set with negative posture difference values, and then the multiple second image frames in the third sample set and the fourth sample set are respectively divided into multiple first sample sets and second sample sets. This allows for accurate and fine division to obtain multiple sample image frame sets, which facilitates the flexibility and effectiveness of sample selection for subsequent model training.
[0097] In other embodiments, the designated part may include at least one of a facial organ and a head, and the posture of the head may include at least one of a yaw angle, a pitch angle, and a roll angle corresponding to the head of the first object in a preset coordinate system.
[0098] Taking the designated parts including the mouth and the head as an example, multiple sample image frames corresponding to the mouth can be obtained according to steps S301-S303, and the model can be trained through the multiple sample image frames corresponding to the mouth, so that the model can learn to reason about the mouth shape with large changes, and then multiple sample image frames corresponding to the head can be obtained according to steps S301-S303, and the model can be trained through the multiple sample image frames corresponding to the head, so that the model can learn to reason about the head posture with large changes. The model can also be trained first through the multiple sample image frames corresponding to the mouth, and then the model can be trained according to the multiple sample image frames corresponding to the head, and the present disclosure does not make specific limitations on this.
[0099] It should be noted that when training the posture of the head, when the posture of the head includes the yaw angle, pitch angle and roll angle corresponding to the head of the first object in the preset coordinate system, multiple corresponding sample image frame sets can be selected for the yaw angle, pitch angle and roll angle, respectively, and the model training can be performed in sequence. For example, multiple sample image frame sets corresponding to the yaw angle can be obtained first for model training, and then multiple sample image frame sets corresponding to the pitch angle can be obtained for model training, and finally multiple sample image frame sets corresponding to the roll angle can be obtained for model training. In this way, the training of each part is decoupled, which can ensure the accuracy and effectiveness of the first model.
[0100] In summary, in the model training process, the present invention first uses more samples with smaller posture difference values for training, and then gradually increases the samples with larger posture difference values as the number of training rounds increases. The training samples in the model training process change dynamically, so that the network model can first learn the reasoning of small posture differences, and then gradually learn the reasoning of large posture differences. Therefore, the first model can be trained stably and accurately, and the trained first model can realize the reasoning of large posture changes.
[0101] Fig.11 is a flow chart of a method for generating a video according to an exemplary embodiment. Fig.11 As shown, the method may include: Step S401, acquiring a first object image frame, a second object image frame and a first audio.
[0102] Step S402: Use the first object image frame, the second object image frame and the first audio as inputs of the first model to obtain multiple third object image frames output by the first model. The first model is trained according to the model training method shown in the embodiment of the present disclosure.
[0103] Step S403: Generate a first video according to a plurality of third object image frames.
[0104] For example, the user may input a first object image frame, a second object image frame, and a first audio through a terminal device, wherein the second object image frame is an image frame obtained by masking the first object image frame, and the masked area may be the area where the specified part is located, and the first object image frame, the second object image frame, and the first audio may have no association. Then the first object image frame, the second object image frame, and the first audio are input into the first model, and the first model may complete the masked area of the second object image frame based on the first object image frame and the first audio to obtain a third object image frame, and then a first video matching the first audio may be generated based on the third object image frame.
[0105] Since the model training method in the embodiment of the present disclosure can stably and accurately train the first model, the authenticity and accuracy of the first video generated by the first model are higher.
[0106] Among them, the training method of the first model has been described in detail in the embodiment of the model training method, and will not be elaborated here.
[0107] In summary, in the model training process, the present invention first uses more samples with smaller posture difference values for training, and then gradually increases the samples with larger posture difference values as the number of training rounds increases. The training samples in the model training process change dynamically, so that the network model can first learn the reasoning of small posture differences, and then gradually learn the reasoning of large posture differences. Therefore, the first model can be trained stably and accurately, and the trained first model can realize the reasoning of large posture changes.
[0108] Fig.12 is a block diagram of a model training device according to an exemplary embodiment. Fig.12 As shown, the device 500 includes: The acquisition module 501 is configured to acquire a first sample image frame. The first sample image frame is an image frame obtained by performing mask processing on a first image frame in an image frame set, and the image frame set includes multiple image frames corresponding to the sample video.
[0109] The training module 502 is configured to perform multiple training rounds of model training on the model according to the first sample image frame to obtain a first model.
[0110] For each of the multiple training rounds, a second sample image frame corresponding to the training round is determined from the second image frame in the multiple image frames, and the first image frame, the first sample image frame, the second sample image frame and the sample audio corresponding to the sample video are used as training samples. The number of third image frames in the second sample image frame is positively correlated with the training round, and the third image frame is an image frame whose posture difference value is greater than a preset difference value, and the posture difference value represents the difference between the posture of the specified part of the first object in the first image frame and the posture of the specified part in the second image frame.
[0111] In some embodiments, the training module 502 is configured to: For each training round, a sample selection probability set corresponding to the training round is determined. Each training round corresponds to a sample selection probability set. The sample selection probability set includes the sample selection probability corresponding to each sample image frame set. The sample selection probability represents the probability that an image frame in the sample image frame set is selected as the second sample image frame.
[0112] According to the sample selection probability set corresponding to the training round, a second sample image frame corresponding to the training round is determined from the plurality of first sample sets or the plurality of second sample sets.
[0113] In other embodiments, the multiple sample image frame sets are obtained by dividing the second image frame according to posture difference values, and the multiple sample image frame sets include multiple first sample sets whose posture difference values are greater than a preset threshold, and multiple second sample sets whose posture difference values are less than the preset threshold.
[0114] In some other embodiments, the training module 502 is configured to: The plurality of sample image frame sets are sorted according to a preset sequence of the posture difference values to obtain a first sequence number corresponding to each sample image frame set.
[0115] The sample selection probability set corresponding to the training round is determined according to the sample selection probability set of the previous training round and the first sequence number.
[0116] In some other embodiments, the training module 502 is configured to: Determine a first selection probability in the sample selection probability set of the previous training round, where the first selection probability is the first sample selection probability greater than a first preset probability value and determined in the sample selection probability set according to a preset arrangement order.
[0117] The sample selection probability set corresponding to the training round is determined according to the sample selection probability set of the previous training round and the second sequence number of the sample image frame set corresponding to the first selection probability.
[0118] In some other embodiments, the training module 502 is configured to: A first probability value is determined according to the number of the sample image frame sets, the second sequence number and the second preset probability value.
[0119] When the preset order is in ascending order of posture difference values, the first selection probability is reduced by a second preset probability value, and the sample selection probability corresponding to the sample image frame set with a sequence number greater than the second sequence number is increased by the first probability value. Or, When the preset order is in descending order of posture difference values, the first selection probability is reduced by a second preset probability value, and the sample selection probability corresponding to the sample image frame set with a sequence number less than the second sequence number is increased by the first probability value.
[0120] In some other embodiments, the plurality of sample image frame sets are determined in the following manner: A first marking point of the first image frame and a second marking point of each second image frame are obtained, where the marking point is used to mark a designated part in the image frame.
[0121] A posture difference value is determined according to the first marking point and the second marking point.
[0122] The second image frame is divided into a plurality of sample image frame sets according to the posture difference value.
[0123] In some other embodiments, determining the posture difference value according to the first marking point and the second marking point includes: A first posture value of a designated part in a first image frame is determined according to the position of the first marking point. The posture value represents a posture feature of the designated part.
[0124] For each second image frame, a second posture value of a designated part in the second image frame is determined according to the position of the second marking point.
[0125] The difference between the first posture value and the second posture value is used as the posture difference value.
[0126] In some other embodiments, dividing the second image frame into a plurality of sample image frame sets according to the posture difference value includes: The second image frame is divided into a third sample set and a fourth sample set according to the posture difference value and a preset threshold.
[0127] The second image frames in the third sample set and the fourth sample set are sorted respectively according to a preset order of the posture difference values to obtain a sorted third sample set and a sorted fourth sample set.
[0128] The sorted third sample set is divided according to the first preset set division quantity to obtain multiple first sample sets, and the sorted fourth sample set is divided according to the second preset set division quantity to obtain multiple second sample sets.
[0129] In other embodiments, the designated part includes at least one of a facial organ and a head, and the posture of the head includes at least one of a yaw angle, a pitch angle, and a roll angle corresponding to the head of the first object in a preset coordinate system.
[0130] In summary, in the model training process, the present invention first uses more samples with smaller posture difference values for training, and then gradually increases the samples with larger posture difference values as the number of training rounds increases. The training samples in the model training process change dynamically, so that the network model can first learn the reasoning of small posture differences, and then gradually learn the reasoning of large posture differences. Therefore, the first model can be trained stably and accurately, and the trained first model can realize the reasoning of large posture changes.
[0131] Fig.13 600 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 600 may be provided as a server. Fig.13 The electronic device 600 includes a processor 622, which may be one or more, and a memory 632 for storing a computer program executable by the processor 622. The computer program stored in the memory 632 may include one or more modules each corresponding to a set of instructions. In addition, the processor 622 may be configured to execute the computer program to perform the above-mentioned video generation method or model training method.
[0132] In addition, the electronic device 600 may further include a power supply component 626 and a communication component 650. The power supply component 626 may be configured to perform power management of the electronic device 600, and the communication component 650 may be configured to implement communication of the electronic device 600, for example, wired or wireless communication. In addition, the electronic device 600 may further include an input / output (I / O) interface 658. The electronic device 600 may operate based on an operating system stored in the memory 632, such as Windows Server 2000. TM , Mac OSX TM , Unix TM , Linux TM etc.
[0133] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned video generation method or model training method are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 632 including program instructions, and the above-mentioned program instructions can be executed by the processor 622 of the electronic device 600 to complete the above-mentioned video generation method or model training method.
[0134] Based on the same concept, an embodiment of the present disclosure also provides a computer-readable medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned video generation method or model training method are implemented.
[0135] Based on the same concept, an embodiment of the present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned video generation method or model training method.
[0136] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.
[0137] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.
[0138] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.
Claims
1. A model training method, characterized in that: The method comprises: Acquire a first sample image frame; the first sample image frame is an image frame obtained by performing mask processing on a first image frame in an image frame set, wherein the image frame set includes a plurality of image frames corresponding to a sample video; According to the first sample image frame, performing model training for a plurality of training rounds on the model to obtain a first model; Among them, for each training round of the multiple training rounds, a second sample image frame corresponding to the training round is determined from the second image frame of the multiple image frames, and the first image frame, the first sample image frame, the second sample image frame and the sample audio corresponding to the sample video are used as training samples; the number of third image frames in the second sample image frames is positively correlated with the training round, and the third image frame is an image frame whose posture difference value is greater than a preset difference value, and the posture difference value represents the difference between the posture of a specified part of the first object in the first image frame and the posture of the specified part in the second image frame.
2. The method according to claim 1, characterized in that The step of determining, for each of the plurality of training rounds, from a second image frame in the plurality of image frames, a second sample image frame corresponding to the training round comprises: For each training round, determine a sample selection probability set corresponding to the training round, each training round corresponds to a sample selection probability set, the sample selection probability set includes a sample selection probability corresponding to each sample image frame set, and the sample selection probability represents the probability that an image frame in the sample image frame set is selected as the second sample image frame; According to the sample selection probability set corresponding to the training round, the second sample image frame corresponding to the training round is determined from multiple first sample sets or multiple second sample sets, and the multiple sample image frame sets include the multiple first sample sets and the multiple second sample sets.
3. The method according to claim 2, characterized in that The multiple sample image frame sets are obtained after the second image frame is divided according to the posture difference value, and the multiple sample image frame sets include the multiple first sample sets whose posture difference values are greater than a preset threshold, and the multiple second sample sets whose posture difference values are less than the preset threshold.
4. The method according to claim 3, characterized in that Determining the sample selection probability set corresponding to the training round includes: Sorting the plurality of sample image frame sets according to a preset order of the posture difference values to obtain a first sequence number corresponding to each sample image frame set; The sample selection probability set corresponding to the training round is determined according to the sample selection probability set of the previous training round and the first sequence number.
5. The method according to claim 4, characterized in that Determining the sample selection probability set corresponding to the training round according to the sample selection probability set of the previous training round of the training round and the first sequence number includes: Determine a first selection probability in the sample selection probability set of the previous training round, where the first selection probability is determined according to a preset arrangement order and is the first sample selection probability in the sample selection probability set that is greater than a first preset probability value; The sample selection probability set corresponding to the training round is determined according to the sample selection probability set of the previous training round and the second sequence number corresponding to the sample image frame set corresponding to the first selection probability.
6. The method according to claim 5, characterized in that Determining the sample selection probability set corresponding to the training round according to the sample selection probability set of the previous training round and the second sequence number corresponding to the sample image frame set corresponding to the first selection probability includes: Determine a first probability value according to the number of the sample image frame sets, the second sequence number, and a second preset probability value; In the case where the preset order is in ascending order of posture difference values, the first selection probability is reduced by the second preset probability value, and the sample selection probability corresponding to the sample image frame set having a sequence number greater than the second sequence number is increased by the first probability value; or When the preset order is in descending order of posture difference values, the first selection probability is reduced by the second preset probability value, and the sample selection probability corresponding to the sample image frame set with a sequence number less than the second sequence number is increased by the first probability value.
7. The method according to claim 2, characterized in that: The plurality of sample image frame sets are determined in the following manner: Acquire a first marking point of the first image frame and a second marking point of each of the second image frames, where the marking point is used to mark the designated part in the image frame; Determining the posture difference value according to the first marking point and the second marking point; The second image frame is divided into the plurality of sample image frame sets according to the posture difference value.
8. The method according to claim 7, characterized in that The determining the posture difference value according to the first marking point and the second marking point comprises: Determining a first posture value of the designated part in the first image frame according to the position of the first marking point; the posture value represents a posture feature of the designated part; For each of the second image frames, determining a second posture value of the designated part in the second image frame according to the position of the second marking point; The difference between the first posture value and the second posture value is used as the posture difference value.
9. The method according to claim 7, characterized in that: The dividing the second image frame into the plurality of sample image frame sets according to the posture difference value comprises: Dividing the second image frame into a third sample set and a fourth sample set according to the posture difference value and a preset threshold; sorting the second image frames in the third sample set and the fourth sample set respectively according to a preset order of the posture difference values to obtain the sorted third sample set and the sorted fourth sample set; The sorted third sample set is divided according to the first preset set division quantity to obtain the multiple first sample sets, and the sorted fourth sample set is divided according to the second preset set division quantity to obtain the multiple second sample sets.
10. The method according to any one of claims 1 to 9, characterized in that The designated part includes at least one of a facial organ and a head, and the posture of the head includes at least one of a yaw angle, a pitch angle, and a roll angle corresponding to the head of the first object in a preset coordinate system.
11. A method for generating a video, characterized in that: The method comprises: Acquire a first object image frame, a second object image frame and a first audio, wherein the second object image frame is an image frame obtained by performing mask processing on the first object image frame; Using the first object image frame, the second object image frame and the first audio as inputs of a first model to obtain a plurality of third object image frames output by the first model, wherein the first model is trained by the model training method according to any one of claims 1 to 10; A first video is generated according to the plurality of third object image frames.
12. A model training device, characterized in that: The device comprises: an acquisition module configured to acquire a first sample image frame; the first sample image frame is an image frame obtained by performing mask processing on a first image frame in an image frame set, wherein the image frame set includes a plurality of image frames corresponding to a sample video; A training module is configured to perform a plurality of training rounds of model training on the model according to the first sample image frame to obtain a first model; Among them, for each training round of the multiple training rounds, a second sample image frame corresponding to the training round is determined from the second image frame of the multiple image frames, and the first image frame, the first sample image frame, the second sample image frame and the sample audio corresponding to the sample video are used as training samples; the number of third image frames in the second sample image frames is positively correlated with the training round, and the third image frame is an image frame whose posture difference value is greater than a preset difference value, and the posture difference value represents the difference between the posture of a specified part of the first object in the first image frame and the posture of the specified part in the second image frame.
13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 10 or 11 are implemented.
14. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 10 or 11.
15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 or 11 are implemented.