A video generation method, device, storage medium, and product

By filling in motion base videos and resting base videos based on target audio and motion parameter lists during 2D digital human video generation, the problem of motion editing is solved, improving the generation effect and user experience of digital human videos.

CN123123660APending Publication Date: 2026-11-17MIGU CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610976621.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-01
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, 2D digital human video generation cannot achieve motion editing, resulting in poor generation effects and a poor user experience.

Method used

By obtaining the target audio and motion parameter list, the insertion time interval of the motion base video is determined, and the first filling process and the second filling process of the resting base video are performed to generate a digital human video that matches the target audio, thereby achieving the specified insertion and connection of motion.

Benefits of technology

It enables motion editing of digital human videos, improving the generation effect and user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123660A_ABST
    Figure CN123123660A_ABST
Patent Text Reader

Abstract

This application provides a video generation method, device, storage medium, and product. The method includes: obtaining a target audio and a list of motion parameters, the list of motion parameters including: motion parameters of multiple motions to be inserted, each motion parameter including: identification information of the motion background video corresponding to each motion and the insertion start time; determining the insertion time interval of multiple motion background videos based on the motion parameters of the multiple motions to be inserted; and generating a digital human video matching the target audio by sequentially performing a first filling process for the multiple motion background videos and a second filling process for the resting background video based on the audio duration of the target audio and the insertion time interval of the multiple motion background videos.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of digital human technology, and in particular to a video generation method, device, storage medium and product. Background Technology

[0002] With the rapid development of artificial intelligence technology, the application of digital human (also known as virtual human, virtual avatar, etc.) technology is becoming increasingly widespread. Two-dimensional (2D) digital humans are digital virtual characters presented in a two-dimensional plane form through computer-generated images and synthesized voice.

[0003] Currently, in related technologies, the playback of audio-matched digital human videos is mainly achieved through a 2D digital lip-syncing driving scheme, which generally only loops the base video and cannot achieve motion editing functions, resulting in poor digital human video generation effects. Summary of the Invention

[0004] This application provides a video generation method, device, storage medium, and product that can solve the problem that digital human solutions in related technologies cannot achieve motion editing and improve the generation effect of digital human videos.

[0005] The technical solution of this application embodiment is implemented as follows: This application provides a video generation method, the method comprising: Obtain the target audio and motion parameter list, the motion parameter list includes: motion parameters of multiple motions to be inserted, and the motion parameters of each motion to be inserted include: the identification information of the motion base video corresponding to each motion to be inserted and the insertion start time; Based on the motion parameters of the multiple actions to be inserted, the insertion time interval of the multiple action base video is determined; Based on the audio duration of the target audio and the insertion time interval of the multiple action platform videos, a digital human video matching the target audio is generated by sequentially performing a first filling process for the multiple action platform videos and a second filling process for the resting platform videos.

[0006] In some exemplary embodiments, the first filling process includes: retaining the filling of the first motion base video whose insertion time interval does not overlap with the previous insertion time interval, and skipping the filling of the second motion base video whose insertion time interval overlaps with the previous insertion time interval; The second filling process includes: filling the time intervals of the audio duration other than the insertion time interval of the first action base video with the resting base video.

[0007] In some exemplary embodiments, the first and last frames of the motion base video and the resting base video are identical.

[0008] In some exemplary embodiments, the process of generating a digital human video matching the target audio by sequentially performing a first padding process for the multiple action platform videos and a second padding process for the resting platform videos, based on the audio duration of the target audio and the insertion time interval of the multiple action platform videos, includes: Based on the audio duration of the target audio, an initial result array is generated, wherein the length of the initial result array is determined according to the audio duration, and the element value of each element of the initial result array is set to a preset initial value; Based on the insertion time intervals of the multiple motion platform videos and the initial result array, a processed result array is obtained through the first filling process; wherein, the first filling process includes: for each motion platform video insertion time interval, determining whether the element values ​​of all target elements in the initial result array corresponding to the insertion time interval are all preset initial values; if yes, filling the element values ​​of all target elements with the index numbers of all image frames of the motion platform video; or, if no, keeping the element values ​​of all target elements unchanged; The digital human video is generated based on the resting base video and the processed result array.

[0009] In some exemplary embodiments, generating the digital human video based on the resting substrate video and the processed result array includes: For each time interval in the processed result array where the element value is a preset value, the digital human video is obtained through one or more of the following processes: If the length of the time interval is less than the video length of the resting background video, a first process is performed based on the resting background video. This first process includes: filling the element value of the first element of the time interval with the index number of the first frame of the resting background video; filling the element value of the last element of the time interval with the index number of the last frame of the resting background video; using the number of elements to be filled in the time interval as the sampling quantity, sampling the index numbers of other image frames of the resting background video (excluding the first and last frames), and filling the element values ​​of the other elements of the time interval with the sampled target index numbers. If the length of the time interval is equal to the video length of the resting base video, a second process is performed, wherein the second process includes: sequentially filling the element values ​​of each element of the time interval with the index numbers of each image frame of the resting base video; If the length of the gap time interval is greater than the video length of the resting base video, a third process is performed, wherein the third process includes: determining the number of fillable items based on the gap time interval and the video length of the resting base video; performing the second process a number of times the number of fillable items is performed in the gap time interval, and performing the first process in the remaining time interval.

[0010] In some exemplary embodiments, the plurality of actions to be inserted includes: a first action and a second action that are different from the resting action, and the insertion time interval of the plurality of action base video includes: a first insertion time interval of the first action base video corresponding to the first action and a second insertion time interval of the second action base video corresponding to the second action, wherein the second insertion time interval overlaps with the first insertion time interval; After generating the digital human video that matches the target audio, the method further includes: In the case where there is a partial overlap between the second insertion time interval and the first insertion time interval, the digital human's action is the first action during the first insertion time interval in which the digital human video is played, and the digital human's action is a different action from the second action during the partial overlap interval in which the digital human video is played. Alternatively, if there is a complete overlap between the second insertion time interval and the first insertion time interval, the digital human's action is the first action during the first insertion time interval in which the digital human video is played, and the digital human's action is a different action from the second action during the target time interval in which the digital human video is played, wherein the target time interval is a time interval with the same length as the second insertion time interval after the complete overlap interval.

[0011] In some exemplary embodiments, determining the insertion time interval of the plurality of action baseboard videos based on the action parameters of the plurality of actions to be inserted includes: For each of the multiple motion parameters to be inserted, perform the following operations to obtain the insertion time intervals for the multiple motion base videos: The video length of the motion base video is determined based on the identification information in the motion parameters; The insertion time interval of the motion base video is determined based on the video length and the insertion start time in the motion parameters.

[0012] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the video generation method provided in the embodiments of this application.

[0013] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, they implement the video generation method provided in this application.

[0014] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the video generation method provided in this application.

[0015] The embodiments of this application have the following beneficial effects: This application provides a video generation method, device, storage medium, and product. After obtaining target audio and a list of motion parameters, including motion parameters for multiple actions to be inserted, and each action parameter including the identifier information of the corresponding action background video and the insertion start time, the insertion time interval of multiple action background videos can be determined based on the motion parameters of the multiple actions to be inserted. Next, based on the audio duration of the target audio and the insertion time interval of the multiple action background videos, a digital human video matching the target audio is generated through a first filling process for the multiple action background videos and a second filling process for the resting background videos. Thus, when the digital human video matching the target audio is played, the action background videos filled by the first filling process allow the playback of a specified action (i.e., the action to be inserted) at a specified time (i.e., the insertion start time) of the target audio. Furthermore, the resting background videos filled by the second filling process allow for the connection between specified actions (i.e., the actions to be inserted). This solves the problem that digital human solutions in related technologies cannot achieve motion editing, thereby improving the generation effect of digital human videos. Attached Figure Description

[0016] Figure 1 A flowchart illustrating a video generation method provided in an embodiment of this application; Figure 2 This is a first display schematic diagram of a video generation method provided in an embodiment of this application; Figure 3 This is a second display schematic diagram of a video generation method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0017] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] In the description of this application, references are made to "some exemplary embodiments," which describe a subset of all possible embodiments. However, it is understood that "some exemplary embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0020] In the description of this application, the terms "first," "second," etc., are used merely to distinguish similar objects and do not represent a specific order of objects, nor should they be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in a sequence other than that illustrated or described herein.

[0021] In the description of this application, "multiple" means two or more. Terms such as "one or more," "one or more items," or "one or more items" indicate any one, any two, or more than two of a plurality. For example, including one or more of A, B, and C can mean including any one, any two, or more elements selected from the set consisting of A, B, and C.

[0022] In the description of this application, the terms "module," "unit," or "component" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented, wholly or partially, using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of a larger module or unit that includes the functionality of that module or unit.

[0023] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0024] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0025] A digital human's baseline video is a ready-to-use, semi-finished video that can be used as basic material to generate a complete digital human video. Baseline videos typically feature a solid-color background (such as gray or green), complete actions (such as explanations, demonstrations, or interactions), and visual consistency. A solid-color background facilitates seamless background replacement or compositing with other visual elements. Complete actions mean that the entire process of the target digital human performing a specified action can be shown. Visual consistency means that the digital human's appearance, lighting, and style remain stable throughout the video, presenting a unified visual effect.

[0026] Currently, in related technologies, the playback of audio-matched digital human videos is mainly achieved through a 2D digital lip-syncing driving scheme. This typically involves simply looping the base video while modifying the lip movements to match the audio, without enabling motion editing. As a result, the generated digital human videos are of poor quality, have low personalization, and provide a poor user experience.

[0027] This application provides a video generation method, device, storage medium, and product that can insert a motion background video corresponding to a specified action (i.e., the action to be inserted) at a specified time (i.e., the insertion start time) of the target audio. After a first filling process for multiple motion background videos, a second filling process for a still background video is performed to connect the multiple motion background videos, resulting in a digital human video that matches the target audio. This solves the problem that digital human solutions in related technologies cannot achieve motion editing, improves the generation effect of digital human videos, and enhances the user experience.

[0028] In some exemplary embodiments, the video generation methods provided in the various embodiments of this application are applicable to various scenarios that require the generation of digital human videos, such as news broadcasting, online education, advertising, customer service, entertainment, and banking services.

[0029] The execution entity of the digital human video generation method provided in this application embodiment can be a digital human video generation device, which can be implemented in one or more ways, such as software and hardware. For example, the digital human video generation device can be an electronic device, or the central processing unit (CPU) of the electronic device, or the control module in the electronic device for generating digital human videos, etc. In this application embodiment, the execution of the digital human video generation method by an electronic device is used as an example to illustrate the digital human video generation method provided in this application embodiment.

[0030] Figure 1 This is a flowchart illustrating a video generation method provided in an embodiment of this application. The following will be combined with... Figure 1 Please provide an explanation. For example... Figure 1 As shown, the video generation method may include: S101, obtain the target audio and motion parameter list. The motion parameter list includes: motion parameters of multiple motions to be inserted. The motion parameters of each motion to be inserted include: the identification information of the motion base video corresponding to each motion to be inserted and the insertion start time. S102, Based on the motion parameters of multiple actions to be inserted, determine the insertion time interval of multiple action base videos; S103, based on the audio duration of the target audio and the insertion time interval of multiple action base videos, a digital human video matching the target audio is generated by sequentially performing a first filling process for the multiple action base videos and a second filling process for the resting base videos.

[0031] The target audio can refer to the digital human audio specified by the user for the action editing to be performed.

[0032] The identification information is used to uniquely identify the motion base video of the digital human; for example, the identification information can be an action ID.

[0033] The action to be inserted can refer to the digital human action specified by the user.

[0034] The insertion start time can refer to the time point at which the digital human action is inserted into the audio sequence, as specified by the user.

[0035] For example, at least some of the actions to be inserted can be the same, or they can be different.

[0036] For example, the insertion start times of multiple actions to be inserted are different.

[0037] In some exemplary embodiments, step S101, obtaining the target audio and motion parameter list, may include: obtaining the target audio and motion parameter list input by the user. For example, when a user wants to insert a specified digital human action (i.e., the action to be inserted) at multiple specified moments (i.e., the insertion start moment) of the target audio, the user can input the target audio and motion parameter list, at which point the target audio and motion parameter list can be obtained.

[0038] For example, each background video, such as a resting background video and different action background videos, can be a pre-set background video. In the resting background video, the digital human performs a resting action, such as breathing while stationary. In the action background video, the digital human performs non-resting actions, such as waving, pointing to the upper left corner, pointing to the lower right corner, making a heart shape with hands, or giving a thumbs up.

[0039] In some exemplary embodiments, the various base videos, such as the resting base video and the different action base videos, can be obtained by pre-recording by shooting the model (the required digital human image) in advance, or by pre-generating by generating a large model based on the video in advance.

[0040] Thus, in this embodiment, after obtaining the target audio and a list of motion parameters, the list of motion parameters includes motion parameters for multiple actions to be inserted. Each motion parameter includes the identifier information of the corresponding motion base video and the insertion start time. Based on the motion parameters of the multiple actions to be inserted, the insertion time interval of the multiple motion base videos can be determined. Next, based on the audio duration of the target audio and the insertion time interval of the multiple motion base videos, a digital human video matching the target audio is generated sequentially through a first filling process for the multiple motion base videos and a second filling process for the resting base videos. In this way, when playing the digital human video matching the target audio, the motion base videos filled by the first filling process allow the playback of a specified action (i.e., the action to be inserted) at a specified time (i.e., the insertion start time) of the target audio. Furthermore, the resting base videos filled by the second filling process allow for the connection between specified actions (i.e., the actions to be inserted). This solves the problem that digital human solutions in related technologies cannot achieve motion editing, thereby improving the generation effect of the digital human video and enhancing the user experience.

[0041] In some exemplary embodiments, the first filling process may include: retaining the filling of the first action background video whose insertion time interval does not overlap with the previous insertion time interval, and skipping the filling of the second action background video whose insertion time interval overlaps with the previous insertion time interval; the second filling process may include: filling the time intervals in the audio duration other than the insertion time interval of the first action background video with rest background video. Thus, in the embodiments of this application, through the first filling process and the second filling process, it can be ensured that the start time of most specified actions (i.e., actions to be inserted) is consistent with the specified time (i.e., the insertion start time), solving the problem in the related art of looping background video schemes where the action start time due to overlapping actions differs more and more from the expected time.

[0042] In some exemplary embodiments, the time intervals in the audio duration other than the insertion time interval of the first motion base video may include: the time interval between two adjacent filled first motion base videos, the time interval between the first filled first motion base video and the start time, and the time interval between the last filled first motion base video and the end time.

[0043] In some exemplary embodiments, the first and last frames of each action platform video and the resting platform video are identical. Thus, in this embodiment, because the first and last frames of all platform videos are consistent, natural transitions can be achieved between the action platform videos and between the action platform videos and the resting platform videos, further enhancing the generation effect of the digital human video.

[0044] In some exemplary embodiments, S103, based on the audio duration of the target audio and the insertion time interval of multiple motion platform videos, sequentially through a first padding process for the multiple motion platform videos and a second padding process for the resting platform videos, generates a digital human video matching the target audio, which may include: S1031, Based on the audio duration of the target audio, generate an initial result array, wherein the length of the initial result array is determined according to the audio duration, and the element value of each element of the initial result array is set to a preset initial value; S1032, based on the insertion time intervals of multiple motion base videos and the initial result array, a processed result array is obtained through a first filling process; wherein, the first filling process includes: for each motion base video insertion time interval, determining whether the element values ​​of all target elements in the initial result array corresponding to the insertion time interval are all preset initial values; if yes, filling the element values ​​of all target elements with the index numbers of all image frames of the motion base video; or, if no, keeping the element values ​​of all target elements unchanged; S1033 generates a digital human video based on the resting base plate video and the processed result array.

[0045] In some exemplary embodiments, S1033, generating a digital human video based on the resting substrate video and the processed result array may include: For each time interval in the processed result array where the element value is a preset value, the digital human video is obtained through one or more of the following processes: If the length of the time interval is less than the length of the resting platform video, a first process is performed based on the resting platform video. The first process may include: filling the element value of the first element of the time interval with the index number of the first frame of the resting platform video; filling the element value of the last element of the time interval with the index number of the last frame of the resting platform video; using the number of elements to be filled in the time interval as the sampling number, sampling the index numbers of other image frames of the resting platform video other than the first and last frames, and filling the element values ​​of other elements in the time interval with the sampled target index numbers. When the length of the time interval is equal to the video length of the resting plate video, a second process is performed, wherein the second process may include: filling the element values ​​of each element of the time interval with the index numbers of each image frame of the resting plate video in sequence; If the length of the gap time interval is greater than the video length of the resting background video, a third process is performed. This third process may include: determining the number of fillable elements N based on the gap time interval and the video length of the resting background video; performing the second process N times within the gap time interval; and performing the first process (i.e., filling the remaining time interval with the sampled resting background video) on the remaining time interval. N is a positive integer greater than or equal to 1.

[0046] In some exemplary embodiments, the multiple actions to be inserted may include: a first action and a second action different from the resting action; the insertion time intervals of the multiple action base videos include: a first insertion time interval of the first action base video corresponding to the first action and a second insertion time interval of the second action base video corresponding to the second action, wherein the second insertion time interval overlaps with the first insertion time interval; after generating a digital human video matching the target audio in S103, the video generation method may further include: S104, when there is a partial overlap between the second insertion time interval and the first insertion time interval, the digital human's action is presented as the first action during the first insertion time interval of playing the digital human video, and the digital human's action is presented as other actions different from the second action during the partial overlap interval of playing the digital human video. Alternatively, in S105, if there is a complete overlap between the second insertion time interval and the first insertion time interval, the digital human's action is presented as a first action during the first insertion time interval of playing the digital human video, and the digital human's action is presented as another action different from the second action during the target time interval of playing the digital human video, where the target time interval is a time interval with the same length as the second insertion time interval after all overlapping intervals.

[0047] The digital human's actions may be different from the second action, including: the digital human's actions may be resting actions, or other non-resting actions inserted after the first action at the start time.

[0048] For example, such as Figure 2 As shown, multiple actions to be inserted can include: a first action and a second action, different from the resting action. The first action is a movement pointing to the upper left corner. The insertion start time t1 of the first action is the 2nd second of the target audio, and the duration of the action background video of the first action is 3 seconds, meaning the insertion time interval of the action background video of the first action is [2s, 5s). The second action is a movement pointing to the lower right corner. The insertion start time t2 of the second action is the 4th second of the target audio, and the duration of the action background video of the second action is 2 seconds, meaning the insertion time interval of the action background video of the second action is [4s, 6s). Figure 2 As shown, within [2s, 5s) of playing the digital human video matching the target audio, the digital human's action can be presented as the first action (i.e., the action pointing to the upper left corner), and within [5s, 6s) of playing the digital human video, the digital human's action can be presented as a resting action, and will not be presented as the second action (i.e., the action pointing to the lower right corner).

[0049] For example, such as Figure 3As shown, the multiple actions to be inserted may include: a first action, a second action, and a third action that are different from the resting action. The first action is an action pointing to the upper left corner. The insertion start time t1 of the first action is the 2nd second of the target audio. The duration of the action background video of the first action is 3 seconds, that is, the insertion time interval of the action background video of the first action is [2s, 5s). The second action is an action pointing to the lower right corner. The insertion start time t3 of the second action is the 3rd second of the target audio. The duration of the action background video of the second action is 2 seconds, that is, the insertion time interval of the action background video of the second action is [3s, 5s]. The third action is a heart-making gesture. The insertion start time t4 of the third action is the 5th second of the target audio. The duration of the action background video of the second action is 2 seconds. That is, the insertion time interval of the action background video of the third action is [5s, 7s). Therefore, within [2s, 5s) of playing the digital human video, the digital human's action can be presented as the first action (i.e., the action pointing to the upper left corner). Within [5s, 7s) of playing the digital human video that matches the target audio, the digital human's action can be presented as the third action (i.e., the heart-making gesture), and will not be presented as the second action (i.e., the action pointing to the lower right corner).

[0050] Thus, in this embodiment of the application, when playing a digital human video that matches the target audio, if two adjacent actions among the multiple actions to be inserted specified by the user overlap, the filling of the latter action (the action with a later start time) among the two adjacent actions will be skipped. The latter action (the action with a later start time) will not continue to play after the former action, thereby ensuring a high degree of matching between the digital human actions and the target audio.

[0051] In some exemplary embodiments, determining the insertion time interval of multiple action baseboard videos based on the action parameters of multiple actions to be inserted may include: For each motion parameter among the multiple motion parameters to be inserted, perform the following operations to obtain the insertion time intervals for the multiple motion base videos: The video length of the motion base plate video is determined based on the identification information in the motion parameters; The insertion time interval of the motion base video is determined based on the video length and the insertion start time in the motion parameters.

[0052] For example, the unit for inserting the start time can be seconds (s).

[0053] As an example, the unit for inserting the motion base video time interval can be seconds or frames.

[0054] As an example, each base video (including moving base videos and resting base videos) can include multiple consecutive image frames. Therefore, in chronological order, the multiple image frames of each base video can be divided into: the first frame, the other image frames, and the last frame. That is, each base video can be represented as the index number of multiple consecutive image frames. The index range for the moving base video can be [index_start, index_end), where index_start represents the index number of the first frame, index_end-1 represents the index number of the last frame, and the length of the moving base video, count = index_end - index_start.

[0055] For example, the motion background video corresponding to a 3-second waving motion can include 75 consecutive image frames. Therefore, the index range of the motion background video can be [200, 275), meaning that the waving motion is represented from frame 200 to frame 274, with frame 200 being the first frame and frame 274 being the last frame. Thus, the total number of frames is count = index_end - index_start = 275 - 200 = 75.

[0056] In some exemplary embodiments, when the video length of the motion base video can be the total number of frames count of the motion base video, the starting endpoint of the insertion time interval is the starting frame number start, and the ending endpoint of the insertion time interval is the sum of the starting frame number start and the total number of frames count. The starting frame number start is determined based on the insertion start time K and a preset frame rate. That is, the insertion time interval of the motion base video can be [start, start + count).

[0057] The preset frame rate is used to characterize the frame rate (frames per second, FPS) of the generated digital human video. For example, the preset frame rate can be 25.

[0058] As an example, the starting frame number start is determined based on the insertion start time K and the preset frame rate, and may include: taking the insertion start time K and the frame rate of the preset frame rate (such as 25) and rounding them down.

[0059] As examples, rounding can include: rounding to the nearest integer, rounding down, and rounding up. Rounding to the nearest integer means: if the decimal part is greater than or equal to 0.5, the integer part is incremented by 1; if it is less than 0.5, the decimal part is discarded. Rounding down means: regardless of the decimal part, it is discarded, retaining only the largest integer not greater than the original number. Rounding up means: regardless of the decimal part, if a decimal exists, the integer part is incremented by 1.

[0060] In some other exemplary embodiments, when the video length of the motion base video can be the total duration ti of the motion base video, the starting endpoint of the insertion time interval is the insertion start time K, and the ending endpoint of the insertion time interval is the sum of the insertion start time K and the total duration ti. That is, the insertion time interval of the motion base video can be [K, K+ti).

[0061] For ease of understanding, the video generation method in this application embodiment will be described below with reference to exemplary application examples.

[0062] In some exemplary embodiments, the video generation method may include the following steps: Step A1: Pre-set a resting base video for presenting resting motions and multiple motion base videos for presenting multiple non-resting motions.

[0063] As an example, a resting floor video, such as a 2-second static breathing state.

[0064] As an example, a motion background video, such as a 3-second waving motion, is represented by a series of 75 consecutive image indices, where the index can be [200, 275), meaning that frames 200 to 274 correspond to the waving motion. Each motion background video has a corresponding motion ID, and each motion ID has its corresponding motion image sequence index.

[0065] In this context, the first and last frames of each action baseboard video and the rest baseboard video are kept consistent.

[0066] Step A2: Obtain the target audio and action parameter list input by the user. The action parameter list includes: action parameters for multiple actions to be inserted. The action parameters for each action to be inserted include: the action ID of the action base video corresponding to each action to be inserted and the insertion start time.

[0067] The unit for the start time is seconds.

[0068] Among them, the action ID can only be the action base video corresponding to a non-resting action.

[0069] For example, the frame rate (FPS) of a video is typically 25, meaning that 25 frames are played per second.

[0070] Step A3: Determine the length of the target audio based on its duration.

[0071] For example, if the target audio duration is 40 seconds and the video frame rate is 25, then the corresponding audio length is 40 seconds. 25 = 1000 frames.

[0072] Step A4: Read the action parameters sequentially from the action parameter list.

[0073] For example, the list of action parameters can be shown in Table 1 below, where the symbol # is used to indicate the content.

[0074] Table 1. Schematic diagram of action parameter list

[0075] Step A5: Based on the target audio length, generate an initial result array `array`. The length of the initial result array `array` is initialized to `length`, and the value of each element in the initial result array is set to a preset initial value (e.g., -1). This results in a result array `array` with a length of `length` and each element having a value of -1.

[0076] The filling process can be to first fill the non-resting actions, and then fill the gaps between the non-resting actions with resting actions.

[0077] Step A6, filling the motion base video for non-resting motions, that is, determining the insertion time interval for each motion base video based on the insertion start time, the video length of the motion base video, and the audio duration of the target audio; if all or part of the insertion time interval of the motion base video has already been filled by motion base videos, then skip the filling; otherwise, fill the current motion base video in the insertion time interval.

[0078] As an example, step A6 may include the following steps: Process the action parameter list sequentially. For each action parameter (insertion start time, action ID), perform the following steps: Step A61: Based on the insertion start time of the motion base video in the motion parameters, determine the starting frame number start of the motion base video; For example, taking the Kth action parameter as an example, the starting frame of action K is start = the insertion start time K. Set a preset frame rate (e.g., 25), then round the start value.

[0079] Step A62: Based on the motion ID of the motion base video in the motion parameters, determine the index range [index_start, index_end] of the motion base video.

[0080] The image frame index range of the motion base video is represented as [index_start, index_end), that is, the motion K is represented from the index_start frame to the index_end - 1 frame.

[0081] Step A63: Determine the video length count of the motion base video based on the image frame index range [index_start, index_end).

[0082] For example, if the image frame index range of the motion base video is represented as [index_start, index_end), then the video length count = index_end - index_start).

[0083] Step A64: Based on the starting frame number `start` and the video length `count` of the motion base video, determine the insertion time interval [start, start + count] of the motion base video. That is, the range where motion K should be filled into the result array `array` is [start, start + count].

[0084] In this way, the insertion time interval of multiple action base videos is determined based on the action parameters of multiple actions to be inserted.

[0085] As an example, step A6 may also include the following steps: In sequence, for each time interval of the motion base video insertion, perform the following steps: Step A65: For the insertion time interval of each motion base video, determine whether the element values ​​of all target elements in the initial result array array corresponding to the insertion time interval are all preset initial values. Step A66: If, i.e., if all elements in array[start: start + count] have a value of -1, it means that the range has not been filled yet. That is, the insertion time interval of the current motion base video does not overlap with the insertion time interval of the previous motion base video. In this case, the image frame index range [index_start, index_end) of the motion base video can be filled into the result array [start, start + count). That is, the element values ​​existing in the result array [start, start + count) are filled with the index numbers of all image frames of the motion base video. In this way, the filling of the first motion base video that retains the insertion time interval that does not overlap with the previous insertion time interval is completed.

[0086] Step A67: If no, that is, if not all elements in array[start: start + count] are -1, it means that this range has already been filled with the image frame indices corresponding to other actions. That is, the insertion time interval of the current action's base video overlaps with the insertion time interval of the previous action's base video. In this case, skip filling the image frame index range of the action's base video, and keep the element values ​​in the result array [start, start + count) unchanged. In this way, the filling of the second action's base video that skips the insertion time interval overlapping with the previous insertion time interval is completed.

[0087] This completes the first filling process for multiple motion base videos. At this point, the motion base videos for the inserted and insertable non-resting motions are filled in the result array `array`. However, there may still be gaps between adjacent non-resting motions that are not filled, which are represented by an element value of -1 in the result array `array`. Therefore, step A7 can be executed to fill the positions in the result array `array` with resting motions at the positions where the element value is -1.

[0088] Step A7, filling the resting base video of the resting action, that is, filling the gaps between non-resting actions with resting actions to obtain the generated digital human video.

[0089] As an example, step A7 may include the following steps: Step A71: Determine the set of time intervals ranges=[(s1,e1),...,(sk, ek),...] in the result array array where the element value is -1. Here, the time interval (sk, ek) means that the element value of array[sk:e_k] is all -1, and the resting action is to be filled.

[0090] Step A72, taking the image frame index range of the resting platform video corresponding to the resting action as [s_sil, e_sil) as an example, for each time interval, perform the following operations to obtain the final result array array, which is the digital human video: Step A721: If the length of the gap time interval (such as the gap duration between two action base videos) is less than the length of the rest base video (e_sil - s_sil), then fill the first and last frames of the rest base video with the elements corresponding to the two endpoints of the gap time interval in the result array array, and fill the gap between the first and last frames with interval sampling.

[0091] As an example, if the length of the time interval (ek – sk) is less than the length of the resting floor video (e_sil - s_sil), the element value of the first element (sk) of the time interval is filled with the index number (s_sil) of the first frame of the resting floor video; the element value of the last element (ek) of the time interval is filled with the index number (e_sil-1) of the last frame of the resting floor video; the number of elements to be filled in the time interval (e.g., [sk-1, ek-1]) is used as the number of samples (e.g., ek-sk-2), and the index numbers of other image frames of the resting floor video (e.g., [s_sil, e_sil-1]) other than the first and last frames are sampled, and the element values ​​of other elements in the time interval are filled with the sampled target index numbers (e.g., ek-sk-2).

[0092] Thus, since the time interval is the time interval between two action base videos, it is possible to fill the adjacent positions of the two action base videos with the first and last frames of the rest base video, and to fill the gap between the two action base videos by sampling between the first and last frames.

[0093] Step A722: If the duration of the gap time interval (such as the gap duration between two action base videos) is equal to the duration of the rest base video, then fill the result array array with the corresponding element of the gap time interval to fill all the image frame indices of the rest base video.

[0094] Step A723: If the duration of the interval time interval (such as the interval between two action base videos) is greater than the duration of the rest base video, then calculate the number of times N of the complete rest base video can be filled, fill the rest base video N times in sequence, and fill the remaining duration according to the aforementioned filling method.

[0095] As an example, if the length of the time interval (ek–sk) is greater than the length of the resting platform video (e_sil - s_sil), meaning the length of the interval to be filled is greater than the length of the resting motion, calculate how many complete resting motions can be filled: N = (sk - ek) / (e_sil - s_sil). This means that all image frame indices of the resting platform video with N resting motions can be filled first. The remaining length of the time interval to be filled is left = (ek-sk) - N. (e_sil - s_sil), at this point, the length of the remaining time interval left is less than the length of the resting base video (e_sil - s_sil), so it can be filled according to step A721.

[0096] In this way, the index of the resting action is assigned to all intervals with a value of -1 in the result array, thus completing the filling of gaps between non-resting actions with resting actions.

[0097] This solves the problem that digital human solutions in related technologies cannot achieve motion editing, thereby improving the generation effect of digital human videos and enhancing the user experience.

[0098] Embodiments of this application provide an electronic device that may include: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in memory, implements one or more of the video generation methods provided in the embodiments of this application.

[0099] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. This electronic device can be applied to... Figure 1 In a corresponding embodiment, a video generation method is provided. For example... Figure 4 As shown, the electronic device 400 may include a processor 401, a memory 402, and a bus 403. The various components in the electronic device 400 are coupled together via the bus 403. Wherein: Bus 403 is used to realize the communication connection between processor 401 and memory 402; Memory 402 is used to store computer-executable instructions or computer programs; When the processor 401 executes computer-executable instructions or computer programs stored in the memory 402, it implements the video generation method in one or more of the above exemplary embodiments.

[0100] In some exemplary embodiments, bus 403 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. For example, bus 403 may be divided into a data bus, a power bus, an address bus, a control bus, a status signal bus, etc. For ease of illustration, in... Figure 4 The bus 403 is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0101] In some exemplary embodiments, the electronic device can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals, or it can be implemented as a server. For example, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0102] In some exemplary embodiments, the processor may be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor may be a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or any conventional processor, etc.

[0103] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the video generation method provided in this application can be implemented. For example, ... Figure 1 The video generation method is shown.

[0104] This application provides a computer program product, which includes a computer program or computer-executable instructions. When the computer-executable instructions or the computer program are executed by a processor, the video generation method provided in this application can be implemented. For example, ... Figure 1 The illustrated video generation method. For example, the computer program or computer-executable instructions are stored in a computer-readable storage medium, the processor of the electronic device reads the computer program or computer-executable instructions from the computer-readable storage medium, and the processor executes the computer program or computer-executable instructions, causing the electronic device to perform the video generation method provided in the embodiments of this application.

[0105] In some exemplary embodiments, the aforementioned computer-readable storage medium / memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; or it may be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0106] In some exemplary embodiments, a computer program or computer-executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0107] As an example, a computer program or computer-executable instructions may, but not necessarily, correspond to a file in a file system. It may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0108] As an example, a computer program or computer-executable instructions may be deployed to execute on one electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected by a communication network.

[0109] It should be noted that the descriptions of the above device, storage medium, or product embodiments are similar to the descriptions of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device, storage medium, or product embodiments of this application, those skilled in the art should refer to the descriptions of the method embodiments of this disclosure for understanding. Further details will not be repeated here.

[0110] The features disclosed in the several methods, devices, storage media or product embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments, devices, storage media or product embodiments.

[0111] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A video generation method, characterized in that, The method includes: Obtain the target audio and motion parameter list, the motion parameter list includes: motion parameters of multiple motions to be inserted, and the motion parameters of each motion to be inserted include: the identification information of the motion base video corresponding to each motion to be inserted and the insertion start time; Based on the motion parameters of the multiple actions to be inserted, the insertion time interval of the multiple action base video is determined; Based on the audio duration of the target audio and the insertion time interval of the multiple action platform videos, a digital human video matching the target audio is generated by sequentially performing a first filling process for the multiple action platform videos and a second filling process for the resting platform videos.

2. The method according to claim 1, characterized in that, The first filling process includes: retaining the filling of the first action base video where the insertion time interval does not overlap with the previous insertion time interval, and skipping the filling of the second action base video where the insertion time interval overlaps with the previous insertion time interval; The second filling process includes: filling the time intervals of the audio duration other than the insertion time interval of the first action base video with the resting base video.

3. The method according to claim 1, characterized in that, The first and last frames of both the action base video and the resting base video are identical.

4. The method according to any one of claims 1 to 3, characterized in that, Based on the audio duration of the target audio and the insertion time interval of the multiple action platform videos, a digital human video matching the target audio is generated sequentially through a first filling process for the multiple action platform videos and a second filling process for the resting platform videos, including: Based on the audio duration of the target audio, an initial result array is generated, wherein the length of the initial result array is determined according to the audio duration, and the element value of each element of the initial result array is set to a preset initial value; Based on the insertion time intervals of the multiple motion platform videos and the initial result array, a processed result array is obtained through the first filling process; wherein, the first filling process includes: for each motion platform video insertion time interval, determining whether the element values ​​of all target elements in the initial result array corresponding to the insertion time interval are all preset initial values; if yes, filling the element values ​​of all target elements with the index numbers of all image frames of the motion platform video; or, if no, keeping the element values ​​of all target elements unchanged; The digital human video is generated based on the resting base video and the processed result array.

5. The method according to claim 4, characterized in that, The process of generating the digital human video based on the resting substrate video and the processed result array includes: For each time interval in the processed result array where the element value is a preset value, the digital human video is obtained through one or more of the following processes: If the length of the time interval is less than the video length of the resting background video, a first process is performed based on the resting background video. This first process includes: filling the element value of the first element of the time interval with the index number of the first frame of the resting background video; filling the element value of the last element of the time interval with the index number of the last frame of the resting background video; using the number of elements to be filled in the time interval as the sampling quantity, sampling the index numbers of other image frames of the resting background video (excluding the first and last frames), and filling the element values ​​of the other elements of the time interval with the sampled target index numbers. If the length of the time interval is equal to the video length of the resting base video, a second process is performed, wherein the second process includes: sequentially filling the element values ​​of each element of the time interval with the index numbers of each image frame of the resting base video; If the length of the gap time interval is greater than the video length of the resting base video, a third process is performed, wherein the third process includes: determining the number of fillable items based on the gap time interval and the video length of the resting base video; performing the second process a number of times the number of fillable items is performed in the gap time interval, and performing the first process in the remaining time interval.

6. The method according to claim 1, characterized in that, The plurality of actions to be inserted include: a first action and a second action that are different from the resting action. The insertion time interval of the plurality of action base video includes: a first insertion time interval of the first action base video corresponding to the first action and a second insertion time interval of the second action base video corresponding to the second action. The second insertion time interval overlaps with the first insertion time interval. After generating the digital human video that matches the target audio, the method further includes: In the case where there is a partial overlap between the second insertion time interval and the first insertion time interval, the digital human's action is the first action during the first insertion time interval in which the digital human video is played, and the digital human's action is a different action from the second action during the partial overlap interval in which the digital human video is played. Alternatively, if there is a complete overlap between the second insertion time interval and the first insertion time interval, the digital human's action is the first action during the first insertion time interval in which the digital human video is played, and the digital human's action is a different action from the second action during the target time interval in which the digital human video is played, wherein the target time interval is a time interval with the same length as the second insertion time interval after the complete overlap interval.

7. The method according to any one of claims 1 to 3, characterized in that, The step of determining the insertion time interval of the multiple action base video based on the action parameters of the multiple actions to be inserted includes: For each of the multiple motion parameters to be inserted, perform the following operations to obtain the insertion time intervals for the multiple motion base videos: The video length of the motion base video is determined based on the identification information in the motion parameters; The insertion time interval of the motion base video is determined based on the video length and the insertion start time in the motion parameters.

8. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by a processor, the method as described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by a processor, the method as described in any one of claims 1 to 7 is implemented.