Digital human video polyphonic ringtone generation method and device, equipment, medium and product
By generating digital human video ringtones, using the action change information in the benchmark audio and video library, the problem of poor music and picture correlation in the existing technology is solved, a more attractive personalized experience is achieved, and the video resources are improved. Utilization.
Patent Information
- Application Number
- CN202411085831.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-08
AI Technical Summary
The poor correlation between music and pictures in existing video ringtones leads to a lack of coherence and appeal in user viewing experience, and the production cost of animation video ringtones is high and lacks authenticity and emotional expression.
By determining the action trajectory of the benchmark character in the benchmark video based on the benchmark audio, filtering the target video from the video library, filtering the action change information based on the lyrics and keywords, generating the action trajectory of the digital person, and combining the benchmark audio, target video and digital person. Form a digital human video ringtone.
Improve the correlation between music and the picture, making the generated digital human video ringtone more interesting and attractive, providing users with a personalized and innovative viewing experience, while improving the rational use of video resources and low-popular videos. Utilization.
Smart Images

Figure CN119071389A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of digital human technology, and in particular to a method, device, equipment, medium and product for generating a digital human video ringback tone. Background Art
[0002] At present, in the existing video ringback tone display, the displayed video ringback tone is mainly based on existing video clips or artificially produced animations. In the edited video ringback tone, due to the limitation of video materials, it is often difficult to fully match the rhythm and emotion of the music, resulting in a lack of coherence and appeal in the viewing experience. In the animated video ringback tone, although a higher degree of creativity and personalization can be achieved, the production cost is high, and it lacks authenticity and emotional expression, making it difficult to resonate with users. In summary, in the existing video ringback tone display scheme, the correlation between music and pictures is poor, and it lacks novelty and experience in terms of user viewing experience. Summary of the invention
[0003] The embodiments of the present application provide a method, device, equipment, medium and product for generating digital human video ringback tone to solve the technical problems in existing video ringback tone display solutions, such as poor correlation between music and pictures, lack of novelty and experience in user viewing experience.
[0004] To solve the above technical problems, the embodiments of the present application provide the following aspects:
[0005] In a first aspect, an embodiment of the present application provides a method for generating a digital human video ringback tone, the method comprising:
[0006] Determine a reference video according to the reference audio, and obtain a reference character motion trajectory in the reference video;
[0007] Filtering a target video from a video library according to the reference character motion trajectory, wherein the soundtrack of each video in the video library is the reference audio;
[0008] determining at least one keyword in the lyrics of the reference audio;
[0009] Screening each video in the video library according to the keyword to obtain action change information of the person in the video frame corresponding to the keyword;
[0010] Determining the motion trajectory of the digital human according to the motion change information;
[0011] The reference audio, the target video and the digital human are combined into a digital human video ringtone.
[0012] Optionally, before determining the motion trajectory of the digital human according to the motion change information, the method further includes:
[0013] Obtaining the digital human image information input by the user;
[0014] The digital human is generated according to the digital human image information.
[0015] Optionally, obtaining a reference character motion trajectory in the reference video includes:
[0016] Obtain the skeleton point coordinates of the person in the reference video, and set the center skeleton point coordinates in the skeleton point coordinates; wherein the skeleton point coordinates include at least one of the following: head skeleton point coordinates, neck skeleton point coordinates, hand skeleton point coordinates, arm skeleton point coordinates, and leg skeleton point coordinates;
[0017] Determine the distances between the coordinates of the remaining skeleton points except the coordinates of the central skeleton point and the coordinates of the central skeleton point, and determine the body movements of the character according to the change information of the distances;
[0018] The reference character motion trajectory is generated according to the body motion.
[0019] Optionally, selecting a target video from a video library according to the reference character motion trajectory includes:
[0020] Respectively determine the character motion trajectory in each video in the video library;
[0021] Comparing the character motion trajectories with the reference character motion trajectories respectively for similarity, so as to determine the similarity of each character motion trajectory with respect to the reference character motion trajectory;
[0022] Determine the video containing the character action trajectory corresponding to the similarity greater than or equal to the preset threshold as the candidate target video;
[0023] According to the user's instruction, the target video is determined from the candidate target videos.
[0024] Optionally, each video in the video library has the same video type as the reference video, and the video type includes at least one of the following: dance type, performance type.
[0025] Optionally, the character's motion change information includes at least one of the following: position change information of the character and speed information of the position change information, each keyword corresponds to a group of video frames, and determining the motion trajectory of the digital human according to the motion change information includes:
[0026] Determine position change information of a person in each video frame and speed information of the position change information in a group of video frames corresponding to the keyword, and calculate average position change information and average speed information of the position change information according to the position change information of the person in each video frame and the speed information of the position change information;
[0027] The motion trajectory of the digital human is determined according to the average position change information and the average speed information of the position change information.
[0028] In a second aspect, an embodiment of the present application provides a device for generating a digital human video ringback tone, the device comprising:
[0029] An acquisition module, used to determine a reference video according to a reference audio, and acquire a reference character motion trajectory in the reference video;
[0030] An execution module, configured to select a target video from a video library according to the reference character motion trajectory, wherein the soundtrack of each video in the video library is the reference audio;
[0031] determining at least one keyword in the lyrics of the reference audio;
[0032] Screening each video in the video library according to the keyword to obtain action change information of characters in the video frames corresponding to the keyword;
[0033] Determining the motion trajectory of the digital human according to the motion change information;
[0034] The reference audio, the target video and the digital human are combined into a digital human video ringtone.
[0035] Optionally, the execution module is further used to obtain the image information of the digital human input by the user before determining the motion trajectory of the digital human according to the motion change information;
[0036] The digital human is generated according to the digital human image information.
[0037] Optionally, the acquisition module is further used to acquire the skeleton point coordinates of the person in the reference video, and set the center skeleton point coordinates in the skeleton point coordinates; wherein the skeleton point coordinates include at least one of the following: head skeleton point coordinates, neck skeleton point coordinates, hand skeleton point coordinates, arm skeleton point coordinates, and leg skeleton point coordinates;
[0038] Determine the distances between the coordinates of the remaining skeleton points except the coordinates of the central skeleton point and the coordinates of the central skeleton point, and determine the body movements of the character according to the change information of the distances;
[0039] The reference character motion trajectory is generated according to the body motion.
[0040] Optionally, the execution module is further used to respectively determine the character action trajectory in each video in the video library;
[0041] Comparing the character motion trajectories with the reference character motion trajectories respectively for similarity, so as to determine the similarity of each character motion trajectory with respect to the reference character motion trajectory;
[0042] Determine the video containing the character action trajectory corresponding to the similarity greater than or equal to the preset threshold as the candidate target video;
[0043] According to the user's instruction, the target video is determined from the candidate target videos.
[0044] Optionally, each video in the video library has the same video type as the reference video, and the video type includes at least one of the following: dance type, performance type.
[0045] Optionally, the character's motion change information includes at least one of the following: the character's position change information and speed information of the position change information, each keyword corresponds to a group of video frames, and the execution module is further used to determine the character's position change information and speed information of the position change information in each video frame in the group of video frames corresponding to the keyword, and calculate the average position change information and the average speed information of the position change information according to the character's position change information and speed information of the position change information in each video frame;
[0046] The motion trajectory of the digital human is determined according to the average position change information and the average speed information of the position change information.
[0047] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of a method for generating a digital human video ringtone as described in the first aspect are implemented.
[0048] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a method for generating a digital human video ringtone as described in the first aspect are implemented.
[0049] In a fifth aspect, an embodiment of the present application provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of a method for generating a digital human video ringback tone as described in the first aspect.
[0050] Therefore, the baseline audio is combined with the target video, and the digital human's action performance is used to make the final generated digital human video ringtone more interesting and attractive, providing users with a more personalized and innovative call experience; the target video is not limited to the baseline video, but is more diverse, and is conducive to improving the rational use and playback of video resources in the resource library and increasing the utilization rate of low-heat videos; the digital human's actions are consistent with the content of the music lyrics, making it more interesting. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0052] Figure 1 A flowchart of a method for generating a digital human video ringback tone provided in an embodiment of the present application;
[0053] Figure 2 A schematic diagram of a character's motion lines in a reference video provided in an embodiment of the present application;
[0054] Figure 3 A flowchart of a method for generating a digital human video ringback tone provided in an embodiment of the present application;
[0055] Figure 4 A schematic diagram of a display effect of a digital human video ringback tone provided in an embodiment of the present application;
[0056] Figure 5 A structural block diagram of a device for generating a digital human video ringback tone provided in an embodiment of the present application;
[0057] Figure 6 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0059] Figure 1 A method for generating a digital human video ringback tone according to an embodiment of the present application is shown. Figure 1 As shown, the method includes:
[0060] Step S101, determining a reference video according to a reference audio, and obtaining a reference character motion trajectory in the reference video;
[0061] Step S102, selecting a target video from a video library according to a reference character motion trajectory;
[0062] Among them, the soundtrack of each video in the video library is the baseline audio;
[0063] Step S103, determining at least one keyword in the lyrics of the reference audio;
[0064] Step S104, screening each video in the video library according to the keyword to obtain action change information of the characters in the video frames corresponding to the keyword;
[0065] Step S105, determining the motion trajectory of the digital human according to the motion change information;
[0066] Step S106: merge the reference audio, the target video and the digital human into a digital human video ringtone.
[0067] In step S101, firstly, a reference audio and a reference video need to be determined. The selection of the reference audio is usually based on specific features of the audio, such as a popular song, a conversation, a music clip, etc. Selecting a reference video corresponding to the reference audio means finding video material related to the audio, which may be a music video, a movie clip, an advertising video, etc. Such pairing can help ensure that the audio and video match each other in content and emotion, so as to better achieve synchronization and fusion of audio and video in subsequent video processing. In the scenario shown in the embodiment of the present application, the reference audio may be a song that the called party user likes, and the corresponding reference video may be the official MV (Music Video) of the song.
[0068] In step S101, a reference character motion track in the reference video is also obtained. In a possible implementation, obtaining the reference character motion track in the reference video includes: obtaining the skeleton point coordinates of the character in the reference video, and setting the center skeleton point coordinates in the skeleton point coordinates; wherein the skeleton point coordinates include at least one of the following: head skeleton point coordinates, neck skeleton point coordinates, hand skeleton point coordinates, arm skeleton point coordinates, and leg skeleton point coordinates; determining the distance from the remaining skeleton point coordinates to the center skeleton point coordinates except the center skeleton point coordinates in the skeleton point coordinates, and determining the body motion of the character according to the distance change information; generating the reference character motion track according to the body motion.
[0069] In a specific application scenario, first, video data containing character joint information in an MV clip can be input: the video data is obtained according to the target detection algorithm and output in the form of a five-dimensional matrix (N, C, T, V, M), where N is the amount of video data, that is, the frame number of the video. If it is the first frame, N=1; C is the joint feature vector, including (x, y, acc), x and y are the coordinates of the human joints, and acc is the coordinate confidence; T is the number of key frames extracted from the video; V represents the number of joints, assuming it is 25 joints; M is the number of people in a video, and then the input data is subjected to BN (Batch Normalization) processing to facilitate subsequent data processing.
[0070] Secondly, the time and space information of the video data is obtained based on the normalized data in the previous step, and the time and space edges of the corresponding trajectory are generated: GCN (Graph Convolutional Network) and TCN (Temporal Convolutional Network) are used alternately to normalize the video joint data input in the above step and then extract features to obtain a feature vector containing time and space information. Spatial information refers to the position coordinates of the character and the coordinate information of the skeleton points in a unit of time, and the time information is mainly the change information of the character's position on the time axis and the change information of the skeleton point position. Finally, the feature vector containing time and space information is used as the skeleton point representation for behavior recognition.
[0071] Then, the video content is calculated by the position change of individual bone points and the position change of the global character: the central bone point of the human body is assumed to be the neck. First, the action of the person is judged based on the change in the distance calculation results between different bone points and the central bone point of the same person; then the change in the situation on the field is judged based on the change in the distance calculation results between the bone points of different people and the central bone point of a certain person. Finally, the action of each person in the video is obtained. Finally, the overall event of the video can be obtained, and the collective action event in the video can be inferred based on the action and position changes of all people. Assuming that the action of several people in the obtained result is dancing, combined with the position change results of all people in the field, it can be concluded that it is a multi-person dance.
[0072] In one possible implementation, action lines can be drawn according to the action. For example, time edges and space edges are generated according to the time and space information obtained in the above steps, and multiple edges can be formed by connecting the obtained bone points, including space edges that conform to the natural connection of joints. At the same time, after combining the time information obtained in the above steps, there will also be time edges connecting the same joint in consecutive time steps, that is, the position information of the same joint will change at different time points. By connecting the coordinates of the same joint at different time points, we can get a line of the same joint in continuous time, which is called a time edge, such as Figure 2 shown.
[0073] It should be noted that the generation process of time edges and space edges is to build a multi-layer spatiotemporal graph convolution based on the previous step. The generation process of time edges has been explained in the above steps, and the space edges are the lines connecting each joint at the same time. By lengthening the time axis and zooming in on the space to the entire screen, the time edge and space edge information of joint changes of multiple people over a period of time can be obtained. It allows information to be integrated along the two dimensions of space and time. The joint points and space edges of a single human body, combined with the time dimension, generate time edges.
[0074] In step S102, target videos may be selected from a video library according to the reference character motion trajectory, wherein the soundtrack of each video in the video library is the reference audio.
[0075] And in a possible implementation, step S102, screening target videos from the video library according to the benchmark character motion trajectory includes: respectively determining the character motion trajectory in each video in the video library; respectively comparing the character motion trajectory with the benchmark character motion trajectory to determine the similarity of each character motion trajectory relative to the benchmark character motion trajectory; determining the video containing the character motion trajectory corresponding to the similarity greater than or equal to a preset threshold as the candidate target video; and determining the target video from the candidate target videos according to the user's instructions.
[0076] Each video in the video library has the same video type as the benchmark video, and the video type includes at least one of the following: dance type and performance type.
[0077] It should be noted that, optionally, the above screening rules are progressive, and first, the video with the soundtrack as the reference audio can be screened out, and then the video of the same type as the reference video can be determined, and finally the video with the matching action trajectory can be determined as the target video according to the above similarity matching method. Optionally, according to the actual needs of the user, one of the three rules can be selected as a screening rule, or two screening rules can be selected at random to combine, and the embodiment of the present application does not limit this.
[0078] Correspondingly, the temporal and spatial information of the candidate video can be used to calculate the similarity with the temporal and spatial information of the music MV obtained in the above steps according to the Frechet distance, and the video clips with the highest similarity ranking according to the system rules can be used as the video clips matched by the system for the music, and can be randomly selected as reference resources for the next digital human dance movement fusion innovation. The final video clips thus selected provide new reference materials for the next digital human dance movements, provide inspiration for the fusion innovation of digital human dance movements, bring new visual experiences to users, and at the same time drive the utilization rate and value of other dance videos in the resource library.
[0079] In step S103 to step S104, at least one keyword in the lyrics of the reference audio is determined, and each video in the video library is screened according to the keyword to obtain the action change information of the character in the video frame corresponding to the keyword.
[0080] It should be noted that keywords can be characters, scenes, actions, emotions, etc. Assuming that the benchmark audio has a total of 3 minutes and there is a keyword in the 30th second, then in the video with the benchmark audio as the soundtrack, when the keyword is played, that is, the benchmark audio is played to the 30th second, the corresponding video frame in the video is the video frame corresponding to the keyword.
[0081] In step S105 to step S106, the character's action change information includes at least one of the following: the character's position change information and the speed information of the position change information. Each keyword corresponds to a group of video frames, such as Figure 3 As shown, step S105, determining the motion trajectory of the digital human according to the motion change information includes:
[0082] Step S301, determining position change information and speed information of the position change information of a person in each video frame in a group of video frames corresponding to the keyword;
[0083] Step S302, calculating average position change information and average speed information of the position change information according to the position change information and speed information of the position change information of each video frame;
[0084] Step S303: Determine the motion trajectory of the digital human according to the average position change information and the average speed information of the position change information.
[0085] In a specific application scenario. All video clips of the music can be selected, and the extracted lyrics can be used as a basis for the next judgment, which is referred to as the "reference data set" below. Split the lyrics content by characters, scenes, actions, and emotional words, and find the character image frame when the word appears at the time point when the word appears in the reference data set in the above steps. In this way, multiple frames of character images of characters, scenes, actions, and emotional words appearing in the corresponding clips will be obtained. Table 1 can be obtained after sorting:
[0086] Table 1
[0087] Word (W) Reference dataset video ID Time point T Image frame F W_1 ID_1 T_1 F_1,F_2,... W_2 ID_1 T_2 F_7,F_8,... ... ... ... ... W_n ID_m T_t F_x1,F_x2,...,F_x
[0088] Table 1 shows the correspondence between keywords and key frames. Key frames can be screened for quality through algorithms such as image quality detection and subject detection to retain clear and high-quality image frames for subsequent data analysis.
[0089] According to the multi-frame character images in the table above, several data can be obtained after statistics: First, the character's limb position information before and after the word (head angle, gesture height, body amplitude, leg movement and other position information) can be obtained based on the image frame. Combining the above information, the change of position information can also be obtained. For example, the character's head in the fragments before and after the word slowly shifts 30° from left to right, or the leg is quickly lifted and kicked out. In summary, the position information and change speed of the key parts of the body corresponding to the characters, scenes, actions, and emotional words can be obtained. Each part has different statistical standards and reference systems. Body parts include but are not limited to the four categories listed in Table 2 below, and the statistics are all average values:
[0090] Table 2
[0091]
[0092]
[0093] Among them, the limb position information and change duration in Table 2 are all average values. There are 7 slow changes, so 7 pieces of head start position information, end position information, change duration and change position information can be obtained. Assuming that the head start position information S_L is calculated, if the angle is 30° to the left, it is recorded as -30, and the angle is 30° to the right, it is recorded as 30. The average value of S_L can be calculated according to the following formula:
[0094]
[0095] In the calculation of the average value of the head start position information S_L, m is 7, |S_L i| is the absolute value of the starting position information value of the i-th head. This is just an example. Each part has different statistical standards and reference systems, which can be set according to actual needs. From this, the position and speed of each part of the character's body corresponding to each word can be obtained. According to the above table, the speed distribution of each word is obtained. The speed distribution of all words is combined, and the average limb position information and position change speed corresponding to the speed with the most speed distribution are taken as the reference value of the rhythm frequency of the digital human's gestures and limb movements.
[0096] Finally, the music selected by the user, the video that matches the audio, and the corresponding movements and rhythm of the digital human can be used to generate the target digital human video ringtone. Finally, a digital human video ringtone that matches the video and music content is presented and displayed on the caller's mobile phone.
[0097] In addition, it should be noted that before controlling the motion trajectory of the digital human according to the motion change information, the method also includes: obtaining the digital human image information input by the user; generating the digital human according to the digital human image information. In other words, the user can upload a personal image photo to the system in advance, and the system generates a corresponding digital human image according to the user's personal image. It is also possible to create a special digital human image according to the video content after matching the video content in step 2, or let the user set the digital human image according to his or her own preferences, for example, the football game can be replaced with a certain team's jersey image, or it can be replaced with a digital human image in a TV series, etc.
[0098] The method shown in the embodiment of the present application can find video content that matches the music according to the favorite music of the called party user, and thus generate digital human movements that match the music rhythm and video content. Before the called party answers the call, the music and video content and the digital human with movements are displayed on the mobile phone screen of the caller, realizing the end-to-end application of digital human video ringback tone. The ringback tone display effect is as follows: Figure 4 shown.
[0099] In summary, the method shown in the embodiment of the present application calculates the similarity of the time table and spatial edges generated by the graph convolution network and the temporal convolution network to find other video resources that match the rhythm of the music MV, and then generates the digital human movements and changing rhythms by analyzing and calculating the character movement positions and changing speeds of the video resources. The similarity calculation of the video content and the auxiliary generation of digital human movement capabilities are used to generate digital human video ringtones, which can improve the user experience.
[0100] The method shown in the embodiment of the present application has the following technical effects: finding other video resources that match the rhythm of the music MV, which are more consistent with the music and the original MV content, so that users have both a new visual experience and a certain relevance to the original MV, which will not make users feel discordant, increase the user's interest and freshness, and improve the user experience; automatically generate digital human movements and changing rhythms according to the ringtone music and video content, which is richer and more harmonious than traditional music and video presentations, and the user experience will be better; the method of generating digital human video ringtones is related to the original video, introduces new video materials, and generates digital human movements that are strongly related to the video content, creating a new application scenario combined with digital humans, and bringing a new visual experience to video ringtone users.
[0101] Figure 5 A device for generating a digital human video ring back tone according to an embodiment of the present application is shown. Figure 5 As shown, the device 50 includes:
[0102] An acquisition module 501 is used to determine a reference video according to a reference audio, and acquire a reference character motion trajectory in the reference video;
[0103] An execution module 502 is used to select a target video from a video library according to a reference character motion trajectory, wherein the soundtrack of each video in the video library is the reference audio;
[0104] determining at least one keyword in the lyrics of the reference audio;
[0105] Screen each video in the video library according to the keyword to obtain the action change information of the characters in the video frames corresponding to the keyword;
[0106] Determine the motion trajectory of the digital human based on the motion change information;
[0107] The reference audio, target video and digital human are combined into a digital human video ringback tone.
[0108] In a possible implementation, the execution module 502 is further used to obtain the digital human image information input by the user before determining the motion trajectory of the digital human according to the motion change information;
[0109] Generate a digital human based on the digital human image information.
[0110] In a possible implementation, the acquisition module 501 is further used to acquire the skeleton point coordinates of the person in the reference video, and set the center skeleton point coordinates in the skeleton point coordinates; wherein the skeleton point coordinates include at least one of the following: head skeleton point coordinates, neck skeleton point coordinates, hand skeleton point coordinates, arm skeleton point coordinates, and leg skeleton point coordinates;
[0111] Determine the distances between the coordinates of the remaining skeleton points except the coordinates of the central skeleton point and the coordinates of the central skeleton point, and determine the body movements of the character according to the distance change information;
[0112] Generate benchmark character motion trajectories based on body movements.
[0113] In a possible implementation, the execution module 502 is further used to respectively determine the character motion trajectory in each video in the video library;
[0114] Comparing the similarity of the character's motion trajectory with the benchmark character's motion trajectory respectively, so as to determine the similarity of each character's motion trajectory with respect to the benchmark character's motion trajectory;
[0115] Determine the video containing the character action trajectory corresponding to the similarity greater than or equal to the preset threshold as the candidate target video;
[0116] According to the user's instruction, a target video is determined from candidate target videos.
[0117] In a possible implementation, each video in the video library has the same video type as the reference video, and the video type includes at least one of the following: dance type and performance type.
[0118] In a possible implementation, the character's motion change information includes at least one of the following: character's position change information and speed information of the position change information, each keyword corresponds to a group of video frames, and the execution module 502 is further used to determine the character's position change information and speed information of the position change information in each video frame in the group of video frames corresponding to the keyword, and calculate the average position change information and the average speed information of the position change information according to the character's position change information and speed information of the position change information in each video frame;
[0119] The motion trajectory of the digital human is determined according to the average position change information and the average speed information of the position change information.
[0120] In summary, it is possible to combine the baseline audio with the target video, and through the action performance of the digital human, the final generated digital human video ringtone is more interesting and attractive, providing users with a more personalized and innovative call experience; the target video is not limited to the baseline video, but is more diverse, and is conducive to improving the rational use and playback of video resources in the resource library, and increasing the utilization rate of low-heat videos; the actions of the digital human are consistent with the content of the music lyrics, which is more interesting.
[0121] The present application embodiment also provides an electronic device 60, such as Figure 6As shown, it includes: a processor 601, a memory 602, and a program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor, the steps of the method for generating a digital human video ringback tone as shown in the above embodiment are implemented.
[0122] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for generating a digital human video ringback tone shown in the above method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0123] The embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the steps of the method for generating a digital human video ringtone shown in the above method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.
[0124] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0125] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0126] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A method for generating a digital human video ringback tone, characterized in that: The method comprises: Determine a reference video according to the reference audio, and obtain a reference character motion trajectory in the reference video; Filtering a target video from a video library according to the reference character motion trajectory, wherein the soundtrack of each video in the video library is the reference audio; determining at least one keyword in the lyrics of the reference audio; Screening each video in the video library according to the keyword to obtain action change information of characters in the video frames corresponding to the keyword; Determining the motion trajectory of the digital human according to the motion change information; The reference audio, the target video and the digital human are combined into a digital human video ringback tone.
2. The method according to claim 1, characterized in that Before determining the motion trajectory of the digital human according to the motion change information, the method further includes: Obtaining the digital human image information input by the user; The digital human is generated according to the digital human image information.
3. The method according to claim 1, characterized in that Obtaining the reference character motion trajectory in the reference video includes: Obtain the skeleton point coordinates of the person in the reference video, and set the center skeleton point coordinates in the skeleton point coordinates; wherein the skeleton point coordinates include at least one of the following: head skeleton point coordinates, neck skeleton point coordinates, hand skeleton point coordinates, arm skeleton point coordinates, and leg skeleton point coordinates; Determine the distances between the coordinates of the remaining skeleton points except the coordinates of the central skeleton point and the coordinates of the central skeleton point, and determine the body movements of the character according to the change information of the distances; The reference character motion trajectory is generated according to the body motion.
4. The method according to claim 1, characterized in that Screening a target video from a video library according to the benchmark character motion trajectory includes: Respectively determine the character motion trajectory in each video in the video library; Comparing the character motion trajectories with the reference character motion trajectories respectively for similarity, so as to determine the similarity of each character motion trajectory with respect to the reference character motion trajectory; Determine the video containing the character action trajectory corresponding to the similarity greater than or equal to the preset threshold as the candidate target video; According to the user's instruction, the target video is determined from the candidate target videos.
5. The method according to claim 1, characterized in that Each video in the video library has the same video type as the reference video, and the video type includes at least one of the following: dance type and performance type.
6. The method according to claim 1, characterized in that The character's motion change information includes at least one of the following: the character's position change information and the speed information of the position change information. Each keyword corresponds to a group of video frames. According to the motion change information, determining the motion trajectory of the digital human includes: Determine position change information of a person in each video frame in a group of video frames corresponding to the keyword and speed information of the position change information; Calculate average position change information and average speed information of the position change information according to position change information of the person in each video frame and speed information of the position change information; The motion trajectory of the digital human is determined according to the average position change information and the average speed information of the position change information.
7. A device for generating a digital human video ringback tone, characterized in that: The device comprises: An acquisition module, used to determine a reference video according to a reference audio, and acquire a reference character motion trajectory in the reference video; An execution module, configured to select a target video from a video library according to the reference character motion trajectory, wherein the soundtrack of each video in the video library is the reference audio; determining at least one keyword in the lyrics of the reference audio; Screening each video in the video library according to the keyword to obtain action change information of characters in the video frames corresponding to the keyword; Determining the motion trajectory of the digital human according to the motion change information; The reference audio, the target video and the digital human are combined into a digital human video ringback tone.
8. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the method for generating a digital human video ringback tone as described in any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the program is executed by the processor, the steps of the method for generating a digital human video ringback tone as described in any one of claims 1 to 6 are implemented.
10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method for generating a digital human video ringback tone as described in any one of claims 1 to 6.
Citation Information
Patent Citations
Music short video generation method and device, electronic equipment and storage medium
CN111935537A
Generation method and device of digital human animation, electronic equipment and storage medium
CN115830193A
Digital human video generation method and device, electronic equipment and storage medium
CN117041664A
Method and system for generating digital human video based on video material
CN117119123A
Video data processing method, apparatus and device, and computer-readable storage medium
WO2021143103A1
