Digital human playing method, device and equipment and computer readable storage medium
By generating target audio and virtual instruments, combining the Merr frequency cepshot coefficient to identify the instrument characteristics, control the movement status of digital people, and realize the performance of virtual instruments by digital people, solving the high cost problem caused by real-person motion capture, reducing production costs and increasing production costs Video production efficiency.
Patent Information
- Application Number
- CN202411254611.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2025-01-07
AI Technical Summary
The method of driving digital people to play musical instruments through real-person motion capture in the prior art leads to high production costs and lacks efficient cost reduction methods.
By obtaining audio demand information, generating target audio and determining virtual instruments, identifying instrument features using the Mel frequency cepshot coefficient, generating virtual instruments, and controlling the movement status of digital people according to tone and loudness, the digital people can use virtual instruments to play target audio.
Digital people can play virtual instruments without real-person motion capture, reducing production costs, speeding up the video production cycle and improving the video nature.
Smart Images
Figure CN119274520A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a digital human playing method, device, equipment and computer-readable storage medium. Background Art
[0002] As digital humans become more and more vivid, the market size is also expanding. When driving digital humans to perform, for example, driving digital humans to play musical instruments, real-life motion capture is usually used to map real-life movements onto digital humans. However, this method of driving digital humans to perform through real-life motion capture will result in higher production costs. Summary of the invention
[0003] In view of this, the present application provides a digital human performance method, device, equipment and computer-readable storage medium for reducing the production cost of driving digital human performance.
[0004] In order to achieve the above-mentioned purpose, in a first aspect, an embodiment of the present application provides a digital human playing method, comprising:
[0005] Get audio demand information;
[0006] Generate target audio corresponding to the audio requirement information;
[0007] Generate a virtual musical instrument and a digital human corresponding to the target audio;
[0008] The digital human is driven to play the target audio using the virtual musical instrument.
[0009] In a possible implementation of the first aspect, the generating the target audio corresponding to the audio requirement information includes:
[0010] The audio requirement information is input into a music generation model to obtain the target audio.
[0011] In a possible implementation manner of the first aspect, the audio demand information includes at least one of the following information: music style, duration, scene, beat, and genre.
[0012] In a possible implementation manner of the first aspect, the audio requirement information includes: preset requirement information and / or requirement information input by a user.
[0013] In a possible implementation manner of the first aspect, generating a virtual musical instrument corresponding to the target audio includes:
[0014] Dividing the target audio into a plurality of sub-audios;
[0015] The instrument used by each sub-audio is determined by the Mel frequency cepstral coefficient;
[0016] A virtual musical instrument corresponding to the determined musical instrument is generated.
[0017] In a possible implementation of the first aspect, when the target audio includes lyrics, the method further includes:
[0018] The digital human's lip shape is controlled to match the lyrics.
[0019] In a possible implementation manner of the first aspect, driving the digital human to play the target audio using the virtual musical instrument includes:
[0020] Obtaining the sounding time, pitch and loudness of each musical instrument in the target audio;
[0021] At the moment of sounding, the motion state of the virtual musical instrument is controlled according to a first corresponding relationship, and the motion state of the digital human is driven according to a second corresponding relationship, wherein the first corresponding relationship is the corresponding relationship between the pitch and loudness and the playing state of the musical instrument, and the second corresponding relationship is the corresponding relationship between the motion state of the digital human and the motion state of the virtual musical instrument.
[0022] In a second aspect, an embodiment of the present application provides a digital human performance device, comprising:
[0023] An acquisition module, used to obtain audio demand information;
[0024] A generating module, used to generate target audio corresponding to the audio requirement information; and to generate a virtual musical instrument and a digital human corresponding to the target audio;
[0025] A driving module is used to drive the digital human to play the target audio using the virtual musical instrument.
[0026] In a third aspect, an embodiment of the present application provides a digital human performance device, including: a memory and a processor, the memory is used to store a computer program; the processor is used to execute the method described in the first aspect or any implementation method of the first aspect when calling the computer program.
[0027] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in the first aspect or any one of the embodiments of the first aspect is implemented.
[0028] The digital human performance method, device, equipment and computer-readable storage medium provided in the embodiments of the present application include: obtaining audio demand information; generating target audio corresponding to the audio demand information; generating virtual musical instruments and digital human corresponding to the target audio; and driving the digital human to play the target audio using the virtual musical instruments. The technical solution provided in the present application can reduce the production cost of driving the digital human performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 A schematic diagram of a method flow of a digital human playing method provided in an embodiment of the present application;
[0030] Figure 2 A schematic flow chart of a method for generating a virtual musical instrument and a digital human provided in an embodiment of the present application;
[0031] Figure 3 A schematic flow chart of a method for driving a digital human to play target audio provided in an embodiment of the present application;
[0032] Figure 4 A schematic diagram of the structure of a digital human performance device provided in an embodiment of the present application;
[0033] Figure 5 A schematic diagram of the structure of a digital human performance device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0034] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.
[0035] Figure 1 A schematic diagram of a method flow of a digital human playing method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method comprises the following steps:
[0036] Step S110: Acquire audio demand information.
[0037] The audio requirement information may be used to indicate the user's personalized requirements for generated audio, and may include one or more of the music style, duration of the generated audio, scene, beat, and genre.
[0038] The audio demand information may be preset demand information, so that the audio demand information can be input into the music generation model, which can reduce the user's operation process and improve efficiency. The audio demand information may also be demand information input by the user, in the form of text, voice, video, picture, audio, etc.
[0039] Step S120: Generate target audio corresponding to the audio requirement information.
[0040] Taking into account that there are many forms in which users can input audio demand information, in order to improve the correlation between the target audio and the audio demand information, in an optional implementation, the music generation model can pre-process the audio demand information after obtaining the audio demand information and then generate the corresponding target audio.
[0041] Exemplarily, when the audio demand information is in text form, the text can be segmented and interference words (such as conjunctions, modal particles, etc.) can be removed to obtain keywords, and then the target audio is generated according to the keywords. When the audio demand information is in the form of pictures, the input pictures can be matched with pictures stored in the database. The pictures stored in the database are marked with keywords (or called tags). Then, the keywords corresponding to the pictures with the highest matching degree can be screened, and the target audio is generated according to the keywords. Among them, the database can be a local database or a database of a cloud server, and the embodiments of the present application do not impose any special restrictions on this.
[0042] It should be noted that the target audio in the embodiment of the present application is automatically generated by an algorithm and is not created by a music producer in a conventional manner.
[0043] The music generation model can select an existing generation model or be constructed through an artificial intelligence algorithm. The format of the target audio can be the Moving Picture Experts Compression Standard Audio Level 3, Digital Music Interface, and lossless audio compression encoding format, etc. This application does not impose any special restrictions on this.
[0044] Step S130: Generate virtual musical instruments and digital humans corresponding to the target audio.
[0045] After generating the target audio, you can determine the instrument used in the target audio so as to generate a virtual instrument corresponding to the instrument. Figure 2 , step S130 may include the following steps:
[0046] Step S131: divide the target audio into multiple sub-audios.
[0047] The target audio is divided into multiple sub-audios and numbered, wherein the duration of each sub-audio can be the same.
[0048] Step S132: Determine the musical instrument used for each sub-audio by using the Mel-frequency cepstral coefficients.
[0049] The characteristics of the musical instrument in each sub-audio can be determined through the Mel-frequency cepstral coefficients, including steps such as fast Fourier transform, Mel filter bank and discrete cosine transform, so as to identify the musical instrument used in the target audio.
[0050] In some embodiments, instrument recognition can also be performed using techniques such as linear prediction coding coefficients, linear prediction cepstral coefficients, and perceptual linear prediction.
[0051] Step S133: Generate a virtual musical instrument corresponding to the determined musical instrument.
[0052] The types of virtual musical instruments are the same as the types of musical instruments appearing in the target audio, and the types of virtual musical instruments may include but are not limited to keyboard instruments, wind instruments, string instruments, and percussion instruments.
[0053] In the embodiment of the present application, the modeling, animation rendering and scene construction of 3D virtual human can be realized through the front-end interaction module, and the digital human image can also be obtained through the database. It can be understood that the number of musical instruments used in the target audio is usually multiple, and optionally, the number of digital humans can be the same as the number of virtual musical instruments, thereby improving the naturalness of the video.
[0054] Step S140: driving the digital human to play the target audio using the virtual musical instrument.
[0055] When driving a digital human to perform, the digital human and the virtual musical instrument can be generated at the same time to improve the naturalness of the video or live broadcast. Of course, the digital human and the virtual musical instrument can also be generated one after the other. Figure 3 , driving the digital human to play the target audio may include:
[0056] Step S141, obtaining the sounding time, pitch and loudness of each musical instrument in the target audio.
[0057] When the Mel-frequency cepstral coefficients are processed for each sub-audio, the sounding moment (including the start time and the end time), tone and loudness (also called pitch or tone level) of each instrument can be determined.
[0058] Step S142: at the moment of sounding, the motion state of the virtual musical instrument is controlled according to the first corresponding relationship, and the motion state of the digital human is driven according to the second corresponding relationship.
[0059] The first corresponding relationship may be a corresponding relationship between the pitch and loudness and the playing state of the musical instrument, and the second corresponding relationship may be a corresponding relationship between the motion state of the digital human and the motion state of the virtual musical instrument.
[0060] For ease of explanation, the following description will be given using a piano as an example.
[0061] The playing state of the piano refers to the position and state of the pressed keys at the time of sound production. There are 88 keys on the piano, and different keys correspond to different tones. When the same key is played with different strengths, the loudness of the sound produced is also different.
[0062] After the digital human is generated, the Openpose network model and bone binding technology can be used to control the digital human to move in a human-like manner.
[0063] Specifically, according to the first corresponding relationship, when it is recognized that the 26th key of the piano is pressed and sounds, the force with which the key is pressed can be determined, thereby determining the pressing speed, and controlling the 26th key of the virtual piano to move according to the pressing speed in the coordinate system, thereby changing the motion state of the 26th key of the virtual piano at that moment. Correspondingly, the position relationship of the digital human's hand relative to the virtual piano and the human body in the coordinate system can be controlled according to the second corresponding relationship, so as to place the digital human's corresponding finger on the 26th key, and control the digital human's corresponding finger to change with the key at the same movement trajectory at that moment, thereby matching the motion state of the digital human with the motion state of the virtual musical instrument. The principle is similar when pressing other keys, thereby achieving the technical effect of the digital human playing the target audio using the virtual piano.
[0064] Through the above implementation, it is possible to realize digital humans playing virtual musical instruments without real-life motion capture, thereby speeding up the video production cycle and reducing costs.
[0065] In an optional implementation, when the target audio contains lyrics, the digital human's lip shape can also be controlled to match the lyrics. For example, Audio2Face can be used to simulate the bending and deformation of the digital human's muscles and skin when speaking, and transition animation frames can be added to simulate the digital human's switching from a closed mouth state to an open mouth state when singing.
[0066] The digital human performance method provided in the embodiment of the present application includes: obtaining audio demand information; generating target audio corresponding to the audio demand information; generating a virtual musical instrument and a digital human corresponding to the target audio; and driving the digital human to play the target audio using the virtual musical instrument. The technical solution provided in the present application can reduce the production cost of driving the digital human performance.
[0067] Those skilled in the art will appreciate that the above embodiments are exemplary and are not intended to limit the present application. Where possible, the execution order of one or more of the above steps may be adjusted, or may be selectively combined to obtain one or more other embodiments. Those skilled in the art may select and combine any of the above steps as needed, and all those that do not depart from the essence of the present application fall within the scope of protection of the present application.
[0068] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present application provides a digital human performance device, and the device embodiment corresponds to the above method embodiment. For ease of reading, the present device embodiment will no longer repeat the details of the above method embodiment one by one, but it should be clear that the device in this embodiment can correspond to and implement all the contents of the above method embodiment.
[0069] Figure 4 A schematic diagram of the structure of a digital human performance device provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the device provided in this embodiment includes:
[0070] An acquisition module 110 is used to acquire audio requirement information;
[0071] A generating module 120, configured to generate target audio corresponding to the audio requirement information; and to generate virtual musical instruments and digital humans corresponding to the target audio;
[0072] The driving module 130 is used to drive the digital human to play the target audio using the virtual musical instrument.
[0073] The digital human performance device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.
[0074] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0075] Based on the same inventive concept, an embodiment of the present application also provides a digital human performance device. Figure 5 A schematic diagram of the structure of a digital human performance device provided in an embodiment of the present application, such as Figure 5 As shown, the digital human performance device provided in this embodiment includes: a memory 210 and a processor 220, the memory 210 is used to store computer programs; the processor 220 is used to execute the method described in the above method embodiment when calling the computer program.
[0076] The digital human performance device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.
[0077] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method described in the above method embodiment is implemented.
[0078] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk or a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.
[0079] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media can include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.
[0080] The naming or numbering of the steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The named or numbered process steps can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.
[0081] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0082] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0083] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0084] In the description of this application, unless otherwise specified, " / " indicates that the objects associated with each other are in an "or" relationship, for example, A / B can represent A or B; "and / or" in this application is merely a description of the association relationship between associated objects, indicating that three relationships may exist, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural.
[0085] Furthermore, in the description of this application, unless otherwise specified, "plurality" means two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.
[0086] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0087] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.
[0088] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.
[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A digital human playing method, characterized in that: include: Get audio demand information; Generating target audio corresponding to the audio requirement information; Generate a virtual musical instrument and a digital human corresponding to the target audio; The digital human is driven to play the target audio using the virtual musical instrument.
2. The method according to claim 1, characterized in that The generating the target audio corresponding to the audio requirement information includes: The audio requirement information is input into a music generation model to obtain the target audio.
3. The method according to claim 1, characterized in that: The audio requirement information includes at least one of the following information: music style, duration, scene, beat and genre.
4. The method according to claim 1, characterized in that: The audio requirement information includes: preset requirement information and / or requirement information input by a user.
5. The method according to claim 1, characterized in that The generating a virtual musical instrument corresponding to the target audio comprises: Dividing the target audio into a plurality of sub-audios; The instrument used by each sub-audio is determined by the Mel frequency cepstral coefficient; A virtual musical instrument corresponding to the determined musical instrument is generated.
6. The method according to claim 1, characterized in that In the case where the target audio includes lyrics, the method further includes: The digital human's lip shape is controlled to match the lyrics.
7. The method according to claim 1, characterized in that The step of driving the digital human to play the target audio using the virtual musical instrument comprises: Obtaining the sounding time, pitch and loudness of each musical instrument in the target audio; At the moment of sounding, the motion state of the virtual musical instrument is controlled according to a first corresponding relationship, and the motion state of the digital human is driven according to a second corresponding relationship, wherein the first corresponding relationship is the corresponding relationship between the pitch and loudness and the playing state of the musical instrument, and the second corresponding relationship is the corresponding relationship between the motion state of the digital human and the motion state of the virtual musical instrument.
8. A digital human performance device, characterized in that: include: An acquisition module, used to acquire audio demand information; A generation module, used to generate target audio corresponding to the audio requirement information; And, generating virtual musical instruments and digital humans corresponding to the target audio; A driving module is used to drive the digital human to play the target audio using the virtual musical instrument.
9. A digital human performance device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the method according to any one of claims 1 to 7 when calling the computer program.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.