Digital human application method and electronic equipment

Abstract

The invention discloses a digital human application method and electronic equipment, and the method mainly comprises the steps: randomly selecting a copywriting in a pre-generated emotional copywriting template through calculating an emotional value of a song; generating a singer copywriting voice corresponding to the emotion value according to the tone of the singer; meanwhile, a digital person is shaped according to the picture of the singer, and the digital person has a facial expression corresponding to the emotion value; driving the digital person to play the singer copywriting voice in the singing waiting time period, and playing the facial animation of the corresponding emotion at the same time; not only are visual presentation and hearing effects enriched, but also copywriting transmission is more vivid and interesting, and the sense of participation and immersion of singers are enhanced; the emotion color in the voice can be more intuitively expressed, the emotion of the song is fully conveyed to a singer, the singer is helped to better understand and feel the emotion behind the song, the overall expressive force and infection of song singing are improved, and the situation that the singer waits for a long time and the entertainment experience of the singer is reduced is avoided.

Description

Technical Field

[0001] The present invention relates to the field of digital audio-visual technology, and in particular to a digital human application method and electronic equipment. Background Art

[0002] In digital audio-visual venues, singers can realize their singing needs through the on-demand system. With the rapid development and continuous innovation of on-demand technology and systems, singers' experience of singing as an entertainment activity is experiencing unprecedented richness and diversity, bringing singers a richer and more diverse singing entertainment experience, and more profoundly affecting singers' expectations and pursuits of singing entertainment.

[0003] The prelude, verse, chorus, bridge, interlude and outro are the basic framework of a song. Among them, the prelude, interlude or outro are usually pure instrumental passages, showing the solo or ensemble skills of the instrument, which is the waiting time period in the song. In the existing on-demand system, the singer needs to finish playing these waiting time periods before starting or continuing to sing. The waiting process can easily make the singer's singing mood gradually dissipate, unable to fully convey the emotions of the song, making the overall expressiveness and appeal of the song greatly reduced. At the same time, the long wait is too monotonous and boring, lacking in fun, which can easily make the singer impatient and reduce their entertainment experience. Summary of the invention

[0004] Therefore, it is necessary to provide a digital human application method to solve the problem that the singer's singing mood gradually dissipates during the waiting process of the song, and the long waiting is too monotonous and lacks interest.

[0005] To achieve the above object, the present invention provides a digital human application method, which comprises the following steps:

[0006] Calculate the sentiment value of the song to be played;

[0007] Matching a corresponding emotional copy template among a plurality of pre-generated emotional copy templates according to the emotional value of the song;

[0008] According to the timbre of the original singer of the song and the matched emotional copy template, a corresponding singer voice is generated in the audio model, and the emotion of the singer voice is adjusted according to the emotional value to generate the singer copy voice;

[0009] Mark the time period for playing the singer's copywriting voice according to the anchor point information of the song;

[0010] Generate a digital human that simulates the singer's image based on the original singer's facial features, and match the digital human's facial animation based on the song's emotional value;

[0011] During the time period marked for playing the singer's copywriting voice, the digital human is driven to play the singer's copywriting voice, and at the same time, the digital human's face displays facial animations corresponding to emotions.

[0012] Furthermore, in the step of calculating the emotion value of the song to be played, the emotion value of the song is calculated using a music emotion value model or a lyrics emotion value model.

[0013] Furthermore, the lyrics sentiment value model includes a text preprocessing module, a text feature extraction module, a sentiment analysis module and a sentiment scoring module;

[0014] The specific method of calculating the emotional value of a song using the lyrics emotional value model is as follows:

[0015] The text preprocessing module performs word segmentation processing on the lyrics of the song, removes stop words, and outputs the preprocessed text;

[0016] Extracting feature words of the text output by the text preprocessing module through the text feature extraction module;

[0017] Analyze the sentiment value corresponding to each feature word through the sentiment analysis module;

[0018] Through the sentiment scoring module, the sum of the sentiment values ​​corresponding to all feature words is counted as the sentiment value of the song.

[0019] Furthermore, the matching of the corresponding emotional copy template among the pre-generated multiple emotional copy templates according to the emotional value of the song specifically includes: associating each of the pre-generated emotional copy templates with an emotional value range, determining the emotional value range into which the emotional value of the song falls, and establishing a matching relationship between the song and the emotional copy template associated with the corresponding emotional value range.

[0020] Furthermore, after matching the corresponding emotional copy template in the pre-generated multiple emotional copy templates according to the emotional value of the song, it also includes: combining the emotional copy template with the on-demand user information of the song to be played, or the venue information of the place where the song to be played is located, or the greeting, to form an updated emotional copy template;

[0021] The step of generating the corresponding singer's voice in the audio model according to the timbre of the original singer of the song and the matched emotional copy template is specifically as follows: generating the corresponding singer's voice in the audio model according to the timbre of the original singer of the song and the updated emotional copy template.

[0022] Furthermore, the time period for playing the singer's copy voice is marked according to the anchor information of the song as follows: the anchor information of the song is obtained by automatically identifying the human voice in the song, the waiting time period for singing the song is obtained according to the anchor information of the song, and one of the waiting time periods for singing is marked as the time period for playing the singer's copy voice.

[0023] Furthermore, marking the time period for playing the singer's copywriting voice according to the anchor point information of the song includes judging whether the duration of the time period for playing the singer's copywriting voice is greater than the duration of the singer's copywriting voice; if greater, playing the singer's copywriting voice; otherwise, not playing it.

[0024] Furthermore, before generating the digital human simulating the singer's image according to the facial feature information of the original singer of the song, the method further includes the step of generating the facial feature information of the original singer of the song, specifically:

[0025] Use OpenCV to load the original singer's photo and convert it into a grayscale image;

[0026] Detect the face of the grayscale image through the face detector and obtain the coordinate points of facial feature information;

[0027] The facial features are classified according to the detected facial feature information coordinate points, and the facial feature information of the original singer of the song is generated.

[0028] Furthermore, adjusting the emotion of the singer's voice according to the emotion value includes: the total emotion value of the song is 1, when the emotion value of the song is lower than 0.5, it is determined to be a sad song, and at least one of the following singer's voice parameters is reduced: speaking speed, pitch or volume; when the emotion value of the song is higher than 0.8, it is determined to be a cheerful song, and at least one of the following singer's voice parameters is increased: speaking speed, pitch or volume.

[0029] Furthermore, when the digital human is driven to play the singer's copywriting voice during the time period marked for playing the singer's copywriting voice, the song playback volume is synchronously reduced, and the emotional copywriting text is displayed on the display screen in the form of animation special effects. When the singer's copywriting voice is finished playing, the song playback volume is restored.

[0030] The electronic device comprises a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the steps of the digital human application method are implemented.

[0031] Different from the existing technology, the above technical solution mainly calculates the emotional value of the song and selects a matching emotional copy template from the pre-generated emotional copy templates; then generates a singer copy voice corresponding to the emotional value according to the singer's timbre; at the same time, a digital human is created according to the singer's photo, which has a facial expression corresponding to the emotional value; during the waiting period for singing, the digital human is driven to play the singer's copy voice, and at the same time plays the facial animation corresponding to the emotion; displaying the singer's copy voice by the digital human not only enriches the visual presentation and auditory effects, but also makes the communication of the copy more vivid and interesting, and enhances the singer's sense of participation and immersion; the combination of digital human and singer's copy voice can more intuitively express the emotional color in the voice, fully convey the emotion of the song to the singer, drive the singer's emotions, help the singer better understand and feel the emotions behind the song, better sing the song, improve the overall expressiveness and appeal of the song, avoid the singer waiting for a long time, and reduce their entertainment experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flow chart of the digital human application method of the present invention. DETAILED DESCRIPTION

[0033] In order to explain the technical content, structural features, achieved objectives and effects of the technical solution in detail, the following is a detailed description in conjunction with specific embodiments and accompanying drawings.

[0034] Reference to "embodiment" herein means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The term "embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or association with other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the various technical features mentioned in the embodiments can be combined in any way to form a corresponding implementable technical solution.

[0035] Unless otherwise defined, the technical terms used in this document have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms in this document is only for describing specific embodiments and is not intended to limit this application.

[0036] In the description of this application, the term "and / or" is an expression used to describe the logical relationship between objects, indicating that three relationships may exist, for example, A and / or B, which means: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" logical relationship.

[0037] In the present application, terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship of quantity, priority or sequence between these entities or operations.

[0038] Without further limitations, in this application, the words "include", "comprises", "has" or other similar expressions used in the sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product including the elements, so that the process, method or product including a series of elements may include not only those limited elements, but also other elements not explicitly listed, or also include elements inherent to such process, method or product.

[0039] Similar to the understanding in the Patent Examination Guidelines, in this application, expressions such as "greater than", "less than", "exceed" and the like are understood to exclude the number itself; expressions such as "above", "below", "within" and the like are understood to include the number itself. In addition, in the description of the embodiments of this application, "multiple" means more than two (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups", "multiple times", etc., unless otherwise clearly and specifically limited.

[0040] In the description of the embodiments of the present application, space-related expressions used, such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "vertical", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicate the orientation or position relationship based on the orientation or position relationship shown in the specific embodiments or drawings, and are only for the convenience of describing the specific embodiments of the present application or facilitating the reader's understanding, and do not indicate or imply that the referred device or component must have a specific position, a specific orientation, or be constructed or operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.

[0041] Unless otherwise expressly specified or limited, in the description of the embodiments of the present application, the terms such as "install", "connect", "connect", "fix", "set", etc. used should be understood in a broad sense. For example, the "connection" can be a fixed connection, a detachable connection, or an integrated setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For technicians in the technical field to which the present application belongs, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0042] See also Figure 1 As shown, the present invention provides a digital human application method, which mainly calculates the emotion value of the song and selects a matching emotion copy template from pre-generated emotion copy templates; then generates a singer copy voice corresponding to the emotion value according to the singer's timbre; at the same time, a digital human is shaped according to the singer's photo, and the digital human has a facial expression corresponding to the emotion value; in the waiting singing period, the digital human is driven to play the singer's copy voice and the facial animation corresponding to the emotion; the digital human displays the singer's copy voice, which not only enriches the visual presentation and auditory effects, but also makes the communication of the copy more vivid and interesting, and enhances the singer's sense of participation and immersion; the combination of the digital human and the singer's copy voice can more intuitively express the emotional color in the voice, fully convey the emotion of the song to the singer, drive the singer's emotions, help the singer better understand and feel the emotions behind the song, better sing the song, improve the overall expressiveness and appeal of the song singing, and avoid the singer waiting for a long time and reduce their entertainment experience.

[0043] To further illustrate a digital human application method of the invention, an implementation method is provided below, which includes the following steps:

[0044] S1, calculating the emotion value of the song to be played;

[0045] S2, matching a corresponding emotional copy template in a plurality of pre-generated emotional copy templates according to the emotional value of the song;

[0046] S3, according to the timbre of the original singer of the song and the matched emotional copy template, generate the corresponding singer's voice in the audio model, and adjust the emotion of the singer's voice according to the emotional value to generate the singer's copy voice;

[0047] S4, marking the time period for playing the singer's copywriting voice according to the anchor point information of the song;

[0048] S5, generating a digital human simulating the image of the singer according to the facial feature information of the original singer of the song, and matching the facial animation of the digital human according to the emotional value of the song;

[0049] S6. During the time period marked for playing the singer's copywriting voice, the digital human is driven to play the singer's copywriting voice, and at the same time, the digital human's face displays facial animations corresponding to the emotions.

[0050] The emotional value of the above-mentioned song can be calculated in real time or pre-calculated. In some embodiments, the emotional value of the song to be sung in the singing list is obtained by extracting the music features of the song, such as the intensity, rhythm speed, melody, harmony, etc. of the music. Specifically, the emotional value of the song can be calculated by training a suitable music emotional value model. In some embodiments, the emotional value of the song in the singing list is obtained by extracting the lyric features of the song, such as verbs, adjectives, nouns, etc. in the lyrics. Specifically, the emotional value of the song can be calculated by training a suitable lyric emotional value model, that is, using the lyric emotional value model to calculate the emotional value of the song. Of course, after calculation, it can also be classified and labeled according to the different emotional values of the songs for direct calling next time.

[0051] In a specific embodiment, a lyric emotional value model is used to calculate the emotional value of the song. The lyric emotional value model can be a natural language processing model (such as snowNLP), which includes a text preprocessing module, a text feature extraction module, an emotional analysis module, an emotional scoring module, etc.

[0052] The text preprocessing module performs word segmentation and stop word removal on the lyrics of the song; the word segmentation can use word segmentation tools (such as jieba word segmenter (Chinese), NLTK (English), etc.) to cut the lyric text into a large number of single words or phrases. By presetting a stop word list, this list usually contains some words that appear frequently in the text but are meaningless for emotional analysis (such as "de", "shi", "he", etc.). After word segmentation, traverse each word in the text. If the word appears in the stop word list, it will be removed from the text.

[0053] The text feature extraction module extracts the feature words output by the text preprocessing module; the feature words are mainly the part-of-speech of the text, such as verbs, adjectives, nouns, etc.

[0054] The emotional analysis module analyzes the emotional value corresponding to each feature word; the emotional analysis module contains a large number of feature words and their corresponding emotional values, that is, each feature word can map to a corresponding emotional value; machine learning or deep learning algorithms (such as naive Bayes, support vector machine, convolutional neural network, etc.) can be used to train the emotional analysis module. During the training process, the model will learn the association between the feature words and the emotional values. After the emotional analysis module is trained, input the feature words into the emotional analysis module, that is, map the feature words to the corresponding emotional values.

[0055] The sentiment value corresponding to all feature words is counted through the sentiment scoring module as the sentiment value of the song; specifically, the feature words output by the text feature extraction module are matched in the sentiment analysis module to obtain the sentiment value of the corresponding feature words, and then the sentiment value of the entire lyrics is calculated, that is, the sentiment value of the song. The calculation method can be a simple summation or weighted average of the sentiment values ​​of the feature words, or a more complex model output. The calculated sentiment value of the song is used as output for subsequent analysis or application.

[0056] The above-mentioned emotional value is a certain value within a preset emotional value interval, and the preset emotional value interval is divided into several ranges, and each range has an emotional copy template associated with it. That is, each of the pre-generated emotional copy templates is associated with an emotional value range, and when matching, the emotional value range into which the emotional value of the song falls is determined, and a matching relationship is established between the song and the emotional copy template associated with the corresponding emotional value range. Of course, there is also the case where the above-mentioned calculated emotional value exceeds the preset emotional value interval and is outside the preset emotional value interval. At this time, emotional words can be added to the lyrics of the song, and then the emotional value of the song can be recalculated. That is, the step of matching the corresponding emotional copy template in the pre-generated multiple emotional copy templates according to the emotional value of the song also includes judging whether the emotional score meets the emotional value range of the pre-generated multiple emotional copy templates; if it does, the song is matched with the corresponding emotional copy template according to the emotional value; otherwise, emotional words are added to the lyrics of the song to recalculate the emotional value of the song. The emotional value of the song calculated above can be directly saved for use in subsequent singing, or it can be not saved and calculated in real time each time it is sung.

[0057] The emotions represented by the above-mentioned emotion value interval can be gradually increasing or gradually decreasing; for example, if the emotion value interval is set to (0, 1), the larger the emotion value, the more positive the emotion, the maximum is 1, indicating that the song is the happiest; the smaller the emotion value, the more negative the emotion, the minimum is 0, indicating that the song is the saddest. The above-mentioned multiple pre-generated emotion copy templates are pre-generated emotion copy templates, each emotion copy template is associated with an emotion value range, each emotion copy template has a number of copy, and the copy in the emotion copy template expresses the same or similar emotion value. When a song is matched with a certain emotion copy template, any copy in the emotion copy template that matches it is selected for the song.

[0058] The following embodiments take four kinds of emotional copy templates as examples to further illustrate the present invention. They are: emotional copy template 1, corresponding to the emotional value of the song 0-0.2; emotional copy template 2: corresponding to the emotional value of the song 0.2-0.5; emotional copy template 3: corresponding to the emotional value of the song 0.5-0.8; emotional copy template 4: corresponding to the emotional value of the song 0.8-1.0;

[0059] Further, the following specific embodiments are provided for ease of understanding:

[0060] Emotional copywriting template 1: Corresponding to the emotional value of the song 0-0.2:

[0061] When the emotional value of a song is in the range of 0-0.2, it is judged that the current song is very sad and the singer is in great pain. At this time, the copy played needs to be more affectionate and sincere.

[0062] Example of copywriting:

[0063] I feel your pain and it's hard, but remember, you are not alone and I am here for you.

[0064] Emotional copywriting template 2: corresponding to the emotional value of the song 0.2-0.5:

[0065] It is judged that the current song is somewhat sad and the singer is in a low mood, so the corresponding playback text should be encouraging.

[0066] Example: I know you may be a little sad now, but please believe that this is only temporary. You are strong and will definitely get through this difficult time.

[0067] Emotional copywriting template 3: Corresponding to the emotional value of the song 0.5-0.8:

[0068] It can be judged that the song is a relaxing and cheerful song, and the text should use warm language.

[0069] Example of copywriting:

[0070] I feel so happy to see your smile. Keep up the good mood and let happiness become the norm in your life!

[0071] Emotional copywriting template 4: Corresponding to the emotional value of the song 0.8-1.0:

[0072] Judging from the song, it is very cheerful and the writing style tends to be happy.

[0073] Example of copywriting:

[0074] Your happiness is contagious to me, and seeing you so happy makes me happy too. Let's celebrate this joy together and enjoy this wonderful time.

[0075] In some embodiments, after matching the corresponding emotional copy template in the pre-generated multiple emotional copy templates according to the emotional value of the song, the method further includes: combining the emotional copy template with the on-demand user information of the song to be played, or the venue information of the place where the song to be played is located, or the greeting, to form an updated emotional copy template;

[0076] The step of generating the corresponding singer's voice in the audio model according to the timbre of the original singer of the song and the matched emotional copy template is specifically as follows: generating the corresponding singer's voice in the audio model according to the timbre of the original singer of the song and the updated emotional copy template.

[0077] The above-mentioned audio model matching the singer with the original singer of the song and the random text in the corresponding emotional text template generates the corresponding singer voice, obtains and adjusts the emotion of the singer voice according to the emotion value, and generates the singer text voice, wherein the singer's audio model is a pre-trained singer's audio model. The training audio data may include audio data of the singer's normal speech, such as interviews, live performances, etc., to ensure that the sound source covers different tones and emotional expressions to improve the generalization ability of the model. The collected training audio data is pre-processed (including format conversion, denoising and reverberation, audio segmentation, feature extraction, etc.), and deep learning models such as gpt-sovits, sovits-svc, etc. are selected for training. The pre-processed training audio data is used as input, the model output is calculated by forward propagation, and then the difference between the output and the true label is calculated by the loss function, and the model parameters are updated by the back propagation algorithm. The learning rate, batch size, number of iterations and other hyperparameters are continuously adjusted during the training process to optimize the performance of the model. Ensure that the audio model of the singer generated after the training is completed can generate the singer voice of the corresponding text after the text is input. Then adjust the singer's voice emotion according to the song's emotional value to generate the final singer's copy voice. The singer's voice emotion can be adjusted by speaking speed, pitch, volume, etc. Sad voices can be adjusted like this: you can lower the pitch, volume, and speed. Conversely, the cheerful emotion is still adjusted. For example, if the emotional value is 0-0.2, the speaking speed is reduced by 4%, the pitch is reduced by 2%, and the volume is reduced by 2%. If the emotional value is 0.2-0.5, the speaking speed is reduced by 2%, the pitch is reduced by 1%, and the volume is reduced by 2%; if the emotional value is 0.5-0.8, no adjustment is made. If the emotional value is 0.8-1.0, the speaking speed is increased by 2%, the pitch is increased by 1%, and the volume is increased by 1%.

[0078] The above-mentioned time period for marking the singer's copywriting voice according to the anchor information of the song is specifically as follows: the anchor information of the song is obtained by automatically identifying the human voice in the song, the waiting singing time period of the song is obtained according to the anchor information of the song, and one of the waiting singing time periods is marked as the time period for playing the singer's copywriting voice. The above-mentioned waiting singing time period refers to the part without vocals in the original song, such as prelude, interlude, and coda. Specifically, it can be identified using online or local audio editing software, or it can be completed using a trained deep learning model. The anchor information can define the time point in the song duration when there is a singing voice, thereby reversely obtaining the time point in the song duration when there is no vocal singing, and then defining the time period without vocal singing as the waiting singing time period. Select one of the waiting singing time periods as the time period for playing the singer's copywriting voice and mark it so that when the song is played to the marked time period, the singer's copywriting voice is played. If the selected waiting singing time period is less than the singer's copywriting voice duration, it can be unmarked to avoid the singer's copywriting voice interfering with the singer's singing. That is, marking the time period for playing the singer's copywriting voice according to the anchor point information of the song includes judging whether the duration of the time period for playing the singer's copywriting voice is greater than the duration of the singer's copywriting voice; if greater, playing the singer's copywriting voice; otherwise, not playing.

[0079] Before generating a digital person simulating the singer's image based on the facial feature information of the original singer of the song, the step of generating the facial feature information of the original singer of the song is further included, specifically:

[0080] Use OpenCV to load the original singer's photo and convert it into a grayscale image;

[0081] Detect the face of the grayscale image through the face detector and obtain the coordinate points of facial feature information;

[0082] The facial features are classified according to the detected facial feature information coordinate points, and the facial feature information of the original singer of the song is generated.

[0083] The above-mentioned original singer's photo can be obtained through a local song library, or through online query of song information, or through search engine search, such as searching with "singer's name" + "picture" as search keywords. Preferred are high-resolution, clear, unobstructed front and / or side photos of the singer, so as to better extract the singer's facial feature information in the singer's photo, thereby making the generated digital person more vivid and lifelike. The above-mentioned facial feature information coordinate points include multiple points of the eyes, nose, mouth, and contours, which are used to classify the facial features and generate the singer's facial feature information. The singer's facial feature information includes face shape, lip shape, eye shape, nose shape, eyebrow shape, hairstyle, facial ornaments, etc. For example, if the contour coordinate points are close to a circle, it is defined as a round face, and the face shape in the facial feature information is a round face;

[0084] The above generates a digital human that simulates the image of the singer based on the facial feature information of the original singer of the song, and matches the facial animation of the digital human according to the emotional value of the song. A digital human image feature library can be established in advance, which mainly includes the digital human image and the facial feature information applied to the digital human, including face shape, lip shape, eye shape, nose shape, eyebrow shape, hairstyle, facial decoration, expression animation, etc. The image feature library can obtain the facial resource file and the feature information corresponding to the facial resource file through the online 3D model library (such as the fbx model resource file of Unity). For example, the face shape includes: round face, square face, and oval face. The lip shape includes: wide mouth and narrow lips. The eye shape includes standard eyes, slender eyes, round eyes (mainly divided by eye height), etc.; and the expression animation is made in advance for each emotional value interval (each emotional copywriting template).

[0085] For example, four facial expression animations are prepared in advance for the four types of scores, including:

[0086] Emotional copy template 1: corresponding to the emotional value 0-0.2, sad expression;

[0087] Emotional copy template 2: corresponds to an emotional value of 0.2-0.5, with a slightly sad expression;

[0088] Emotional copy template 3: corresponds to an emotional value of 0.5-0.8, a slightly cheerful expression;

[0089] Emotional copy template 4: corresponding to the emotional value of 0.8-1.0, a cheerful expression.

[0090] When the digital human plays the singer's copywriting voice during the time period marked for playing the singer's copywriting voice, the song playback volume is simultaneously reduced, and the emotional copywriting text is displayed on the display screen in the form of animation special effects. When the singer's copywriting voice is played, the song playback volume is restored. Automatically adjusting the song volume when playing the singer's copywriting voice avoids the conflict between the singer's copywriting voice and the background music of the song, allowing users to clearly hear the voice content without manually adjusting the volume, which improves the convenience of use.

[0091] The present invention also provides an electronic device, which includes a memory and a processor, wherein a computer program is stored on the memory, and the steps of the above method are implemented when the computer program is executed by the processor. The memory includes, but is not limited to, a disk, a tape, a magnetic card, a floppy disk, a flash memory, an optical disk, an optical card, a read-only memory (ROM), a random access memory (RAM), an erasable programmable ROM (EPROM), and an electrically erasable programmable ROM (EEPROM), etc., and also includes other biological, physical or chemical structures that can realize functions similar to or equivalent to the above-listed memories, such as DNA, RNA, protein, etc., which have information storage capabilities. In a specific embodiment, the memory involved can be one of the above-mentioned memory types, or a combination of the above-mentioned memories. In different embodiments, the computer program involved in the embodiment can be stored in a single memory in a centralized manner, or it can be stored in multiple memories in a distributed manner. The memory containing a computer device readable storage medium can be a non-volatile memory or a random access memory. These computer device readable memories can be built into the device, or they can be connected to the device involved in the embodiment as an external device or a part of an external device. In some embodiments, the memory is deployed locally; in other embodiments, a solution of deploying the memory away from the processor may also be adopted, such as a network attached memory accessed via an RF circuit or an external port and a communication network, wherein the communication network may be the Internet, one or more intranets, a local area network (LAN), a wide area network (WLAN), a storage area network (SAN), etc., or a suitable combination thereof, as long as the computer device can access the memory. In addition, the computer program involved in the embodiment may be stored in plaintext / ciphertext form, or may be designed as training data, which may be integrated and reorganized through model training and implicitly stored in the parameter state of a deep neural network or other machine learning model.The processor described in the embodiments of the present application can be implemented by hardware, firmware, software or a combination thereof, and can use a circuit, a single or multiple application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), a digital signal processor (Digital Signal Processor, DSP), a digital signal processing device (Digital Signal Processing Device, DSPD), a programmable logic device (Programmable Logic Device, PLD), a field programmable gate array (Field Programmable Gate Array, FPGA), a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, at least one of a microprocessor, and also includes other physical, biological or chemical structures that can achieve similar or equivalent functions to the processors listed above, such as biological neurons, quantum computing units, DNA computing units, etc., so that the processor can execute some or all of the steps in the computer program or method involved in the various embodiments of the present application, or any combination of the steps mentioned therein.

[0092] It should be noted that, although the above embodiments have been described in this article, the patent protection scope of the present invention is not limited thereby. Therefore, based on the innovative concept of the present invention, changes and modifications made to the embodiments described herein, or equivalent structures or equivalent process changes made using the contents of the present invention specification and drawings, directly or indirectly applying the above technical solutions to other related technical fields, are all included in the patent protection scope of the present invention.

Claims

1. A digital human application method, characterized in that: The following steps are involved: Calculate the sentiment value of the song to be played; Matching a corresponding emotional copy template among a plurality of pre-generated emotional copy templates according to the emotional value of the song; According to the timbre of the original singer of the song and the matched emotional copy template, a corresponding singer voice is generated in the audio model, and the emotion of the singer voice is adjusted according to the emotional value to generate the singer copy voice; Mark the time period for playing the singer's copywriting voice according to the anchor point information of the song; Generate a digital human that simulates the singer's image based on the original singer's facial features, and match the digital human's facial animation based on the song's emotional value; During the time period marked for playing the singer's copywriting voice, the digital human is driven to play the singer's copywriting voice, and at the same time, the digital human's face displays facial animations corresponding to emotions.

2. The digital human application method according to claim 1, characterized in that: In the step of calculating the emotion value of the song to be played, the emotion value of the song is calculated using a music emotion value model or a lyrics emotion value model.

3. The digital human application method according to claim 2, characterized in that: The lyrics sentiment value model includes a text preprocessing module, a text feature extraction module, a sentiment analysis module and a sentiment scoring module; The specific method of calculating the emotional value of a song using the lyrics emotional value model is as follows: The text preprocessing module performs word segmentation processing on the lyrics of the song, removes stop words, and outputs the preprocessed text; Extracting feature words of the text output by the text preprocessing module through the text feature extraction module; Analyze the sentiment value corresponding to each feature word through the sentiment analysis module; Through the sentiment scoring module, the sum of the sentiment values ​​corresponding to all feature words is counted as the sentiment value of the song.

4. The digital human application method according to claim 1 or 3, characterized in that: The method of matching the corresponding emotional copy template in multiple pre-generated emotional copy templates according to the emotional value of the song specifically includes: associating each of the pre-generated emotional copy templates with an emotional value range, determining the emotional value range into which the emotional value of the song falls, and establishing a matching relationship between the song and the emotional copy template associated with the corresponding emotional value range.

5. The digital human application method according to claim 1, characterized in that: After matching the corresponding emotional copy template in the pre-generated multiple emotional copy templates according to the emotional value of the song, the method further includes: combining the emotional copy template with the on-demand user information of the song to be played, or the venue information of the place where the song to be played is located, or the greeting, to form an updated emotional copy template; The step of generating the corresponding singer's voice in the audio model according to the timbre of the original singer of the song and the matched emotional copy template is specifically as follows: generating the corresponding singer's voice in the audio model according to the timbre of the original singer of the song and the updated emotional copy template.

6. The digital human application method according to claim 1, characterized in that: The time period for playing the singer's voice copy according to the anchor information of the song is specifically marked as follows: the anchor information of the song is obtained by automatically identifying the human voice in the song, the waiting time period for singing the song is obtained according to the anchor information of the song, and one of the waiting time periods for singing is marked as the time period for playing the singer's voice copy.

7. The digital human application method according to claim 1, characterized in that: The time period for playing the singer's copywriting voice is marked according to the anchor point information of the song, including judging whether the duration of the time period for playing the singer's copywriting voice is greater than the duration of the singer's copywriting voice; if greater, playing the singer's copywriting voice; otherwise, not playing.

8. The digital human application method according to claim 1, characterized in that: Before generating a digital person simulating the singer's image based on the facial feature information of the original singer of the song, the step of generating the facial feature information of the original singer of the song is further included, specifically: Use OpenCV to load the original singer's photo and convert it into a grayscale image; Detect the face of the grayscale image through the face detector and obtain the coordinate points of facial feature information; The facial features are classified according to the detected facial feature information coordinate points, and the facial feature information of the original singer of the song is generated.

9. The digital human application method according to claim 1, characterized in that: The method of adjusting the emotion of the singer's voice according to the emotion value includes: the total emotion value score of the song is 1. When the emotion value of the song is lower than 0.5, it is determined to be a sad song, and at least one of the following singer's voice parameters is reduced: speaking speed, pitch or volume; when the emotion value of the song is higher than 0.8, it is determined to be a cheerful song, and at least one of the following singer's voice parameters is increased: speaking speed, pitch or volume.

10. The digital human application method according to claim 1, characterized in that: When the digital human is driven to play the singer's copywriting voice during the time period marked for playing the singer's copywriting voice, the song playback volume is synchronously reduced, and the emotional copywriting text is displayed on the display screen in the form of animation special effects. When the singer's copywriting voice is finished playing, the song playback volume is restored.

11. An electronic device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the digital human application method as claimed in any one of claims 1 to 10 are implemented.

Images