Virtual human driving method and device, equipment, medium and product thereof

Abstract

The application relates to a virtual person driving method and device, equipment, medium and product, the method comprising: obtaining driving text submitted by an anchor user in a network live broadcast; generating voice data of a virtual person in a live broadcast room of the anchor user based on the driving text, obtaining time sequence alignment information about words between the driving text and the voice data; matching material segments corresponding to each word in the driving text from an action description material library, the material segments being suitable for driving the virtual person to perform corresponding actions; and splicing the material segments corresponding to each word into an action description file suitable for driving the virtual person in the live broadcast room of the anchor user to perform a series of actions according to a time sequence relationship between the words as specified by the time sequence alignment information. The application can drive virtual person movement by text, avoid completely relying on reasoning ability learned by a machine learning model, effectively control technical implementation cost, and achieve efficient and high-quality processing effect.

Description

TECHNICAL FIELD

[0001] The present application relates to virtual human control technology, and in particular to a virtual human driving method and device, equipment, medium, and product. BACKGROUND

[0002] With the rise of the metaverse, the introduction of virtual humans in live streaming scenarios has become a standard feature of live streaming applications. Major manufacturers and live streaming platforms have successively introduced virtual virtual human functions. Live streaming through virtual humans has become a popular choice for many anchors. In live streaming scenarios, virtual humans have many new play options, and driving virtual humans through external materials such as sound, movement, and music has become a very novel and popular function.

[0003] Currently, in the scheme of driving virtual humans through external materials, a machine learning method is generally used to generate a motion file that controls the movement of virtual humans through external materials. The motion file contains the data of each frame of the movement of the virtual human, and each frame of data contains the coordinate information of each skeletal node of the human body. The pose of the virtual human is indicated by the coordinate information of each skeletal node, and multiple frames of images are continuously drawn to drive the virtual human to present a movement effect.

[0004] Because the motion file generated by machine learning often has semantic understanding ambiguity, the motion of the virtual human driven by the motion file often exhibits abnormal phenomena such as misalignment, unsmoothness, stillness, and rhythm, and thus an alternative solution is needed. SUMMARY

[0005] The present application aims to solve the above problems and provide a virtual human driving method and corresponding device, equipment, non-volatile readable storage medium, and computer program product.

[0006] According to one aspect of the present application, a virtual human driving method is provided, comprising the following steps:

[0007] Obtaining driving text submitted by an anchor user in a network live streaming;

[0008] Generating speech data of a virtual human in a live streaming room of the anchor user based on the driving text, and obtaining time sequence alignment information between the driving text and the speech data regarding words;

[0009] Matching material segments corresponding to each word in the driving text from an action description material library, the material segments being suitable for driving the virtual human to perform corresponding actions;

[0010] According to the time sequence relationship between the words as defined by the time sequence alignment information, the material segments corresponding to the respective words are spliced into a motion description file suitable for driving the virtual human in the live room of the anchor user to perform a series of actions.

[0011] Optionally, after splicing the material segments corresponding to the respective words into a motion description file suitable for driving the virtual human to perform a combined action according to the time sequence relationship between the words as defined by the time sequence alignment information, the method comprises:

[0012] driving the virtual human to move by applying the motion description file to generate a motion image stream of the virtual human;

[0013] generating a subtitle text corresponding to the driving text;

[0014] integrating the motion image stream, the voice data and the subtitle text into a video stream according to the time sequence alignment information;

[0015] pushing the video stream to the live room for playing.

[0016] Optionally, generating a subtitle text corresponding to the driving text comprises:

[0017] obtaining a purified text obtained by preprocessing the driving text in a process of generating voice data of the virtual human in the live room of the anchor user based on the driving text;

[0018] performing sentence segmentation on the driving text and the purified text respectively according to punctuation marks and counting the number of corresponding sentence segmentation;

[0019] determining whether the number of sentence segmentation of the driving text and the purified text is consistent, and when the number is consistent, setting the driving text as the subtitle text according to the purified text, otherwise, setting the purified text as the subtitle text.

[0020] Optionally, matching the material segments corresponding to the respective words in the driving text from a motion description material library, the material segments being suitable for driving the virtual human to perform corresponding actions, comprises:

[0021] performing sequential word segmentation on the driving text to obtain a word segmentation sequence composed of the respective words in the driving text;

[0022] determining the action labels corresponding to the respective words in the word segmentation sequence by using a preset action mapping model / label mapping library to obtain a label sequence;

[0023] determining a material segment under the corresponding action label from a motion description material library based on the action labels corresponding to the respective words in the label sequence.

[0024] Optionally, based on the action label corresponding to each word in the label sequence, a material segment under the corresponding action label is determined from the action description material library, including:

[0025] Based on the current word in the label sequence, a style feature vector of the corresponding speech segment in the speech data is determined, the style feature vector is used to represent the style invariant features of the corresponding speech segment, and the style includes rhythm style and / or emotional style;

[0026] The style feature vectors of each material segment corresponding to the action label of the current word are obtained from the action description material library;

[0027] The similarity between the style feature vector of the corresponding speech segment and the style feature vector of each material segment is calculated, and the material segment with the maximum similarity of the style feature vector is determined as the selected target material segment.

[0028] Optionally, the action label corresponding to each word in the word sequence is determined, and a label sequence is obtained, including:

[0029] The current word is taken as a query word, and it is queried from the label mapping library whether there is a corresponding action label, when there is, the corresponding action label is determined as the action label of the current word, and is added to the label sequence;

[0030] When the action label corresponding to the current word does not exist in the label mapping library, and the action label of the word with the current word as the prefix exists in the label mapping library, the current word and the next word are spliced to form a new query word, it is queried from the label mapping library whether there is an action label corresponding to the new query word, when there is a corresponding action label, the new query word is replaced by each word spliced, and the action label of the new query word is added to the label sequence;

[0031] When there is no action label corresponding to the new query word, the action label of the current word is set to point to a preset material segment belonging to a variable rhythm action, and is added to the label sequence.

[0032] Optionally, each word corresponding to the material segment is spliced into an action description file suitable for driving the virtual person in the live room of the host user to implement a series of actions, including:

[0033] A material segment corresponding to a word in the label sequence is obtained as a current segment, and the action type of the current segment is identified as a fixed semantic action or a variable rhythm action;

[0034] If the action type of the current segment is a variable rhythm action, the action description file is implanted according to the time zone occupied by the corresponding word of the current segment in the time sequence alignment information, wherein if the action type of the next word corresponding material segment is a fixed semantic action, the current segment is truncated to avoid the next word corresponding material segment;

[0035] If the action type of the current segment is a fixed semantic action, the current segment is implanted in the action description file in its entirety, and is allowed to span the time zone corresponding to the subsequent word;

[0036] The above process is repeated until each word in the label sequence is traversed.

[0037] According to another aspect of the present application, a virtual person driving device is provided, comprising:

[0038] A text acquisition module is configured to acquire driving text submitted by an anchor user in a network live broadcast;

[0039] A voice conversion module is configured to generate voice data of a virtual person in a live broadcast room of the anchor user based on the driving text, and obtain time sequence alignment information about words between the driving text and the voice data;

[0040] A material matching module is configured to match material segments corresponding to each word in the driving text from an action description material library, the material segments being suitable for driving the virtual person to perform corresponding actions;

[0041] A file construction module is configured to splice the material segments corresponding to each word into an action description file suitable for driving the virtual person in the live broadcast room of the anchor user to perform a series of actions according to the time sequence relationship between the words as specified by the time sequence alignment information.

[0042] According to another aspect of the present application, a virtual person driving device is provided, comprising a central processing unit and a memory, the central processing unit being configured to invoke a computer program stored in the memory to execute the steps of the virtual person driving method described in the present application.

[0043] According to another aspect of the present application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the virtual person driving method in the form of computer readable instructions, the computer program being invoked and run by a computer to execute the steps included in the method.

[0044] According to another aspect of the present application, a computer program product is provided, comprising computer programs / instructions, the computer programs / instructions being executed by a processor to implement the steps of the method described in any one of the embodiments of the present application.

[0045] Compared with the prior art, the application generates corresponding voice data and timing alignment information according to a given driving text, matches material segments that are suitable for words in the driving text from a motion description material library, and splices the material segments corresponding to each word into the same motion description file under the guidance of the timing alignment information, which can be used to drive a virtual person to perform a series of actions and can keep timing coordination and synchronization with the voice data. Since the material segments are usually preferred in advance and are determined by using the mapping relationship between words and material segments, the semantic correspondence is better, which can ensure that the matched material segments are closely corresponding to the corresponding words, and the voice data and the material segments have a timing synchronization relationship. Therefore, the effect of driving a virtual person to move and make sound synchronously by using a text can be achieved. The above process has low computational complexity, clear action semantic understanding, and is more efficient. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0047] Figure 1 A network architecture schematic diagram of an exemplary application environment of the application;

[0048] Figure 2 A flowchart schematic diagram of an embodiment of the virtual person driving method of the application;

[0049] Figure 3 A flowchart schematic diagram of generating a video stream according to a motion description file in an embodiment of the application;

[0050] Figure 4 A flowchart schematic diagram of generating a subtitle text in an embodiment of the application;

[0051] Figure 5 A flowchart schematic diagram of matching material segments for a driving text in an embodiment of the application;

[0052] Figure 6 A flowchart schematic diagram of determining a target material segment of a word based on similar matching in an embodiment of the application;

[0053] Figure 7 A flowchart schematic diagram of constructing a label sequence in an embodiment of the application;

[0054] Figure 8 A flowchart schematic diagram of constructing a motion description file in an embodiment of the application;

[0055] Figure 9A principle block diagram of a virtual human driving device of the present application;

[0056] Figure 10 A structural schematic diagram of a virtual human driving device adopted by the present application. DETAILED DESCRIPTION

[0057] Referring to Figure 1 , a network architecture adopted by an exemplary application scenario of the present application includes a terminal device 80, a media server 81 and an application server 82, the application server 82 can be used to deploy a network live broadcast service, the media server 81 can be used to generate a video stream corresponding to a virtual human according to an action description file, when a host user accesses the network live broadcast service provided by the application server 82 from the terminal device 80, starts to control the video stream corresponding to the virtual human, driving text can be input in the terminal device 80, the media server 81 generates an action description file and voice data for controlling the motion of the virtual human in the live broadcast room of the host user according to the driving text, then the media server 81 generates a motion image stream of the virtual human according to the action description file, and combines the motion image stream with the voice data to form a corresponding video stream, and pushes the video stream to the live broadcast room under the coordination of the application server 82, so as to provide a virtual human live broadcast service through the network live broadcast. Of course, subtitle text generated according to the driving text can also be added in the video stream.

[0058] In another exemplary application scenario of the present application, the functions of the media server can be implemented by a terminal device, after the user inputs driving text, the terminal device completes all the business logic of the media server, and finally generates a corresponding video stream, and pushes the video stream to a remote computer device, so that further applications related to calls and live broadcasts can be realized.

[0059] Referring to Figure 2 , a virtual human driving method provided by the present application includes the following steps in one embodiment thereof:

[0060] Step S1100, obtaining driving text submitted by a host user in a network live broadcast;

[0061] In combination with the application scenarios of the foregoing examples of the present application, the terminal device can obtain the driving text provided by the anchor user. The driving text can be entered by the user or specified from relevant materials such as a script file. After the anchor user provides the driving text, the driving text is submitted to the media server for further processing. The media server can generate an action description file for controlling the motion of the virtual human in the live room of the anchor user according to the driving text. According to the action description file, the motion image stream of the virtual human can be generated, and the voice data corresponding to the driving text can be provided, and even the subtitle text corresponding to the driving text can be provided. Finally, the motion image stream, the voice data, the subtitle text, and the like can be synthesized into a video stream about the virtual human in the live room on demand, and the video stream is played in the live room to present the audio-visual effect of controlling the motion of the virtual human. The driving text is usually expressed in the form of natural language, and the style can be flexibly selected, for example, it can be a commentary, a daily conversation, lyrics, lines, and the like.

[0062] In step S1200, voice data of the virtual human in the live room of the anchor user is generated based on the driving text, and time sequence alignment information about words between the driving text and the voice data is obtained.

[0063] Based on the driving text, a text-to-speech (TTS) technology can be applied to generate voice data corresponding to the driving text. According to the needs of the actual scene, the voice data can be pushed to the live room of the anchor user for playing to present the effect of the virtual human speaking. When it is played, the effect of reading the driving text by a human voice is presented. The text-to-speech technology is a relatively mature technology. In the process of converting the driving text into voice data, the driving text is usually preprocessed, for example, redundant words are removed, Arabic numerals are converted into Chinese characters, and redundant punctuation marks are removed, and the like, to obtain a purified text.

[0064] In an embodiment, the voice data can be generated based on voice conversion of the purified text, and of course, it can be generated based on voice conversion of the driving text. Since the difference between the driving text and the purified text is generally small, whether the above-mentioned preprocessing is needed can be determined according to the actual quality requirements and the text quality of the driving text. For example, when the driving text itself has high text quality, the driving text can be used as the purified text, and no additional text preprocessing is needed. The voice data of the driving text is generated directly based on the driving text. Otherwise, the purified text is obtained by preprocessing the driving text, and the corresponding voice data is generated based on the purified text.

[0065] After the speech conversion on the driving text or the purified text thereof, in addition to the speech data, timing alignment information is also obtained, in which timing information of each word in the driving text or the purified text thereof in the speech data is represented, usually including the corresponding start timestamp or end timestamp of each word in the speech data. It can be seen that the timing alignment information specifies the timing information of each word in the driving text (or the purified text thereof), and also specifies the timing relationship between the words. Through this timing information and timing relationship, the corresponding speech segment of the word in the speech data can also be located and extracted.

[0066] For Chinese, the word in the timing alignment information can be a single Chinese character or a word. For the convenience of understanding, it can be understood as a word with an independent ideographic unit. This word can be a single Chinese character or composed of multiple Chinese characters. In actual operation, since the timing information of each word is given in the timing alignment information, the timing information can be read in units of words or words, and flexible operation can be performed according to actual needs.

[0067] Step S1300, matching the material segment corresponding to each word in the driving text from the action description material library, the material segment is suitable for driving the virtual person to implement the corresponding action;

[0068] The present application is prepared with an action description material library, in which a large number of material segments are stored. Each material segment can be used to control the virtual person to implement the corresponding action, and therefore it is encapsulated and generated according to the format of the action file of the virtual person. Usually, each material segment is used to control the virtual person to implement a corresponding action.

[0069] In an embodiment, the posture action control of the virtual person is implemented by controlling the position transformation of the skeleton key points of the virtual person in the three-dimensional space. In this case, the material segment can be represented according to the control information of the skeleton key points of the virtual person. Or the facial action control of the virtual person is implemented by controlling the deformation target of the facial expression of the virtual person. In this case, the material segment can be represented according to the control information of the deformation target of the virtual person. When the material segment is applied, the actual action transformation of the posture or face of the virtual person is controlled through the control information, so as to realize the control of the movement of the virtual person.

[0070] In order to match the material segment corresponding to each word in the driving text or the purified text thereof from the action description material library, in an embodiment, the corresponding material segment of each word can be called as the target material segment adapted to each word according to the semantic mapping relationship between each word and the material segment. In another embodiment, in order to facilitate the distinction of various material segments, a label library is also predefined, and a plurality of action labels are provided in the label library. The action types of the virtual person are distinguished by different action labels. Then, the material segments belonging to the same action type are marked as the same action label, so that each action label can be associated and mapped to one or more material segments. These material segments driving the actions implemented by the virtual person have certain similarities, such as different actions such as waving both hands or waving a single hand to represent “goodbye”, or various dance actions to represent an excited state, and the like. In this case, for the words in the driving text or the purified text thereof, the corresponding action label of each word can be determined by named entity recognition or keyword mapping, and a material segment under the action label can also be called as the target material segment.

[0071] It should be noted that in the case of text preprocessing of the driving text to obtain the purified text thereof, matching the target material segment corresponding to each word from the action description material library according to the words in the purified text is equivalent to matching the target material segment corresponding to each word from the action description material library according to the words in the driving text. Since the driving text and the purified text belong to the same source, the purified text is only the product of the text preprocessing of the driving text, so the two ways are actually completely equivalent and do not affect the embodiment of the inventive spirit of the present application.

[0072] Step S1400, according to the time sequence relationship between the words defined by the time sequence alignment information, the material segments corresponding to each word are spliced into an action description file suitable for driving the virtual person in the live room of the anchor user to implement a series of actions.

[0073] Since the time sequence information of each word in the driving text or the purified text thereof is defined in the time sequence alignment information, the time sequence relationship between each word is embodied. According to this time sequence relationship, the time sequence relationship of each material segment matched from the action description material library is also embodied. In this case, the material segments corresponding to each word are sequentially spliced according to the time sequence relationship between each word, and an action description file can be obtained. The action description file can be used to drive the virtual person to implement a series of actions. The action description file can be encapsulated together with the voice data and published as an action control template, and stored to a specified location for reuse.

[0074] Since the time length of each material segment is usually long or short, the splicing of each material segment can be flexibly processed. For example, if the time length of the current word material segment is greater than the time zone occupied by the time sequence information of the current word, the current word material segment can be truncated; if the time length of the current word material segment is less than the time zone occupied by the time sequence information of the current word, the same material segment can be appropriately repeated to fill the time zone.

[0075] In some embodiments, when the semantics of the current word and the next word are similar, the material segment corresponding to the current word can also span the time zone occupied by the next word, thereby omitting the material segment of the next word.

[0076] In some embodiments, the action type of the material segment can be identified in advance. If the action corresponding to the material segment is a coherent action for complete expression of a semantic, this case can be marked as a fixed semantic action; if the action corresponding to the material segment is designed to adapt to a cyclic rhythm and does not need to rely on the action of the entire process to express semantics, this case can be marked as a variable rhythm action. Thus, when splicing the material segments of adjacent words, for the material segment of the fixed semantic action, it can be allowed to span the time zone occupied by the next word, and if the material segment of the current word belongs to the variable rhythm action and its length spans the time zone occupied by the next character, the material segment can be truncated to avoid the material segment of the next word and avoid spanning the time zone occupied by the next word. Especially when the material segment of the next word belongs to the fixed semantic action, the material segment of the current word can be forcibly truncated as described above to make the material segment of the next word fully function.

[0077] The above processing of splicing the material segments is embodied in the modification and setting of the time sequence instruction information of the description information of each material segment in actual operation. Finally, the action description information of each word corresponding to the material segment is stored in an action description file, which can be used to drive the virtual person to perform a series of actions. These series of actions are composed of the actions corresponding to each word corresponding to the material segment. However, the time of the action of each material segment is coordinated and synchronized with the pronunciation time length of each word in the voice data through the foregoing processing process. That is, the rhythm synchronization is realized based on the time sequence alignment information generated in the process of converting the driving text into voice data and the action description file between the driving text and the voice data. When the action description file is applied to the virtual person to generate a motion image stream of the virtual person and combined with the voice data to form a video stream for playing, it can be ensured that the virtual person keeps coordinated and synchronized in action and pronunciation.

[0078] The generated video stream can be directly played in the player of the terminal device or pushed to the live room of the anchor user and played by the terminal device of the audience user.

[0079] According to the above embodiment, the application generates corresponding voice data and timing alignment information according to the given driving text, matches the material segments that adapt to the words in the driving text from the action description material library, and splices the material segments corresponding to each word into the same action description file under the guidance of the timing alignment information. It can be used to drive the virtual person to perform a series of actions, and can maintain timing coordination and synchronization with the voice data. Since these material segments are usually pre-selected and determined by the mapping relationship between words and material segments, the semantic correspondence is better, which can ensure that the matched material segments and the corresponding words are closely corresponding, and the voice data and the material segments have a timing synchronization relationship. Therefore, the effect of driving the virtual person to move and make sound synchronously by text can be achieved. The above process has low computational complexity, clear action semantic understanding, and is more efficient.

[0080] On the basis of any embodiment of the application, referring to Figure 3 After the material segments corresponding to each word are spliced into the action description file suitable for driving the virtual person to perform a combination action according to the timing relationship between the words specified by the timing alignment information, the method comprises:

[0081] Step S1500, applying the action description file to drive the virtual person to move, and generating a motion image stream of the virtual person;

[0082] After the action description file is completed, it can be used to control the specified virtual person to move. Specifically, the action description file is applied to the three-dimensional model of the pre-specified virtual person. Through the application of the control information of the virtual person in the action description file, the virtual person produces a motion effect of changing its action in time sequence. In the application process, the texture of each motion state of the virtual person is added and rendered to obtain the corresponding image frame. The image frames obtained by applying the entire action description file in time sequence are sequentially organized, which constitutes the motion image stream of the virtual person corresponding to the action description file.

[0083] Step S1600, generating a subtitle text corresponding to the driving text;

[0084] Further, the corresponding subtitle text of the driving text can be generated. When generating the subtitle text, considering that the driving text and its purified text are basically consistent in text content, the driving text itself can be used to generate the subtitle text, or the purified text of the driving text can be used to generate the subtitle text, depending on the actual needs.

[0085] The subtitle text can be obtained by punctuating the driving text or the purified text thereof according to punctuation marks, and time sequence information of each sentence is marked by time sequence alignment information, so as to facilitate alignment and synthesis with the voice data and the motion image stream.

[0086] Step S1700, synthesizing the motion image stream, the voice data and the subtitle text into a video stream according to the time sequence alignment information;

[0087] Since the motion image stream of the virtual person is generated at a certain frame rate, and the time sequence information of each action therein is associated by the corresponding motion description text, the voice data and the subtitle text also establish a time sequence corresponding relationship based on the time sequence alignment information, in this case, the motion image stream, the voice data and the subtitle text have actually established an association according to the time sequence alignment information, and thus, the motion image stream, the voice data and the subtitle text can be synthesized into the same video stream according to the association corresponding relationship.

[0088] Step S1800, pushing the video stream to the live room for playing.

[0089] When the generation of the video stream is completed, the video stream can be directly played in the terminal device, or can be pushed to the live room of the anchor user to be played by the terminal device of the audience user side of the live room according to the needs of the network live scene. When the video stream is played, not only the continuous actions of the virtual person can be played, but also the voice data can be synchronously played and the corresponding subtitle text can be displayed. The voice data and the subtitle text are basically synchronous and consistent in content, and the actions of the virtual person and the content of the voice data and the subtitle text at the time are highly consistent in voice.

[0090] According to the above embodiments, the motion image stream of the virtual person is generated based on the action description file obtained in the application, and the voice data, the subtitle text and the motion image stream are synthesized to obtain the video stream according to the time sequence alignment information generated in the process of converting the voice data from the driving text in the application, which can ensure the synchronization of the video stream, the text and the sound multimedia, and obtain the natural and coordinated virtual person activity image effect.

[0091] On the basis of any embodiment of the application, please refer to Figure 4 , generating the subtitle text corresponding to the driving text, comprising:

[0092] Step S1610, obtaining the purified text obtained by preprocessing the driving text in the process of generating the voice data of the virtual person in the live room of the anchor user based on the driving text;

[0093] As mentioned above, in the process of converting the driving text into speech data by using the text-to-speech technology, the time alignment information is automatically generated, in which the start time stamp and the end time stamp of the corresponding speech segment of each word in the driving text in the speech data are marked.

[0094] In the embodiment, the driving text is pre-processed on demand before conversion, and after removing the redundant words, redundant punctuation marks, correcting the wrong words and other operations, the purified text is obtained. The subsequent operation can be performed on the basis of the purified text.

[0095] In step S1620, the driving text and the purified text are respectively punctuated according to punctuation marks, and the corresponding punctuation number is counted.

[0096] The driving text generally represents the original meaning expressed by the user, while the purified text may have slight differences with the original meaning after pre-processing. In this case, the subtitle text can be made from the driving text or the purified text. Therefore, the driving text and the purified text can be respectively punctuated according to punctuation marks, and the corresponding sentence sequence of each is obtained, and the punctuation number of each sentence sequence is counted.

[0097] In step S1630, it is judged whether the punctuation number of the driving text and the punctuation number of the purified text are consistent. When they are consistent, the driving text is set as the subtitle text corresponding to the purified text, otherwise, the purified text is set as the subtitle text.

[0098] When the punctuation number of the driving text and the punctuation number of the purified text are compared and judged to be consistent, it can be considered that the purified text has little difference with the driving text. In this case, it is more inclined to use the sentence in the driving text to make the subtitle text to express the original meaning as much as possible. Therefore, the driving text can be set as the subtitle text corresponding to the purified text, so as to ensure that the corresponding time sequence of each sentence of the driving text is accurately matched when the video stream is generated.

[0099] When the punctuation number of the driving text and the punctuation number of the purified text are inconsistent, considering that the speech data is generated according to the purified text, in order to avoid the difference between the speech content and the subtitle content, the purified text is directly set as the subtitle text, and the driving text is not used as the subtitle text.

[0100] According to the above embodiments, when setting the subtitle text, the quality difference between the driving text and the purified text generated in the preprocessing of the converted speech data is comprehensively considered, the number of sentences is compared, the driving text is preferentially applied when the number of sentences is consistent to apply the original meaning expression, and the purified text is preferentially used when the number of sentences is inconsistent, so that the quality of the subtitle text is comprehensively ensured to be optimal through automatic weighing means.

[0101] On the basis of any embodiment of the present application, refer to Figure 5 The material segment corresponding to each word in the driving text is matched from the action description material library, and the material segment is suitable for driving the virtual person to perform the corresponding action, and includes:

[0102] In step S1310, the driving text is sequentially segmented, and a segmentation sequence composed of each word in the driving text is obtained;

[0103] The driving text is sequentially segmented, which can be based on the driving text itself or its purified text. For example, the driving text is segmented based on various feasible segmentation methods, including but not limited to N-Gram segmentation algorithm, bag-of-words segmentation algorithm, etc. The driving text is sequentially divided into multiple words, each of which can be a single word or a multi-word vocabulary, which is determined by the segmentation algorithm result. These words form a segmentation sequence according to their order, which facilitates subsequent operations based on the segmentation sequence.

[0104] In step S1320, a preset action mapping model / tag mapping library is used to determine the action tag corresponding to each word in the segmentation sequence, and a tag sequence is obtained;

[0105] Based on the obtained segmentation sequence, the action tag corresponding to each word in the segmentation sequence can be further determined, which can be determined in the following any way:

[0106] In one embodiment, a tag mapping library is prepared, and the mapping relationship data between the action tags and the words is stored in the tag mapping library. Thus, the action tag corresponding to the word in the segmentation sequence can be determined by searching the word in the mapping tag library. Generally, the mapping relationship between the word and the action tag can be established according to the same or similar semantics, so that the action tag can distinguish different types of actions.

[0107] In another embodiment, a neural network model based on deep learning can be used as an action mapping model, and corresponding data samples are used to train the model, so that the model can map the word to the corresponding action tag. Thus, after the word in the segmentation sequence is input into the action mapping model, the action tag corresponding to the word can be obtained.

[0108] After each word in the word sequence obtains its corresponding action label, all the action labels are organized in word order to obtain a label sequence.

[0109] It should be noted that for individual rare words, if there is no corresponding action label corresponding to it, a default action label representing null value can be specified for identification, and subsequent matching of material segments for the corresponding word can be based on this default action label.

[0110] Step S1330, based on the action label corresponding to each word in the label sequence, a material segment under the corresponding action label is determined from the action description material library.

[0111] In the action description material library, a large number of material segments are stored, and each material segment is pre-labeled with one or more corresponding action labels. In this case, one action label may correspond to multiple material segments, so for the action label corresponding to each word in the label sequence, when multiple material segments are queried from the action description material library, one of them can be randomly determined as the target material segment matched with the word in an embodiment. Of course, in other embodiments, the action label of the word can also be matched with its corresponding target material segment based on similarity.

[0112] According to the above embodiments, it can be understood that by sequentially segmenting the driving text, determining the action label based on the segmentation, and then determining the corresponding material segment based on the action label, this process does not need to rely on machine learning models, is more efficient, and because the action label has been associated with high-quality material segments through semantics, these material segments have clear semantics due to prior selection, so the entire driving text can ultimately be matched with all target material segments corresponding to each word, and a high-quality action description file can be generated accordingly.

[0113] On the basis of any embodiment of the present application, please refer to Figure 6 , based on the action label corresponding to each word in the label sequence, a material segment under the corresponding action label is determined from the action description material library, including:

[0114] Step S1331, based on the current word in the label sequence, a style feature vector of the corresponding speech segment in the speech data is determined, the style feature vector is used to represent the style invariant features of the corresponding speech segment, and the style includes rhythm style and / or emotional style;

[0115] Each word in the driving text has its corresponding speech segment in the speech data generated according to the driving text, and the speech segment can be intercepted according to the time alignment information obtained in the process of generating the speech data, so that when the target material segment of the action label corresponding to the current word in the label sequence is determined, the corresponding speech segment can be intercepted from the speech data based on the current word and the start timestamp and the end timestamp corresponding to the current word in the time alignment information.

[0116] Then, a pre-trained speech feature representation model suitable for representing features of the speech segment is used to extract a style feature vector of the speech segment, and the style-invariant features in the speech segment are represented by the style feature vector. The style referred to herein can be any one or any combination of rhythm style, emotional style, etc., which is determined by the ability learned by the speech feature representation model during training.

[0117] As can be seen, in this way, the style feature vector of the speech segment corresponding to the current word in the speech data can be obtained.

[0118] Step S1332, obtaining the style feature vectors of the material segments corresponding to the action label of the current word from the action description material library;

[0119] For each material segment in the action description material library, a corresponding material feature representation model can also be used to pre-extract its corresponding style feature vector. This style feature vector can be obtained based on the motion image stream generated by controlling the virtual human to generate motion based on the corresponding material segment, and thus can represent the relatively stable features of the rhythm style and / or emotional style.

[0120] In an exemplary training architecture, the speech feature representation model and the material feature representation model can be built into a double-tower model for joint training, and a large number of data samples are prepared, and each data sample is called in turn to implement training. In each data sample, a speech segment and a material segment are included to generate a motion image stream, which is respectively subjected to feature representation by the speech feature representation model and the material feature representation model to obtain corresponding style feature vectors, and then the two style feature vectors are spliced into a comprehensive feature vector, which is mapped to a classifier through a fully connected layer, the similarity of the two style feature vectors is calculated in the classifier, then a supervision label pre-labeled for the data sample is used, the supervision label represents whether the speech segment and the motion image stream of the material segment are semantically corresponding, the loss value of the similarity is calculated, and then the loss value is used to correct the parameters of each link of the entire double-tower model through back propagation, so as to realize gradient update. Through such iterative training, the entire double-tower model is trained to a converged state, and then it can be put into use. The speech feature representation model in the double-tower model can be used to extract a style feature vector for a speech segment, and the material feature representation model can be used to extract a corresponding style feature vector for a material segment.

[0121] In step S1333, the similarity between the style feature vector of the corresponding speech segment and the style feature vector of each material segment is calculated, and the material segment with the maximum similarity of the style feature vector is determined as the selected target material segment.

[0122] As described above, the speech feature representation model and the material feature representation model in the double-tower model can respectively obtain the style feature vector of the speech segment and the style feature vector of the material segment, so that the style feature vector of the speech segment of the current word and the style feature vector of each matched material segment are spliced into a comprehensive feature vector in the double-tower model, and the corresponding similarity can be calculated by the classifier thereof. Each material segment is calculated with the speech segment to obtain the corresponding similarity, and the similarities of the material segments can be further compared.

[0123] Generally speaking, for the plurality of material segments under the action label determined for the current word, when one of the material segments has the highest similarity, this material segment can be considered as the target material segment that is closest in semantics to the speech segment, and can be used to construct an action description file.

[0124] In the above manner, the target material segment corresponding to each word in the driving text is determined one by one, a material segment most associated with semantics is determined for each word, and then an action description file is constructed based on the target material segments. This action description file is undoubtedly of the highest quality.

[0125] According to the above embodiments, the semantic correlation between the style feature vectors of the voice segment and the material segment preferably matches each word with a target material segment, which can ensure that the action of the target material segment is closely related to the word in semantics, and the generated action description file is of high quality.

[0126] Based on any of the embodiments of the present application, refer to Figure 7 , determine the action label corresponding to each word in the word segmentation sequence, obtain a label sequence, including:

[0127] Step S1321, taking the current word as a query word, querying whether there is a corresponding action label in the label mapping library, when there is, determining that the corresponding action label is the action label of the current word, and adding it to the label sequence;

[0128] When determining the action label corresponding to each word based on the word segmentation sequence, each word in it can be matched as the current word. Specifically, taking the current word as a query word, first query whether there is a corresponding action label in the label mapping library, if there is, the action label can be directly determined as the action label of the current word, and added to the corresponding position in the label sequence.

[0129] Step S1322, when the label mapping library does not have an action label corresponding to the current word, and the label mapping library has an action label of a word with the current word as a prefix, concatenate the next word with the current word to form a new query word, query whether there is an action label corresponding to the new query word in the label mapping library, when there is a corresponding action label, replace the concatenated words with the new query word, and add the action label of the new query word to the label sequence;

[0130] If the label mapping library does not have an action label corresponding to the current word, but the label mapping library has an action label of a word with the current word as a prefix, in this case, the current word can be expanded. Specifically, according to the order in the word segmentation sequence, the current word and its next word can be sequentially concatenated into a new query word, and then the new query word can be queried in the label mapping library. If there is an action label corresponding to the new query word in the label mapping library, the new query word can replace the concatenated words in the word segmentation sequence, and the action label becomes the action label of the replaced word, which can be added to the corresponding position in the label sequence. In this way, the semantic association of the action label can be maximized to match the driving text, and the semantic of individual words can be avoided from becoming an island.

[0131] Step S1323, when there is no action label corresponding to the new query word, the action label of the current word is set to point to a preset material segment belonging to a variable rhythm action and is added to the label sequence.

[0132] If the corresponding action label cannot be queried from the label mapping library with the new query word, the iterative query for expansion can not be continued, in which case, the action label of the current word can be set as a default action label representing a null value, for which the corresponding material segment is a preset material segment belonging to a variable rhythm action, and the material segment corresponding to the corresponding word is a preset default material segment. The default action label is added to the corresponding position of the label sequence, and the subsequent default material segment is set according to this principle.

[0133] As can be seen from the above embodiments, in the process of matching the action label of each word in the word segmentation sequence, when there is no corresponding action label for the current word, it is merged with the next word for expansion and query to determine its action label, so as to avoid the semantic of individual words being isolated as much as possible, and to make the determined action label correspond to more words as much as possible, so that the finally determined material segments can be more closely associated with the original meaning of the driving text, and the quality of the action description file is more optimal.

[0134] On the basis of any embodiment of the present application, refer to Figure 8 Splicing the material segments corresponding to each word into an action description file suitable for driving the virtual human to perform a series of actions in the live room of the anchor user, comprising:

[0135] Step S1410, obtaining a material segment corresponding to a word in the label sequence as a current segment, and identifying that the action type of the current segment is a fixed semantic action or a variable rhythm action;

[0136] As described above, different material segments corresponding to the same action label can have an action type of a fixed semantic action, referred to as a fixed semantic action, or an action associated with rhythm, referred to as a variable rhythm action, which can be represented by the action type.

[0137] Each word of the driving text in the label sequence has a corresponding action label, and the corresponding target material segment is determined for the action label. In this case, each target material segment can be spliced based on the action description file.

[0138] To this end, the label sequence can be sequentially indexed, and for each material segment, i.e., target material segment, the current segment is taken, and then it is identified whether the action type belongs to a fixed semantic action or a variable rhythm action, i.e., the action type of the current segment is determined.

[0139] Step S1420: If the action type of the current segment is a variable rhythm action, the action description file is inserted according to the time zone occupied by the corresponding words in the timing alignment information of the current segment. If the action type of the material segment corresponding to the next word is a fixed semantic action, the current segment is truncated to avoid the material segment corresponding to the next word.

[0140] If the action type of the current segment is a variable rhythm action, the current segment can be inserted into the corresponding position in the action description file. The insertion position mainly refers to the temporal position, which can be the time zone position occupied by the corresponding words in the temporal alignment information of the current segment.

[0141] Since the duration of a clip may span the time zone of its corresponding word, if the action type of the clip corresponding to the next word is a fixed semantic action, it will disrupt the semantic integrity of the action in the next clip. In this case, considering that the current clip is a variable-rhythm action, it can be truncated, cutting off the portion whose duration exceeds the time zone of its corresponding word to avoid the clip corresponding to the next word. Of course, if the action type of the clip of the next word is also a variable-rhythm type, it can be handled more flexibly. For example, the current clip can be fully embedded in the action description file, even if its duration exceeds the time zone corresponding to the current word. The next word whose time zone is occupied can adapt to the fact that it is occupied and automatically avoid the current clip.

[0142] Step S1430: If the action type of the current segment is a fixed semantic action, the current segment is fully embedded into the action description file, and it is allowed to cross the time zone corresponding to subsequent words;

[0143] If the action type of the current segment is detected to be a fixed semantic action, in order to maintain the semantic integrity of the current segment (e.g., ensuring the integrity of a waving gesture indicating "goodbye"), the current segment can be fully embedded into the action description file, even if its duration in the time sequence exceeds the time zone occupied by the current character and enters the time zone corresponding to the next word. Similarly, for the material segment corresponding to the next word, the current segment can be time-avoided to avoid conflicts.

[0144] Step S1440: Repeat the above process until the traversal of each word in the label sequence is completed.

[0145] When the above steps are executed, it is determined whether to iterate from the next word to step S1410 to step S1430 according to whether the current word is the last word. For each word in the driving text, it is processed as a current word in steps S1410 to S1430, and the traversal process is decided in this step. Finally, the construction of the entire action description file can be completed.

[0146] According to the above embodiment, different action types are distinguished, and the time sequence position of the material segment belonging to the fixed semantic action in the action description file is preferentially ensured. The material segment belonging to the variable rhythm action is avoided for the former, the semantic integrity of the virtual person action is ensured, the action description file of the virtual person is more excellent, and the semantics between the picture, sound and word in the generated video stream is more coordinated.

[0147] Please refer to Figure 9 According to an aspect of the present application, a virtual person driving device is provided, which comprises a text acquisition module 1100, a voice conversion module 1200, a material matching module 1300, and a file construction module 1400. The text acquisition module 1100 is configured to acquire driving text submitted by an anchor user in a network live broadcast. The voice conversion module 1200 is configured to generate voice data of a virtual person in a live broadcast room of the anchor user based on the driving text, and obtain time sequence alignment information about words between the driving text and the voice data. The material matching module 1300 is configured to match material segments corresponding to each word in the driving text from an action description material library, and the material segments are suitable for driving the virtual person to perform corresponding actions. The file construction module 1400 is configured to splice the material segments corresponding to each word into an action description file suitable for driving the virtual person in the live broadcast room of the anchor user to perform a series of actions according to the time sequence relationship between the words specified by the time sequence alignment information.

[0148] On the basis of any embodiment of the present application, the virtual person driving device further comprises an action application module configured to apply the action description file to drive the virtual person to move and generate a moving image stream of the virtual person; a subtitle generation module configured to generate a subtitle text corresponding to the driving text; a video synthesis module configured to synthesize the moving image stream, the voice data and the subtitle text into a video stream according to the time sequence alignment information; and a video playing module configured to push the video stream to the live broadcast room for playing.

[0149] On the basis of any embodiment of the present application, the subtitle generation module comprises: a text purification unit configured to obtain purified text obtained by preprocessing the driving text in the process of generating voice data of the virtual person in the live room of the anchor user based on the driving text; a text segmentation unit configured to segment the driving text and the purified text according to punctuation marks respectively, and count the number of corresponding segmentations; and a subtitle optimization unit configured to determine whether the number of segmentations of the driving text and the purified text is consistent, and when the number of segmentations is consistent, set the driving text corresponding to the purified text as a subtitle text, otherwise, set the purified text as a subtitle text.

[0150] On the basis of any embodiment of the present application, the material matching module 1300 comprises: a word segmentation processing unit configured to sequentially segment the driving text to obtain a word segmentation sequence composed of each word in the driving text; a label determination unit configured to determine the action label corresponding to each word in the word segmentation sequence by using a preset action mapping model / label mapping library, and obtain a label sequence; and a segment matching unit configured to determine a material segment corresponding to the action label from an action description material library based on the action label corresponding to each word in the label sequence.

[0151] On the basis of any embodiment of the present application, the segment matching unit comprises: a voice interception subunit configured to determine the style feature vector of the corresponding voice segment in the voice data based on the current word in the label sequence, wherein the style feature vector is used to represent the style invariant feature of the corresponding voice segment, and the style comprises rhythm style and / or emotional style; a material calling subunit configured to obtain the style feature vector of each material segment corresponding to the action label of the current word from the action description material library; and a similarity matching subunit configured to calculate the similarity between the style feature vector of the corresponding voice segment and the style feature vector of each material segment, and determine the material segment with the maximum similarity of the style feature vector as the selected target material segment.

[0152] Based on any embodiment of this application, the tag determination unit includes: a word query subunit, configured to use the current word as the query word, query the tag mapping library to see if there is a corresponding action tag, and if so, determine the corresponding action tag as the action tag of the current word and add it to the tag sequence; a word expansion query subunit, configured to, when there is no action tag corresponding to the current word in the tag mapping library, but there is an action tag of a word with the current word as a prefix in the tag mapping library, concatenate the current word with the next word to form a new query word, query the tag mapping library to see if there is an action tag corresponding to the new query word, and if there is a corresponding action tag, replace each concatenated word with the new query word and add the action tag of the new query word to the tag sequence; and a default processing subunit, configured to, when there is no action tag corresponding to the new query word, set the action tag of the current word to point to a preset material segment belonging to variable rhythm action and add it to the tag sequence.

[0153] Based on any embodiment of this application, the file construction module 1400 includes: a type identification unit, configured to acquire a material fragment corresponding to a word in the tag sequence as the current fragment, and identify whether the action type of the current fragment is a fixed semantic action or a variable rhythm action; a variable splicing unit, configured to, if the action type of the current fragment is a variable rhythm action, implant an action description file according to the time zone occupied by the corresponding word in the temporal alignment information, wherein, if the action type of the material fragment corresponding to the next word is a fixed semantic action, truncate the current fragment to avoid the material fragment corresponding to the next word; a fixed splicing unit, configured to, if the action type of the current fragment is a fixed semantic action, fully implant the current fragment into the action description file, and allow it to cross the time zone corresponding to subsequent words; and an iterative decision unit, configured to repeat the above process until the traversal of each word in the tag sequence is completed.

[0154] Another embodiment of this application provides a virtual human driving device. For example... Figure 10 The diagram shows the internal structure of a virtual human driving device. This virtual human driving device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable, non-volatile storage medium stores an operating system, a database, and computer-readable instructions. The database can store information sequences, and when executed by the processor, these computer-readable instructions enable the processor to implement a virtual human driving method.

[0155] The processor of the virtual human driving device is configured to provide computing and control capabilities to support the operation of the entire virtual human driving device. The memory of the virtual human driving device can store computer readable instructions, which, when executed by the processor, can cause the processor to execute the virtual human driving method of the present application. The network interface of the virtual human driving device is configured to communicate with the terminal.

[0156] Those skilled in the art can understand that, Figure 10 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the virtual human driving device to which the scheme of the present application is applied. A specific virtual human driving device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0157] The processor in the embodiment is configured to execute the specific functions of each module in Figure 9 The memory stores program codes and various data required for executing the above-mentioned modules or sub-modules. The network interface is configured to realize data transmission between the user terminal or the server. The non-volatile readable storage medium in the embodiment of the present application stores program codes and data required for executing all modules in the virtual human driving device of the present application. The server can call the program codes and data of the server to execute the functions of all modules.

[0158] The present application also provides a non-volatile readable storage medium storing computer readable instructions, which, when executed by one or more processors, cause the one or more processors to execute the steps of the virtual human driving method of any embodiment of the present application.

[0159] The present application also provides a computer program product, including computer programs / instructions, which, when executed by one or more processors, implement the steps of the method described in any embodiment of the present application.

[0160] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of each method. The storage medium can be a computer readable storage medium such as a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM).

[0161] In summary, the application can realize text-driven virtual human motion, obtain each material segment spliced into a motion description file for controlling virtual human motion by matching the words and material segments of the driving text, avoid completely relying on the reasoning ability learned by the machine learning model, effectively control the technical implementation cost, and at the same time realize the efficient and high-quality processing effect, when the motion description file of the application is used to control the virtual human to generate a motion image, more natural and smooth virtual human motion effect can be obtained, which can serve various application scenarios such as virtual human network live broadcast and virtual human music singing, and pave the way for the meta-universe technology era.

Claims

1. A virtual human driving method, characterized by, The method comprises the following steps: obtaining driving text submitted by a host user in a network live broadcast; generating speech data of a virtual person in a live broadcast room of the host user based on the driving text, and obtaining time sequence alignment information between the driving text and the speech data regarding words; matching a material segment corresponding to each word in the driving text from an action description material library, the material segment being suitable for driving the virtual person to perform corresponding actions; splicing the material segment corresponding to each word into an action description file suitable for driving the virtual person in the live broadcast room of the host user to perform a series of actions according to a time sequence relationship between the words specified by the time sequence alignment information, comprising: obtaining a material segment corresponding to a word in a label sequence as a current segment, and identifying that the action type of the current segment is a fixed semantic action or a variable rhythm action; if the action type of the current segment is a variable rhythm action, implanting the action description file according to the time zone occupied by the word corresponding to the current segment in the time sequence alignment information, wherein if the action type of the material segment corresponding to the next word is a fixed semantic action, the current segment is truncated to avoid the material segment corresponding to the next word; if the action type of the current segment is a fixed semantic action, implanting the current segment into the action description file completely, and allowing it to cross the time zone corresponding to the subsequent word; repeating the above process until each word in the label sequence is traversed.

2. The virtual human driving method of claim 1, wherein, After splicing the material segment corresponding to each word into an action description file suitable for driving the virtual person to perform a combined action according to the time sequence relationship between the words specified by the time sequence alignment information, comprising: applying the action description file to drive the virtual person to move, and generating a moving image stream of the virtual person; generating a subtitle text corresponding to the driving text; combining the moving image stream, the speech data and the subtitle text into a video stream according to the time sequence alignment information; pushing the video stream to the live broadcast room for playing.

3. The virtual human driving method of claim 2, wherein, Generating a subtitle text corresponding to the driving text comprises: obtaining a purified text obtained by preprocessing the driving text in the process of generating speech data of a virtual person in a live broadcast room of the host user based on the driving text; performing sentence segmentation on the driving text and the purified text respectively according to punctuation marks, and counting the number of corresponding sentence segmentation; determining whether the number of sentence segmentation of the driving text and the purified text is consistent, when consistent, setting the driving text corresponding to the purified text as a subtitle text according to the sentence, otherwise, setting the purified text as a subtitle text.

4. The virtual human driving method according to any one of claims 1 to 3, characterized in that, Matching a material segment corresponding to each word in the driving text from an action description material library, the material segment being suitable for driving the virtual person to perform corresponding actions, comprising: performing sequential word segmentation on the driving text to obtain a word sequence composed of each word in the driving text; determining the action label corresponding to each word in the word sequence by using a preset action mapping model / label mapping library, and obtaining a label sequence; determining a material segment under a corresponding action label from an action description material library based on the action label corresponding to each word in the label sequence.

5. The virtual human driving method of claim 4, wherein, determine a material segment corresponding to the action label of the current word from an action description material library based on the action label corresponding to each word in the label sequence, including: determine a style feature vector of the corresponding speech segment in the speech data based on the current word in the label sequence, the style feature vector being used to represent style-invariant features of the corresponding speech segment, the style including rhythm style and / or emotional style; obtain the style feature vectors of each material segment corresponding to the action label of the current word from the action description material library; calculate the similarity between the style feature vector of the corresponding speech segment and the style feature vectors of each material segment, and determine the material segment with the maximum similarity as the target material segment.

6. The virtual human driving method of claim 4, wherein, determine the action label corresponding to each word in the word sequence to obtain a label sequence, including: query whether there is a corresponding action label in the label mapping library with the current word as a query word, and when there is, determine the corresponding action label as the action label of the current word and add it to the label sequence; when there is no action label corresponding to the current word in the label mapping library, and there is an action label of a word with the current word as a prefix in the label mapping library, concatenate the next word with the current word to form a new query word, query whether there is an action label corresponding to the new query word in the label mapping library, when there is a corresponding action label, replace each word by the new query word, and add the action label of the new query word to the label sequence; when there is no action label corresponding to the new query word, set the action label of the current word to point to a preset material segment belonging to a variable rhythm action, and add it to the label sequence.

7. A virtual human driving apparatus, characterized by, including: a text acquisition module configured to acquire driving text submitted by a host user in a network live broadcast; a speech conversion module configured to generate speech data of a virtual person in a live broadcast room of the host user based on the driving text, and to obtain time sequence alignment information about words between the driving text and the speech data; a material matching module configured to match material segments corresponding to each word in the driving text from an action description material library, the material segments being suitable for driving the virtual person to perform corresponding actions; a file construction module configured to concatenate the material segments corresponding to each word into an action description file suitable for driving the virtual person in the live broadcast room of the host user to perform a series of actions according to the time sequence relationship between the words specified by the time sequence alignment information; the file construction module includes a type identification unit configured to obtain a material segment corresponding to a word in a label sequence as a current segment, and identify the action type of the current segment as a fixed semantic action or a variable rhythm action; a variable concatenation unit configured to, if the action type of the current segment is a variable rhythm action, implant the action description file according to the time zone occupied by the corresponding word of the current segment in the time sequence alignment information, wherein if the action type of the material segment corresponding to the next word is a fixed semantic action, the current segment is truncated to avoid the material segment corresponding to the next word. The fixed splicing unit is configured to, if the action type of the current segment is a fixed semantic action, completely implant the current segment into the action description file and allow it to span the time zone corresponding to the subsequent word; The iterative decision unit is configured to loop the above process until the traversal of each word in the label sequence is completed.

8. The virtual human driving apparatus according to claim 7, wherein The method comprises the following steps: The original word query subunit is configured to take the current word as a query word to query whether there is a corresponding action label in the label mapping library, and when there is, determine that the corresponding action label is the action label of the current word and add it to the label sequence; The expanded word query subunit is configured to, when there is no action label corresponding to the current word in the label mapping library and there is an action label of a word with the current word as a prefix in the label mapping library, concatenate the next word with the current word to form a new query word, query whether there is an action label corresponding to the new query word in the label mapping library, when there is a corresponding action label, replace each word that is spliced with the new query word, and add the action label of the new query word to the label sequence; The default processing subunit is configured to, when there is no action label corresponding to the new query word, set the action label of the current word to point to a preset material segment belonging to a variable rhythm action and add it to the label sequence.

9. A virtual human driving apparatus comprising a central processing unit and a memory, characterized by, The central processing unit is configured to call and run a computer program stored in the memory to perform the steps of the method according to any one of claims 1 to 6.

10. A non-volatile readable storage medium, characterized by The computer program is stored in the form of computer readable instructions and is implemented according to the method of any one of claims 1 to 6. When the computer program is called and run by a computer, the steps included in the corresponding method are performed.

Images