Driving method and device of virtual biological model, video generation method and device, equipment, medium and computer program product

Abstract

The embodiment of the invention provides a driving method of a virtual biological model, a video generation method and device, equipment, a medium and a computer program product, and the driving method of the virtual biological model comprises the steps: obtaining driving information used for driving the virtual biological model; the driving information comprises at least one of text information and voice information; converting the driving information into a phoneme sequence; determining action information corresponding to each phoneme in the phoneme sequence to obtain an action information sequence; at least one piece of action information in the action information sequence at least contains facial action information of the virtual biological model; and driving the virtual biological model to act according to the action information sequence.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and more specifically, to a driving method, video generation method, device, equipment, medium and computer program product of a virtual biological model. Background Art

[0002] With the development of computer technology, the application of virtual models is becoming more and more extensive, especially virtual biological models (Avatar) are used in various scenarios. For example, virtual character models / digital human models are used to replace real people in online live broadcasts or video conferences, virtual biological models are used as actors in animation videos, and virtual character models are used to convert / translate the natural speech of real people into sign language for the convenience of hearing-impaired / speech-impaired people.

[0003] In order to realize the application of virtual biological models in various scenarios, virtual biological model driving technology has also emerged. The so-called virtual biological model driving technology refers to the technology that drives the virtual biological model to move, or in layman's terms, the technology that forces the virtual biological model to "move". Summary of the invention

[0004] In the related art, the virtual biological model is mainly driven by manual driving. Specifically, the staff sets the spatial coordinate values ​​of each vertex / bone point in the virtual biological model at different times in space, and controls each vertex / bone point in the virtual biological model to move to the spatial position marked by the corresponding coordinate value set by the staff at the corresponding time, thereby realizing the driving of the virtual biological model. However, the manual driving method is not flexible enough and the manual workload is large. Therefore, the embodiments of the present application provide a virtual biological model driving method, a video generation method, a device, an electronic device, a storage medium and a computer program product for driving a virtual biological model, which is more flexible than the manual driving method in the related art and can reduce the manual workload to a certain extent.

[0005] The first aspect of an embodiment of the present application provides a method for driving a virtual biological model, the method comprising: obtaining driving information for driving the virtual biological model; the driving information comprising at least one of text information and voice information; converting the driving information into a phoneme sequence; determining the action information corresponding to each phoneme in the phoneme sequence to obtain an action information sequence; at least one action information in the action information sequence contains at least facial action information of the virtual biological model; and driving the virtual biological model to perform actions according to the action information sequence.

[0006] A second aspect of an embodiment of the present application provides a method for generating a video, the method comprising: in the process of driving the virtual biological model to perform an action according to the driving method of the virtual biological model described in the first aspect, synchronously outputting phonemes in a target phoneme sequence to generate a video; the target phoneme sequence is a phoneme sequence obtained by converting driving information in the process of driving the virtual biological model to perform an action according to the driving method of the virtual biological model described in the first aspect.

[0007] The third aspect of an embodiment of the present application provides a driving device for a virtual biological model, the device comprising: a first acquisition module, used to obtain driving information for driving the virtual biological model; the driving information includes at least one of text information and voice information; a first conversion module, used to convert the driving information into a phoneme sequence; a first determination module, used to determine the action information corresponding to each phoneme in the phoneme sequence to obtain an action information sequence; there is at least one action information in the action information sequence that contains at least facial action information of the virtual biological model; and a first driving module, used to drive the virtual biological model to perform actions according to the action information sequence.

[0008] The fourth aspect of an embodiment of the present application provides a video generating device, the device comprising: the driving device described in the third aspect; a first generating module, used to synchronously output the phonemes in the target phoneme sequence to generate a video during the process of the driving device driving the virtual biological model to perform an action; the target phoneme sequence is a phoneme sequence obtained by converting the driving information during the process of the driving device driving the virtual biological model to perform an action.

[0009] The fifth aspect of an embodiment of the present application provides an electronic device, comprising: a processor and a memory, the memory storing programs or instructions that can be run on the processor, and the program or instructions, when executed by the processor, implement the steps of the driving method of the virtual biological model as described in the first aspect or the steps of the video generation method as described in the second aspect.

[0010] The sixth aspect of an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the driving method of the virtual biological model as described in the first aspect or the steps of the video generation method as described in the second aspect are implemented.

[0011] The seventh aspect of the embodiments of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the driving method of the virtual biological model as described in the first aspect or the steps of the video generation method as described in the second aspect.

[0012] In the embodiment of the present application, after obtaining the driving information in the form of text or voice, it is converted into a phoneme sequence, the actions corresponding to each phoneme in the phoneme sequence are determined, and the action information sequence is obtained, and the virtual biological model is driven to act according to the action information sequence. And there is at least one action information in the action information sequence that contains at least the facial action information of the virtual biological model. Therefore, when driving the virtual biological model to act, at least in one action, the face of the virtual biological model may be driven to act, that is, in the process of driving the biological model, the facial muscles / facial organs of the virtual biological model may also be driven to act in certain actions to change the facial expression of the virtual biological model. Compared with the manual driving method in the related art, the virtual biological model driving method provided in the embodiment of the present application can drive the virtual biological model based on the driving information in the form of text or voice, and the driving method is more flexible, which can reduce the manual workload to a certain extent.

[0013] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0015] Figure 1 A schematic diagram of the steps of a method for driving a virtual biological model provided in an embodiment of the present application;

[0016] Figure 2 A schematic diagram of a scenario in which a virtual human body model is used to broadcast live an online live broadcast scenario provided in an embodiment of the present application;

[0017] Figure 3 A schematic diagram of a scenario for customizing a virtual anchor in an online live broadcast scenario provided in an embodiment of the present application;

[0018] Figure 4 A schematic diagram of a scenario for obtaining driving information in text form in an online live broadcast scenario provided by an embodiment of the present application;

[0019] Figure 5 A speech rate setting interface provided in an embodiment of the present application;

[0020] Figure 6 A schematic diagram of the structure of a driving device for a virtual biological model provided in an embodiment of the present application;

[0021] Figure 7 A schematic diagram of the structure of a video generation device provided in an embodiment of the present application;

[0022] Figure 8 A schematic diagram of the hardware structure of an electronic device for implementing an embodiment of the present application;

[0023] Fig. 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0025] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0026] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0027] The embodiment of the present application provides a driving method, video generation method, device, equipment, medium and computer program product of a virtual biological model, which has a more flexible driving mode and can reduce the manual workload to a certain extent. The following is a description of a driving method, video generation method, device, equipment, medium and computer program product of a virtual biological model provided by the embodiment of the present application in conjunction with the accompanying drawings.

[0028] like Figure 1 As shown, Figure 1 The figure is a schematic diagram of the steps of a method for driving a virtual biological model provided in an embodiment of the present application. The method for driving a virtual biological model can be applied to various electronic devices, such as computers, smart phones, tablet computers, wearable smart devices, in-vehicle information system on chip (IVI SoC) on smart cars, and other devices, and can also be servers of distributed systems, cloud servers, intelligent cloud computing servers with artificial intelligence technology, or intelligent cloud hosts. Please refer to Figure 1 The driving method of the virtual biological model comprises the following steps S102-S108:

[0029] S102: Acquire driving information for driving the virtual biological model.

[0030] As is known to all, organisms refer to living things with metabolic functions in the objective world, including plants, animals (humans are also animals) and microorganisms. The virtual biological model described in the embodiments of the present application refers to a two-dimensional (2D) or three-dimensional (3D) virtual biological / biological model made based on computer modeling technology. Of course, the virtual biological model does not have metabolic functions, but is just a role object with "life" in the virtual world.

[0031] Among them, the biomimicking model is a virtual biological model that is made by imitating animals, plants or microorganisms in the objective world and has the same or similar appearance as the plants, animals or microorganisms in the objective world, such as virtual human models, virtual animal models, virtual plant models, etc.; the quasi-biological model is a virtual model that is created out of thin air by engineers through their imagination or based on artificial intelligence technology to play the role of a living organism in the virtual world, such as the virtual model of a tree lady and the virtual model of an elf made by the staff, the virtual model of an alien made by an artificial intelligence model, etc. The quasi-biological model is a model of an "imaginary creature".

[0032] It should be noted that the virtual biological model described in the embodiments of the present application generally includes a trunk, limbs and / or a head. Among them, the limbs and the head are all connected to the trunk, and the head and limbs can be driven to perform actions in a set virtual space, and a face can also be made on the head, and facial organs and facial muscles can also be driven to show different facial expressions. For example, a virtual human body model includes a trunk, limbs and a head, and the head and limbs are connected to the trunk, and the head has facial features and hair. For another example, a virtual model of a tree grandmother includes a tree trunk (including a trunk, branches, and leaves, flowers and / or fruits connected to the branches, etc.), facial organs and facial muscles on the trunk, etc.

[0033] For example, Figure 2 As shown, Figure 2 Schematic diagram of a scenario in which a virtual human model is used to broadcast live online in an embodiment of the present application. Figure 2 As shown in A, a virtual anchor (i.e., virtual human model) configuration interface is provided for live broadcast users, and users can freely and flexibly configure the virtual anchor in this interface. For example, at least one virtual human model can be pre-set in the system, and the user can click Figure 2 After clicking the "virtual anchor" control on the interface shown in A, the virtual anchor selection interface is entered, on which preset virtual human models are displayed. The user can click on the virtual human model he likes to select the "virtual anchor" for this live broadcast, such as Figure 2 B shows a schematic diagram of the interface of a virtual anchor selected by a user. The system can also pre-set at least one headgear model for decorating the virtual anchor, such as a hat headgear model, a wig headgear model, a hairpin decoration model, etc. The user clicks Figure 2 After clicking the "headgear effect" control on the interface shown in A, the headgear configuration interface is entered, and the preset headgear models are displayed on the interface. Users can click their favorite headgear model to dress up the selected virtual anchor. The system can also preset the "face pinching" function in advance, and users can click Figure 2 After clicking the "Automatic Face Pinching" control on the interface shown in A, the face pinching interface is entered, on which the user can configure the face of the selected virtual anchor, similar to "pinching" the face of the virtual anchor like pinching a clay doll, so as to change the facial shape / contour of the virtual anchor.

[0034] For example, Figure 3 As shown, Figure 3 Schematic diagram of a scenario for customizing a virtual anchor in an online live broadcast scenario provided by an embodiment of the present application. Figure 2 Click each custom function control on the interface shown to enter Figure 3 The virtual anchor customization interface shown in the figure. Figure 3 As shown in A, the user can also click the image control in the interface to switch to the image customization window, in which the user can customize the gender, facial details, hair, clothing, etc. of the virtual anchor; Figure 3 As shown in B, the user can also click the background control in the interface to switch to the background customization window, in which the user can customize the background image behind the virtual anchor; Figure 3 As shown in C, the user can also click the action control in the interface to switch to the appearance action customization window, in which the user can customize the body movements and / or facial expressions of the virtual anchor when he / she appears on the stage; Figure 3As shown in FIG. 3 , the user can also click the perspective control in the interface to switch to the perspective customization window, in which the user can customize the perspective of the virtual anchor, so that the live broadcast screen watched by the audience watching the live broadcast contains the whole body, upper body, or head, shoulders and neck of the virtual anchor.

[0035] The driving information is information used to determine the actions of the trunk, limbs, head and / or face of the virtual biological model, and can be at least one of text information and voice information.

[0036] In some optional implementations, the system is pre-configured with a text input box dedicated to detecting driving information in text form. The user can enter driving information in text form in the text input box. The system can monitor the text input box in real time, and when it is detected that there is text information in the text input box, the monitored text information is used as the driving information.

[0037] For example, Figure 4 As shown, Figure 4 is a schematic diagram of a scenario in which text-based driving information is obtained in an online live broadcast scenario provided by an embodiment of the present application. Figure 4 A. When the user clicks the "STA" control in the interface, a text input box for inputting text-based driver information pops up on the interface (for example, at the bottom of the interface). After the user clicks the input box, a virtual keyboard pops up, which jumps to Figure 4 B. The user can input the driving information in text form in the text input box through the virtual keyboard.

[0038] In some optional embodiments, the electronic device that executes the driving method of the virtual biological model provided in the embodiments of the present application has a microphone integrated inside. The microphone can collect sounds from the environment to generate voice audio. The system uses the voice audio as driving information, that is, obtains driving information in the form of voice.

[0039] Of course, the driving information may also be obtained by the electronic device that executes the driving method of the virtual biological model provided in the embodiment of the present application from other electronic devices. For example, the electronic device that executes the driving method of the virtual biological model provided in the embodiment of the present application sends a driving information acquisition request to other electronic devices, and the other electronic devices send the pre-stored driving information or the real-time generated driving information to the electronic device that executes the driving method of the virtual biological model provided in the embodiment of the present application.

[0040] S104, converting the driving information into a phoneme sequence.

[0041] Phoneme is the smallest speech unit divided according to the natural properties of speech. It is the smallest unit or smallest speech segment that constitutes a syllable.

[0042] If the driving information obtained is in text form, it can be input into the speech engine for conversion processing. The speech engine is a program / model for converting text into speech (Text To Speech, TTS). After the text is converted by the speech engine, it will be converted into a phoneme sequence, that is, a queue composed of at least one phoneme. These phonemes will be presented as sound when they are output in sequence by the audio output hardware of the electronic device (including speakers, speaker controllers, etc.). The arrangement order of the phonemes in the phoneme sequence remains consistent with the text arrangement order in the original text. For example, if the text is in Chinese form "We will go hiking on Sunday", a phoneme sequence in Mandarin form is converted, and the obtained phoneme sequence will obtain the natural pronunciation "We will go hiking on Sunday" in Mandarin form when it is output in sequence. Optionally, the speech engine can be implemented based on artificial intelligence (AI) technology. For details, please refer to the relevant technology of natural language processing, which will not be repeated in this application.

[0043] Please continue to see Figure 4 B. After the user inputs the driving information in text form, he clicks the "Synthesis" control next to the text input box. The program will automatically call the speech engine to perform TTS processing on the text entered in the text input box to obtain the phoneme sequence of the text entered in the text input box.

[0044] If the driving information obtained is in the form of voice, the obtained voice information (i.e., driving information) can be directly segmented, that is, the voice information is segmented into phonemes one by one, but the arrangement order of these phonemes is not changed, that is, the phonemes in the obtained phoneme sequence are still arranged according to their original order in the voice information.

[0045] S106, determining the action information corresponding to each phoneme in the phoneme sequence to obtain an action information sequence.

[0046] Action information is direct information when driving a virtual biological model to perform an action, and may be the posture coordinates of each vertex / bone point in the virtual biological model in a preset virtual space. For example, the action information corresponding to the preset phoneme A is information for driving the virtual anchor to stretch his arms straight on both sides of the body so that the arms are in the shape of a "one" as a whole. Then the action information converted by phoneme A is the posture coordinates of each vertex / bone point in the virtual anchor in the preset virtual space when the virtual anchor stretches his arms straight on both sides of the body in the shape of a "one" in the preset virtual space. When each vertex / bone point in the virtual anchor is driven to move to the corresponding posture coordinates in the preset virtual space, the virtual anchor presents a posture of stretching his arms straight on both sides of the body in the shape of a "one".

[0047] Obviously, when a series of phonemes (i.e., phoneme sequences) are converted into corresponding action information, an action information sequence is obtained. By driving the virtual biological model to present a series of postures according to the action information sequence, the virtual biological model can be driven to perform actions.

[0048] In some optional implementations, the motion information may include: limb motion information, trunk motion information, head motion information and / or facial motion information. As the name implies, limb motion information is information for driving the limbs of the virtual biological model to move; trunk motion information is information for driving the trunk of the virtual biological model to move; head motion information is information for driving the head of the virtual biological model to move; and facial motion information is information for driving the face of the virtual biological model to move and present different facial expressions.

[0049] It should be noted that the action information corresponding to each phoneme may contain at least one of the four types of action information, namely, limb action information, trunk action information, head action information, and facial action information. That is, a phoneme can correspond to the action information of multiple body organs that drive the virtual biological model at the same time, and is not limited to the action information of a certain organ or organ in a certain area such as limbs / face. In an embodiment of the present application, at least one action information in the action information sequence obtained by converting the phoneme sequence contains at least facial action information of the virtual biological model. That is, at least one action information in the converted action information sequence will drive the face of the virtual biological model to change its facial expression.

[0050] In some optional implementations, the seed neural network model is trained in advance using sample phonemes based on AI technology to obtain a first target neural network model that can predict the action information corresponding to each phoneme. Therefore, each phoneme in the phoneme sequence can be input into the first target neural network for prediction to obtain the action information corresponding to each phoneme in the phoneme sequence, thereby obtaining an action information sequence.

[0051] In some optional implementations, the action information corresponding to each phoneme is preset, that is, the mapping relationship between various factors and various action information is preset. Therefore, after obtaining the phoneme sequence, the action information corresponding to each phoneme in the phoneme sequence can be determined according to the preset mapping relationship between various phonemes and various action information to obtain the action information sequence. That is, the action information corresponding to each phoneme in the phoneme sequence is searched in the preset mapping relationship to obtain the action information sequence.

[0052] Optionally, the step of "determining the action information corresponding to each phoneme in the phoneme sequence according to the preset mapping relationship between various phonemes and various action information to obtain the action information sequence" can be implemented by the following two steps:

[0053] The first step is to determine the action coefficient corresponding to each phoneme in the phoneme sequence according to a preset first mapping relationship between various phonemes and various action coefficients, and obtain an action coefficient sequence;

[0054] In the second step, according to a preset second mapping relationship between various action coefficients and various action information, the action information corresponding to each action coefficient in the action coefficient sequence is determined to obtain an action information sequence.

[0055] A specific action can be considered as a combination of multiple basic actions. For example, a specific facial expression (i.e., facial action) can be a combination of multiple expression base units (i.e., basic facial actions, which are pre-defined facial deformations, such as eyes wide open, mouth open, eyebrows raised, etc.). Therefore, some basic actions can be pre-set, and each specific action is composed of these pre-set basic actions.

[0056] Optionally, at least one basic action is preset, and each specific action is synthesized based on these basic actions. It should be noted that the weight of each basic action in the basic action selected when each specific action is synthesized is different, so each specific action will set a corresponding weight value for each basic action when synthesized based on the basic action. For example, a specific action is composed of a basic action a, a basic action b, and a basic action c, wherein the weight of the basic action one is 0.1, the weight coefficient of the basic action two is 0.7, and the weight coefficient of the basic action three is 0.2, then the action coefficient of the specific action one is {0.1a, 0.7b, 0.2c}. In other words, the action coefficient represents the basic action required to synthesize the action and the weights of each basic action required. Various basic action selection methods and weight configuration methods can be exhaustively enumerated to obtain all the specific actions that can be synthesized and the action coefficients corresponding to each specific action.

[0057] Based on the above, it can be known that the action information of each basic action can be determined by the spatial coordinates of each vertex / bone point of the virtual biological model in the preset virtual space. Therefore, the above exhaustive list of various basic action selection methods and weight configuration methods, so that the action information of all specific actions that can be synthesized is also determined. Correspondingly, the second mapping relationship between the action information corresponding to each of all specific actions that can be synthesized and the action coefficients corresponding to each of all specific actions that can be synthesized is also set.

[0058] Furthermore, a first mapping relationship between various factors and various action coefficients may be preset. For example, a first mapping relationship may be established between phoneme A and action coefficients {0.1a, 0.7b, 0.2c}.

[0059] In this embodiment, a first mapping relationship between various phonemes and various action coefficients and a second mapping relationship between various action coefficients and various actions are preset. Therefore, the action information corresponding to each phoneme in the phoneme sequence can be searched in the first mapping relationship to obtain an action coefficient sequence, and then the action information corresponding to each action coefficient in the action coefficient sequence can be searched in the second mapping relationship to obtain an action information sequence.

[0060] For example, various facial expressions can be synthesized based on the avatar52 expression base (including eyes, jaw, mouth, eyebrows, cheeks, nose, etc.), and the facial action information of each expression base is known. Multiple expression bases can be weighted in advance based on the pronunciation of Chinese and English phonemes to obtain the expression base coefficients (BS values) and facial action information of various expressions that can be synthesized by the expression base, that is, to establish a second mapping relationship between the BS values ​​of each specific expression that can be synthesized and the corresponding facial action information. And establish a first mapping relationship between various phonemes and various BS values ​​to obtain a BS value table of the corresponding relationship between phonemes and BS values. After obtaining a certain phoneme, the BS value corresponding to each phoneme can be searched in the pre-established BS value table, and the facial action information corresponding to the found BS value can be determined based on the second mapping relationship, thereby obtaining the facial action information corresponding to the phoneme.

[0061] S108, driving the virtual biological model to perform actions according to the action information sequence.

[0062] Control the virtual biological model to perform actions in sequence according to the action information in the action information sequence. For example, control each vertex / bone point of the virtual biological model to move to the coordinate position of each vertex / bone point in each action information in the action information sequence, so as to drive the virtual biological model to perform actions.

[0063] In the embodiment of the present application, after obtaining the driving information in the form of text or voice, it is converted into a phoneme sequence, the actions corresponding to each phoneme in the phoneme sequence are determined, and the action information sequence is obtained, and the virtual biological model is driven to act according to the action information sequence. And there is at least one action information in the action information sequence that contains at least the facial action information of the virtual biological model. Therefore, when driving the virtual biological model to act, at least in one action, the face of the virtual biological model may be driven to act, that is, in the process of driving the biological model, the facial muscles of the virtual biological model may also be driven to act in certain actions to change the facial expression of the virtual biological model. Compared with the manual driving method in the related art, the virtual biological model driving method provided in the embodiment of the present application can drive the virtual biological model based on the driving information in the form of text or voice, and the driving method is more flexible, which can reduce the manual workload to a certain extent.

[0064] After the above step S106, the driving method of the virtual biological model further includes: performing smooth transition processing on the action information in the action information sequence to obtain an adjusted action information sequence, and then executing step S108: driving the virtual biological model to perform actions according to the adjusted action information sequence.

[0065] The purpose of smoothing the action information in the action information sequence is to avoid the sudden jump between the previous and next actions, which would appear unnatural. For example, the mouth is tightly closed in the previous action information, and is wide open in the next action information. If smoothing is not performed, the mouth of the virtual biological model will suddenly jump from being tightly closed to being wide open, which would appear unnatural. In fact, the mouth should be slowly opened from being tightly closed to be more in line with the laws of nature.

[0066] The purpose of performing smooth transition processing on the action information in the action information sequence is to make the transition between the action information in the action information sequence smoother and reduce the possibility of jumps between the two action information. After performing smooth transition processing on the action information in the action information sequence, an adjusted action information sequence is obtained. Based on the adjusted action information sequence, the virtual biological model is driven to perform actions, so that the process of the virtual biological model performing actions appears more natural and real.

[0067] In some optional implementations, a seed neural network model may be trained based on the sample action information pairs to obtain a target neural network model for smoothing the action information pairs. Therefore, after obtaining the action information sequence, every two adjacent action information in the action information sequence may be input into the target neural network model for smoothing, and the action information in the original action information sequence may be replaced with the smoothed action information to obtain an adjusted action information sequence.

[0068] In some optional implementations, smoothing processing may be performed on the basis of the action information sequence determined based on the first mapping relationship and the second mapping relationship, specifically including the following steps A and B:

[0069] Step A: after determining the action coefficient corresponding to each phoneme in the phoneme sequence according to the first mapping relationship between various preset phonemes and various action coefficients and obtaining the action coefficient sequence, smooth transition processing is performed on the action coefficients corresponding to all adjacent front and back phonemes in the phoneme sequence to obtain an adjusted action coefficient sequence; the front phoneme in the front and back phonemes includes at least one first phoneme, and the rear phoneme in the front and back phonemes includes at least one second phoneme, and the first phoneme is different from the second phoneme.

[0070] For example, there is a phoneme sequence AAABBCCC...ZZ. A, B, C...Z each represent a phoneme. There is a high probability of a jump phenomenon when transitioning between the first segment of phonemes AAA and the second segment of phonemes BB, so it is necessary to smooth the action coefficients corresponding to the two segments of phonemes; accordingly, it is also necessary to adjust the action coefficients corresponding to the phoneme segment BB and the phoneme segment CCC... until the action coefficients corresponding to all adjacent segments of the phonemes in the phoneme sequence AAABBCCC...ZZ are smoothed, and the adjusted action coefficient sequence is obtained.

[0071] In some optional implementations, the step of "performing smooth transition processing on the action coefficients corresponding to all adjacent front and rear segments of the phoneme sequence to obtain an adjusted action coefficient sequence" includes the following steps S:

[0072] Step S: Based on the cubic Hermite spline curve, the action coefficients corresponding to the two adjacent segments of phonemes in the phoneme sequence are fitted, and the first output duration is used as a constraint condition for fitting the action coefficients corresponding to each phoneme in the first segment of phonemes, and the second output duration is used as a constraint condition for fitting the action coefficients corresponding to each phoneme in the second segment of phonemes, to obtain an adjusted action coefficient sequence; the first output duration is the total output duration of all first phonemes in the first segment of phonemes, and the second output duration is the total output duration of all second phonemes in the second segment of phonemes.

[0073] The output duration of each phoneme is determined according to the speaking speed. For example, the default output duration of each phoneme is 5ms (milliseconds), so the total output duration of the first segment of phonemes AAA in the aforementioned example phoneme sequence AAABBCCC...ZZ is 15ms, and the total output duration of the second segment of phonemes BB is 10ms. When fitting the action coefficients of the first segment of phonemes AAA and the action coefficients of the second segment of phonemes BB based on the cubic Hermite spline curve, 15ms is used as the constraint condition for fitting the action coefficients of each phoneme in the first segment of phonemes AAA, and 10ms is used as the constraint condition for fitting the action coefficients of each phoneme in the second segment of phonemes BB. That is, when fitting, it is ensured that the presentation duration of the action / posture corresponding to each phoneme is consistent with the output duration corresponding to each phoneme.

[0074] Step B: According to the preset second mapping relationship between various action coefficients and various action information, the action information corresponding to each action coefficient in the adjusted action coefficient sequence is determined to obtain an adjusted action information sequence.

[0075] The action information corresponding to each action coefficient in the adjusted action coefficient sequence is searched from the preset second mapping relationship to obtain the adjusted action information sequence.

[0076] For example, for the facial actions corresponding to phoneme A and phoneme B, direct transition may cause jumps. The obtained phoneme segment is AAABB. Therefore, after finding the BS value of phoneme A and the BS value of phoneme B, based on 15ms as the constraint value of phoneme segment AAA and 10ms as the constraint value of phoneme segment BB, the action coefficient of phoneme AAA and the BS value of phoneme BB are fitted based on the cubic Hermite spline curve, so that the BS values ​​corresponding to each phoneme in the phoneme segment AAABB present a smooth transition. The facial expressions corresponding to each phoneme in the phoneme segment AAABB are determined based on the smoothed BS value and the avatar expression.

[0077] In some optional implementations, the speech rate can be freely set to change the output duration of each phoneme. Therefore, the driving method of the virtual biological model further includes the following steps:

[0078] Get speech rate information;

[0079] Determine the output duration of each phoneme in the phoneme sequence according to the speech rate information;

[0080] Driving the virtual biological model to perform actions according to the action information sequence includes:

[0081] The virtual biological model is driven to perform actions according to the action information sequence, and each action is kept for the same duration as the output duration of the corresponding phoneme.

[0082] like Figure 5 As shown, Figure 5 shows a speech rate setting interface. Figure 5 A. The system provides a speech rate setting interface, on which the user can drag the speech rate setting control to change the speech rate. Specifically, the system has a default speech rate, and the user drags the preset setting control to change the actual output speech rate relative to the default speech rate to set the speech rate.

[0083] The preset default speech rate is equivalent to the preset default output duration of each phoneme, and changing the multiple of the actual output speech rate relative to the default speech rate is equivalent to changing the multiple of the actual output duration of each phoneme relative to the default output duration. For example, the default output duration of a phoneme is 5ms, and it is adjusted to 2 times the output speed, then the output duration of each phoneme becomes 2.5ms.

[0084] The virtual biological model is driven to perform actions according to the action information sequence, and each action is kept the same duration as the output duration of the corresponding phoneme, so that the action of the virtual biological model and the driving information appear to be more matched. For example, the phoneme sequence AAABBC..., each phoneme output duration is 5ms, phoneme A corresponds to action one, phoneme B corresponds to action two, and phoneme C corresponds to action three, then the virtual biological model is controlled to first present action one for 15ms, then action two for 10ms, and then action three for 5ms...

[0085] See also Figure 5 B. The system also provides the function of setting the tone and timbre. Users can click the "Tone" control or "Timbre" control to enter the tone and timbre setting window.

[0086] In some optional implementations, the virtual biological model can be set to follow the limbs of the real biological body to perform physical movements, while the face is driven according to the driving information, that is, the face and limbs are driven separately. Therefore, the driving method of the virtual biological model also includes the following steps:

[0087] acquiring an image containing a control subject;

[0088] Identifying body movement information of the control subject in the image;

[0089] driving the limbs of the virtual biological model to move according to the limb movement information;

[0090] The step of driving the virtual biological model to perform actions according to the action information sequence includes:

[0091] The face of the virtual biological model is driven to perform actions according to the action information sequence.

[0092] The controlling subject is a physical creature in the objective world, such as humans, animals, etc.

[0093] Collect videos or images containing the control subject, identify the body movement information of the control subject in the video or image, control the limbs of the virtual biological model to move according to the identified body movement information of the control subject, and synchronously control the face of the virtual biological model to move according to the action information sequence determined by the driving information. In this way, the face and limbs of the virtual biological model can be driven separately, making the driving method more flexible and changeable.

[0094] The second aspect of the embodiment of the present application further provides a method for generating a video, which can be applied to various electronic devices, such as computers, smart phones, tablet computers, wearable smart devices, in-vehicle information systems on chips (IVI SoC) on smart cars, and other devices, and can also be servers such as servers, cloud servers, intelligent cloud computing servers with artificial intelligence technology, or intelligent cloud hosts in distributed systems. The method for generating a video includes the following steps:

[0095] The driving method of the virtual biological model according to any of the aforementioned embodiments drives the virtual biological model to perform actions, and in the process of driving the virtual biological model to perform actions, the phonemes in the target phoneme sequence are synchronously output to generate a video; the target phoneme sequence is a phoneme sequence obtained by converting the driving information in the process of driving the virtual biological model according to any of the aforementioned embodiments.

[0096] The virtual biological model is driven to perform actions to generate video images, and the phonemes in the target phoneme sequence are synchronously output to dub the video images, thereby generating a video with sound.

[0097] The driving method of the virtual biological model provided in the embodiment of the present application can be executed by a driving device of the virtual biological model. In the embodiment of the present application, the driving method of the virtual biological model is taken as an example to illustrate the driving device of the virtual biological model provided in the embodiment of the present application.

[0098] like Figure 6 FIG. 1 is a schematic diagram showing a structure of a driving device for a virtual biological model provided in an embodiment of the present application. Figure 6 The driving device 20 of the virtual biological model comprises:

[0099] A first acquisition module 201 is used to acquire driving information for driving the virtual biological model; the driving information includes at least one of text information and voice information;

[0100] A first conversion module 202, used to convert the driving information into a phoneme sequence;

[0101] The first determination module 203 is used to determine the action information corresponding to each phoneme in the phoneme sequence to obtain an action information sequence; wherein at least one action information in the action information sequence contains at least facial action information of the virtual biological model;

[0102] The first driving module 204 is used to drive the virtual biological model to perform actions according to the action information sequence.

[0103] In some optional embodiments, the driving device 20 further includes:

[0104] A first adjustment module, used for performing smooth transition processing on the action information in the action information sequence to obtain an adjusted action information sequence;

[0105] The first driving module 204 includes:

[0106] The first driving submodule is used to drive the virtual biological model to perform actions according to the adjusted action information sequence.

[0107] In some optional implementations, the first determining module 203 includes:

[0108] The first determination submodule is used to determine the action information corresponding to each phoneme in the phoneme sequence according to the preset mapping relationship between various phonemes and various action information, so as to obtain an action information sequence.

[0109] In some optional implementations, the first determining submodule includes:

[0110] A first determining unit, configured to determine the action coefficient corresponding to each phoneme in the phoneme sequence according to a preset first mapping relationship between various phonemes and various action coefficients, to obtain an action coefficient sequence;

[0111] The second determining unit is used to determine the action information corresponding to each action coefficient in the action coefficient sequence according to a second mapping relationship between various preset action coefficients and various action information, so as to obtain an action information sequence.

[0112] In some optional implementations, the first adjustment module includes:

[0113] A first adjustment submodule is used for determining the action coefficient corresponding to each phoneme in the phoneme sequence according to the first mapping relationship between various preset phonemes and various action coefficients, and after obtaining the action coefficient sequence, performing smooth transition processing on the action coefficients corresponding to all adjacent front and back phonemes in the phoneme sequence to obtain an adjusted action coefficient sequence; the front phoneme of the front and back phonemes includes at least one first phoneme, the rear phoneme of the front and back phonemes includes at least one second phoneme, and the first phoneme is different from the second phoneme;

[0114] The second determination submodule is used to determine the action information corresponding to each action coefficient in the adjusted action coefficient sequence according to a second mapping relationship between various preset action coefficients and various action information, so as to obtain an adjusted action information sequence.

[0115] In some optional implementations, the output duration of each phoneme in the phoneme sequence is preset;

[0116] The first adjustment submodule includes:

[0117] The first adjustment unit is used to fit the action coefficients corresponding to the two adjacent segments of phonemes in the phoneme sequence based on a cubic Hermite spline curve, and use the first output duration as a constraint condition for fitting the action coefficients corresponding to each phoneme in the first segment of phonemes, and use the second output duration as a constraint condition for fitting the action coefficients corresponding to each phoneme in the second segment of phonemes, to obtain an adjusted action coefficient sequence; the first output duration is the total output duration of all first phonemes in the first segment of phonemes, and the second output duration is the total output duration of all second phonemes in the second segment of phonemes.

[0118] In some optional embodiments, the device 20 further includes:

[0119] The second acquisition module is used to acquire speech rate information;

[0120] A second determination module, configured to determine the output duration of each phoneme in the phoneme sequence according to the speech rate information;

[0121] The first determining module 204 includes:

[0122] The second driving submodule is used to drive the virtual biological model to perform actions according to the action information sequence, and to make each action maintain the same duration as the output duration of the corresponding phoneme.

[0123] In some optional embodiments, the apparatus 20 method further includes:

[0124] A third acquisition module is used to acquire an image containing a control subject;

[0125] A first recognition module, used to recognize body movement information of the control subject in the image;

[0126] A second driving module, used for driving the limbs of the virtual biological model to move according to the limb movement information;

[0127] The first determining module 204 includes:

[0128] The third driving submodule is used to drive the face of the virtual biological model to move according to the action information sequence.

[0129] like Figure 7, which shows a schematic diagram of the structure of a video generation device provided by an embodiment of the present application. Figure 7 , the video generation device 30 comprises:

[0130] Driving device 20;

[0131] The first generating module 301 is used to synchronously output the phonemes in the target phoneme sequence to generate a video when the driving device drives the virtual biological model to perform an action; the target phoneme sequence is a phoneme sequence obtained by converting the driving information when the driving device drives the virtual biological model to perform an action.

[0132] The driving device 20 of the virtual biological model and the video generating device 30 in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmentedreality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personaldigital assistant, PDA), etc., and can also be a server, a network attached storage (Network AttachedStorage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0133] The driving device 20 of the virtual biological model and the video generating device 30 in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0134] The driving device 20 and the video generating device 30 of the virtual biological model provided in the embodiment of the present application can realize Figures 1 to 5 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0135] In some optional embodiments, such as Figure 8As shown, an embodiment of the present application further provides an electronic device 130, including a first processor 131, a CPU 132 and a memory 133, wherein the memory 133 stores programs or instructions that can be executed on at least one of the first processor 131 and the CPU 132, and the programs or instructions stored in the memory 133 contain various steps of the embodiments of the driving method of the virtual biological model and the video generation method when the programs or instructions are executed by the first processor, and can achieve the same technical effect. To avoid repetition, they are not repeated here.

[0136] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0137] Fig. 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0138] The electronic device 170 includes, but is not limited to, components such as a radio frequency unit 171, a network module 172, an audio output unit 143, an input unit 144, a sensor 145, a display unit 146, a user input unit 147, an interface unit 148, a memory 149, and a processor 1410. Those skilled in the art will appreciate that the electronic device 140 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1410 through a power management system, so as to implement functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0139] The processor 1410 is configured to:

[0140] Acquiring driving information for driving the virtual biological model; the driving information includes at least one of text information and voice information;

[0141] converting the driving information into a phoneme sequence;

[0142] Determine the action information corresponding to each phoneme in the phoneme sequence to obtain an action information sequence; wherein at least one action information in the action information sequence contains at least facial action information of the virtual biological model;

[0143] driving the virtual biological model to perform actions according to the action information sequence;

[0144] Or, in the process of driving the virtual biological model to perform actions, the phonemes in the target phoneme sequence are synchronously output to generate a video; the target phoneme sequence is a phoneme sequence obtained by converting the driving information in the process of driving the virtual biological model to perform actions.

[0145] It should be understood that in the embodiment of the present application, the input unit 144 may include a graphics processor (Graphics Processing Unit, GPU) 1441 and a microphone 1442, and the graphics processor 1441 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 146 may include a display panel 1461, and the display panel 1461 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 147 includes a touch panel 1471 and at least one of other input devices 1472. The touch panel 1471 is also called a touch screen. The touch panel 1471 may include two parts: a touch detection device and a touch controller. Other input devices 1472 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0146] The memory 149 can be used to store software programs and various data. The memory 149 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 149 may include a volatile memory or a non-volatile memory, or the memory 149 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory Read-Only Memory, ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 149 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0147] The processor 1410 may include one or more processing units; optionally, the processor 1410 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1410.

[0148] Any of the above-mentioned product embodiments can implement the various processes of the above-mentioned method embodiment for measuring vibration test parameters through its own processor operation, and can achieve the same technical effect. In order to avoid repetition, they will not be described one by one.

[0149] The embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by the processor, each process of the above-mentioned driving method of the virtual biological model and the video generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the processor is the processor in the electronic device or electronic system described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a disk or an optical disk, etc.

[0150] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned virtual biological model driving method and video generation method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0151] In the implementation methods provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0152] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0153] In addition, each functional unit in each implementation of the embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0154] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each implementation method of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0155] The above description is only an implementation method of the embodiment of the present application, and does not limit the patent scope of the embodiment of the present application. The above specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can make many forms of equivalent structures or equivalent process changes made by using the description and drawings of the embodiment of the present application, or directly or indirectly apply them in other related technical fields without departing from the scope of protection of the purpose of the present application and the claims, which are also included in the patent protection scope of the embodiment of the present application.

Claims

1. A method for driving a virtual biological model, characterized in that: The method comprises: Acquiring driving information for driving the virtual biological model; the driving information includes at least one of text information and voice information; converting the driving information into a phoneme sequence; Determine the action information corresponding to each phoneme in the phoneme sequence to obtain an action information sequence; wherein at least one action information in the action information sequence contains at least facial action information of the virtual biological model; The virtual biological model is driven to perform actions according to the action information sequence.

2. The method according to claim 1, characterized in that After determining the action information corresponding to each phoneme in the phoneme sequence to obtain the action information sequence, the method further includes: Performing smooth transition processing on the action information in the action information sequence to obtain an adjusted action information sequence; The step of driving the virtual biological model to perform actions according to the action information sequence includes: The virtual biological model is driven to perform actions according to the adjusted action information sequence.

3. The method according to claim 2, characterized in that The step of determining the action information corresponding to each phoneme in the phoneme sequence to obtain the action information sequence includes: According to the preset mapping relationship between various phonemes and various action information, the action information corresponding to each phoneme in the phoneme sequence is determined to obtain an action information sequence.

4. The method according to claim 3, characterized in that The step of determining the action information corresponding to each phoneme in the phoneme sequence according to the preset mapping relationship between various phonemes and various action information to obtain the action information sequence includes: According to a preset first mapping relationship between various phonemes and various action coefficients, determining the action coefficient corresponding to each phoneme in the phoneme sequence, and obtaining an action coefficient sequence; According to the preset second mapping relationship between various action coefficients and various action information, the action information corresponding to each action coefficient in the action coefficient sequence is determined to obtain the action information sequence.

5. The method according to claim 4, characterized in that The step of performing smooth transition processing on the action information in the action information sequence to obtain an adjusted action information sequence includes: After determining the action coefficient corresponding to each phoneme in the phoneme sequence according to the first mapping relationship between various preset phonemes and various action coefficients to obtain the action coefficient sequence, smooth transition processing is performed on the action coefficients corresponding to all adjacent front and back phonemes in the phoneme sequence to obtain an adjusted action coefficient sequence; the front phoneme of the front and back phonemes includes at least one first phoneme, and the back phoneme of the front and back phonemes includes at least one second phoneme, and the first phoneme is different from the second phoneme; According to the preset second mapping relationship between various action coefficients and various action information, the action information corresponding to each action coefficient in the adjusted action coefficient sequence is determined to obtain the adjusted action information sequence.

6. The method according to claim 5, characterized in that The output duration of each phoneme in the phoneme sequence is preset; The step of performing smooth transition processing on the action coefficients corresponding to the two adjacent segments of phonemes in the phoneme sequence to obtain an adjusted action coefficient sequence includes: Based on the cubic Hermite spline curve, the action coefficients corresponding to the two adjacent segments of phonemes in the phoneme sequence are fitted, and the first output duration is used as a constraint condition for fitting the action coefficients corresponding to each phoneme in the first segment of phonemes, and the second output duration is used as a constraint condition for fitting the action coefficients corresponding to each phoneme in the second segment of phonemes, to obtain an adjusted action coefficient sequence; the first output duration is the total output duration of all first phonemes in the first segment of phonemes, and the second output duration is the total output duration of all second phonemes in the second segment of phonemes.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Get speech rate information; Determine the output duration of each phoneme in the phoneme sequence according to the speech rate information; Driving the virtual biological model to perform actions according to the action information sequence includes: The virtual biological model is driven to perform actions according to the action information sequence, and each action is kept for the same duration as the output duration of the corresponding phoneme.

8. The method according to any one of claims 1 to 6, characterized in that The method further comprises: acquiring an image containing a control subject; Identifying body movement information of the control subject in the image; driving the limbs of the virtual biological model to move according to the limb movement information; The step of driving the virtual biological model to perform actions according to the action information sequence includes: The face of the virtual biological model is driven to perform actions according to the action information sequence.

9. A method for generating a video, characterized in that: The method comprises: In the process of driving the virtual biological model to perform actions according to the driving method of the virtual biological model according to any one of claims 1 to 8, the phonemes in the target phoneme sequence are synchronously output to generate a video; the target phoneme sequence is a phoneme sequence obtained by converting the driving information during the process of driving the virtual biological model to perform actions according to the driving method of the virtual biological model according to any one of claims 1 to 8.

10. A driving device for a virtual biological model, characterized in that: The device comprises: A first acquisition module, used to acquire driving information for driving the virtual biological model; the driving information includes at least one of text information and voice information; A first conversion module, used for converting the driving information into a phoneme sequence; A first determination module is used to determine the action information corresponding to each phoneme in the phoneme sequence to obtain an action information sequence; wherein at least one action information in the action information sequence contains at least facial action information of the virtual biological model; The first driving module is used to drive the virtual biological model to perform actions according to the action information sequence.

11. A video generation device, characterized in that: The device comprises: The driving device according to claim 10; The first generating module is used to synchronously output the phonemes in the target phoneme sequence to generate a video when the driving device drives the virtual biological model to perform an action; the target phoneme sequence is a phoneme sequence obtained by converting the driving information when the driving device drives the virtual biological model to perform an action.

12. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the driving method of the virtual biological model as described in any one of claims 1 to 8 or the steps of the video generation method as described in claim 9 are implemented.

13. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the program or instruction implements the steps of the method for driving a virtual biological model as described in any one of claims 1 to 8 or the steps of the method for generating a video as described in claim 9.

14. A computer program product, characterized in that The method comprises a program or an instruction, which, when executed by a processor, implements the steps of the method for driving a virtual biological model as claimed in any one of claims 1 to 8 or the steps of the method for generating a video as claimed in claim 9.

Images