Expression sequence generation method and device, computer device and storage medium

Abstract

The application relates to an expression sequence generation method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: determining an expression state transition graph corresponding to the emotion category of target audio; determining a node link matched with an emotion key frame sequence from the expression state transition graph based on the audio features of each key frame in the emotion key frame sequence; the node link comprises target nodes corresponding to each key frame one by one and directed edges connecting the target nodes, and the audio features of the key frame corresponding to the target node pointed by the directed edge satisfy the node connection condition of the directed edge; the expression bases corresponding to each target node are respectively acquired, and the expression bases corresponding to each target node are combined into an expression base sequence according to the node connection order represented by the directed edge; and the expression sequence of the target audio is determined based on the expression base sequence. The expression sequence generated by the method can improve the accuracy of emotion expression.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer device, storage medium, and computer program product for facial expression sequencing. Background Technology

[0002] With the development of computer technology, audio-to-facial expression generation technology has emerged. This technology allows input of an audio clip to generate a corresponding facial expression sequence based on the emotions expressed in the audio. This facial expression sequence can then be used to drive a virtual object, allowing the virtual object to display the facial expression corresponding to the audio clip.

[0003] In related technologies, an end-to-end network training can be performed using a large amount of paired facial expression and audio data to obtain an audio-to-facial expression model. However, due to the weak correlation between audio and facial expression information, the current model learning effect is poor, resulting in low accuracy of the generated facial expression sequence in expressing the emotions in the audio. Summary of the Invention

[0004] Therefore, it is necessary to provide an expression sequence method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve the accuracy of the generated expression sequence in expressing emotions in audio, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a method for generating facial expression sequences. The method includes:

[0006] Identify the facial expression state transition diagram corresponding to the emotion type of the target audio;

[0007] Based on the audio features of each key frame in the emotional key frame sequence of the target audio, a node link matching the emotional key frame sequence is determined from the expression state transition graph; the node link includes a target node corresponding to each of the key frames and a directed edge connecting the target node, wherein the audio features of the key frame corresponding to the target node pointed to by the directed edge satisfy the node connection condition of the directed edge.

[0008] Obtain the expression base corresponding to each target node, and form an expression base sequence according to the node connection order represented by the directed edges.

[0009] Based on the expression base sequence, the expression sequence of the target audio is determined.

[0010] Secondly, this application also provides an apparatus for generating facial expression sequences. The apparatus includes:

[0011] The state transition diagram determination module is used to determine the facial expression state transition diagram corresponding to the emotion type of the target audio.

[0012] The node link determination module is used to determine the node link matching the emotional key frame sequence from the expression state transition graph based on the audio features of each key frame in the emotional key frame sequence of the target audio. The node link includes a target node corresponding to each key frame and a directed edge connecting the target node. The audio features of the key frame corresponding to the target node pointed to by the directed edge satisfy the node connection condition of the directed edge.

[0013] The expression base acquisition module is used to acquire the expression base corresponding to each target node respectively, and to form an expression base sequence according to the node connection order represented by the directed edges.

[0014] An expression sequence determination module is used to determine the expression sequence of the target audio based on the expression base sequence.

[0015] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0016] Identify the facial expression state transition diagram corresponding to the emotion type of the target audio;

[0017] Based on the audio features of each key frame in the emotional key frame sequence of the target audio, a node link matching the emotional key frame sequence is determined from the expression state transition graph; the node link includes a target node corresponding to each of the key frames and a directed edge connecting the target node, wherein the audio features of the key frame corresponding to the target node pointed to by the directed edge satisfy the node connection condition of the directed edge.

[0018] Obtain the expression base corresponding to each target node, and form an expression base sequence according to the node connection order represented by the directed edges.

[0019] Based on the expression base sequence, the expression sequence of the target audio is determined.

[0020] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0021] Identify the facial expression state transition diagram corresponding to the emotion type of the target audio;

[0022] Based on the audio features of each key frame in the emotional key frame sequence of the target audio, a node link matching the emotional key frame sequence is determined from the expression state transition graph; the node link includes a target node corresponding to each of the key frames and a directed edge connecting the target node, wherein the audio features of the key frame corresponding to the target node pointed to by the directed edge satisfy the node connection condition of the directed edge.

[0023] Obtain the expression base corresponding to each target node, and form an expression base sequence according to the node connection order represented by the directed edges.

[0024] Based on the expression base sequence, the expression sequence of the target audio is determined.

[0025] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0026] Identify the facial expression state transition diagram corresponding to the emotion type of the target audio;

[0027] Based on the audio features of each key frame in the emotional key frame sequence of the target audio, a node link matching the emotional key frame sequence is determined from the expression state transition graph; the node link includes a target node corresponding to each of the key frames and a directed edge connecting the target node, wherein the audio features of the key frame corresponding to the target node pointed to by the directed edge satisfy the node connection condition of the directed edge.

[0028] Obtain the expression base corresponding to each target node, and form an expression base sequence according to the node connection order represented by the directed edges.

[0029] Based on the expression base sequence, the expression sequence of the target audio is determined.

[0030] The aforementioned expression sequence generation method, apparatus, computer device, storage medium, and computer program product, by determining an expression state transition diagram corresponding to the emotion type of the target audio, and based on the audio features of each keyframe in the emotion keyframe sequence of the target audio, determine the node links matching the emotion keyframe sequence from the expression state transition diagram. Each node link includes target nodes corresponding to each keyframe and directed edges connecting the target nodes. The audio features of the keyframes corresponding to the target nodes pointed to by the directed edges satisfy the node connection conditions of the directed edges. Furthermore, the expression bases corresponding to each target node are obtained separately, and the expression base sequences corresponding to each target node are formed according to the node connection order represented by the directed edges. This achieves automatic generation of expression sequences for the target audio. Since the node links can be determined from the expression state transition diagram in combination with specific audio features, and the expression base sequence is determined according to the connection order of each node in the node link after obtaining the expression bases corresponding to each target node in the node link, a rhythmic expression sequence can be generated, making the obtained expression sequence more flexible and vivid, and improving the accuracy of the expression sequence in expressing the emotions in the audio. Attached Figure Description

[0031] Figure 1 This is a diagram illustrating the application environment of an expression sequence generation method in one embodiment;

[0032] Figure 2 This is a flowchart illustrating a facial expression sequence generation method in one embodiment;

[0033] Figure 3 Here is an example of an expression state transition diagram in one embodiment;

[0034] Figure 4 Here is an example of an emotion expression library in one embodiment;

[0035] Figure 5 This is a flowchart illustrating the steps for extracting an emotion keyframe sequence in one embodiment.

[0036] Figure 6 This is a schematic diagram of the volume analysis of the target audio in one embodiment;

[0037] Figure 7 This is a schematic diagram of the interpolation curve in one embodiment;

[0038] Figure 8 This is a schematic diagram illustrating the changes in eyelid state description values ​​in a blinking template in one embodiment.

[0039] Figure 9 This is a schematic diagram illustrating the range of eye movement in one embodiment;

[0040] Figure 10Here is a system framework diagram corresponding to the facial expression sequence generation method in one embodiment;

[0041] Figure 11 This is a structural block diagram of an expression sequence generation device in one embodiment;

[0042] Figure 12 This is an internal structural diagram of a computer device in one embodiment;

[0043] Figure 13 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0046] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0047] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0048] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, smart customer service, vehicle networking, and intelligent transportation. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0049] The solutions provided in this application relate to machine learning technology in artificial intelligence, and are specifically illustrated through the following embodiments:

[0050] The facial expression sequence method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. The data storage system can store the data that server 104 needs to process, such as audio features of the target audio, expression bases corresponding to various emotion types, etc. The data storage system can be integrated onto server 104, or it can be located in the cloud or on other servers. Server 104 can receive target audio uploaded by terminal 102, determine the expression state transition diagram corresponding to the emotion type of the target audio, and determine the node link matching the emotion keyframe sequence from the expression state transition diagram based on the audio features of each keyframe in the emotion keyframe sequence of the target audio. The node link includes target nodes corresponding to each keyframe and directed edges connecting the target nodes. The audio features of the keyframes corresponding to the target nodes pointed to by the directed edges satisfy the node connection conditions of the directed edges. Server 104 further obtains the expression base corresponding to each target node, and forms an expression base sequence according to the node connection order represented by the directed edges. Based on the expression base sequence, the expression sequence of the target audio is determined. Server 104 can further send the generated expression sequence to terminal 102. Terminal 102 can drive the virtual character through the expression sequence to obtain an expression animation that can express the emotion in the target audio.

[0051] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0052] In one embodiment, such as Figure 2 As shown, an expression sequence method is provided, which can be applied to... Figure 1The terminal 102 or server 104 mentioned above can also be applied to a system composed of terminal 102 and server 104. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0053] Step 202: Determine the facial expression state transition diagram corresponding to the emotion type of the target audio.

[0054] The target audio refers to the audio from which the facial expression sequence needs to be generated. The target audio can express a certain emotion, and the emotion category to which the target audio primarily expresses is its emotion type. The emotion type can be any one of neutral, happy, sad, angry, or questioning. "Neutral" can be seen as the initial emotion for other emotions; any audio with the emotion categories of "happy," "sad," "angry," or "questioning" can contain the emotion of "neutral." This can be understood as follows: when an audio expresses only the emotion of "neutral," its emotion type is "neutral"; when an audio expresses any one of the emotions of "happy," "sad," "angry," or "questioning," that particular emotion type is the primary emotion expressed by the audio, and therefore, that particular emotion type is the emotion type of the target audio.

[0055] Each emotion can manifest as multiple facial expressions, and different expressions can be represented by different expression bases. The expression state transition graph corresponding to an emotion category describes the possible transition order between the various expression states within that emotion category. The expression state transition graph can include multiple nodes connected by directed edges, where the node pointed to by the directed edge represents the direction of expression state transition. Each node can correspond to one or more expression bases expressing the same expression state within that emotion category. The directed edges between nodes represent the node connection conditions, which are the conditions that must be satisfied for the expression state to transition from the source node of the directed edge to the target node of the directed edge. Expression state transition graphs can be generated based on expert experience, which is derived from analyzing and summarizing a large amount of historical audio data.

[0056] In a specific embodiment, each emotion category can be further divided into multiple emotion subcategories based on the possible facial expressions. Each emotion subcategory presents a specific facial expression. For example, for the emotion of anger, four different facial expressions can be presented: "slight frown," "frowning and raising eyebrows," "frowning and lowering eyebrows," and "frowning and lowering eyebrows (more pronounced)." Based on these four facial expressions, four emotion subcategories can be defined as "frown," "frustrated," "upset," and "angry." Except for neutral emotions, the other four emotion categories each have corresponding emotion state transition diagrams. Multiple emotion subcategories belonging to the same emotion category have varying degrees of intensity. Some emotion subcategories express stronger emotions, while others express relatively weaker emotions. For example, among the several emotion subcategories of anger, the emotion subcategory "angry" is clearly more intense than the others. During the normal duration of an emotion, the emotional state usually undergoes a change, which is reflected in facial expressions. An expression state transition diagram can illustrate the possible sequence of changes in expression states within the same emotion category due to different emotion subcategories. In a specific embodiment, when emotions are divided into emotion subcategories, the expression state transition diagram can use the emotion subcategories as node names, thereby more clearly expressing the possible transition methods between expression states. Figure 3 The diagram shown is a schematic representation of the emotion transition of the expression state of anger in one embodiment. It should be noted that this schematic diagram is only used to illustrate the form of the expression state transition diagram and is not intended to limit the expression state transition diagram in actual application.

[0057] Specifically, the server can acquire the target audio, and after acquiring the target audio, it can obtain an expression state transition diagram corresponding to the emotion type of the target audio. The target audio can be audio uploaded by the terminal, audio stored in the server's audio database, or audio acquired by the server from other computer devices. This application does not restrict the source of the target audio.

[0058] In one specific embodiment, the target audio may carry an emotion tag, and the server may determine the emotion type of the target audio based on the emotion tag. For example, the target audio may carry an "angry" tag, so the server can determine the emotion type of the target audio as angry based on the tag.

[0059] In other embodiments, after acquiring the target audio, the server can input the target audio into a pre-trained audio classification model. This model then classifies the target audio and determines its emotional category based on the classification results. The audio classification model is a machine learning model, which can be trained in a supervised manner using labeled audio samples.

[0060] Step 204: Based on the audio features of each keyframe in the emotional keyframe sequence of the target audio, determine the node links that match the emotional keyframe sequence from the expression state transition graph; the node links include target nodes that correspond one-to-one with each keyframe, and directed edges connecting the target nodes, and the audio features of the keyframes corresponding to the target nodes pointed to by the directed edges satisfy the node connection conditions of the directed edges.

[0061] The audio features of a keyframe refer to the characteristic information of the keyframe in the target audio. In specific applications, the audio features of a keyframe can be features related to rhythm and tempo, such as one or more of the keyframe's pitch, volume, pitch variation information, and volume variation information. It is understood that in other embodiments, the audio features of a keyframe may also include other information, as long as it can express the characteristics of the keyframe in the target audio; this application does not limit the specific form of the audio features. An emotion keyframe sequence refers to a frame sequence composed of keyframes in the target audio arranged in temporal order. Keyframes in the target audio refer to audio frames representing key emotional information in the target audio; keyframes may include, for example, emotion start frames, emotion climax frames, and emotion end frames. An emotion keyframe sequence can express the rhythmic changes in facial expressions in the target audio. A node link matching the emotion keyframe sequence, i.e., the rhythmic changes in facial expressions described by that node link, is consistent with the rhythmic changes in facial expressions in the target audio. A node link includes a target node that corresponds one-to-one with each keyframe. That is, for each keyframe, the node link includes a corresponding target node, and the connection order between the target nodes is consistent with the order of the corresponding keyframes in the keyframe sequence. Multiple target nodes in a node link are connected by directed edges, and the direction of the directed edges indicates the connection order between the nodes.

[0062] Specifically, the server can extract keyframes from the target audio. The sequence of extracted keyframes is called the keyframe sequence. For each keyframe, the server can obtain its audio features, which are extracted from the target audio. Optionally, the server can perform pitch and volume analysis on the target audio and extract the audio features of the keyframes based on the pitch and volume analysis results. In other embodiments, the server can input the target audio into a pre-trained audio feature extraction model, which extracts features from each keyframe to obtain the audio features of each keyframe. The audio feature extraction model is a machine learning model obtained through supervised training. After obtaining the audio features of each keyframe, the server can further search the facial expression state transition graph to determine the node links that match the emotion keyframe sequence from the facial expression state transition graph.

[0063] It should be noted that in some other embodiments, the extraction of keyframes and their audio features can be performed by other computer devices. The server can send the target audio to other computer devices and receive the keyframes and audio features returned by those devices. Alternatively, the server can directly receive the emotion type of the target audio and the audio features of the keyframes sent by other computer devices, determine the facial expression state transition graph based on the emotion type, and search the facial expression state transition graph based on the audio features to determine node links. In this case, the server does not need to obtain the source file of the target audio.

[0064] Step 206: Obtain the expression base corresponding to each target node, and form an expression base sequence according to the node connection order represented by the directed edges.

[0065] An expression base is a static base representing a facial expression. This base can represent a certain expression state. Expression bases can be represented numerically, and the numerical value of the expression base is the expression state description value corresponding to the expression base. The expression state description value can be a mesh point value or a face controller value. In practical applications, when the expression state description value is a face controller value, one expression base can be bound to one or more face controllers. Different face controllers can control different facial parts.

[0066] Specifically, there is a correspondence between target nodes and expression bases. Based on this correspondence, after determining the node link, the server can obtain the expression base corresponding to each target node in the node link. These expression bases, according to the node connection order represented by the directed edges in the node link, can form the expression base sequence corresponding to the emotion keyframe sequence. For example, suppose the node link is... Figure 3In the sequence "disappointment -> anger -> disappointment -> frown", the expression base obtained for the first target node in the link is expression base A, the expression base obtained for the second target node in the link is expression base B, the expression base obtained for the third target node in the link is expression base C, and the expression base obtained for the fourth target node in the link is expression base D. Then the final expression base sequence is: expression base A -> expression base B -> expression base C -> expression base D.

[0067] Step 208: Determine the expression sequence of the target audio based on the expression base sequence.

[0068] Specifically, the server can determine the expression base of each audio frame in the target audio based on the expression base sequence. The expression bases corresponding to all audio frames in the target audio are arranged into an initial expression sequence according to the temporal order of the audio frames. The server can further determine the expression sequence of the target audio based on the initial expression sequence.

[0069] In one specific embodiment, the server can obtain the lip shape sequence of the target audio frame, merge the initial expression sequence and the lip shape sequence, and finally obtain the audio sequence of the target audio. In a specific application, assuming that the expression state description value of the expression base is the value of the face controller, merging the initial expression sequence and the lip shape sequence can be done by adding a value that matches the corresponding lip shape in the lip shape sequence to the controller bound to each expression base in the initial expression sequence.

[0070] In one specific embodiment, the server can determine the audio duration segment corresponding to the emotion type of the target audio based on the emotion keyframes. For the audio frames within the audio duration segment, interpolation processing is performed based on the expression base sequence to obtain the expression base of each audio frame within the time segment. For audio outside the time segment, the expression base corresponding to the neutral emotion type can be used as the expression base of these audios.

[0071] In the above-mentioned expression sequence generation method, by determining the expression state transition graph corresponding to the emotion type of the target audio, and based on the audio features of each keyframe in the emotion keyframe sequence of the target audio, the node links matching the emotion keyframe sequence are determined from the expression state transition graph. The node links include target nodes corresponding to each keyframe and directed edges connecting the target nodes. The audio features of the keyframes corresponding to the target nodes pointed to by the directed edges satisfy the node connection conditions of the directed edges. Furthermore, the expression bases corresponding to each target node are obtained respectively. The expression bases corresponding to each target node are combined into an expression base sequence according to the node connection order represented by the directed edges, thereby realizing the automatic generation of the expression sequence of the target audio. Since the node links can be determined from the expression state transition graph in combination with specific audio features, after obtaining the expression bases corresponding to each target node in the node links, the expression base sequence is determined according to the connection order of each node in the node links, which can generate an expression sequence with a certain rhythm, thereby making the obtained expression sequence more flexible and vivid, and improving the accuracy of the expression sequence in expressing the emotion in the audio.

[0072] In one embodiment, based on the audio features of each keyframe in the emotional keyframe sequence of the target audio, determining the node links matching the emotional keyframe sequence from the expression state transition graph includes: acquiring multiple audio features extracted from the target audio, each audio feature corresponding one-to-one with each keyframe in the emotional keyframe sequence of the target audio; matching the audio features of each keyframe with the node connection conditions of each directed edge in the expression state transition graph, and determining the target node corresponding to each keyframe according to the node connection conditions satisfied by the audio features of each keyframe; and determining the node links formed by each target node and the directed edges between each target node as the node links matching the emotional keyframe sequence.

[0073] Specifically, the server can obtain the audio features corresponding to each keyframe, and match the audio features of each keyframe with the node connection conditions of each directed edge in the expression state transition graph. Through matching, the node connection conditions satisfied by the audio features of each keyframe can be determined from the expression state transition graph. The directed edges where the node connection conditions satisfied by the audio features of each keyframe are located are determined as the target edges corresponding to each keyframe, and the nodes pointed to by each target edge are determined as the target nodes corresponding to each keyframe. After determining the target edges and target nodes corresponding to each keyframe, these target edges and target nodes can be used to determine a node link according to the order in the state transition graph. The server can determine this node link as the node link that matches the emotion keyframe sequence.

[0074] In practical implementation, if a keyframe matches the node connection conditions of multiple directed edges in the expression state transition graph, the order of these directed edges in the expression state transition graph can be further determined. The directed edges whose order matches the order of the keyframe in the emotion keyframe sequence are identified as the target edges of that keyframe. The order of the directed edges in the expression state transition graph is determined based on their distance from the initial node, for example, in... Figure 3 In the given context, the directed edges between "neutral" and "disappointed" are in the first order, while the directed edges between "disappointed" and "angry" are in the second order.

[0075] For example, please refer to [the relevant documentation / reference]. Figure 3 Assuming that the four keyframes in the emotion keyframe sequence correspond to the target nodes "disappointment", "anger", "disappointment" and "frowning" respectively in the expression state transition graph, then the node link matching the emotion keyframe sequence is: disappointment → anger → disappointment → frowning.

[0076] In the above embodiments, by matching the audio features of the keyframes with the node connection conditions of each directed edge in the facial expression state transition graph, the target node can be determined. Furthermore, based on the directed edges between the target nodes, the node links matching the emotion keyframe sequence can be quickly determined, thereby improving the generation efficiency of the facial expression sequence.

[0077] In one embodiment, the audio features of each keyframe are matched with the node connection conditions of each directed edge in the expression state transition graph, and the target node corresponding to each keyframe is determined according to the node connection conditions satisfied by the audio features of each keyframe. This includes: determining the directed edges of each starting node from the expression state transition graph; determining the target edge corresponding to the node connection conditions satisfied by the audio features of the first keyframe in the emotion keyframe sequence from the directed edges of each starting node, and determining the node pointed to by the target edge as the target node corresponding to the first keyframe; for each current keyframe starting from the second keyframe, determining the target edge corresponding to the node connection conditions satisfied by the audio features of the current keyframe from the directed edges of the target node corresponding to the previous keyframe, and determining the target node pointed to by the determined target edge as the target node corresponding to the current keyframe, until the target node corresponding to the last keyframe in the emotion keyframe sequence is determined.

[0078] The first keyframe refers to the first keyframe in the emotion keyframe sequence, the second keyframe refers to the second keyframe in the emotion keyframe sequence, and the last keyframe refers to the last keyframe in the emotion keyframe sequence.

[0079] Specifically, the server can start from the first keyframe in the emotion keyframe sequence and sequentially determine the target edges and target nodes corresponding to each keyframe. For each keyframe with a target node, all directed edges originating from that target node are used as the matching range for the audio features of the next keyframe. Thus, when matching the node connection conditions for the next keyframe, the audio features of the next keyframe are matched with the node connection conditions of each directed edge within the matching range. This avoids matching the audio features of each keyframe with the node connection conditions of all directed edges. Furthermore, since the order of each directed edge within the matching range is consistent with the order of the next keyframe, it also ensures that accurate directed edges are matched quickly. This process is repeated sequentially for each keyframe in the emotion keyframe sequence until the target edge and target node corresponding to the last keyframe are determined.

[0080] In the above embodiments, since the target edges and target nodes of each frame are determined sequentially starting from the first frame, the matching range of each audio frame can be gradually narrowed, thereby improving the efficiency and accuracy of node link determination.

[0081] In one embodiment, obtaining the expression base corresponding to each target node and assembling the expression bases corresponding to each target node into an expression base sequence according to the node connection order represented by the directed edges includes: determining the emotion subclass represented by each target node from the set of emotion subclasses corresponding to the emotion types of the target audio; for each determined emotion subclass, obtaining the target expression base used to represent the emotion subclass, and determining the obtained target expression base as the expression base corresponding to the target node representing the emotion subclass; determining the order of each target node according to the node connection order represented by the directed edges, and assembling the expression bases corresponding to each target node into an expression base sequence according to the order of each target node.

[0082] In this embodiment, each emotion category may include multiple emotion subclasses, and multiple emotion subclasses belonging to the same emotion category constitute the emotion subclass set of that emotion category. Each node in the facial expression state transition graph can represent an emotion subclass, and each emotion subclass can be expressed using one or more expression bases. A pre-established correspondence exists between emotion subclasses and the expression bases that express those emotion subclasses. In a specific implementation, an emotion expression library can be pre-established to store information such as the emotion subclasses and expression bases corresponding to each emotion category. For example, refer to [reference needed]. Figure 4 This is a schematic diagram of an emotion expression library in one embodiment. Figure 4 As can be seen, there are four emotion subclasses for the emotion of anger, and each emotion subclass stores multiple expression templates.

[0083] Specifically, after obtaining the node links that match the emotion keyframe sequence, the server determines the emotion subclass represented by each target node in the node link. Then, from the expression bases that correspond to the emotion subclass, the server determines an expression base as the expression base corresponding to the target node. Finally, the order of each target node is determined according to the node connection order represented by the directed edges, and the expression bases corresponding to each target node are combined into an expression base sequence according to the order of each target node.

[0084] In the above embodiments, by further dividing the types of emotions into multiple emotion subclasses and establishing the correspondence between emotion subclasses and expression bases, the expression bases of each target node can be accurately obtained, thereby obtaining a more accurate expression base sequence.

[0085] In one embodiment, obtaining a target expression base for representing an emotion subclass includes: determining a set of expression bases for the emotion subclass; selecting an expression base from the set of expression bases that matches the audio features corresponding to the target node representing the emotion subclass, and determining the selected expression base as the target expression base; or randomly selecting an expression base from the set of expression bases and determining the selected expression base as the target expression base.

[0086] In this embodiment, the expression base set of each emotion subclass may include multiple expression bases. Thus, for each target node, after determining the emotion subclass represented by the target node, the server can obtain the expression base set of that emotion subclass and select the target expression base from the expression base set.

[0087] Optionally, the server can randomly select an expression base from the expression base set of the emotion subclass as the target expression base. Alternatively, the server can match the audio features of the target node with the selection conditions corresponding to each expression base in the expression base set, and determine the expression base corresponding to the matching selection conditions as the target expression base.

[0088] In the above embodiments, multiple expression bases can be selected for each emotion subclass, which can make the generated animation sequence more expressive.

[0089] The above embodiments describe the process of determining node links and obtaining corresponding expression bases for target nodes in the node links in this application. In this application, when the server obtains the source file of the target audio, the server needs to extract the emotion keyframe sequence for the target audio. The following embodiments will specifically describe the process of extracting the emotion keyframe sequence from the target audio.

[0090] In one embodiment, reference Figure 5 Extracting an emotion keyframe sequence from target audio can specifically include the following steps:

[0091] Step 502: Extract the pitch feature map and volume feature map from the target audio respectively.

[0092] Among them, the pitch feature map is a map that can characterize the pitch features of each audio frame in the target audio, and the volume feature map is a map that can characterize the volume features of each audio frame in the target audio.

[0093] Specifically, the server can analyze the target audio to extract pitch feature maps and volume feature maps.

[0094] In one specific embodiment, extracting a pitch feature map and a volume feature map from the target audio includes: obtaining a volume curve based on the volume of each frame in the target audio and a pitch curve based on the pitch of each frame in the target audio; determining the extreme points of the volume curve and merging the extreme points of the volume curve to obtain a volume feature map of the target audio; determining the extreme points of the pitch curve and merging the extreme points of the pitch curve to obtain a pitch feature map of the target audio.

[0095] The process of merging extreme points refers to unifying them, meaning using a single value to represent each extreme point. By merging these extreme points, the areas of emotional change can be identified from the pitch curve. For example... Figure 6 The image shown is a schematic diagram of the volume analysis of the target audio. Figure 6 The horizontal axis represents the frame number, and the vertical axis represents the volume value. Curve 602, composed of solid lines, is the volume curve; curve 604, composed of low-density dashed lines, is the volume characteristic map; and curve 606, composed of high-density dashed lines, is the volume envelope. Figure 6 As can be seen from the curved section within circle 608, during the process of merging extreme values, the extreme points of this curve can be unified into values ​​on the envelope curve. This flattens the section of the curve containing extreme points into a straight line, ultimately resulting in a trapezoidal region representing the area of ​​emotional change. For example... Figure 6 The two trapezoidal curve regions in the middle curve 604 represent the areas of emotional change.

[0096] Step 504: Determine the volume change information of each audio frame in the target audio based on the volume feature map, and determine the first emotion keyframe sequence based on the volume change information.

[0097] Volume variation information refers to information that reflects the intensity of volume changes in audio frames. In audio containing non-neutral emotions, the emotion typically shifts from neutral to other non-neutral emotions, and then back to neutral. This shift can be reflected in the volume variation information of the volume feature map. For example, in target audio containing the emotion of anger, if the volume of audio frames is initially low and relatively stable, and then suddenly changes dramatically and becomes very high, it indicates that the speaker may be in a very angry state. Therefore, this audio frame is a keyframe that expresses the emotion of anger. Based on this, after obtaining the audio feature map, the server can determine the volume variation information of each audio frame in the target audio based on the volume feature map, determine the first emotion keyframe sequence based on the volume variation information, determine the tone variation information of each audio frame in the target audio based on the tone feature map, and determine the second emotion keyframe sequence based on the tone variation information.

[0098] In one specific embodiment, determining the volume change information of each audio frame in the target audio based on the volume feature map, and determining the first emotion key frame sequence based on the volume change information, includes: calculating the derivative of each audio frame in the target audio in the volume feature map; determining the curve rising start point, at least one curve rising maximum point after the curve rising start point, and curve falling end point after the curve rising maximum point in the volume feature map based on the derivative; obtaining the target frame sequence number corresponding to the curve rising start point, at least one curve rising maximum point, and curve falling end point respectively; and determining the audio frame corresponding to the target frame sequence number as the emotion key frame to obtain the first emotion key frame sequence.

[0099] Since facial expressions typically begin in a neutral state, transition to an emotional expression state, and then return to a neutral state after a period of time, at least three audio frames can be extracted from the target audio as keyframes: the emotion initiation frame, indicating the transition from a neutral to a non-neutral emotional state; the emotion climax frame, the audio frame with the strongest emotional expression; and the emotion termination frame, indicating the transition from a non-neutral to a neutral emotional state. These keyframes are represented on the audio feature curve as the curve's starting point, at least one peak after the starting point, and the curve's descent termination point. Therefore, after obtaining the volume feature map, the server can calculate the derivative of each audio frame in the target audio within the volume feature map. Based on the derivative, the server can determine the curve's starting point, at least one peak after the starting point, and the curve's descent termination point in the volume feature map. The server can then obtain the target frame sequence corresponding to these curve points and identify the audio frames corresponding to the target frame numbers as emotion keyframes, thus obtaining the first emotion keyframe sequence.

[0100] In practical applications, to better express changes in emotional state in audio, four keyframes can be extracted from the target audio: the emotion start frame (start), the emotion climax start frame (start_keep), the emotion climax end frame (end_keep), and the emotion end frame (end). Correspondingly, the server can perform change point detection on the volume feature map from left to right. By comparing the derivative with a set threshold, when the curve begins to rise, the frame corresponding to that start point is designated as start; when the curve begins to stabilize, the frame corresponding to that start point is designated as start_keep. Similarly, end and end_keep are obtained from right to left. By calculating the derivative to detect change points on the volume feature map, keyframes can be quickly determined.

[0101] For example, suppose the volume feature map is... Figure 6 As shown in the curve 604, the section with the horizontal coordinate from 100 to 200 is represented. The audio frames corresponding to the four vertices of the trapezoidal region in this section of the curve are the four keyframes, which are start, start_keep, end_keep and end from left to right.

[0102] Step 506: Determine the pitch change information of each audio frame in the target audio based on the pitch feature map, and determine the second emotion key frame sequence based on the pitch change information.

[0103] Among them, pitch change information refers to information that can reflect the intensity of pitch change in an audio frame.

[0104] In a specific embodiment, determining the pitch change information of each audio frame in the target audio based on the pitch feature map, and determining the first emotion key frame sequence based on the pitch change information, includes: calculating the derivative of each audio frame in the target audio in the pitch feature map; determining the curve rising start point, at least one curve rising maximum point after the curve rising start point, and curve falling termination point after the curve rising maximum point in the pitch feature map based on the derivative; obtaining the target frame sequence number corresponding to the curve rising start point, at least one curve rising maximum point, and curve falling termination point respectively; and determining the audio frame corresponding to the target frame sequence number as the emotion key frame to obtain the first emotion key frame sequence.

[0105] Step 508: Calculate the average value of the frame numbers of the corresponding emotional keyframes in the first and second emotional keyframe sequences, and determine the emotional keyframe sequence of the target audio based on the calculation result.

[0106] Specifically, the server can calculate the average of the frame numbers of the corresponding emotional keyframes in the first and second emotional keyframe sequences. When the calculated average is an integer, the result is the keyframe number at the corresponding position. When the calculated average is not an integer, the result can be rounded to obtain the keyframe number at the corresponding position. The audio frames corresponding to the keyframe numbers at each corresponding position form the emotional keyframe sequence. For example, assuming the frame numbers of the first emotion keyframe are 120, 130, 150, and 160, and the frame numbers of the second emotion keyframe are 122, 132, 152, and 162, then the frame number of the emotion start frame is (120+122) / 2, the frame number of the emotion climax start frame is (130+132) / 2, the frame number of the emotion climax end frame is (150+152) / 2, and the frame number of the emotion end frame is (160+162) / 2. Finally, based on the calculation results, the frame numbers of the emotion keyframes of the target audio are 121, 131, 151, and 161.

[0107] In the above embodiments, an emotion keyframe sequence is determined based on two different feature maps, and the average value of the frame numbers of the keyframes at corresponding positions in the two obtained emotion keyframe sequences is calculated. The emotion keyframe sequence of the target audio is determined based on the average value calculation result. The obtained emotion keyframe sequence can more accurately express the key emotion information of the target audio.

[0108] In one embodiment, determining the expression sequence of a target audio based on an expression base sequence includes: obtaining the expression state description value corresponding to each expression base in the expression base sequence; performing interpolation processing on the expression state description value corresponding to each expression base to obtain a target interpolation curve; determining the expression state description value of each target audio frame in the audio duration segment corresponding to the emotion type from the target interpolation curve; determining the expression base of each target audio frame according to the expression state description value of each target audio frame in the audio duration segment; and determining the expression sequence of the target audio based on the expression base of each target audio frame in the audio duration segment.

[0109] The audio duration segment corresponding to an emotion type refers to the duration of the emotion in the target audio that corresponds to the emotion type of the target audio. For example, assuming the emotion type of the target audio is anger, the audio duration segment corresponding to the emotion type refers to the duration of anger in the target audio. In practical applications, if the emotion keyframe sequence includes an emotion start frame and an emotion end frame, then the time from the emotion start frame to the emotion end frame is the audio duration segment corresponding to the emotion type.

[0110] Specifically, since the expression base sequence is obtained from the emotion keyframe sequence, which contains only a few keyframes of expression base, in order to obtain the expression base of other audio frames within the audio duration of the emotion type, the server can obtain the expression state description value corresponding to each expression base in the expression base sequence. By interpolating the expression state description value corresponding to each expression base, the server can obtain the expression base of each target audio frame within the emotion duration of the target audio.

[0111] During interpolation, since different facial features in the expression base are controlled by different values, interpolation is required for the expression state description values ​​describing the same facial feature. In a specific embodiment, when the expression state description value of the expression base is a facial controller value, for each facial controller bound to the expression base, the server can obtain the value of each expression base corresponding to that facial controller in the expression base sequence, and then perform interpolation on the value of that facial controller. In a specific implementation, the server can use linear interpolation to interpolate the value of the facial controller. Alternatively, the server can use non-linear interpolation to interpolate the value of the facial controller.

[0112] After the server performs interpolation processing on the expression state description values ​​corresponding to each expression base, it can obtain the target interpolation curve. From the target interpolation curve, the server can further determine the expression state description values ​​of each target audio frame in the audio duration segment corresponding to the emotion type. Based on the expression state description values ​​of each target audio frame in the audio duration segment, the server can determine the expression base of each target audio frame. Thus, the expression sequence of the target audio can be determined based on the expression base of each target audio frame.

[0113] In one embodiment, interpolating the expression state description values ​​corresponding to each expression base to obtain a target interpolation curve includes: performing linear interpolation based on the expression state description values ​​corresponding to each expression base to obtain a first initial interpolation curve; performing nonlinear interpolation based on the expression state description values ​​corresponding to each expression base to obtain a second initial interpolation curve; and merging the first initial interpolation curve and the second initial interpolation curve to obtain the target interpolation curve.

[0114] Specifically, in order to make the facial expressions more coherent, this embodiment can use two interpolation methods, linear interpolation and nonlinear interpolation, respectively, and then merge the difference curves obtained by the two different interpolation methods to obtain the target interpolation curve.

[0115] In practical implementation, second-order Bezier interpolation based on control points can be used. For an example, see [reference needed]. Figure 7 This is a schematic diagram of the difference curve in one embodiment. Figure 7In Figure (a), curve 702 is the first initial interpolation curve obtained by linear interpolation. During the linear interpolation process, the linear interpolation curve is obtained by connecting the numerical points of each keyframe through the linear interpolation curve. Figure 7 Curve 704 in (a) is the second initial interpolation curve obtained by interpolation using a first-order Bezier curve. Figure 7 Curve 706 in (b) is the target interpolation curve obtained by merging the first initial interpolation curve and the second initial interpolation curve. This target interpolation curve is a combination of a linear interpolation curve and a first-order Bezier curve. During the merging process, the midpoint of the linear interpolation curve can be taken, and then the midpoints can be connected by a second-order Bezier curve. The resulting curve can fit the state of each keyframe very closely. After connecting the curves, the expression base of each target audio frame in the audio duration under the controller can be obtained.

[0116] In the above embodiments, linear interpolation is performed based on the expression state description values ​​corresponding to each expression base to obtain a first initial interpolation curve. Nonlinear interpolation is performed based on the expression state description values ​​corresponding to each expression base to obtain a second initial interpolation curve. The first initial interpolation curve and the second initial interpolation curve are merged to obtain the target interpolation curve, which can make the obtained expression sequence more coherent and natural in its emotional expression.

[0117] To more realistically mimic the details of human facial expressions, blinking and eye perturbations need to be added to the generated expression sequence, making the resulting expressions more lifelike and vivid. The following examples will describe the specific process of adding blinking and eye perturbations.

[0118] In one embodiment, determining the expression sequence of the target audio based on the expression base sequence includes: determining the expression base of each audio frame in the target audio based on the expression base sequence; assembling the expression bases of each audio frame into an initial expression sequence according to the temporal order of the audio frames; obtaining a first blink template and a second blink template; wherein the change amplitude of the eyelid state description value in the first blink template is greater than the change amplitude of the eyelid state description value in the second blink template; adding the first blink template to the first target expression base of the initial expression sequence, and randomly adding the second blink template to each of the first target expression bases of the initial expression sequence to obtain the expression sequence of the target audio.

[0119] The first and second blink templates are different templates. Each template records the changes in eyelid state description values ​​used to control blinking during a single blink. If the change in eyelid state description values ​​in the first blink template is greater than that in the second blink template, then the second blink template can describe a natural blink, while the first blink template can describe a large-amplitude blink. For example... Figure 8The image shown is a schematic diagram illustrating the changes in eyelid state description values ​​in a blinking template in one embodiment. Figure 8 (a) in the figure represents the curve showing the change in the eyelid state description value in the first blink template. Figure 8 (b) in the figure represents the curve showing the change in eyelid state description values ​​in the second blink template. Figure 8 It can be seen that the variation range of the eyelid state description value in the first blink template is greater than that in the first blink template. Here, the eyelid state description value refers to the value in the expression base used to describe the eyelid state, such as the value of the controller corresponding to the eyelid.

[0120] The first target expression base is the forward neighbor expression base corresponding to the expression base corresponding to the emotion type of the target audio. Here, the forward neighbor expression base refers to all expression bases adjacent to and preceding the expression base corresponding to the emotion type of the target audio. For example, it could be an expression base preceding the expression base corresponding to the emotion start frame. The second target expression base is the expression base other than the expression base corresponding to the emotion type of the target audio.

[0121] Specifically, after determining the expression base sequence corresponding to the emotion keyframe, the server can determine the expression base of each audio frame in the target audio based on the expression base sequence. Specifically, the server can perform interpolation processing on each target audio frame within the audio duration segment corresponding to the emotion type to obtain the expression base. For other audio frames in the target audio, an expression base can be randomly selected from the set of expression bases corresponding to the neutral emotion type. The expression bases of each audio frame in the target audio are arranged according to the temporal order of the audio frames to obtain the expression base sequence of the entire target audio. This expression base sequence is the initial expression sequence. After that, the server can add the first blink template and the second blink template to the initial expression sequence.

[0122] Normally, blinking doesn't occur naturally during the duration of emotions, such as anger or sadness, and these emotions are often preceded by significant blinking. Therefore, in this embodiment, blinking is added using dynamic template matching. The server first determines a first target expression base and a second target expression base. A first blinking template is added to the first target expression base of the initial expression sequence, and a second blinking template is randomly added to each of the first target expression bases in the initial expression sequence to obtain the expression sequence of the target audio.

[0123] In the implementation, the server randomly adds a second blink template to each of the first target expression bases in the initial expression sequence. This can be done by randomly selecting an audio frame at a certain blinking frequency, binding the expression base of the audio frame to the controller value or mesh point value corresponding to eyelid control, and setting it according to the value in the second blink template.

[0124] In the above embodiments, a first blinking template is added to the first target expression base of the initial expression sequence, and a second blinking template is randomly added to each of the first target expression bases of the initial expression sequence to obtain the expression sequence of the target audio. This can more realistically imitate the details of human facial expressions, making the emotional expression of the obtained expression sequence more realistic and natural.

[0125] In one embodiment, the method for determining the expression sequence of the target audio based on the expression base sequence includes: determining the expression base of each audio frame in the target audio based on the expression base sequence; assembling the expression bases of each audio frame into an initial expression sequence according to the temporal order of the audio frames; selecting target expression bases in the initial expression sequence at preset intervals; and adding random eye state description values ​​to each selected target expression base to obtain the expression sequence of the target audio; wherein the eye state description values ​​meet the requirements of the eye perturbation range.

[0126] Eye perturbations are achieved by adding random perturbations of a certain frequency to the eyeball. The area and size of the perturbation are as follows: Figure 9 As shown, the dark boxes represent the maximum achievable range of the eyeball, while the light boxes represent the range of eyeball perturbation. The range of eyeball perturbation is a reduction of the eyeball's maximum achievable range. Within this perturbation range, the eyeball moves randomly, as indicated by the arrows in the light boxes, creating a simulated random perturbation of the human eye. Eyeball state description values ​​refer to the values ​​in the expression base used to describe the state of eyeball movement. These can be, for example, the values ​​of the controller corresponding to the eyeball or the values ​​of mesh points at the eyeball's location. For eyeball state description values ​​to meet the eyeball perturbation range requirement, the perturbation point described by the eyeball state description values ​​must be within the range of eyeball perturbation.

[0127] In this embodiment, after obtaining the initial expression sequence, the server selects target expression bases from the initial expression sequence at a preset interval. The preset interval is used to simulate the frequency of eye movement and can correspond to the normal frequency of eye movement. For each selected target expression base, the value of the controller or mesh point bound to the target expression base for controlling eye movement is set to a random value, so that the eye moves randomly within the range of eye movement.

[0128] In the above embodiments, target expression bases are selected in the initial expression sequence at preset intervals. For each selected target expression base, random eye state description values ​​are added, which can more realistically imitate the details of human facial expressions, making the emotional expression of the obtained expression sequence more realistic and natural.

[0129] The facial expression sequence generation method provided in this application automates the process of generating facial expressions based on audio. It can quickly produce facial expression animations without human intervention, replacing the process of motion capture and manual animation generation by animators. This saves significant manpower and time costs, and the generated effect can reach the level of professional animator's manual rendering. It can be applied in practical scenarios such as game dialogues and virtual human live streaming. The following sections further illustrate the facial expression sequence generation method in this application using game dialogues and virtual human live streaming as examples.

[0130] In one specific embodiment, this application also provides an application scenario in which the expression sequence generation method provided by this application is applied to virtual human live streaming. The live streaming client of the terminal can collect the target audio of the anchor during the live streaming process and send the target audio to the server. The server uses the expression sequence generation method provided in this application embodiment to generate an expression sequence for the target audio. The generated expression sequence drives the virtual human to obtain the animation video corresponding to the target audio. Furthermore, the animation video can be sent to the live streaming clients of each viewer in the live streaming room to realize virtual human live streaming.

[0131] refer to Figure 10 This is a system framework diagram of the facial expression sequence generation method in this embodiment. It mainly includes four modules: audio analysis module, facial expression search module, curve connection module, and micro-expression module. The input consists of audio, emotion tags, and lip-shape sequences generated upstream. There are five main emotion tags: neutral, happy, sad, angry, and questioning. The output is a facial expression sequence that integrates the lip shape (lower half of the face) and the facial expression (upper half of the face), i.e., the values ​​of each controller on the lower face in each frame.

[0132] The above process framework generates a four-step process, where the input audio sequence (wav file format) serves as the basis for audio analysis, emotion tags serve as the basis for expression search, and the lip-shape sequence is used to merge with the generated expression sequence to form a full-face expression. Specifically: Step 1, Audio Analysis. The audio sequence enters the audio analysis module, which extracts features from the audio and establishes empirical rules for these features, thereby finding keyframes for the onset, climax, and end of emotions in the audio temporal sequence. The audio analysis module outputs the positions and types of these keyframes in the audio temporal sequence, including the emotion start frame, the start frame of the emotion climax, the end frame of the emotion climax, and the emotion end frame, as well as audio features, including feature maps of volume and pitch. Step 2, Expression Search. The expression search module receives emotion tags and determines the emotion category under which to search for expression bases. It receives keyframes and audio feature maps from step one to determine the selected expression base type and transfer method, including inter-class and intra-class transfers. Each emotion subclass's expression base is a type; transfers occur from one emotion subclass to another, while intra-class transfers select the expression base from within the set of expression bases for a given emotion subclass. This outputs the searched expression base for each keyframe. Step three involves curve connection. The curve connection module receives the expression base values ​​for the corresponding frames from step two and uses second-order Bezier interpolation based on control points to interpolate sequences without keyframes. This allows the expression to gradually change from a neutral state to the beginning, climax, and end of the emotion, then back to a neutral state. After step three, the upper half of the face's expression sequence is formed. Superimposing the lower half's lip shape sequence creates the basic facial expression sequence, the initial sequence. To make the expressions more realistic, this framework system also includes a fourth module, step four: superimposing micro-expressions. By incorporating a dynamic blinking template and setting perturbation rules, micro-expressions involving blinking and perturbation are added to the facial expression sequence, making the generated expressions more detailed and realistic. Finally, a facial expression sequence that includes emotional expressions, lip movements, and micro-expression variations is output.

[0133] In another specific embodiment, this application also provides another application scenario. In this scenario, the facial expression sequence generation method provided by this application is applied to game dialogue. The game client of the terminal can collect the target audio of the game user during the game and send the target audio to the server. The server uses the facial expression sequence generation method provided in this embodiment to generate facial expression sequences for the target audio and sends the facial expression sequences to each terminal participating in the game. The game client of the terminal can drive the virtual objects in the game through the received facial expression sequences to obtain game animations corresponding to the target audio. In this embodiment, the facial expression sequence generation method specifically includes the following steps:

[0134] 1. Determine the facial expression state transition diagram corresponding to the emotional type of the target audio.

[0135] 2. Obtain the volume curve based on the volume of each frame in the target audio, and the pitch curve based on the pitch of each frame in the target audio; determine the extreme points of the volume curve, merge the extreme points of the volume curve to obtain the volume feature map of the target audio; determine the extreme points of the pitch curve, merge the extreme points of the pitch curve to obtain the pitch feature map of the target audio.

[0136] 3. Determine the volume change information of each audio frame in the target audio based on the volume feature map, and determine the first emotion keyframe sequence based on the volume change information; determine the tone change information of each audio frame in the target audio based on the tone feature map, and determine the second emotion keyframe sequence based on the tone change information; calculate the average of the frame numbers of the corresponding emotion keyframes in the first emotion keyframe sequence and the second emotion keyframe sequence, and determine the emotion keyframe sequence of the target audio based on the calculation result.

[0137] The process involves determining the volume change information of each audio frame in the target audio based on the volume feature map, and determining the first emotion keyframe sequence based on the volume change information. This includes: calculating the derivative of each audio frame in the target audio in the volume feature map; determining the curve rising start point, at least one curve rising maximum point after the curve rising start point, and the curve falling end point after the curve rising maximum point in the volume feature map based on the derivative; obtaining the target frame sequence number corresponding to the curve rising start point, at least one curve rising maximum point, and curve falling end point, respectively; and determining the audio frame corresponding to the target frame sequence number as the emotion keyframe to obtain the first emotion keyframe sequence.

[0138] The pitch change information of each audio frame in the target audio is determined based on the pitch feature map, and the first emotion key frame sequence is determined based on the pitch change information. This includes: calculating the derivative of each audio frame in the target audio in the pitch feature map; determining the curve rising start point, at least one curve rising maximum point after the curve rising start point, and curve falling termination point after the curve rising maximum point in the pitch feature map based on the derivative; obtaining the target frame sequence number corresponding to the curve rising start point, at least one curve rising maximum point, and curve falling termination point respectively; and determining the audio frame corresponding to the target frame sequence number as the emotion key frame to obtain the first emotion key frame sequence.

[0139] 4. Obtain multiple audio features extracted from the target audio, with each audio feature corresponding one-to-one with each keyframe in the emotional keyframe sequence of the target audio.

[0140] 5. Determine the directed edges of each starting node from the expression state transition graph.

[0141] 6. From the directed edges of each starting node, determine the target edge corresponding to the node connection condition satisfied by the audio features of the first key frame in the emotion key frame sequence, and determine the node pointed to by the target edge as the target node corresponding to the first key frame.

[0142] 7. For each current keyframe starting from the second keyframe, from the directed edges of the target node corresponding to the target node of the previous keyframe, determine the target edge corresponding to the node connection condition satisfied by the audio features of the current keyframe, and determine the target node pointed to by the determined target edge as the target node corresponding to the current keyframe, until the target node corresponding to the last keyframe in the emotion keyframe sequence is determined.

[0143] 8. Determine the node links formed by each target node and the directed edges between each target node as the node links that match the emotion keyframe sequence.

[0144] 9. Determine the emotion subclass represented by each target node in the node link from the set of emotion subclasses corresponding to the emotion types of the target audio.

[0145] 10. For each determined emotion subclass, obtain the target expression base used to represent the emotion subclass, and determine the obtained target expression base as the expression base corresponding to the target node representing the emotion subclass.

[0146] The process of obtaining the target expression base for representing the emotion subclass includes: determining the expression base set for the emotion subclass; selecting an expression base from the expression base set that matches the audio features corresponding to the target node representing the emotion subclass, and determining the selected expression base as the target expression base; or randomly selecting an expression base from the expression base set and determining the selected expression base as the target expression base.

[0147] 11. Determine the order of each target node according to the node connection order represented by the directed edges, and form an expression basis sequence according to the expression basis corresponding to each target node.

[0148] 12. Obtain the expression state description value corresponding to each expression base in the expression base sequence; perform interpolation processing on the expression state description value corresponding to each expression base to obtain the target interpolation curve; determine the expression state description value of each target audio frame in the audio duration segment corresponding to the emotion type from the target interpolation curve; determine the expression base of each target audio frame according to the expression state description value of each target audio frame in the audio duration segment.

[0149] The process of interpolating the expression state description values ​​corresponding to each expression base to obtain a target interpolation curve includes: performing linear interpolation based on the expression state description values ​​corresponding to each expression base to obtain a first initial interpolation curve; performing nonlinear interpolation based on the expression state description values ​​corresponding to each expression base to obtain a second initial interpolation curve; and merging the first initial interpolation curve and the second initial interpolation curve to obtain the target interpolation curve.

[0150] 13. For each non-target audio frame in the target audio, randomly select an expression base from the set of expression bases corresponding to neutral emotions as the expression base of that audio frame. The expression bases of all audio frames in the target audio are arranged into an initial expression sequence according to the temporal order of the audio frames.

[0151] Non-target audio frames refer to audio frames outside the duration segment corresponding to the emotional type of the target audio.

[0152] 14. Add blinking. Obtain the first blinking template and the second blinking template; add the first blinking template to the first target expression base of the initial expression sequence, and randomly add the second blinking template to each of the second target expression bases of the initial expression sequence.

[0153] Among them, the change range of the eyelid state description value in the first blink template is greater than the change range of the eyelid state description value in the second blink template; the first target expression base is the forward neighboring expression base of the expression base corresponding to the emotion type of the target audio; the second target expression base is an expression base other than the expression base corresponding to the emotion type of the target audio.

[0154] 15. Add eye perturbation. Select target expression bases at preset intervals in the initial expression sequence, and add random eye state description values ​​for each selected target expression base.

[0155] Among them, the eye state description value meets the requirements for the range of eye perturbation.

[0156] 16. The initial expression sequence obtained after adding blinking and eye perturbation is combined with the lip shape sequence corresponding to the target audio to obtain an expression sequence that has both emotional expression and lip shape as well as micro-expression changes.

[0157] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0158] Based on the same inventive concept, this application also provides an expression sequence generation apparatus for implementing the expression sequence generation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more expression sequence generation apparatus embodiments provided below can be found in the limitations of the expression sequence generation method described above, and will not be repeated here.

[0159] In one embodiment, such as Figure 11 As shown, an expression sequence generation device 1100 is provided, comprising:

[0160] The state transition diagram determination module 1102 is used to determine the expression state transition diagram corresponding to the emotion type of the target audio.

[0161] The node link determination module 1104 is used to determine the node link matching the emotion key frame sequence from the expression state transition graph based on the audio features of each key frame in the emotion key frame sequence of the target audio. The node link includes the target node corresponding to each key frame and the directed edge connecting the target node. The audio features of the key frame corresponding to the target node pointed to by the directed edge satisfy the node connection condition of the directed edge.

[0162] The expression base acquisition module 1106 is used to acquire the expression base corresponding to each target node respectively, and to form an expression base sequence according to the node connection order represented by the directed edges.

[0163] The facial expression sequence determination module 1108 is used to determine the facial expression sequence of the target audio based on the facial expression base sequence.

[0164] The aforementioned facial expression sequence generation device determines the facial expression state transition diagram corresponding to the emotion type of the target audio. Based on the audio features of each keyframe in the emotional keyframe sequence of the target audio, it determines the node links matching the emotional keyframe sequence from the facial expression state transition diagram. The node links include target nodes corresponding to each keyframe and directed edges connecting the target nodes. The audio features of the keyframes corresponding to the target nodes pointed to by the directed edges satisfy the node connection conditions of the directed edges. Furthermore, it obtains the facial expression base corresponding to each target node and forms a facial expression base sequence according to the node connection order represented by the directed edges. This realizes the automatic generation of facial expression sequences of the target audio. Since the node links can be determined from the facial expression state transition diagram in combination with specific audio features, after obtaining the facial expression base corresponding to each target node in the node links, the facial expression base sequence is determined according to the connection order of each node in the node links. This can generate facial expression sequences with a certain rhythm, making the obtained facial expression sequences more flexible and vivid, and improving the accuracy of the facial expression sequences in expressing the emotions in the audio.

[0165] In one embodiment, the node link determination module is further configured to acquire multiple audio features extracted from the target audio, each audio feature corresponding one-to-one with each keyframe in the emotional keyframe sequence of the target audio; match the audio features of each keyframe with the node connection conditions of each directed edge in the expression state transition graph, and determine the target node corresponding to each keyframe according to the node connection conditions satisfied by the audio features of each keyframe; and determine the node links formed by each target node and the directed edges between each target node as the node links matching the emotional keyframe sequence.

[0166] In one embodiment, the node link determination module is further configured to determine the directed edges of each starting node from the expression state transition graph; determine the target edge corresponding to the node connection condition satisfied by the audio features of the first key frame in the emotion key frame sequence from the directed edges of each starting node, and determine the node pointed to by the target edge as the target node corresponding to the first key frame; for each current key frame starting from the second key frame, determine the target edge corresponding to the node connection condition satisfied by the audio features of the current key frame from the directed edges of the target node corresponding to the previous key frame, and determine the target node pointed to by the determined target edge as the target node corresponding to the current key frame, until the target node corresponding to the last key frame in the emotion key frame sequence is determined.

[0167] In one embodiment, the expression base acquisition module is further configured to: determine the emotion subclass represented by each target node from the set of emotion subclasses corresponding to the emotion types of the target audio; for each determined emotion subclass, acquire the target expression base used to represent the emotion subclass, and determine the acquired target expression base as the expression base corresponding to the target node representing the emotion subclass; determine the order of each target node according to the node connection order represented by the directed edges, and form an expression base sequence according to the order of each target node's expression base.

[0168] In one embodiment, the expression base acquisition module is further configured to determine the expression base set of the emotion subclass; select an expression base from the expression base set that matches the audio features corresponding to the target node representing the emotion subclass, and determine the selected expression base as the target expression base; or randomly select an expression base from the expression base set and determine the selected expression base as the target expression base.

[0169] In one embodiment, the above-mentioned expression sequence generation method further includes: a keyframe determination module, used to extract a pitch feature map and a volume feature map from the target audio respectively; determine the volume change information of each audio frame in the target audio according to the volume feature map, and determine a first emotion keyframe sequence according to the volume change information; determine the pitch change information of each audio frame in the target audio according to the pitch feature map, and determine a second emotion keyframe sequence according to the pitch change information; calculate the average value of the frame numbers of the corresponding emotion keyframes in the first emotion keyframe sequence and the second emotion keyframe sequence, and determine the emotion keyframe sequence of the target audio according to the calculation result.

[0170] In one embodiment, the keyframe determination module is further configured to acquire a volume curve obtained based on the volume of each frame in the target audio and a pitch curve obtained based on the pitch of each frame in the target audio; determine the extreme points of the volume curve, merge the extreme points of the volume curve to obtain a volume feature map of the target audio; determine the extreme points of the pitch curve, merge the extreme points of the pitch curve to obtain a pitch feature map of the target audio.

[0171] In one embodiment, the keyframe determination module is further configured to calculate the derivative of each audio frame in the target audio in the volume feature map; determine the starting point of the curve rise, at least one maximum point of the curve rise after the starting point of the curve rise, and the ending point of the curve fall after the maximum point of the curve rise in the volume feature map based on the derivative; obtain the target frame number corresponding to the starting point of the curve rise, at least one maximum point of the curve rise, and the ending point of the curve fall respectively; determine the audio frame corresponding to the target frame number as the emotion keyframe, and obtain the first emotion keyframe sequence.

[0172] In one embodiment, the expression sequence determination module is further configured to: obtain the expression state description value corresponding to each expression base in the expression base sequence; perform interpolation processing on the expression state description value corresponding to each expression base to obtain a target interpolation curve; determine the expression state description value of each target audio frame in the audio duration segment corresponding to the emotion type from the target interpolation curve; determine the expression base of each target audio frame according to the expression state description value of each target audio frame in the audio duration segment; and determine the expression sequence of the target audio based on the expression base of each target audio frame in the audio duration segment.

[0173] In one embodiment, the expression sequence determination module is further configured to perform linear interpolation processing based on the expression state description values ​​corresponding to each expression base to obtain a first initial interpolation curve; perform nonlinear interpolation processing based on the expression state description values ​​corresponding to each expression base to obtain a second initial interpolation curve; and merge the first initial interpolation curve and the second initial interpolation curve to obtain a target interpolation curve.

[0174] In one embodiment, the above-mentioned expression sequence generation method further includes: a blink control module, configured to determine the expression base of each audio frame in the target audio based on the expression base sequence; assemble the expression bases of each audio frame into an initial expression sequence according to the temporal order of the audio frames; obtain a first blink template and a second blink template; wherein the change amplitude of the eyelid state description value in the first blink template is greater than the change amplitude of the eyelid state description value in the second blink template; add the first blink template to the first target expression base of the initial expression sequence, and randomly add the second blink template to each second target expression base of the initial expression sequence to obtain the expression sequence of the target audio; wherein the first target expression base is the forward neighbor expression base of the expression base corresponding to the emotion type of the target audio; and the second target expression base is an expression base other than the expression base corresponding to the emotion type of the target audio.

[0175] In one embodiment, the above-mentioned expression sequence generation method further includes: an eye perturbation control module, configured to determine the expression base of each audio frame in the target audio based on the expression base sequence; to assemble the expression bases of each audio frame into an initial expression sequence according to the temporal order of the audio frames; to select target expression bases in the initial expression sequence at preset intervals; and to add random eye state description values ​​to each selected target expression base to obtain the expression sequence of the target audio; wherein the eye state description values ​​meet the eye perturbation range requirements.

[0176] Each module in the aforementioned facial expression sequence generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0177] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores facial expression-related data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements the facial expression sequence generation method provided in any embodiment of this application.

[0178] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 13 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements the facial expression sequence generation method provided in any embodiment of this application. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0179] Those skilled in the art will understand that Figure 12 , Figure 13 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0180] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the facial expression sequence generation method in any of the above embodiments.

[0181] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the facial expression sequence generation method in any of the above embodiments.

[0182] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the facial expression sequence generation method in any of the above embodiments.

[0183] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0184] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0185] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0186] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for generating facial expression sequences, characterized in that, The method includes: Identify the facial expression state transition diagram corresponding to the emotion type of the target audio; Based on the audio features of each key frame in the emotional key frame sequence of the target audio, a node link matching the emotional key frame sequence is determined from the expression state transition graph; the node link includes a target node corresponding to each of the key frames and a directed edge connecting the target node, wherein the audio features of the key frame corresponding to the target node pointed to by the directed edge satisfy the node connection condition of the directed edge. Obtain the expression base corresponding to each target node, and form an expression base sequence according to the node connection order represented by the directed edges. Based on the expression base sequence, the expression sequence of the target audio is determined.

2. The method according to claim 1, characterized in that, The audio features of each keyframe in the emotion keyframe sequence based on the target audio are used to determine the node links matching the emotion keyframe sequence from the expression state transition graph, including: Multiple audio features extracted from the target audio are obtained, and each audio feature corresponds one-to-one with each keyframe in the emotional keyframe sequence of the target audio. The audio features of each key frame are matched with the node connection conditions of each directed edge in the facial expression state transition graph, and the target node corresponding to each key frame is determined according to the node connection conditions satisfied by the audio features of each key frame. The node links formed by each target node and the directed edges between each target node are determined as node links that match the emotion keyframe sequence.

3. The method according to claim 2, characterized in that, The step of matching the audio features of each keyframe with the node connection conditions of each directed edge in the facial expression state transition graph, and determining the target node corresponding to each keyframe based on the node connection conditions satisfied by the audio features of each keyframe, includes: The directed edges of each starting node are determined from the expression state transition graph; From the directed edges of each starting node, determine the target edge corresponding to the node connection condition satisfied by the audio features of the first key frame in the emotion key frame sequence, and determine the node pointed to by the target edge as the target node corresponding to the first key frame. For each current keyframe starting from the second keyframe, from the directed edges originating from the target node corresponding to the previous keyframe, determine the target edge corresponding to the node connection condition satisfied by the audio features of the current keyframe, and determine the target node pointed to by the determined target edge as the target node corresponding to the current keyframe, until the target node corresponding to the last keyframe in the emotion keyframe sequence is determined.

4. The method according to claim 1, characterized in that, The step of obtaining the expression base corresponding to each target node and assembling the expression base sequence according to the node connection order represented by the directed edges includes: The emotion subclass represented by each target node is determined from the set of emotion subclasses corresponding to the emotion types of the target audio. For each determined emotion subclass, obtain the target expression base used to represent the emotion subclass, and determine the obtained target expression base as the expression base corresponding to the target node representing the emotion subclass; The order of each target node is determined according to the node connection order represented by the directed edges, and the expression bases corresponding to each target node are combined into an expression base sequence according to the order of each target node.

5. The method according to claim 4, characterized in that, The step of obtaining the target expression base for representing the emotion subclass includes: Determine the expression base set for the emotion subclass; From the set of expression bases, select an expression base that matches the audio features corresponding to the target node representing the emotion subclass, and determine the selected expression base as the target expression base; or An expression base is randomly selected from the set of expression bases, and the selected expression base is determined as the target expression base.

6. The method according to claim 1, characterized in that, The method further includes: Pitch feature map and volume feature map are extracted from the target audio respectively; Based on the volume feature map, determine the volume change information of each audio frame in the target audio, and determine the first emotion keyframe sequence based on the volume change information; Based on the pitch feature map, determine the pitch change information of each audio frame in the target audio, and determine the second emotion keyframe sequence based on the pitch change information; The average value of the frame numbers of corresponding emotional keyframes in the first and second emotional keyframe sequences is calculated, and the emotional keyframe sequence of the target audio is determined based on the calculation result.

7. The method according to claim 6, characterized in that, The extraction of pitch feature map and volume feature map from the target audio includes: Obtain a volume curve based on the volume of each frame in the target audio, and a pitch curve based on the pitch of each frame in the target audio; The extreme points of the volume curve are determined, and the extreme points of the volume curve are merged to obtain the volume feature map of the target audio. The extreme points of the pitch curve are determined, and the extreme points of the pitch curve are merged to obtain the pitch feature map of the target audio.

8. The method according to claim 6, characterized in that, The step of determining the volume change information of each audio frame in the target audio based on the volume feature map, and determining the first emotion keyframe sequence based on the volume change information, includes: Calculate the derivative of each audio frame in the target audio in the volume feature map; The derivative is used to determine the starting point of the curve rise in the volume feature graph, at least one highest point of the curve rise after the starting point of the curve rise, and the ending point of the curve fall after the highest point of the curve rise. Obtain the target frame number corresponding to the curve's rising start point, the at least one curve's rising highest point, and the curve's falling end point, respectively. The audio frame corresponding to the target frame number is determined as the emotion key frame, thus obtaining the first emotion key frame sequence.

9. The method according to claim 1, characterized in that, Determining the expression sequence of the target audio based on the expression base sequence includes: Obtain the facial expression state description value corresponding to each facial expression base in the facial expression base sequence; Interpolate the expression state description values ​​corresponding to each expression base to obtain the target interpolation curve; From the target interpolation curve, determine the expression state description value of each target audio frame in the audio duration segment corresponding to the emotion type; Based on the facial expression state description value of each target audio frame in the audio duration segment, the facial expression base of each target audio frame is determined respectively; The expression sequence of the target audio is determined based on the expression base of each target audio frame in the audio duration segment.

10. The method according to claim 9, characterized in that, The step of interpolating the expression state description values ​​corresponding to each expression base to obtain the target interpolation curve includes: Linear interpolation is performed based on the facial expression state description values ​​corresponding to each facial expression base to obtain the first initial interpolation curve; A second initial interpolation curve is obtained by performing nonlinear interpolation processing based on the facial state description values ​​corresponding to each facial base; The first initial interpolation curve and the second initial interpolation curve are merged to obtain the target interpolation curve.

11. The method according to any one of claims 1 to 10, characterized in that, Determining the expression sequence of the target audio based on the expression base sequence includes: Based on the expression base sequence, the expression base of each audio frame in the target audio is determined; The facial expression bases of each audio frame are arranged into an initial facial expression sequence according to the temporal order of the audio frames; Obtain a first blink template and a second blink template; the change range of the eyelid state description value in the first blink template is greater than the change range of the eyelid state description value in the second blink template; The first blinking template is added to the first target expression base of the initial expression sequence, and the second blinking template is randomly added to each of the second target expression bases of the initial expression sequence to obtain the expression sequence of the target audio; Wherein, the first target expression base is the forward neighbor expression base of the expression base corresponding to the emotion type of the target audio; the second target expression base is an expression base other than the expression base corresponding to the emotion type of the target audio.

12. The method according to any one of claims 1 to 10, characterized in that, Determining the expression sequence of the target audio based on the expression base sequence includes: Based on the expression base sequence, the expression base of each audio frame in the target audio is determined; The facial expression bases of each audio frame are arranged into an initial facial expression sequence according to the temporal order of the audio frames; In the initial expression sequence, target expression bases are selected at preset intervals. For each selected target expression base, random eye state description values ​​are added to obtain the expression sequence of the target audio. The eye state description values ​​meet the eye perturbation range requirements.

13. An expression sequence generation device, characterized in that, The device includes: The state transition diagram determination module is used to determine the facial expression state transition diagram corresponding to the emotion type of the target audio. The node link determination module is used to determine the node link matching the emotional key frame sequence from the expression state transition graph based on the audio features of each key frame in the emotional key frame sequence of the target audio. The node link includes a target node corresponding to each key frame and a directed edge connecting the target node. The audio features of the key frame corresponding to the target node pointed to by the directed edge satisfy the node connection condition of the directed edge. The expression base acquisition module is used to acquire the expression base corresponding to each target node respectively, and to form an expression base sequence according to the node connection order represented by the directed edges. An expression sequence determination module is used to determine the expression sequence of the target audio based on the expression base sequence.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

Images