Facial action generation method and device, electronic equipment and readable storage medium

Abstract

The invention provides a facial action generation method and device, electronic equipment and a readable storage medium. The method comprises the steps of obtaining an emotion category and an audio text of a target emotion; determining an emotion change parameter corresponding to the emotion category, and generating first facial action data expressing the target emotion based on the emotion change parameter; determining second facial action data for reading the audio text; and based on the first expression locator and the second expression locator, fusing the first facial action data and the second facial action data to obtain target facial action data. According to the method and the device, the first facial action data for expressing the target emotion can be fused into the second facial action data for reading in the facial action.

Description

Technical Field

[0001] The present application relates to computer technology, and in particular to a facial action generation method, device, electronic device and readable storage medium. Background Art

[0002] Robotics is an interdisciplinary field that integrates knowledge and skills from key disciplines such as mechanical engineering, electronic engineering, computer science, and artificial intelligence. Bionic robotics is a key branch of robotics. It drives the robot's facial expression system by creating a blended shape data of the target object, making the robot's facial expressions more natural and diverse.

[0003] In the related technology, fine control of the expression of the target object is achieved by creating a blended deformation (BlendShape) model of the target object and motion capture technology. However, this method requires a lot of manpower and time costs to produce expression keyframes, and also has high requirements for hardware configuration. In addition, the motion data of the target object must be recorded in advance, which means that in actual applications, the target object can only mechanically repeat fixed movements and expressions, lacks the ability to adapt to changes, and thus it is difficult to achieve a natural and smooth interactive experience. Summary of the invention

[0004] The embodiments of the present application provide a facial action generation method, device, electronic device and readable storage medium, which can integrate first facial action data expressing a target emotion into second facial action data used for reading aloud in facial actions.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a facial action generation method, the method comprising:

[0007] Get the emotion category and audio text of the target emotion;

[0008] Determine an emotion change parameter corresponding to the emotion category, and based on the emotion change parameter, generate first facial action data expressing the target emotion, wherein the emotion change parameter is used to control the emotion change of the first facial action data;

[0009] Determining second facial action data for reading the audio text aloud, wherein a first expression locator included in the first facial action data is partially identical to a second expression locator included in the second facial action data;

[0010] Based on the first expression locator and the second expression locator, the first facial action data and the second facial action data are fused to obtain target facial action data.

[0011] An embodiment of the present application provides a facial action generation device, the device comprising:

[0012] A data collection module is used to obtain the emotion category and audio text of the target emotion;

[0013] A facial action data generation module, used to determine the emotion change parameter corresponding to the emotion category, and based on the emotion change parameter, generate first facial action data expressing the target emotion, wherein the emotion change parameter is used to control the emotion change of the first facial action data;

[0014] The facial action data generating module is further used to determine second facial action data for reading the audio text, wherein the first expression locator included in the first facial action data is partially identical to the second expression locator included in the second facial action data;

[0015] The data fusion module is used to fuse the first facial action data and the second facial action data based on the first expression locator and the second expression locator to obtain target facial action data.

[0016] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0017] A memory for storing computer executable instructions or computer programs;

[0018] The processor is used to implement the facial action generation method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the facial action generation method provided in the embodiment of the present application when executed by a processor.

[0020] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the facial action generation method provided in the embodiment of the present application is implemented.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] Through the emotion change parameters corresponding to the emotion category of the target emotion, the first facial action data expressing the target emotion is generated, so as to realize more flexible and simple control of the first facial action data of the target emotion. Through the audio text, the second facial action data for reading the audio text is determined. Based on the first expression locator and the second expression locator, the first facial action data and the second facial action data are fused to obtain the target facial action data, so as to integrate the first facial action data expressing the target emotion into the second facial action data for reading in the facial action, so as to realize a more natural and immersive human-computer communication experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a structural schematic diagram of the facial action generation system architecture provided by an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of the structure of an electronic device for facial action generation provided by an embodiment of the present application;

[0025] Figure 3 is a first flow chart of the facial action generation method provided by an embodiment of the present application;

[0026] Figure 4 is a second flow chart of the facial action generation method provided in an embodiment of the present application;

[0027] Figure 5 is a third flow chart of the facial action generation method provided in an embodiment of the present application;

[0028] Figure 6 is a fourth flow chart of the facial action generation method provided in an embodiment of the present application;

[0029] Figure 7 is a fifth flow chart of the facial action generation method provided in an embodiment of the present application;

[0030] Figure 8 is a sixth flow chart of the facial action generation method provided in an embodiment of the present application;

[0031] Fig. 9 It is a first principle schematic diagram of the facial action generation method provided by an embodiment of the present application;

[0032] Fig.10 It is a second principle schematic diagram of the facial action generation method provided in an embodiment of the present application.

[0033] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0035] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0036] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0037] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0038] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0039] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0040] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0041] 1) BlendShape model: It is a technology used in computer graphics and animation production. It allows complex facial expressions to be created by mixing the weight values ​​corresponding to multiple expression locators. It is usually used to simulate the facial movements of organisms, so that virtual characters can show rich emotions and expressions.

[0042] 2) BlendShape data: In computer graphics and animation production, BlendShape data refers to facial deformation data used to create BlendShape models. These data usually include a series of expression locators (Expression Targets) and corresponding weights (Weights). By adjusting the weights of the expression locators, countless different facial expressions can be created.

[0043] 3) Expression Targets: refers to the key points or markers used to identify and control specific facial muscles or areas in facial animation. These locators can be virtual points or physical markers. In the blendshape model system based on 52 expression locators provided by the Augmented Reality Kit (ARKit), the following are examples of some expression locators: left eye blink (eyeBlinkLeft), left eye looking down (eyeLookDownLeft), left eye looking at the tip of the nose (eyeLookInLeft), left eye looking left (eyeLookOutLeft), left eye looking up (eyeLookUpLeft).

[0044] 4) Frame Rate Value: It is a metric used to measure the update speed of a video, animation or image sequence. The unit is frames per second (fps), that is, the number of frames displayed per second. Frame rate value is an important concept in video production and game development, and is crucial to ensuring the smoothness of video or animation playback.

[0045] Embodiments of the present application provide a facial action generation method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can incorporate first facial action data expressing a target emotion into second facial action data used for reading aloud in facial actions.

[0046] The electronic device for facial action generation provided in the embodiments of the present application may be various types of terminals or servers, wherein the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services; the terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart TV, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and the present application does not limit this.

[0047] See also Figure 1 , Figure 1 It is a structural diagram of the facial action generation system architecture provided in the embodiment of the present application. In the facial action generation system 10 provided in the embodiment of the present application, in order to support a facial action generation application, the terminal 400 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0048] Terminal 400 may be used to obtain a facial action generation request including an emotion category for obtaining a target emotion and an audio text.

[0049] In some embodiments, a facial action generation plug-in may be implanted in the client running in terminal 400 to implement a facial action generation method locally on the client. For example, terminal 400 calls the facial action generation plug-in to implement the facial action generation method, generates first facial action data expressing the target emotion through the emotion change parameters corresponding to the emotion category of the target emotion, determines second facial action data for reading the audio text through the audio text, and fuses the first facial action data and the second facial action data based on the first expression locator and the second expression locator to obtain the target facial action data.

[0050] In some embodiments, after terminal 400 obtains a facial action generation request including an emotion category for obtaining a target emotion and an audio text, it calls a facial action generation interface of server 200 (which can be provided in the form of a cloud service), and server 200 implements a facial action generation method through a facial action generation plug-in, generates first facial action data expressing the target emotion through emotion change parameters corresponding to the emotion category of the target emotion, determines second facial action data for reading the audio text through the audio text, and fuses the first facial action data and the second facial action data based on the first expression locator and the second expression locator to obtain the target facial action data.

[0051] The facial action generation method provided in the embodiment of the present application can be implemented independently by a terminal or a server, or can be implemented jointly by a terminal and a server. For example, the terminal independently undertakes the facial action generation method described below, or the terminal sends a facial action generation request containing an emotion category for obtaining a target emotion and an audio text to the server. The server executes the facial action generation method based on the received facial action generation request containing an emotion category for obtaining a target emotion and an audio text, generates first facial action data expressing the target emotion through emotion change parameters corresponding to the emotion category of the target emotion, determines second facial action data for reading the audio text through the audio text, and fuses the first facial action data and the second facial action data based on the first expression locator and the second expression locator to obtain the target facial action data.

[0052] In some embodiments, the terminal or server can implement the facial action generation method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a live broadcast application; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0053] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device for generating facial movements provided by an embodiment of the present application, Figure 2 The electronic device 500 shown may be Figure 1 The terminal 400 or the server 200 in the electronic device 500 includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 2 Various buses are labeled as bus system 540 .

[0054] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0055] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0056] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, a hard disk drive, an optical disk drive, etc. The memory 550 may optionally include one or more storage devices that are physically located away from the processor 510.

[0057] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0058] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0059] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0060] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0061] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripherals and displaying content and information);

[0062] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions.

[0063] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The facial action generating device 555 stored in the memory 550 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a data acquisition module 5551, a facial action data generating module 5552, a data fusion module 5553 and a facial action data application module 5554. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0064] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the facial action generation method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0065] See also Figure 3 , Figure 3 is a first flow chart of the facial action generation method provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown are used for illustration. The facial action generation method provided in the embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. Therefore, the execution entity of each step will not be repeated below.

[0066] In step 101, the emotion category and audio text of the target emotion are obtained.

[0067] As an example, obtain audio data corresponding to the facial action to be generated of the target object, perform speech recognition processing on the audio data to obtain an audio text, perform emotion recognition processing on the audio text, obtain K emotion recognition results of the facial action to be generated, and determine the emotion category of the target emotion from the K emotion recognition results. Wherein, the emotion recognition result includes the emotion category and the duration corresponding to the emotion category. Determining the emotion category of the target emotion from the K emotion recognition results can be achieved in the following way: take the kth emotion recognition result as the target emotion, and take the emotion category of the kth emotion recognition result as the emotion category of the target emotion, where K is an integer greater than 0, k is an integer that increases in sequence, 1≤k≤K, speech recognition processing and emotion recognition processing can be achieved through services provided by a third-party service provider, or through an artificial intelligence model that has been trained locally, and there is no restriction here. Emotion category is a classification of human emotional experience, including happiness, sadness, anger, and sadness, etc. The target object can be a digital virtual object or a robot object, and there is no restriction here.

[0068] For example, audio data corresponding to the facial action to be generated is obtained, and speech recognition processing is performed on the audio data to obtain the audio text "The weather is really nice today, but I can't go out for an outing." Emotion recognition processing is performed on the audio text, and the emotion recognition result is that the emotion category of "The weather is really nice today" is happy, and the duration is 2 seconds, and the emotion category of "But I can't go out for an outing" is sad, and the duration is 3 seconds. The emotion category of "The weather is really nice today" corresponding to happy in the emotion recognition result is determined as the emotion category of the target emotion.

[0069] In step 102, an emotion change parameter corresponding to the emotion category is determined, and based on the emotion change parameter, first facial action data expressing the target emotion is generated.

[0070] As an example, the emotion change parameters corresponding to different emotion categories may be the same or different, and are not limited here. The first facial action data includes at least one frame of mixed shape data (blendshape data) arranged in a chronological order, wherein the blendshape data includes at least one-dimensional expression locator and a weight value corresponding to the expression locator, and the weight value is a real number greater than or equal to 0 and less than or equal to 1.

[0071] In some embodiments, the first facial action data includes N expression segments, where N is an integer greater than 1, see Figure 4 , Figure 4 is a second flow chart of the facial action generation method provided in an embodiment of the present application, Figure 3The "generating first facial action data expressing the target emotion based on the emotion change parameter" in step 102 shown can be achieved through the following steps 1021 to 1022, which are described in detail below.

[0072] In step 1021, based on the emotion category, an nth initial expression segment of the first facial action data is determined.

[0073] Wherein, n is an integer that increases successively, 1≤n≤N.

[0074] As an example, the same emotion category includes at least one emotion expression method, and different emotion expressions of the same emotion category correspond to different standard expression templates, that is, one emotion category corresponds to at least one standard expression template, and from at least one standard expression template corresponding to the emotion category, a standard expression template for constructing the nth initial expression segment is determined, wherein the initial expression segment includes at least one frame of standard expression templates arranged in chronological order, and at least one frame of standard expression template included in the initial expression segment is the same standard expression template, and the sum of the number of frames of the N expression segments is less than or equal to the number of continuous frames expressing the target emotion.

[0075] The standard expression template is a specific blendshape data used to express a specific emotion category. For example, in the blendshape model system based on 52 expression locators provided by the augmented reality kit (ARKit), the standard expression template corresponding to the sad category can be represented as a 52-dimensional blendshape data [0, 0.7, 0.7, 0.8, ..., 0, 0]. The standard expression template corresponding to the sad category uses the expression locators with indexes 1, 2, and 3 among the 52 expression locators, wherein the initial weight value of the expression locator with index 1 is 0.7, the initial weight value of the expression locator with index 2 is 0.7, and the initial weight value of the expression locator with index 3 is 0.8 (the weight values ​​of the expression locators corresponding to the remaining indices are 0). The list of relevant dimensions of the standard expression template can be represented as [1, 2, 3].

[0076] In some embodiments, the sum of the triggering probabilities of multiple standard expression templates corresponding to the same emotion category is 1. When the emotion category corresponds to multiple standard expression templates, "determining the standard expression template used to construct the nth initial expression segment" can be achieved through the following steps: separately determine the triggering probabilities of multiple standard expression templates corresponding to the emotion category; based on the triggering probabilities of multiple standard expression templates, select a target standard expression template; and use the target standard expression template as the standard expression template for constructing the nth initial expression segment.

[0077] For example, the emotion of the sad category corresponds to two different standard expression templates, namely standard expression template 1 and standard expression template 2. The triggering probability of standard expression template 1 is 0.6, and the triggering probability of standard expression template 2 is 0.4. Based on the triggering probabilities of standard expression template 1 and standard expression template 2, the selected target standard expression template has a 0.6 probability of being standard expression template 1 and a 0.4 probability of being standard expression template 2.

[0078] In the embodiment of the present application, the same emotion category includes at least one emotion expression method, and different emotion expression methods of the same emotion category correspond to different standard expression templates. Different standard expression templates provide different expression forms for each basic emotion category. The different expression forms of the standard expression templates are reflected in that, for the emotion category of anger, standard expression template A expresses anger by frowning and squinting, and standard expression template B expresses anger by staring and squinting, thereby making the target object more delicate and real in emotional expression, and increasing the richness of expression.

[0079] In some embodiments, the "continuous number of frames expressing the target emotion" can be determined by the following steps: obtaining the frame rate value of the target facial action playback; based on the frame rate value and the duration of the target emotion, determining the continuous number of frames expressing the target emotion.

[0080] As an example, the frame rate value of the target facial action playback refers to the number of frames rendered per second for the target object's facial action. The unit of the frame rate value is frames per second (fps). The number of frames that last to express the target emotion is determined based on the product of the frame rate value and the duration of the target emotion.

[0081] For example, when the frame rate value is 60fps and the duration of the target emotion is 2 seconds, the number of frames that express the target emotion is the product of the frame rate value and the duration, which is 120 frames.

[0082] In step 1022, based on the emotion change parameter, the nth initial expression segment is deformed to obtain the nth expression segment.

[0083] As an example, the initial expression segment includes at least one frame of standard expression templates arranged in chronological order. Based on the emotion change parameters, each standard expression template included in the nth initial expression segment is deformed to obtain the nth expression segment, wherein the nth expression segment includes at least one frame of standard expression templates arranged in chronological order.

[0084] In the embodiment of the present application, by using standardized standard expression templates, the rendering performance of the expression action of the target object can be optimized, the utilization efficiency of computing resources can be improved, and at the same time, there is no need to create a new template from scratch for each new emotion category. The standard expression template of a new emotion category can be quickly created by adjusting the existing standard expression template, thereby saving development time and human resources. In addition, by adjusting the emotion change parameters, more flexible and simple control of the expression change of the nth expression segment can be achieved.

[0085] In some embodiments, the emotion change parameters include the emotion change degree, emotion change amplitude, emotion change frame number and emotion retention frame number. The nth expression segment includes the emotion change segment and the emotion retention segment. Figure 5 , Figure 5 is a third flow chart of the facial action generation method provided in an embodiment of the present application, Figure 4 The illustrated step 1022 can be implemented by executing the following steps 201 to 203 for the standard expression template included in the nth initial expression segment, which will be described in detail below.

[0086] In step 201, based on the degree of emotion change, the amplitude of emotion change and the number of emotion change frames, the weight value of the third expression locator included in the I-frame standard expression template is determined.

[0087] Among them, I is the number of emotion change frames.

[0088] As an example, an emotion change segment includes an emotion change segment from weak to strong and an emotion change segment from strong to weak, and the emotion change frame number includes the emotion change frame number from weak to strong and the emotion change frame number from strong to weak. In the case where the emotion change segment is an emotion change segment from weak to strong, the weight value of the third expression locator included in the I-frame standard expression template is determined based on the emotion change degree, the emotion change amplitude and the emotion change frame number; in the case where the emotion change segment is an emotion change segment from strong to weak, the weight value of the third expression locator included in the I-frame standard expression template is determined based on the emotion change degree, the emotion change amplitude and the emotion change frame number.

[0089] In some embodiments, the third expression locator includes a fifth expression locator, a sixth expression locator, and a seventh expression locator, there is a facial symmetry relationship between the fifth expression locator and the sixth expression locator, the seventh expression locator is an expression locator in the third expression locator that is different from the fifth expression locator and the sixth expression locator, Figure 5The shown step 201 can be implemented by the following steps: based on the emotion change amplitude and the emotion change frame number, determine the weight value of the fifth expression locator and the weight value of the seventh expression locator included in the I frame standard expression template; based on the weight value of the fifth expression locator, determine the weight value of the sixth expression locator.

[0090] As an example, facial symmetry relationship refers to the left-right symmetry relationship of the facial structure of a biological being. For example, in the blendshape model system based on 52 expression locators provided by ARKit, the expression locator "left eye blink" and the expression locator "right eye blink" have a left-right symmetric relationship in the facial structure. After determining the weight value of the fifth expression locator "left eye blink" included in the I-frame standard expression template based on the amplitude of emotion change and the number of frames of emotion change, there is no need to calculate the weight value of the sixth expression locator "right eye blink", and the weight value of the fifth expression locator "left eye blink" can be directly used as the weight value of the sixth expression locator "right eye blink".

[0091] The embodiment of the present application ensures that the left and right sides of the target object's face remain consistent by making the weight values ​​of the expression locators in the existing facial symmetry relationship consistent, thereby avoiding visual unnaturalness caused by asymmetrical expressions, and at the same time reducing the number of expression locators that need to be independently controlled, thereby saving computing resources and improving the efficiency of expression control.

[0092] In some embodiments, see Figure 6 , Figure 6 is a fourth flow chart of the facial action generation method provided in an embodiment of the present application, Figure 5 The illustrated step 201 may be implemented by the following steps 2011 to 2013, which are described in detail below.

[0093] In step 211, based on the number of emotion change frames, an I-frame standard expression template is determined from the nth initial expression segment.

[0094] As an example, in the case where the emotion change segment is a segment where the emotion changes from weak to strong, obtain the first emotion change frame number range, and randomly select I frame of emotion change from weak to strong from the first emotion change frame number range; in the case where the emotion change segment is a segment where the emotion changes from strong to weak, obtain the second emotion change frame number range, and randomly select I frame of emotion change from strong to weak from the second emotion change frame number range, and determine I frame standard expression template from the nth initial expression segment, wherein the first emotion change frame number range and the second emotion change frame number range may be the same or different, and there is no limitation here.

[0095] For example, when the emotion change segment is a segment where the emotion changes from weak to strong, obtain the first emotion change frame number range [4, 10], randomly select 6 frames of emotion change from weak to strong from the first emotion change frame number range, and determine 6 frames of standard expression templates from the nth initial expression segment.

[0096] In step 2012, the range of emotion change is determined based on the degree of emotion change, the amplitude of emotion change, and the initial weight value of the third expression locator included in the I-frame standard expression template.

[0097] As an example, during an emotion change process, the emotion change range refers to the range of change of the weight value of the third expression locator between the emotion minimum value and the emotion maximum value, the emotion minimum value is the minimum value that the weight value of the third expression locator may reach, the emotion maximum value is the maximum value that the weight value of the third expression locator may reach, and the emotion change amplitude is the numerical value used to determine the emotion maximum value.

[0098] In the embodiment of the present application, by randomly determining the number of emotion change frames within the range of emotion change frames, the emotion change can be made more natural and unpredictable, thus avoiding mechanical and stereotyped animation performance.

[0099] In some embodiments, Figure 6 The step 2012 shown can be achieved through the following steps: based on the degree of emotion change, the amplitude of emotion change and the initial weight value, determine the emotion maximum change range; determine the random number determined from the emotion maximum change range as the emotion maximum; determine the weight value of the expression locator included in the last frame standard expression template of the n-1th expression segment as the emotion minimum; construct the emotion change range based on the emotion maximum and the emotion minimum.

[0100] As an example, a random number determined from the emotion maximum variation range is determined as the emotion maximum, and the weight value of the expression locator included in the standard expression template of the last frame of the n-1th expression segment is determined as the emotion minimum. Based on the emotion maximum and the emotion minimum, an emotion variation range is constructed, wherein for the first expression segment, the emotion minimum is a random number that is 0 or close to 0.

[0101] For example, when the target emotion category is sadness, the standard expression template corresponding to the sadness category is [0, 0.7, 0.7, 0.8, ..., 0, 0]. Taking the third expression locator with index 1 as an example, the minimum value of the maximum emotion change range of the third expression locator is the initial weight value 0.7, and the emotion change range is a coefficient sequence composed of 52 coefficients [1.0, 0.4, ..., 0.5, ..., 0.2, 0.2, ..., 0.5, 0.5]. Calculate the initial weight value 0.7 and the emotion change range indexed by 1. The product of the coefficient 0.4 is determined, and the above product 0.28 is determined as the minimum value of the emotion maximum variation range of the third expression locator, and the emotion maximum variation range is obtained from the minimum value 0.28 of the emotion maximum variation range to the maximum value 0.7 of the emotion maximum variation range. The emotion maximum value is randomly determined as 0.5 from the emotion maximum variation range [0.28, 0.7], and the weight value 0.2 of the expression locator of the standard expression template of the last frame of the n-1th expression fragment is determined as the emotion minimum value, and the emotion variation range [0.2, 0.5] is obtained.

[0102] In some embodiments, "determining the maximum emotional change range based on the degree of emotional change, the amplitude of emotional change and the initial weight value" can be achieved through the following steps: when the degree of emotional change is less than the threshold value of the degree of emotional change, the product of the amplitude of emotional change, the initial weight value and the first coefficient is determined as the first minimum value, and the first maximum value is determined based on the first product, the first ratio and the first minimum value, and the maximum emotional change range is constituted based on the first maximum value and the first minimum value; when the degree of emotional change is greater than or equal to the threshold value of the degree of emotional change, the product of the amplitude of emotional change and the initial weight value is determined as the second minimum value, and the second maximum value is determined based on the product of the first difference and the second difference, the threshold value of the degree of emotional change and the second minimum value, and the maximum emotional change range is constituted based on the second maximum value and the second minimum value.

[0103] Among them, the first product is the product of the emotion change amplitude, the initial weight value and the second coefficient, the first ratio is the ratio of the emotion change degree and the emotion change degree threshold, the first difference is the difference between the initial weight value and the second minimum value, and the second difference is the difference between the emotion change degree and the emotion change degree threshold.

[0104] As an example, the maximum emotion change range is determined by comparing the relationship between the emotion change degree and the emotion change degree threshold, wherein the emotion change threshold is set to distinguish the strength of the emotion change degree.

[0105] When the degree of emotion change is less than the emotion change threshold, the emotion maximum value change range is composed of the first minimum value and the first maximum value. The product of the emotion change amplitude, the initial weight value and the first coefficient is determined as the first minimum value by the first minimum value calculation formula (1.1). The formula (1.1) is as follows:

[0106] min_val=scale*value*α (1.1)

[0107] Among them, min_val is the first minimum value, scale is the amplitude of emotion change, value is the initial weight value, α is the first coefficient, and α can be set to 0.8.

[0108] The first maximum value is determined by the first maximum value calculation formula (1.2) based on the first product, the first ratio and the first minimum value. The formula (1.2) is as follows:

[0109]

[0110] Among them, max_val is the first maximum value, min_val is the first minimum value, (1-α)*scale*value is the first product, scale is the amplitude of emotion change, value is the initial weight value, and (1-α) is the second coefficient; is the first ratio, emo_level is the degree of emotion change, and emo_level_cut is the threshold of emotion change degree.

[0111] When the degree of emotion change is greater than or equal to the emotion change threshold, the emotion maximum value change range is composed of the second minimum value and the second maximum value. The second minimum value is calculated by the second minimum value calculation formula (1.3), and the product of the emotion change amplitude and the initial weight value is determined as the second minimum value. The formula (1.3) is as follows:

[0112] min_val = scale * value (1.3)

[0113] Among them, min_val is the second minimum value, scale is the amplitude of emotion change, and value is the initial weight value.

[0114] The second maximum value is determined by the second maximum value calculation formula (1.4) based on the product of the first difference and the second difference, the emotion change degree threshold and the second minimum value. The formula (1.4) is as follows:

[0115]

[0116] Among them, max_val is the second maximum value, min_val is the second minimum value, value-min_val is the first difference, value is the initial weight value, emo_level-emo_level_cut is the second difference, emo_level is the degree of emotion change, and emo_level_cut is the threshold value of the degree of emotion change.

[0117] In step 2013, based on the emotion variation range, a weight value of the third expression locator included in the I-frame standard expression template is determined.

[0118] As an example, based on the range of emotion changes, I weight values ​​of each third expression locator included in the I-frame standard expression template are determined frame by frame.

[0119] In some embodiments, Figure 6 Step 2013 shown can be implemented by performing the following steps for the third expression locator of each dimension: determining I random values ​​from the range of emotion changes; sorting the I random values ​​according to the order of emotion changes, and determining the sorted I random values ​​as the weight value of the third expression locator.

[0120] As an example, the order of emotion change includes the order of emotion change from weak to strong and the order of emotion change from strong to weak, I random values ​​are determined from the range of emotion change, and when the order of emotion change is the order of emotion change from weak to strong, the I random values ​​are sorted from small to large; when the order of emotion change is the order of emotion change from strong to weak, the I random values ​​are sorted from large to small, and the sorted I random values ​​are determined as the weight value of the third expression locator.

[0121] For example, six random values ​​0.3, 0.33, 0.40, 0.25, 0.29, and 0.48 are determined from the emotion change range [0.2, 0.5]. When the emotion change order is from weak to strong, the six random values ​​are sorted from small to large to obtain 0.25, 0.29, 0.3, 0.33, 0.40, and 0.48. The sorted six random values ​​are determined in turn as the weight values ​​of the third expression locator.

[0122] In the embodiment of the present application, the randomness of the severity of the emotion change is increased by randomly determining the emotion maximum value from the emotion maximum value variation range and randomly determining the weight value of the third expression locator from the emotion variation range.

[0123] In step 202, based on the number of emotion-preserving frames, a weight value of a fourth expression locator included in the J-frame standard expression template is determined.

[0124] Among them, J is the number of emotion-preserving frames.

[0125] As an example, an emotion holding segment includes an emotion peak holding segment and an emotion silence holding segment, and the emotion holding frame number includes an emotion peak holding frame number and an emotion silence holding frame number. When the emotion holding segment is an emotion peak holding segment, the weight value of the fourth expression locator included in the J frame standard expression template is determined based on the emotion peak holding frame number; when the emotion holding segment is an emotion silence holding segment, the weight value of the fourth expression locator included in the J frame standard expression template is determined based on the emotion silence holding frame number, wherein emotion holding refers to a state in which the weight value of the fourth expression locator remains unchanged, emotion peak holding refers to a state in which the weight value of the fourth expression locator remains unchanged at the emotion maximum value, and emotion silence holding refers to a state in which the weight value of the fourth expression locator remains unchanged at the emotion minimum value.

[0126] In some embodiments, see Figure 7 , Figure 7 is a fifth flow chart of the facial action generation method provided in an embodiment of the present application, Figure 5 The illustrated step 202 may be implemented by following steps 2021 to 2023, which are described in detail below.

[0127] In step 2021, based on the number of emotion retention frames, J frames of standard expression templates are determined from the nth initial expression segment.

[0128] As an example, when the emotion holding segment is an emotion peak holding segment, the emotion peak holding frame number range is obtained, and J emotion peak holding frame number frames are randomly selected from the emotion peak holding frame number range; when the emotion holding segment is an emotion silence holding segment, the emotion silence holding frame number range is obtained, and J emotion silence holding frame number frames are randomly selected from the emotion silence holding frame number range, and J frames of standard expression templates are determined from the nth initial expression segment.

[0129] For example, when the emotion holding segment is an emotion peak holding segment, the emotion peak holding frame number range [8, 20] is obtained, 15 emotion peak holding frames are randomly selected from the emotion peak holding frame number range, and 15 frames of standard expression templates are determined from the nth initial expression segment; when the emotion holding segment is an emotion silence holding segment, the emotion silence holding frame number range [30, 120] is obtained, 60 emotion silence holding frames are randomly selected from the emotion silence holding frame number range, and 60 frames of standard expression templates are determined from the nth initial expression segment.

[0130] In step 2022, a target standard expression template that is located before the J-frame standard expression template and adjacent to the J-frame standard expression template is determined.

[0131] As an example, when the emotion holding segment is an emotion peak holding segment, the target standard expression template is the standard expression template of the last frame of the nth expression segment where the emotion changes from weak to strong; when the emotion holding segment is an emotion silence holding segment, the target standard expression template is the standard expression template of the last frame of the nth expression segment where the emotion changes from strong to weak.

[0132] In step 2023, the weight value of the expression locator included in the target standard expression template is determined as the weight value of the fourth expression locator included in the J frame standard expression template.

[0133] As an example, the weight value of each expression locator included in the target standard expression template is determined frame by frame as the weight value of the corresponding fourth expression locator in the J-frame standard expression template.

[0134] The embodiment of the present application makes the expression changes more natural and unpredictable by randomly selecting the number of frames within a specific emotion retention frame range, avoids fixed patterns, and thus enhances the randomness of the emotion change segments. While randomly selecting the number of frames to retain the emotion, the naturalness of the emotion transition is maintained by selecting adjacent target standard expression templates, thereby ensuring the continuity of the emotion change.

[0135] In step 203, based on the weight value of the third expression locator, an emotion change segment is formed, and based on the weight value of the fourth expression locator, an emotion retention segment is formed.

[0136] See also Fig. 9 , Fig. 9 This is a first principle schematic diagram of the facial action generation method provided in an embodiment of the present application. Based on the weight value of the third expression locator (the expression locator with an index value equal to 0), an emotion change segment 901 from weak to strong and an emotion change segment 902 are respectively constituted. Based on the weight value of the fourth expression locator, an emotion peak holding segment 903 and an emotion silence holding segment 904 are respectively constituted.

[0137] In step 103, second facial action data for reading an audio text is determined.

[0138] The first expression locator included in the first facial action data is partially identical to the second expression locator included in the second facial action data.

[0139] As an example, all expression locators used for facial action generation are divided to obtain expression locators related only to lip-type drive, expression locators related only to emotion drive, expression locators related to both lip-type drive and emotion drive, and expression locators related to neither lip-type drive nor emotion drive. The first facial action data includes expression locators related only to emotion drive and expression locators related to both lip-type drive and emotion drive. The second facial action data is used to read audio text and includes expression locators related only to lip-type drive and expression locators related to both lip-type drive and emotion drive. The first expression locators included in the first facial action data are partially the same as the second expression locators included in the second facial action data, and both include expression locators related to both lip-type drive and emotion drive.

[0140] In step 104, the first facial action data and the second facial action data are fused based on the first expression locator and the second expression locator to obtain target facial action data.

[0141] In some embodiments, see Figure 8 , Figure 8 is a sixth flow chart of the facial action generation method provided in an embodiment of the present application, Figure 3 The illustrated step 104 may be implemented by following the steps 1041 to 1042 , which are described in detail below.

[0142] In step 1041, the sum of the first weight value and the second weight value is determined as the target weight value of the first target expression locator.

[0143] Among them, the first target expression locator is the expression locator in the first expression locator that is the same as the second expression locator, the first weight value is the weight value of the first target expression locator included in the first facial action data, and the second weight value is the weight value of the first target expression locator included in the second facial action data.

[0144] As an example, when the sum of the first weight value and the second weight value is less than or equal to 1, the sum is determined as the target weight value of the first target expression locator; when the sum of the first weight value and the second weight value is greater than 1, the target weight value of the first target expression locator is determined to be 1.

[0145] For example, the first facial action data includes expression locator 1, expression locator 2 and expression locator 3, the second facial action data includes expression locator 3, expression locator 4 and expression locator 5, the first target expression locator is expression locator 3, the first weight value is the weight value sequence of expression locator 3 included in the first facial action data [1.0, 0.4, 0.5, 0.2, 0.2, 0.5, 0.5], the second weight value is the weight value of expression locator 3 included in the second facial action data [0.2, 0.5, 0.5, 1.0, 0.4, 0.5, 0.2], the first weight value and the second weight value are added frame by frame to obtain the target weight value sequence of the first target expression locator [1.0, 0.9, 1.0, 1.0, 0.6, 1.0, 0.7].

[0146] In step 1042, the target weight value of the first target expression locator and the weight value of the second target expression locator are combined to determine the target facial action data.

[0147] The second target expression locator is an expression locator among the first expression locator and the second expression locator except the first target expression locator.

[0148] For example, the first facial action data includes expression locator 1, expression locator 2 and expression locator 3, the second facial action data includes expression locator 3, expression locator 4 and expression locator 5, the first target expression locator is expression locator 3, the second target expression locator includes expression locator 1 and expression locator 2 of the first facial action data, expression locator 4 and expression locator 5 of the second facial action data, the target weight value of the first target expression locator and the weight value of the second target expression locator are combined to determine the target facial action data, the target facial action data includes expression locator 1 and expression locator 2 of the first facial action data, expression locator 4 and expression locator 5 of the second facial action data and the target weight value of the first target expression locator.

[0149] The embodiment of the present application achieves more flexible emotional expression by combining the weight values ​​of two different facial action data, so that the target object can also show rich emotional changes when speaking, thereby improving the flexibility of emotional changes.

[0150] In some embodiments, Figure 3 After step 104 shown, the following steps may also be performed: smoothing the target facial action data to obtain smoothed target facial action data; and applying the smoothed target facial action data to the face of the target object.

[0151] As an example, obtain the frame rate value of facial action playback, determine the smoothing frame number m based on the frame rate value, wherein the frame rate value is proportional to the smoothing frame number; determine the target frame in the target facial action data, and based on the smoothing frame number m, smooth the standard expression templates of the target frame, the first m frames of the target frame, and the last m frames of the target frame to obtain smoothed target facial action data, and apply the smoothed target facial action data to the face of the target object, wherein the target frame is any frame of data in the target facial action data.

[0152] In the embodiment of the present application, by smoothing the target facial action data and applying weighted smoothing of the previous and next frames in each dimension, it can be ensured that each frame of action can utilize the information of the previous and next frames, thereby achieving a smoother and more natural transition. Not only can the virtual human's movements be more natural and smooth, but also the movements of the driving hardware can be more natural and smooth, thereby improving the driving quality of the target object's facial movements, making facial movements more natural and smooth, thereby enhancing the user experience. In addition, it also avoids damage to the hardware motor due to sudden and large movements.

[0153] The following describes an exemplary application of the embodiment of the present application in a blendshape model system based on 52 expression locators provided by an actual ARKit.

[0154] With the continuous advancement of artificial intelligence technology, the application scope of digital humans is gradually expanding. Whether it is animation production, movie special effects, game characters, or advertising, customer service, live broadcast interaction and other fields, digital humans are playing an increasingly important role. However, in actual application scenarios, relying solely on model construction is far from enough. Digital humans also need to have the ability to capture movements and expressions in real time so that they can communicate and interact with humans more naturally and effectively. In addition, robotics technology is booming, and the field of bionic robots has become a key branch, enabling robots to perform lively movements and show rich emotions, which has become one of the important trends in the research and development of bionic robots.

[0155] However, in related technologies, the motion data of virtual humans must be recorded in advance, which means that in actual applications, virtual humans can only mechanically repeat fixed movements and expressions and lack the ability to adapt to changing circumstances, making it difficult to achieve a natural and smooth interactive experience. It also requires a lot of manpower and time to produce expression keyframes, and has high requirements for hardware configuration.

[0156] In order to solve the above problems, the embodiment of the present application proposes a facial action generation method, which can drive the virtual human face to show corresponding emotional expressions, and through integration and connection with hardware, it can achieve precise control of the robot's facial emotions.

[0157] 1. Incomplete decoupling of content and emotion.

[0158] The content and emotion of the 52 blendshape weights are not completely decoupled, that is, there is a dimension common to the content and emotion (the dimension is equivalent to the emotion identifier above), and the 52-dimensional blendshape is split into four categories, respectively:

[0159] 1) Dimensions that are only related to the content of speech, that is, dimensions related to lip-sync driving (equivalent to the expression locators that are only related to lip-sync driving mentioned above).

[0160] 2) Dimensions related only to emotions, namely, dimensions related to eyebrows, eyes, and cheeks (equivalent to the expression locators related only to emotional drive mentioned above).

[0161] 3) Dimensions that are somewhat related to both speech content and emotion, namely, dimensions related to mouth corners (equivalent to the expression locators mentioned above that are related to both lip shape drive and emotion drive).

[0162] 4) Dimensions that are not closely related to speech content and emotions, that is, blendshape dimensions corresponding to actions that cannot be performed by the robot hardware (equivalent to the expression locators mentioned above that are not related to lip-driven and emotion-driven). Even if there are data in these dimensions, the robot's facial movements and expressions will not be much different.

[0163] Since different emotional expressions show inconsistent facial change areas and change intensities, different emotional expressions may require the use of different dimension combinations.

[0164] 2. Obtain the dimension set used by the standard template expression (equivalent to the standard expression template mentioned above).

[0165] Create multiple standard template expressions according to the required emotional categories.

[0166] Since the same emotion may involve multiple different forms of expression and different degrees of emotional expression, one emotion can be used to create multiple standard template expressions with different facial expressions. The realization of different degrees of emotion will be introduced later.

[0167] Among them, different expressions of the same emotion may use different blendshape dimensions. Therefore, the list of relevant dimensions of emotions is a set of dimensions used by standard template expressions corresponding to multiple different expressions of an emotion category, and the dimension set of standard template expressions is a set of dimensions of the list of relevant dimensions of multiple emotion categories.

[0168] For example, for the three emotion categories of happy, sad, and angry, happy emotion uses a standard template expression in one form of expression, and the list of related dimensions of happy includes: [0, 1, 4, 7, 8, 15, 16, 20, 21] dimensions; sad emotion has two different forms of standard template expressions, and the list of related dimensions of sad includes: [0, 1, 4, 35, 36, 40, 41] and [1, 2, 3, 40, 41] dimensions; angry emotion uses a standard template expression in one form of expression, and the list of related dimensions of angry includes: [8, 9, 10, 15, 16, 20, 21] dimensions.

[0169] The dimension set of the standard template expression is the set of the above dimensions, that is, [0, 1, 4, 7, 8, 15, 16, 20, 21, 35, 36, 40, 41, 2, 3, 9, 10]. The index values ​​corresponding to the above dimensions do not represent the actual index values ​​of the 52-dimensional blendshape in ARKit. They are only used to illustrate the relationship between the dimensions used to make standard template expressions and the dimensions used by a single expression.

[0170] 3. Construct standard template expression data.

[0171] A total of 4 standard template expressions of 3 emotion categories were produced.

[0172] Based on the blendshape dimension set used by the standard template expressions, the values ​​corresponding to each dimension when the emotional degree is the maximum under different expression forms of each category of emotion are obtained (equivalent to the weight value above). Among them, the blendshape data of the standard template expressions is 52 dimensions, and the value range corresponding to the used dimensions is [0, 1.0], and the values ​​corresponding to the unused dimensions are set to 0, thus obtaining 4 52-dimensional standard template expression data.

[0173] 4. Control of changes in emotion-related dimensions.

[0174] The change control of the weight values ​​corresponding to the emotion-related dimensions can be achieved through the amplitude and range of emotion changes, the number of frames for emotions to change from weak to strong, the change curve from weak to strong, the number of frames maintained in a strong emotion state, the number of frames for emotions to change from strong to weak, the change curve from strong to weak, the frequency of emotion triggering and the triggering probability of different expressions of the same emotion.

[0175] See also Fig. 9 , emotion-related dimensions include 42 dimensions such as [0, 1, ..., 41], Fig. 9The horizontal direction of the coordinate system represents the number of frames of the blendshape data, and the vertical direction represents the value of the corresponding dimension. The expression of emotion includes emotion enhancement, emotion retention, emotion weakening, and no emotion state. The duration of emotion expression is different, and the blendshape data change curve is also different.

[0176] For example, for the emotion in the sad category, there are two different expressions, using the [0, 1, 4, 35, 36, 40, 41] dimension and the [1, 2, 3, 40, 41] dimension respectively;

[0177] The standard template expression data of the sad emotion is data1 = [0.6, 0.7, ..., 0.9, ..., 0.8, 0.8, ..., 0.5, 0.5, ...] with a total of 52 dimensions, and the values ​​of the omitted dimensions are all 0.

[0178] The standard template expression data 2 for the sad category of emotion is data2 = [0, 0.7, 0.7, 0.8, ..., 0.8, 0.8, ...] with a total of 52 dimensions, and the values ​​of the omitted dimensions are all 0.

[0179] See also Fig.10 , Fig.10 This is a second principle diagram of the facial action generation method provided in an embodiment of the present application. Taking the standard template expression data1 of the sad emotion category as an example, the numerical changes of the emotion-related dimensions in the 52-dimensional data included in data1 (hereinafter referred to as emotion-related dimensions) are explained.

[0180] Step 1, set the amplitude and range of emotion change: emotion changes between the minimum and maximum values, that is, emotion-related dimensions have minimum and maximum values ​​respectively, the range of emotion change is from the minimum to the maximum value, the range of emotion maximum value change is [data1*s1, data1*1.0], where s1 is the coefficient (equivalent to the emotion change amplitude above), s1 = [1.0, 0.4, ..., 0.5, ..., 0.2, 0.2, ..., 0.5, 0.5, ...] a total of 52 dimensions, the emotion minimum value is the value at the end of the last emotion trigger (when the emotion is triggered for the first time, the emotion minimum value is 0).

[0181] Step 2, set the number of frames from weak to strong emotion: set the range of nframe1, the number of frames from weak to strong emotion, to [4, 10]. When nframe1=6, 6 frames of data are required to transition from the minimum value of emotion to the maximum value of that emotion. Then the number of transition frames for emotion-related dimensions is 6 frames. Through these 6 frames of transition, the emotion-related dimensions reach the maximum value of that emotion expression. The maximum value of each emotion expression is not necessarily the same, thereby enhancing the diversity of emotion changes.

[0182] Step 3, set the change curve data from weak to strong: when it takes 6 frames for the emotion to change from weak to strong, the value corresponding to each emotion-related dimension is a random number between the minimum and maximum values. The 6 random numbers of each dimension of the emotion-related dimension are arranged from small to large to form a change curve from weak to strong for that dimension.

[0183] Step 4, setting the number of frames to be maintained in a strong emotion state: setting the range of the number of frames nframe_emo to be maintained in a strong emotion state is [8, 20]. When nframe_emo=15, the value corresponding to the emotion-related dimension is maintained at 15 frames.

[0184] Step 5, setting the number of frames from strong to weak emotions: setting the number of frames nframe2 from strong to weak emotions to a range of [8, 16], see the steps for setting the number of frames from weak to strong emotions above, which will not be repeated here.

[0185] Step 6, setting the change curve from strong to weak: refer to the steps for setting the change curve data from weak to strong above, which will not be repeated here.

[0186] Step 7, set the frequency of emotion triggering: The frequency of emotion triggering refers to the frequency at which the emotion becomes weak and remains for a certain number of frames, and then changes from no emotion to emotion. The range of nframe_normal is set to [30, 120]. When nframe_normal = 60, the value corresponding to the emotion-related dimension is maintained for 60 frames after the last emotion ends, that is, the emotion-related dimensions in the 52-dimensional data are maintained for 60 frames respectively, among which the data maintained by the emotion uses the data after the last emotion ends.

[0187] You can also set the triggering probability of different expressions of the same emotion: for example, the emotion of sadness has two different expressions, and the probabilities are set to p1 = 0.4 and p2 = 0.6 respectively. Assuming that the same emotion has multiple different expressions, the sum of the probabilities is 1. This makes different expressions of the same emotion appear with a certain probability.

[0188] Through the above steps 1 to 7, emotion-related dimension data, namely emotion data (equivalent to the first facial action data above), is constructed.

[0189] 5. Control the numerical changes of emotion-related dimensions and adjust them symmetrically.

[0190] Since a normal human face is basically symmetrical on the left and right, the control of facial expressions must also take into account left-right symmetry. Based on this, from the emotion-related dimensions, determine those dimensions a (equivalent to the fifth expression locator mentioned above) and dimension b (equivalent to the sixth expression locator mentioned above) that include both the left and right sides. You can only control the value corresponding to dimension a (the value on one side of the face), and the value corresponding to dimension b directly reuses the value corresponding to dimension a (the data on the other side uses the same value).

[0191] 6. Add emotional expression to basic lip shape data.

[0192] From the 52-dimensional blendshape data that drives the virtual human's facial expressions, the dimensions that are only related to the speech content are split out. When the algorithm is processed, when the same text is spoken under different emotions, the corresponding lip shape data (equivalent to the second facial action data above) is constructed. Among them, the dimensions included in the lip shape data that are only related to the speech content remain unchanged, and then the basic lip shape is maintained to ensure the consistency between the speech content and the lip shape. Among them, the lip shape data is also 52-dimensional, and the value range corresponding to the dimension related to the speech content is [0, 1.0], and the value of other dimensions is 0.

[0193] The lip-sync data is fused with the data of emotion-related dimensions, wherein the emotion-related dimensions include dimensions that are only related to emotions and dimensions that are somewhat related to both speech content and emotions.

[0194] For dimensions that are related to both speech content and emotion, the values ​​corresponding to the above dimensions are the superposition of the dimensional data related to emotion and the dimensional data only related to speech content, that is, the values ​​corresponding to the above dimensions of lip shape data and the above dimensions of emotion data are added together. When the added result exceeds 1.0, it is set to 1.0.

[0195] For example, when the dimensions used for lip shape data are [0, 1, 48, 49], and the emotion-related dimensions include [0, 1, 4, 7, 8, 15, 16, 20, 21, 35, 36, 40, 41, 2, 3, 9, 10], it can be seen that the data of the basic lip shape used in the 48th and 49th dimensions remain unchanged, the data of the 0th and 1st dimensions use the superposition of the two, and the remaining [4, 7, 8, 15, 16, 20, 21, 35, 36, 40, 41, 2, 3, 9, 10] dimensions use the values ​​corresponding to the emotion-related dimensions.

[0196] 7. Expression and control of emotions when there is no speaking content.

[0197] This is the same as the process of processing the lip shape related dimensions when speaking, which is equivalent to a special case where the values ​​of the lip shape related dimensions are all 0, and will not be elaborated here.

[0198] 8. Data smoothing.

[0199] The numerical value corresponding to each emotion-related dimension and the numerical value corresponding to the lip shape-related dimension in the dimension set of the standard template expression are smoothed.

[0200] For example, emotion-related dimensions include [0, 1, 4, 35, 36, 40, 41], and these seven dimensions are smoothed separately. Considering that some of the dimensions are symmetrical, the dimension on one side of the symmetrical dimension pair can also be smoothed, and the corresponding dimension on the other side takes the same value.

[0201] The smoothing method is: get the frame rate value of the blendshape data, and determine the number of frames for smoothing filtering based on the frame rate value. That is, when the frame rate value is high, more frames need to be smoothed at the same time, and when the frame rate value is low, fewer frames need to be smoothed. When the frame rate value is 30pfs, a total of 7 frames of data are used for smoothing, and when the frame rate value is 10pfs, 3 frames of data are used for smoothing.

[0202] Taking the smoothing process using the 7 frames of data before and after when the frame rate value is 30 as an example, smoothing is performed from the first frame data. For the nth frame data, the n-3th frame data, n-2th frame data, n-1th frame data, nth frame data, n+1th frame data, n+2th frame data, and n+3th frame data are set and smoothed according to weights 0.04, 0.07, 0.22, 0.34, 0.22, 0.07, and 0.04.

[0203] For the first few frames of blendshape data:

[0204] When n=1, the first frame data, the second frame data, the third frame data, and the fourth frame data are smoothed according to weights of 0.45, 0.3, 0.15, and 0.1.

[0205] When n=2, the first frame data, the second frame data, the third frame data, the fourth frame data, and the fifth frame data are smoothed according to weights of 0.2, 0.4, 0.2, 0.1, and 0.1.

[0206] When n=3, the first frame data, the second frame data, the third frame data, the fourth frame data, the fifth frame data, and the sixth frame data are smoothed according to weights of 0.1, 0.2, 0.3, 0.2, 0.1, and 0.1.

[0207] In the last few frames of the blendshape data, use the above similar weight setting steps to swap the order of the weights to ensure that the weight sum of the smoothed data used is 1, the middle data has the highest weight, and the farther away the data is, the lower the weight is.

[0208] Through the facial action generation method of the embodiment of the present application, a simple and easy-to-configure implementation mechanism is used to achieve the purpose of multi-dimensional emotional control of the target object without interfering with the changes in the speaking mouth shape, and it is possible to generate rich and delicate emotional fluctuations, giving the target object a more real and natural emotional expression. In addition, the facial action generation method of the embodiment of the present application is also highly scalable, allowing new emotion categories to be easily introduced. Only by creating corresponding standard template expressions can the emotion library be quickly expanded. At the same time, the design of the adjustment parameters is flexible, and detailed adjustments can be made according to the actual effect of the target object's emotional expression to adapt to the diverse emotional change control needs, thereby improving the target object's expression expression and interactive authenticity.

[0209] The following is a description of an exemplary structure of the facial action generating device 555 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the facial action generating device 555 of the memory 550 may include:

[0210] The data collection module 5551 is used to obtain the emotion category and audio text of the target emotion.

[0211] The facial action data generation module 5552 is used to determine the emotion change parameters corresponding to the emotion category, and based on the emotion change parameters, generate the first facial action data expressing the target emotion, wherein the emotion change parameters are used to control the emotion change expressed by the first facial action data.

[0212] The facial action data generating module 5552 is further used to determine second facial action data for reading the audio text, wherein the first expression locator included in the first facial action data is partially identical to the second expression locator included in the second facial action data.

[0213] The data fusion module 5553 is used to fuse the first facial action data and the second facial action data based on the first expression locator and the second expression locator to obtain target facial action data.

[0214] In some embodiments, the first facial action data includes N expression segments, where N is an integer greater than 1, and the facial action data generation module 5552 is further used to determine the nth initial expression segment of the first facial action data based on the emotion category, where n is a sequentially increasing integer, 1≤n≤N; based on the emotion change parameter, the nth initial expression segment is deformed to obtain the nth expression segment.

[0215] In some embodiments, the emotion change parameters include the degree of emotion change, the amplitude of emotion change, the number of emotion change frames and the number of emotion retention frames, the nth expression segment includes an emotion change segment and an emotion retention segment, and the facial action data generation module 5552 is also used to perform the following processing on the standard expression template included in the nth initial expression segment: based on the degree of emotion change, the amplitude of emotion change and the number of emotion change frames, determine the weight value of the third expression locator included in the I frame standard expression template, wherein I is the number of emotion change frames; based on the number of emotion retention frames, determine the weight value of the fourth expression locator included in the J frame standard expression template, wherein J is the number of emotion retention frames; based on the weight value of the third expression locator, construct the emotion change segment, and based on the weight value of the fourth expression locator, construct the emotion retention segment.

[0216] In some embodiments, the facial action data generation module 5552 is also used to determine the I-frame standard expression template from the nth initial expression segment based on the number of emotion change frames; determine the range of emotion change based on the degree of emotion change, the amplitude of emotion change and the initial weight value of the third expression locator included in the I-frame standard expression template; and determine the weight value of the third expression locator included in the I-frame standard expression template based on the range of emotion change.

[0217] In some embodiments, the facial action data generation module 5552 is also used to determine the range of emotion maximum variation based on the degree of emotion change, the amplitude of emotion change and the initial weight value; determine the random number determined from the range of emotion maximum variation as the emotion maximum; determine the weight value of the expression locator included in the last frame standard expression template of the n-1th expression segment as the emotion minimum; and construct the emotion variation range based on the emotion maximum and the emotion minimum.

[0218] In some embodiments, the facial action data generating module 5552 is also used to, when the degree of emotion change is less than the threshold value of the degree of emotion change, determine the product of the emotion change amplitude, the initial weight value and the first coefficient as a first minimum value, determine a first maximum value based on the first product, the first ratio and the first minimum value, and constitute the emotion maximum value change range based on the first maximum value and the first minimum value, wherein the first product is the product of the emotion change amplitude, the initial weight value and the second coefficient, and the first ratio is the ratio of the emotion change amplitude to the emotion change amplitude threshold; when the degree of emotion change is greater than or equal to the threshold value of the emotion change amplitude, determine the product of the emotion change amplitude and the initial weight value as a second minimum value, determine a second maximum value based on the product of the first difference and the second difference, the emotion change amplitude threshold and the second minimum value, and constitute the emotion maximum value change range based on the second maximum value and the second minimum value, wherein the first difference is the difference between the initial weight value and the second minimum value, and the second difference is the difference between the emotion change amplitude and the emotion change amplitude threshold.

[0219] In some embodiments, the facial action data generation module 5552 is also used to perform the following processing for the third expression locator of each dimension: determine I random values ​​from the emotion change range; sort the I random values ​​according to the order of emotion change, and determine the sorted I random values ​​as the weight value of the third expression locator.

[0220] In some embodiments, the facial action data generation module 5552 is also used to determine the J frame standard expression template from the nth initial expression segment based on the number of emotion retention frames; determine the target standard expression template that is located before the J frame standard expression template and adjacent to the J frame standard expression template; and determine the weight value of the expression locator included in the target standard expression template as the weight value of the fourth expression locator included in the J frame standard expression template.

[0221] In some embodiments, the data fusion module 5553 is further used to determine the sum of the first weight value and the second weight value as the target weight value of the first target expression locator, wherein the first target expression locator is the expression locator in the first expression locator that is the same as the second expression locator, the first weight value is the weight value of the first target expression locator included in the first facial action data, and the second weight value is the weight value of the first target expression locator included in the second facial action data; the target weight value of the first target expression locator and the weight value of the second target expression locator are combined to determine the target facial action data, wherein the second target expression locator is the expression locator in the first expression locator and the second expression locator except the first target expression locator.

[0222] In some embodiments, the facial action data application module 5554 is used to smooth the target facial action data to obtain the smoothed target facial action data; and apply the smoothed target facial action data to the face of the target object.

[0223] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device performs the facial action generation method described in the embodiment of the present application.

[0224] The present application embodiment provides a computer-readable storage medium in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will be caused to execute the facial action generation method provided by the present application embodiment, for example, Figures 3 to 8 The facial action generation method is shown.

[0225] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0226] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0227] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0228] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0229] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A facial action generation method, characterized in that: The method comprises: Get the emotion category and audio text of the target emotion; Determine an emotion change parameter corresponding to the emotion category, and based on the emotion change parameter, generate first facial action data expressing the target emotion, wherein the emotion change parameter is used to control the emotion change expressed by the first facial action data; Determining second facial action data for reading the audio text aloud, wherein a first expression locator included in the first facial action data is partially identical to a second expression locator included in the second facial action data; Based on the first expression locator and the second expression locator, the first facial action data and the second facial action data are fused to obtain target facial action data.

2. The method according to claim 1, characterized in that The first facial action data includes N expression segments, where N is an integer greater than 1; The step of generating first facial action data expressing the target emotion based on the emotion change parameter comprises: Based on the emotion category, determining an nth initial expression segment of the first facial action data, wherein n is an integer that increases in sequence, 1≤n≤N; Based on the emotion change parameter, the nth initial expression segment is deformed to obtain the nth expression segment.

3. The method according to claim 2, characterized in that The emotion change parameters include the emotion change degree, emotion change amplitude, emotion change frame number and emotion retention frame number, and the nth expression segment includes an emotion change segment and an emotion retention segment; The step of performing deformation processing on the nth initial expression segment based on the emotion change parameter to obtain the nth expression segment comprises: The following processing is performed on the standard expression template included in the nth initial expression segment: Determine a weight value of a third expression locator included in an I-frame standard expression template based on the degree of emotion change, the amplitude of emotion change and the number of emotion change frames, wherein I is the number of emotion change frames; Based on the emotion retention frame number, determining the weight value of the fourth expression locator included in the J-frame standard expression template, wherein J is the emotion retention frame number; The emotion change segment is formed based on the weight value of the third expression locator, and the emotion retention segment is formed based on the weight value of the fourth expression locator.

4. The method according to claim 3, characterized in that The step of determining the weight value of the third expression locator included in the I-frame standard expression template based on the emotion change degree, the emotion change amplitude and the emotion change frame number includes: Based on the number of emotion change frames, determining the I-frame standard expression template from the n-th initial expression segment; Determining the range of emotion change based on the emotion change degree, the emotion change amplitude, and the initial weight value of the third expression locator included in the I-frame standard expression template; Based on the emotion variation range, a weight value of a third expression locator included in the I-frame standard expression template is determined.

5. The method according to claim 4, characterized in that The determining of the emotion change range based on the emotion change degree, the emotion change amplitude and the initial weight value of the third expression locator included in the I-frame standard expression template comprises: Determine the maximum emotion change range based on the emotion change degree, the emotion change amplitude and the initial weight value; Determine a random number determined from the emotion maximum value variation range as the emotion maximum value; The weight value of the expression locator included in the standard expression template of the last frame of the n-1th expression segment is determined as the minimum value of emotion; The emotion variation range is constructed based on the emotion maximum value and the emotion minimum value.

6. The method according to claim 5, characterized in that The step of determining the maximum emotion value variation range based on the emotion variation degree, the emotion variation amplitude and the initial weight value comprises: When the emotion change degree is less than the emotion change degree threshold, the product of the emotion change amplitude, the initial weight value and the first coefficient is determined as the first minimum value, and the first maximum value is determined based on the first product, the first ratio and the first minimum value, and the emotion maximum value change range is constructed based on the first maximum value and the first minimum value, wherein the first product is the product of the emotion change amplitude, the initial weight value and the second coefficient, and the first ratio is the ratio of the emotion change amplitude to the emotion change degree threshold; When the degree of emotion change is greater than or equal to the emotion change degree threshold, the product of the emotion change amplitude and the initial weight value is determined as the second minimum value, and the second maximum value is determined based on the product of the first difference and the second difference, the emotion change degree threshold and the second minimum value, and the emotion maximum change range is constructed based on the second maximum value and the second minimum value, wherein the first difference is the difference between the initial weight value and the second minimum value, and the second difference is the difference between the emotion change degree and the emotion change degree threshold.

7. The method according to claim 4, characterized in that The step of determining the weight value of the third expression locator included in the I-frame standard expression template based on the emotion variation range includes: The following processing is performed for the third expression locator of each dimension: Determine a random value from the emotion variation range; The I random values ​​are sorted according to the order of emotion changes, and the sorted I random values ​​are determined as the weight value of the third expression locator.

8. The method according to claim 3, characterized in that The step of determining the weight value of the fourth expression locator included in the J-frame standard expression template based on the emotion retention frame number includes: Based on the emotion retention frame number, determining the J-frame standard expression template from the n-th initial expression segment; Determine a target standard expression template that is located before the J-frame standard expression template and adjacent to the J-frame standard expression template; The weight value of the expression locator included in the target standard expression template is determined as the weight value of the fourth expression locator included in the J-frame standard expression template.

9. The method according to any one of claims 1 to 8, characterized in that: The step of fusing the first facial action data and the second facial action data based on the first expression locator and the second expression locator to obtain target facial action data includes: Determine the sum of the first weight value and the second weight value as the target weight value of the first target expression locator, wherein the first target expression locator is the same as the second expression locator in the first expression locators, the first weight value is the weight value of the first target expression locator included in the first facial action data, and the second weight value is the weight value of the first target expression locator included in the second facial action data; The target weight value of the first target expression locator and the weight value of the second target expression locator are combined to determine the target facial action data, wherein the second target expression locator is an expression locator between the first expression locator and the second expression locator except the first target expression locator.

10. The method according to any one of claims 1 to 8, characterized in that: After the first facial action data and the second facial action data are fused based on the first expression locator and the second expression locator to obtain target facial action data, the method further includes: Performing smoothing processing on the target facial action data to obtain smoothed target facial action data; The smoothed target facial motion data is applied to the face of the target object.

11. A facial action generation device, characterized in that: The device comprises: A data collection module is used to obtain the emotion category and audio text of the target emotion; A facial action data generation module, used to determine the emotion change parameter corresponding to the emotion category, and based on the emotion change parameter, generate first facial action data expressing the target emotion, wherein the emotion change parameter is used to control the emotion change expressed by the first facial action data; The facial action data generating module is further used to determine second facial action data for reading the audio text, wherein the first expression locator included in the first facial action data is partially identical to the second expression locator included in the second facial action data; The data fusion module is used to fuse the first facial action data and the second facial action data based on the first expression locator and the second expression locator to obtain target facial action data.

12. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, configured to implement the facial action generation method according to any one of claims 1 to 10 when executing the computer executable instructions or computer program stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the facial action generation method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the facial action generation method according to any one of claims 1 to 10 is implemented.

Images