Virtual artist expression adjusting system with emotion perception and method thereof

Abstract

The invention relates to a virtual artist expression adjusting system with emotion perception and a method thereof, which are characterized in that a voice signal and corresponding emotion information are loaded as training data at the initial time, then the training data are input into an artificial intelligence model for training to generate an emotion recognition model, then the voice of a user is received for feature acquisition, standardization and dimension reduction processing, and the emotion recognition model is obtained. The method comprises the following steps: inputting an emotion recognition model to obtain an emotion state, then executing facial expression generation operation to generate corresponding facial key points based on the emotion state, and adjusting facial model parameters of the virtual artist according to the facial key points in real time to dynamically display the facial expression of the virtual artist. Therefore, the technical effect of improving the authenticity and richness of the expression of the virtual artist is achieved.

Description

Technical Field

[0001] The present invention relates to an expression adjustment system and method thereof, in particular to an expression adjustment system and method thereof for a virtual artist with emotion perception. Background Art

[0002] In recent years, with the popularization and vigorous development of virtual technology, various applications of virtual technology have sprung up like mushrooms after rain. Among them, the economic value of virtual artists has attracted the most attention.

[0003] Generally speaking, traditional virtual artists usually obtain movements and even expressions through motion acquisition technology. However, although this method can instantly allow virtual artists to make the same movements and expressions as real people, the equipment required for motion acquisition technology is very complex, the synchronization requirements are also high, and it may even require post-processing, so the usability is greatly limited.

[0004] In view of this, some manufacturers have proposed a technical means of facial expression simulation, which directly simulates various human expressions in advance and applies them to virtual artists. However, this method can only change expressions according to a set process, so the flexibility and usability of expressions are insufficient. For example, the expressions are not rich enough and are prone to rigidity. In addition, this method cannot be coordinated with voice at the same time, which easily produces a sense of disharmony and lacks realism, such as an angry voice with a happy expression. Therefore, it is still unable to effectively solve the problem of insufficient authenticity and richness of virtual artists' expressions.

[0005] In summary, it can be seen that the existing technology has long had the problem of insufficient authenticity and richness of virtual artists' expressions, so it is necessary to propose improved technical means to solve this problem. Summary of the invention

[0006] The invention discloses a virtual artist expression adjustment system with emotion perception and a method thereof.

[0007] First, the present invention discloses a virtual artist expression adjustment system with emotion perception, which includes: an emotional voice database and a computer host. The emotional voice database is used to store multiple voice signals and the emotional information corresponding to each of the voice signals. Then, in the part of the computer host, it is connected to the emotional voice database, and the computer host includes: a non-transitory computer-readable storage medium and a hardware processor. The non-transitory computer-readable storage medium is used to store multiple computer-readable instructions; and the hardware processor is electrically connected to the non-transitory computer-readable storage medium to execute the multiple computer-readable instructions, so that the hardware processor is configured to: initially, load the voice signal and its corresponding emotional information from the emotional voice database as training data, and input the training data into the artificial intelligence model for training to generate an emotion recognition model; receive user voice for feature acquisition, standardization and dimensionality reduction processing, and input the processed user voice into the emotion recognition model to obtain the emotional state; and perform facial expression generation operation to generate corresponding multiple facial landmarks based on the emotional state, and adjust multiple facial model parameters of the virtual artist according to the facial landmarks in real time to dynamically display the facial expressions of the virtual artist.

[0008] In addition, the present invention also discloses a method for adjusting the expression of a virtual artist with emotion perception, the steps of which include: connecting an emotional speech database to a computer host, wherein the emotional speech database stores multiple voice signals and the emotion information corresponding to each of the voice signals; at the initial stage, the computer host loads the voice signal and its corresponding emotion information from the emotional speech database as training data, and inputs the training data into an artificial intelligence model for training to generate an emotion recognition model; the computer host receives the user's voice for feature acquisition, standardization and dimensionality reduction processing, and inputs the processed user's voice into the emotion recognition model to obtain the emotional state; and the computer host executes a facial expression generation operation to generate a corresponding plurality of facial key points based on the emotional state, and adjusts the virtual artist's multiple facial model parameters in real time according to the facial key points to dynamically display the virtual artist's facial expression.

[0009] The system and method disclosed in the present invention are as described above. The difference from the prior art is that the present invention initially loads the voice signal and its corresponding emotional information as training data, and then inputs it into the artificial intelligence model for training to generate an emotion recognition model. Next, the user's voice is received for feature acquisition, standardization and dimensionality reduction processing, and then input into the emotion recognition model to obtain the emotional state. Then, a facial expression generation operation is performed to generate corresponding facial key points based on the emotional state, and the facial model parameters of the virtual artist are adjusted in real time according to the facial key points to dynamically display the facial expressions of the virtual artist.

[0010] Through the above-mentioned technical means, the present invention can achieve the technical effect of improving the authenticity and richness of the expressions of virtual artists. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a system block diagram of the virtual artist expression adjustment system with emotion perception of the present invention.

[0012] Figure 2A and Figure 2B The present invention is a flowchart of a method for adjusting the expression of a virtual artist with emotion perception.

[0013] Figure 3 Schematic diagram of key points of a human face according to the present invention.

[0014] Figure 4 A schematic diagram of adjusting the expression of a virtual artist using the present invention.

[0015] Description of reference numerals:

[0016] 110: Emotional Speech Database

[0017] 120: Computer host

[0018] 121: Non-transitory computer readable storage medium

[0019] 122: Hardware Processor

[0020] 300: Face

[0021] 311a~311n:Face key points

[0022] 400: Happy face image

[0023] 411a~411n:Face key points

[0024] 421: Initial Expression

[0025] 422: Happy Expression

[0026] Step 210: Connecting an emotional speech database to a computer host, wherein the emotional speech database stores a plurality of speech signals and the emotional information corresponding to each of the speech signals.

[0027] Step 220: The computer host initially loads the speech signal and the corresponding emotion information from the emotional speech database as training data, and inputs the training data into the artificial intelligence model for training to generate an emotion recognition model.

[0028] Step 230: The computer host receives the user's speech to perform feature acquisition, standardization and dimension reduction processing, and inputs the processed user's speech into the emotion recognition model to obtain the emotional state.

[0029] Step 240: The computer host executes a facial expression generation operation to generate a plurality of corresponding facial landmarks based on the emotional state, and adjusts a plurality of facial model parameters of the virtual artist according to the facial landmarks in real time to dynamically display the facial expression of the virtual artist.

[0030] Step 241: The computer host performs a facial expression generation operation to generate a plurality of corresponding facial key points based on the emotional state.

[0031] Step 242: Detect whether the facial key points meet the inappropriate expression characteristics

[0032] Step 243: Immediately adjust the facial model parameters of the virtual artist according to the facial key points

[0033] Step 244: prohibiting the use of the facial key points and initializing the facial model parameters

[0034] Step 245: Dynamically display the facial expression of the virtual artist according to the facial model parameters DETAILED DESCRIPTION

[0035] The following will describe the implementation methods of the present invention in detail with reference to the accompanying drawings and embodiments, so that the implementation process of how the present invention applies technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0036] First, please refer to the Figure 1 ”, “ Figure 1 ” is a system block diagram of the virtual artist expression adjustment system with emotion perception of the present invention, and the system includes: an emotional speech database 110 and a computer host 120. The emotional speech database 110 is used to store multiple voice signals and the emotional information corresponding to each of the voice signals. For example, an angry voice will correspond to an emotional information, and the emotional information can be of a multimodal type, such as: the text "angry", an image or video of an angry face, etc.

[0037] Next, in the part of the computer host 120, it is connected to the emotional speech database 110, and the computer host 120 includes: a non-transitory computer-readable storage medium 121 and a hardware processor 122. Among them, the non-transitory computer-readable storage medium 121 is used to store a plurality of computer-readable instructions. In actual implementation, the non-transitory computer-readable storage medium 121 may include: a hard disk, an optical disk, a flash memory or the like. In addition, the computer-readable instructions are executed by the hardware processor 122, and the computer-readable instructions may be assembly language instructions, instruction set architecture instructions, machine instructions, machine-related instructions, microinstructions, firmware instructions, or source code or object code (Object Code) written in any combination of one or more programming languages, and the programming languages ​​include object-oriented programming languages, such as: Common Lisp, Python, C++, Objective-C, Smalltalk, Delphi, Java, Swift, C#, Perl, Ruby and PHP, etc., as well as conventional procedural programming languages, such as: C language or similar programming languages.

[0038] The hardware processor 122 is electrically connected to the non-transitory computer-readable storage medium 121 to execute the plurality of computer-readable instructions, so that the hardware processor 122 is configured to: initially, load the speech signal and its corresponding emotion information from the emotional speech database as training data, and input the training data into the artificial intelligence model for training to generate an emotion recognition model; receive the user's speech for feature acquisition, standardization and dimensionality reduction, and input the processed user's speech into the emotion recognition model to obtain the emotional state; and perform facial expression generation operation to generate a plurality of corresponding facial key points based on the emotional state, and adjust the plurality of facial model parameters of the virtual artist according to the facial key points in real time to dynamically display the facial expressions of the virtual artist. In actual implementation, the emotion recognition model may include a convolutional neural network (CNN) and a recurrent neural network (RNN), and during the training process, it is allowed to receive the emotion information including text, pictures, images, sounds and combinations thereof as multimodal training data. In addition, the facial expression generation operation may include generating a facial image corresponding to the emotional state through at least one of Generative Adversarial Network (GAN), Deep Learning (DL) and Reinforcement Learning (RL), and then obtaining facial key points from the facial image. In addition, the virtual artist may have a predefined facial expression model, which includes facial model parameters. When the facial key points are generated, the hardware processor 122 will use a filter to smooth the changes of the facial key points, and perform boundary checks to delete unnatural expressions, and then map the processed facial key points to the facial model parameters to change the facial expression model, thereby dynamically presenting the facial expression of the virtual artist. Furthermore, the hardware processor 122 can also detect whether the facial key points meet the characteristics of inappropriate expressions. When they meet the characteristics of inappropriate expressions, it is prohibited to use the facial key points to adjust the facial model parameters of the virtual artist, and the facial model parameters are initialized at the same time, effectively preventing the virtual artist from showing frightening or unnatural expressions.

[0039] It should be particularly noted that in actual implementation, the present invention can be partially or completely implemented based on hardware. For example, the hardware processor 122 in the system can be implemented by an integrated circuit chip, a system on chip (System on Chip, SoC), a complex programmable logic device (Complex Programmable Logic Device, CPLD), and a field programmable gate array (Field Programmable Gate Array, FPGA). The non-transitory computer-readable storage medium 121 described in the present invention carries computer-readable instructions (or computer program instructions) for enabling the hardware processor 122 to implement various aspects of the present invention. The non-transitory computer-readable storage medium 121 can be, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of the non-transitory computer-readable storage medium 121 include: a hard disk, a random access memory, a read-only memory, a flash memory, an optical disk, a floppy disk, and any suitable combination of the above. The non-transitory computer-readable storage medium 121 used herein is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., an optical signal through a fiber optic cable), or an electrical signal transmitted through a wire. In addition, the computer-readable instructions described herein can be downloaded from the non-transitory computer-readable storage medium 121 to each computing / processing device, or downloaded to an external computer device or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, hubs, and / or doorways. The network card or network interface in each computing / processing device receives the computer-readable instruction from the network and forwards the computer-readable instruction for storage in the non-transitory computer-readable storage medium 121 in each computing / processing device.

[0040] Please refer to " Figure 2A "and" Figure 2B ”, “ Figure 2A "and" Figure 2B” is a method flow chart of the method for adjusting the expression of a virtual artist with emotion perception of the present invention, and the steps include: connecting the emotional speech database 110 to the computer host 120, wherein the emotional speech database 110 stores a plurality of speech signals and the emotion information corresponding to each of the speech signals (step 210); the computer host 120 initially loads the speech signals and the corresponding emotion information from the emotional speech database 110 as training data, and inputs the training data into the artificial intelligence model for training to generate an emotion recognition model (step 220); the computer host 120 receives the user's speech for feature acquisition, standardization and dimensionality reduction processing, and inputs the processed user's speech into the emotion recognition model to obtain the emotional state (step 230); and the computer host The machine 120 performs a facial expression generation operation to generate a plurality of corresponding facial key points based on the emotional state, and adjusts a plurality of facial model parameters of the virtual artist according to the facial key points in real time to dynamically display the facial expressions of the virtual artist (step 240). Through the above steps, the voice signal and its corresponding emotional information are loaded as training data at the initial stage, and then input into the artificial intelligence model for training to generate an emotion recognition model, then, after receiving the user's voice for feature acquisition, standardization and dimensionality reduction processing, it is input into the emotion recognition model to obtain the emotional state, and then the facial expression generation operation is performed to generate the corresponding facial key points based on the emotional state, and the facial model parameters of the virtual artist are adjusted according to the facial key points in real time to dynamically display the facial expressions of the virtual artist.

[0041] It should be particularly noted that step 240 performed by the computer host 120 may be as follows: Figure 2B ", including: performing facial expression generation operations to generate a plurality of corresponding facial key points based on the emotional state (step 241); detecting whether the facial key points meet the inappropriate expression characteristics (step 242); if not, adjusting the facial model parameters of the virtual artist according to the facial key points in real time (step 243); if so, prohibiting the use of the facial key points and initializing the facial model parameters (step 244); finally, dynamically displaying the facial expression of the virtual artist according to the facial model parameters (step 245). In other words, compared with " Figure 2A ” Step 240 further includes steps 242 and 244 for performing corresponding processing when the key points of the face meet the inappropriate expression characteristics. In this way, the virtual character can be effectively prevented from showing inappropriate expressions, such as unnatural or frightening expressions.

[0042] The following matches " Figure 3 "and" Figure 4 ” is described as follows by way of example, such as “ Figure 3 " Figure 3” is a schematic diagram of the facial key points of the present invention. First, the facial key points (311a-311n) of the present invention refer to specific areas of great significance on the face 300, such as the corners of the eyes, eyebrows, nose tips, edges of lips, etc., which are usually used to describe the structure and expression of the face. The number of these facial key points (311a-311n) can be N (N is a positive integer), which can help identify and analyze the shape, movement and emotion of the face. Therefore, it has a wide range of applications in computer vision and image processing. In actual implementation, a standard facial key point detection model (such as Dlib, OpenFace, etc.) can be used to identify the face in the image, picture, etc. and generate facial key points (311a-311n). In fact, the detection of facial key points (311a-311n) can usually adopt machine learning (such as support vector machine or regression model), deep learning (such as convolutional neural network, hourglass network, etc.) or similar methods, such as PIPNet (Pixel-in-Pixel Net). During the training process, the model will learn how to automatically identify the location of feature points from the image, and generate corresponding facial key points (311a~311n) and their coordinates. It should be particularly noted that the emotional state output by the emotion recognition model of the present invention can be multimodal data such as text, pictures, and images. Therefore, if pictures or images are output, facial key points can be generated in the above manner; if text is output, the facial key points representing different emotions can be matched with the text in advance, for example: the text "anger" is matched with the facial key points of the face when angry, the text "happy" is matched with the facial key points presented on the face when happy, and so on, and the corresponding facial key points can be generated according to the text in the future. It should be supplemented that, for the convenience of illustration, the facial key points are presented using circles with the same dots.

[0043] like" Figure 4 " Figure 4 ” is a schematic diagram of applying the present invention to adjust the expression of a virtual artist. In actual implementation, initially, the computer host 120 loads the voice signal and its corresponding emotion information from the emotional voice database 110 as training data, and inputs the training data into the artificial intelligence model for training to generate an emotion recognition model. The emotion recognition model allows the input of the user's voice to identify or predict the emotional state of the user's voice, which can be text, pictures or images. Take text as an example, such as: "happy"; take pictures as an example, it can be a happy face picture; take images as an example, it can be a happy face image.

[0044] When the computer host 120 receives the user's voice, it will perform feature acquisition, standardization and dimensionality reduction processing, and input the processed user voice into the emotion recognition model to obtain the emotional state, such as: happy facial image 400. Then, the computer host 120 will perform a facial expression generation operation to generate a corresponding plurality of facial key points (411a~411n) according to the emotional state (such as: happy facial image 400), the number of which can be N (N is a positive integer), and adjust the virtual artist's multiple facial model parameters according to these facial key points (411a~411n) in real time. For example, assuming that the virtual artist has an initial expression 421, after the facial key points (411a~411n) are adjusted, the virtual artist's expression will be like " Figure 4 ” as shown, from the initial expression 421 to the happy expression 422. At this point, the expression adjustment of the virtual artist is completed. It should be added that, in order to avoid unnatural expressions, the facial key points (411a~411n) can be smoothed by a filter to smooth the changes of the facial key points (411a~411n), and a boundary check can be performed to delete unnatural expressions, and then the processed facial key points (411a~411n) are mapped to the facial model parameters to change the facial expression model of the virtual artist, and finally presented as “ Figure 4 ” indicates a happy expression 422.

[0045] To sum up, it can be seen that the difference between the present invention and the prior art lies in that the voice signal and its corresponding emotional information are loaded as training data at the initial stage, and then input into the artificial intelligence model for training to generate an emotion recognition model. Then, the user's voice is received for feature acquisition, standardization and dimensionality reduction processing, and then input into the emotion recognition model to obtain the emotional state, and then the facial expression generation operation is performed to generate the corresponding facial key points based on the emotional state, and the facial model parameters of the virtual artist are adjusted in real time according to the facial key points to dynamically display the facial expressions of the virtual artist. This technical means can solve the problems existing in the prior art, and thus achieve the technical effect of improving the authenticity and richness of the expressions of the virtual artist.

[0046] Although the present invention is disclosed as above with the aforementioned embodiments, it is not intended to limit the present invention. Any person skilled in the art may make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of patent protection of the present invention shall be subject to the scope defined by the claims attached to this specification.

Claims

1. A virtual artist expression adjustment system with emotion perception, the system comprising: An emotional speech database, used to store a plurality of speech signals and the emotion information corresponding to each of the speech signals; and A computer host, used to connect to the emotional speech database, the computer host comprising: a non-transitory computer-readable storage medium for storing a plurality of computer-readable instructions; and A hardware processor is electrically connected to the non-transitory computer-readable storage medium and is used to execute the plurality of computer-readable instructions, so that the hardware processor is configured to: Initially, the speech signal and the corresponding emotion information are loaded from the emotional speech database as training data, and the training data is input into the artificial intelligence model for training to generate an emotion recognition model; receiving user speech to perform feature acquisition, standardization and dimension reduction processing, and inputting the processed user speech into the emotion recognition model to obtain the emotional state; and A facial expression generation operation is performed to generate a corresponding plurality of facial key points based on the emotional state, and a plurality of facial model parameters of the virtual artist are adjusted in real time according to the facial key points to dynamically display the facial expression of the virtual artist.

2. The emotion-aware virtual artist expression adjustment system as claimed in claim 1, wherein the emotion recognition model comprises a convolutional neural network and a recurrent neural network, and during the training process, it is allowed to receive the emotion information comprising text, pictures, images, sounds and combinations thereof as the multimodal training data.

3. The emotion-aware virtual artist expression adjustment system as described in claim 1, wherein the facial expression generation operation includes generating a facial image corresponding to the emotional state through at least one of a generative adversarial network, deep learning and reinforcement learning, and then obtaining the facial key points from the facial image.

4. The emotion-aware virtual artist expression adjustment system as described in claim 1, wherein the virtual artist has a predefined facial expression model, the facial expression model includes the facial model parameters, when the facial key points are generated, the changes of the facial key points are smoothed by a filter, and a boundary check is performed to delete unnatural expressions, and the processed facial key points are then mapped to the facial model parameters to change the facial expression model.

5. The emotion-aware virtual artist expression adjustment system as described in claim 1, wherein the hardware processor further includes detecting whether the facial key points meet the inappropriate expression characteristics. When the inappropriate expression characteristics are met, the use of the facial key points to adjust the facial model parameters of the virtual artist is prohibited, and the facial model parameters are initialized at the same time.

6. A method for adjusting the expression of a virtual artist with emotion perception, the steps comprising: Connecting an emotional speech database to a computer host, wherein the emotional speech database stores a plurality of speech signals and emotional information corresponding to each of the speech signals; The computer host initially loads the speech signal and the corresponding emotion information from the emotional speech database as training data, and inputs the training data into the artificial intelligence model for training to generate an emotion recognition model; The computer host receives the user's speech to perform feature acquisition, standardization and dimension reduction processing, and inputs the processed user's speech into the emotion recognition model to obtain the emotional state; and The computer host executes a facial expression generation operation to generate a plurality of corresponding facial key points based on the emotional state, and instantly adjusts a plurality of facial model parameters of the virtual artist according to the facial key points to dynamically display the facial expression of the virtual artist.

7. The method for adjusting expression of a virtual artist with emotion perception as described in claim 6, wherein the emotion recognition model includes a convolutional neural network and a recurrent neural network, and during the training process, it is allowed to receive the emotion information including text, pictures, images, sounds and combinations thereof as the multimodal training data.

8. The method for adjusting expression of a virtual artist with emotion perception as described in claim 6, wherein the facial expression generation operation includes generating a facial image corresponding to the emotional state through at least one of a generative adversarial network, deep learning and reinforcement learning, and then obtaining the facial key points from the facial image.

9. The method for adjusting expression of a virtual artist with emotion perception as described in claim 6, wherein the virtual artist has a predefined facial expression model, the facial expression model includes the facial model parameters, and when the facial key points are generated, the changes of the facial key points are smoothed by a filter, and a boundary check is performed to delete unnatural expressions, and the processed facial key points are then mapped to the facial model parameters to change the facial expression model.

10. The method for adjusting the expression of a virtual artist with emotion perception as described in claim 6, wherein the computer host further includes detecting whether the facial key points meet the inappropriate expression characteristics. When the inappropriate expression characteristics are met, the use of the facial key points to adjust the facial model parameters of the virtual artist is prohibited, and the facial model parameters are initialized at the same time.

Images