Data processing apparatus, data processing method, and program

Abstract

To provide a data processing apparatus, a data processing method, and a program capable of outputting a voice suitable for the appearance of an avatar while making the movement of the avatar natural when outputting the voice of a chatbot by the avatar.SOLUTION: The data processing apparatus acquires facial data of an avatar realized by computer graphics, acquires skeletal data corresponding to the facial data, and acquires musculoskeletal model data representing a musculoskeletal system corresponding to the skeletal data. The data processing apparatus sets an avatar model having a face and a musculoskeletal system as elements, generates an acoustic feature representing a feature of a voice emitted by the avatar model, and converts text data of an object into voice data. The data processing apparatus controls the display of an avatar model on a display unit while operating each unit of the face and each unit of the musculoskeletal system as elements of the avatar model in accordance with the voice data, and outputs the voice data from the voice output unit.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The technology disclosed herein relates to a data processing device, a data processing method, and a program. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, when a user interacts with a chatbot, it is assumed that the interaction will take place through an avatar. In this case, text data output from the chatbot is converted into voice data, and the voice data is output in accordance with the mouth movements of the avatar.

[0005] When an avatar is realized by computer graphic animation, simply moving the avatar's mouth and outputting voice data often looks unnatural, and the user may find the movement creepy.

[0006] The technology disclosed herein has been made in consideration of the above circumstances, and aims to provide a data processing device, a data processing method, and a program that, when outputting the speech of a chatbot through an avatar realized by computer graphics, can make the avatar's movements natural and output a voice that suits the avatar's appearance. [Means for solving the problem]

[0007] A first aspect of the technology of the present disclosure is a data processing device including: a face acquisition unit that acquires face data of an avatar realized by computer graphics; a skeletal acquisition unit that acquires skeletal data corresponding to the face data; a musculoskeletal acquisition unit that acquires musculoskeletal model data representing a musculoskeletal structure corresponding to the skeletal data; a setting unit that sets an avatar model having as elements a face represented by the face data and the musculoskeletal structure represented by the musculoskeletal model data; a feature generation unit that generates acoustic features representing characteristics of a voice uttered via the avatar model based on at least one of the face data and the skeletal data; a conversion unit that converts text data into voice data having the acoustic features; and an output unit that controls the face represented by the face data and the musculoskeletal structure represented by the musculoskeletal model data to operate in accordance with the voice data, and to output the voice data from a voice output unit.

[0008] A second aspect of the technology of the present disclosure is a data processing method in which a computer executes processes to acquire facial data of an avatar realized by computer graphics, acquire skeletal data corresponding to the facial data, acquire musculoskeletal model data representing a musculoskeletal structure corresponding to the skeletal data, set an avatar model having a face represented by the facial data and the musculoskeletal structure represented by the musculoskeletal model data as elements, generate acoustic features representing characteristics of voice uttered via the avatar model based on at least one of the facial data and the skeletal data, convert text data into voice data having the acoustic features, operate the face represented by the facial data and the musculoskeletal structure represented by the musculoskeletal model data in accordance with the voice data, and control the voice data to be output from an audio output unit.

[0009] A third aspect of the technology of the present disclosure is a program for causing a computer to execute a process including acquiring facial data of an avatar realized by computer graphics, acquiring skeletal data corresponding to the facial data, acquiring musculoskeletal model data representing a musculoskeletal structure corresponding to the skeletal data, setting an avatar model having a face represented by the facial data and the musculoskeletal structure represented by the musculoskeletal model data as elements, generating acoustic features representing characteristics of a voice uttered via the avatar model based on at least one of the facial data and the skeletal data, converting text data into voice data having the acoustic features, operating the face represented by the facial data and the musculoskeletal structure represented by the musculoskeletal model data in accordance with the voice data, and controlling the voice data to be output from a voice output unit. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a conceptual diagram illustrating an example of a configuration of a data processing system. [Diagram 2] FIG. 2 is a conceptual diagram showing an example of main functions of a data processing device and a smart device. [Figure 3A] FIG. 1 is a diagram for explaining the present embodiment. [Figure 3B] FIG. 4 is a diagram illustrating an example of musculoskeletal model data. [Figure 3C] FIG. 2 is a diagram for explaining an avatar model. [Figure 4] 2 shows a schematic functional configuration of a specific processing unit of the data processing device. [Diagram 5] 13 illustrates an example of an operational flow of a specific process performed by a data processing device. [Figure 6A] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 6B] 1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an example of an embodiment of a data processing device, a data processing method, and a program according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0012] First, the terms used in the following description will be explained.

[0013] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0014] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0015] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0016] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0017] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0018] FIG. 1 shows an example of the configuration of a data processing system 10 according to the embodiment.

[0019] As shown in Fig. 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server. An example of the smart device 14 is a smartphone. In this embodiment, the data processing device 12 is an example of a "data processing device" according to the technology of the present disclosure.

[0020] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0021] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0022] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0023] The output device 40 includes a display 40A and a speaker 40B, and presents data to the person 20 by outputting the data in a form of expression that the person 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0024] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0025] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0026] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0027] The storage 32 stores face data 57A, a skeleton generation model 57B, musculoskeletal model data 57C, an acoustic feature generation model 57D, a data generation model 58, and an emotion identification model 59. The face data 57A, the skeleton generation model 57B, the musculoskeletal model data 57C, the acoustic feature generation model 57D, the data generation model 58, and the emotion identification model 59 are used by the identification processing unit 290.

[0028] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0029] Next, the processing of the specific processing unit 290 in the data processing device 12 will be described.

[0030] When outputting the chatbot's speech via the avatar, the data processing device 12 generates a voice according to the face or skeleton of the avatar. The data processing device 12 also generates facial movement of the avatar in accordance with the speech.

[0031] When a user interacts with a chatbot, it may be assumed that the interaction will be via an avatar. In this case, the chatbot's speech is converted into voice data, and the voice is output in accordance with the avatar's mouth movements.

[0032] When audio is output in sync with the movements of a two-dimensional cartoon-style avatar, users often do not find it unnatural even if the avatar's mouth simply opens and closes in sync with the audio. In contrast, when an avatar is realized by computer graphic animation, users often find it unnatural to simply move the avatar's mouth and output audio, and often find the movements creepy.

[0033] One of the reasons why users feel uneasy is thought to be that an actual human face is made up of bones and muscles, whereas avatars created using computer graphic animation lack information about bones and muscles.

[0034] More specifically, although the appearance of an avatar created through computer graphic animation looks very similar to a real human, unlike a real human, the avatar lacks bones and muscles, and so the avatar's movements are subtly different from those of a real human. Even if the avatar's mouth moves in time with the audio, it is likely that users will find the avatar creepy.

[0035] Even if skeletal and muscular information is reflected in an avatar realized by computer graphic animation, the user may find the avatar creepy depending on the combination of the avatar's appearance and the output voice. For example, if a male avatar has a high-pitched voice, the user may find the avatar creepy.

[0036] Thus, in order to reproduce the act of speaking, it is not enough to move the muscles of the tongue and the bones that form the mouth; all of the human facial muscles, blinking, neck movements, etc. must be natural, otherwise the user will find it creepy.

[0037] Therefore, the data processing device 12 of this embodiment estimates the skeleton of the avatar's head from the avatar's face, and generates a voice suitable for the avatar from the avatar's face or skeleton. The data processing device 12 of this embodiment also generates a musculoskeletal model consisting of a skeleton and muscles from the avatar's skeleton. The data processing device 12 of this embodiment then generates facial expressions for the avatar by moving each part of the avatar's face and each part of the musculoskeletal model in accordance with the generated voice data.

[0038] FIG. 3A is a diagram for explaining this embodiment. As shown in FIG. 3A, in this embodiment, a combination of actual human face data F, human head skeleton data G, and voice data A emitted by the human is collected in advance. Then, as shown in FIG. 3A, these combinations are used as learning data to train a machine learning model, thereby generating a trained model. In this embodiment, a skeleton generation model for generating skeleton data from face data and an acoustic feature generation model for generating acoustic features of voice data from face data are used.

[0039] Specifically, in this embodiment, when a user specifies facial data of an avatar, the facial data is input to a skeleton generation model, whereby skeleton data corresponding to the facial data of the avatar is generated. Furthermore, the facial data is input to an acoustic feature generation model, whereby acoustic features of voice data corresponding to the facial data of the avatar are generated. These acoustic features are used to generate voice emitted from the avatar. Note that the skeleton generation model and the acoustic feature generation model can be realized by a known machine learning model and a known machine learning algorithm.

[0040] Next, musculoskeletal model data of the avatar's head is generated based on the skeletal data corresponding to the avatar's face data. Fig. 3B is a diagram showing an example of the musculoskeletal model data. As shown in Fig. 3B, the musculoskeletal model data AM is composed of skeletal data S and muscle data M. Each part of the avatar's face is configured to move in accordance with the movement of each part of this musculoskeletal model data AM.

[0041] Then, as shown in FIG. 3C, an avatar model is set, the components of which are the face data FD and the musculoskeletal model data AM. Specifically, the avatar model is set on a computer so that the musculoskeletal model data AM is arranged behind the face data FD. In this embodiment, the avatar model is made to move in accordance with the voice. Specifically, the face and musculoskeletal structure of the avatar model are moved to directly simulate the mechanism by which a human makes facial expressions in accordance with speaking. As described above, the voice uttered through the avatar model is generated in accordance with the acoustic feature quantity corresponding to the face of the avatar model. As a result, when the utterance of a chatbot is output as voice through an avatar realized by computer graphics, it is possible to output voice suitable for the appearance of the avatar while making the avatar's movements natural.

[0042] As shown in FIG. 4, the identification processing unit 290 includes a face acquisition unit 292, a skeleton acquisition unit 294, a musculoskeletal acquisition unit 296, a setting unit 298, a feature generation unit 300, a dialogue unit 302, a conversion unit 304, and an output unit 306.

[0043] The face acquisition unit 292 acquires face data representing the face of an avatar that will interact with a user. The face of the avatar in this embodiment is realized by computer graphics. For example, the face acquisition unit 292 outputs a plurality of pieces of face data 57A stored in the storage 32 to the smart device 14. When the control unit 46A of the smart device 14 receives the plurality of pieces of face data 57A, the control unit 46A displays the plurality of pieces of face data 57A on the display 40A. The display 40A is an example of a display unit.

[0044] The user operates touch panel 38A of smart device 14 to select the face of an avatar with which the user wishes to have a conversation from the plurality of face data 57A displayed on display 40A.

[0045] The control unit 46A of the smart device 14 transmits to the data processing device 12 the face data of the avatar selected by the user.

[0046] The face acquisition unit 292 of the data processing device 12 acquires face data of an avatar selected by a user, and sets the face data as the face data of the avatar.

[0047] The skeleton acquisition unit 294 acquires skeleton data corresponding to the face data acquired by the face acquisition unit 292, based on the face data. In this embodiment, the skeleton acquisition unit 294 generates the skeleton data from the face data by using the skeleton generation model 57B stored in the storage 32. The skeleton data is three-dimensional model data.

[0048] The skeleton generation model 57B is a machine learning model that outputs skeleton data representing a skeleton corresponding to a face when face data, which is a three-dimensional image, is input. The skeleton generation model 57B is trained in advance based on a plurality of learning data in which face data for learning and skeleton data for learning are associated with each other. The skeleton generation model 57B may be any model that can be generated by a known machine learning algorithm.

[0049] For this reason, the skeleton acquisition unit 294 inputs the face data acquired by the face acquisition unit 292 into the skeleton generation model 57B stored in the storage 32, and acquires the skeleton data output from the skeleton generation model 57B.

[0050] The musculoskeletal acquisition unit 296 acquires musculoskeletal model data representing a musculoskeletal structure corresponding to the skeletal data acquired by the skeletal acquisition unit 294, based on the skeletal data. As described above, Fig. 3B is a diagram showing an example of the musculoskeletal model data. As shown in Fig. 3B, the musculoskeletal model data AM is a three-dimensional model composed of skeletal data S and muscle (and fascia) data M.

[0051] It is possible to estimate the muscle structure of the human head to some extent based on the skeleton. For this reason, the musculoskeletal acquisition unit 296 of this embodiment selects musculoskeletal model data of a skeleton similar to the skeletal data acquired by the skeleton acquisition unit 294 from the multiple musculoskeletal model data 57C stored in the storage 32. For example, the musculoskeletal acquisition unit 296 calculates the similarity between the skeletal data acquired by the skeleton acquisition unit 294 and each skeletal part of the multiple musculoskeletal model data 57C using a known similarity calculation algorithm. Then, the musculoskeletal acquisition unit 296 selects the musculoskeletal model data having the skeletal part with the largest similarity as the musculoskeletal model data of the avatar. It is also possible to use a machine learning model when acquiring the musculoskeletal model data. In this case, for example, the musculoskeletal model data may be acquired using a musculoskeletal model generation model that outputs musculoskeletal model data corresponding to at least one of face data and skeletal data when at least one of face data and skeletal data is input.

[0052] The setting unit 298 sets an avatar model having as its elements a face represented by the face data acquired by the face acquisition unit 292 and a musculoskeletal structure represented by the musculoskeletal model data acquired by the musculoskeletal acquisition unit 296. A conversation takes place between the user and the chatbot via this avatar model.

[0053] The feature generator 300 generates acoustic features representing the features of the voice uttered through the avatar model based on the face data acquired by the face acquirer 292. As the acoustic features, for example, features such as a fundamental frequency and a Mel-frequency cepstrum coefficient can be adopted. Voice data, which will be described later, is generated according to the values ​​of these acoustic features. This is because the acoustic features of a voice uttered by a human can be determined by the facial expression of the human.

[0054] Specifically, the feature generating unit 300 uses the acoustic feature generation model 57D stored in the storage 32 to generate acoustic features from face data.

[0055] The acoustic feature generation model 57D is a machine learning model that outputs an acoustic feature when, for example, face data, which is a three-dimensional image, is input. The acoustic feature generation model 57D is trained in advance based on a plurality of training data in which training face data and training acoustic features are associated with each other. The acoustic feature generation model 57D may be any model that can be generated by a known machine learning algorithm.

[0056] For this reason, the feature generation unit 300 inputs face data acquired by the face acquisition unit 292 into the acoustic feature generation model 57D stored in the storage 32, and acquires the acoustic features output from the acoustic feature generation model 57D.

[0057] The acoustic feature of a voice uttered by a human is a physical quantity that may be determined by the facial features of the human, but may also be determined by the human skeleton. Since the voice reverberates in the facial skeleton and is output to the outside, people with similar skeletons tend to have similar voices. For this reason, the acoustic feature generation model 57D may be a machine learning model that outputs an acoustic feature when skeleton data is input. In this case, the acoustic feature generation model 57D is trained in advance based on a plurality of training data in which training skeleton data and training acoustic features are associated with each other. The acoustic feature generation model 57D may be any model that can be generated by a known machine learning algorithm.

[0058] The acoustic feature generation model 57D may be a machine learning model that outputs an acoustic feature when face data and skeletal data are input. It may be a machine learning model that outputs an acoustic feature when skeletal data is input. In this case, the acoustic feature generation model 57D is trained in advance based on a plurality of training data in which training face data, training skeletal data, and training acoustic features are associated with each other.

[0059] The dialogue unit 302 dialogues with the user who operates the smart device 14. Specifically, the dialogue unit 302 acquires the user's voice input received by the smart device 14, converts the voice input into text data, and inputs the text data to the data generation model 58. The data generation model 58 outputs a response in text data according to the input text data. The text data output from the data generation model 58 corresponds to the target text data issued via the avatar model. The dialogue unit 302 and the data generation model 58 realize the function of a chatbot capable of dialogue with the user.

[0060] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0061] The conversion unit 304 converts the text data output from the data generation model 58 into speech data having acoustic features generated by the feature generation unit 300. For example, if the fundamental frequency, which is an acoustic feature, is X, the text data is converted into speech data having a fundamental frequency of X.

[0062] The output unit 306 controls the avatar model to be displayed on the display 40A of the smart device 14 while moving each part of the face and each part of the musculoskeletal system, which are elements of the avatar model set by the setting unit 298, in accordance with the voice data obtained by the conversion unit 304. The output unit 306 also controls the voice data obtained by the conversion unit 304 to be output from the speaker 40B of the smart device 14. The speaker 40B is an example of a sound output unit. The way in which each part of the face and each part of the musculoskeletal system, which are elements of the avatar model, move can be simulated according to the speech represented by the voice data. For this reason, the output unit 306 simulates the movement of the musculoskeletal muscles and the movement of the face in accordance with the speech represented by the voice data, calculates the propagation of the force on the face surface associated with the movement, and generates a natural movement of the face of the avatar model. As a result, each part of the face and each part of the musculoskeletal system, which are elements of the avatar model, move in accordance with the voice data obtained by the conversion unit 304.

[0063] The dialogue unit 302 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion and the data generation model 58.

[0064] The output unit 306 transmits a control signal corresponding to the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. Specifically, the control unit 46A causes the display 40A to display the movement of the avatar model and the speaker 40B to output audio data based on the control signal output from the output unit 306 of the data processing device 12. The microphone 38B acquires audio indicating a user input for the result of the specific processing. The control unit 46A transmits the audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0065] Next, the operation of the data processing system 10 will be described with reference to FIG.

[0066] First, an example of the flow of the specific processing will be described with reference to Fig. 5. Note that the flow of the specific processing shown in Fig. 5 is an example of a "data processing method" according to the technology of the present disclosure.

[0067] In the specification process shown in FIG. 5, first, in step S100, the face acquisition unit 292 acquires face data of an avatar selected by a user, and sets the face data as the face data of the avatar.

[0068] In step S102, the skeleton acquisition unit 294 inputs the face data acquired by the face acquisition unit 292 in step S100 into the skeleton generation model 57B stored in the storage 32, and acquires the skeleton data output from the skeleton generation model 57B.

[0069] In step S104, the musculoskeletal acquisition unit 296 uses a known similarity calculation algorithm to calculate the similarity between the skeletal data acquired by the skeletal acquisition unit 294 in step S102 and each skeletal part of the multiple musculoskeletal model data 57C stored in the storage 32. Then, the musculoskeletal acquisition unit 296 acquires the musculoskeletal model data having the skeletal part with the greatest similarity as the musculoskeletal model data of the avatar.

[0070] In step S106, the setting unit 298 sets an avatar model having as its elements the face represented by the face data acquired in step S100 and the musculoskeletal structure represented by the musculoskeletal model data acquired in step S104. A conversation takes place between the user and the chatbot via this avatar model.

[0071] In step S108, the feature generation unit 300 inputs the face data acquired in step S100 into the acoustic feature generation model 57D stored in the storage 32, and acquires acoustic features output from the acoustic feature generation model 57D.

[0072] In step S110, the dialogue unit 302 acquires a voice input of the user that is accepted by dialogue with the user who operates the smart device 14, converts the voice input into text data, and inputs the text data to the data generation model 58. The data generation model 58 outputs a response in the form of text data according to the input text data. The dialogue unit 302 acquires the text data output from the data generation model 58.

[0073] In step S112, the conversion unit 304 converts the text data acquired in step S110 into voice data having the acoustic features generated in step S108.

[0074] In step S114, the output unit 306 controls the avatar model to be displayed on the display 40A of the smart device 14 while causing each part of the face and each part of the musculoskeletal system, which are elements of the avatar model set by the setting unit 298 in step S106, to move in accordance with the voice data obtained by the conversion unit 304 in step S112, and also controls the voice data obtained by the conversion unit 304 in step S112 to be output from the speaker 40B of the smart device 14.

[0075] As described above, the data processing device 12 acquires face data of an avatar realized by computer graphics. The data processing device 12 acquires skeletal data corresponding to the face data. The data processing device 12 acquires musculoskeletal model data representing a musculoskeletal structure corresponding to the skeletal data. The data processing device 12 sets an avatar model having a face represented by the face data and a musculoskeletal structure represented by the musculoskeletal model data as elements. The data processing device 12 generates an acoustic feature representing the characteristics of a voice uttered through the avatar model based on at least one of the face data and the skeletal data. The data processing device 12 converts target text data uttered through the avatar model into voice data having an acoustic feature. The data processing device 12 controls the avatar model to be displayed on the display unit while moving each part of the face and each part of the musculoskeletal structure, which are elements of the avatar model, in accordance with the voice data, and controls the voice data to be output from the voice output unit. Thereby, when the chatbot's speech is output as voice via an avatar realized by computer graphics, it is possible to output a voice suitable for the appearance of the avatar while making the movement of the avatar natural.

[0076] In the above embodiment, the avatar model is configured by musculoskeletal model data, but the present invention is not limited to this. For example, the avatar model may be configured to further include a skin model representing skin and a fat model representing fat in addition to the musculoskeletal model data. In this case, the output unit 306 of the data processing device 12 controls the skin and fat, which are elements of the avatar model, to move in accordance with the voice data. Specifically, the output unit 306 of the data processing device 12 calculates the propagation of force to the fat and skin accompanying the movement of the musculoskeletal model, and generates a natural movement of the avatar model.

[0077] In this way, to generate the avatar's facial expressions, the human skull, muscles, fascia, fat, and skin and the propagation of force are simulated, and the text data output from the chatbot is automatically generated in a voice that matches the avatar's bones, making it possible to output natural speaking movements while keeping the avatar's movements natural, even when the chatbot's speech is output as audio through an avatar realized by computer graphics.

[0078] The emotion identification model 59 may determine the emotion of the user according to a specific mapping. Specifically, the emotion identification model 59 may determine the emotion of the user according to an emotion map (see FIG. 6A), which is a specific mapping.

[0079] FIG. 6A is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0080] These emotions are distributed in the three o'clock direction of emotion map 400 and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0081] The inside of emotion map 400 represents what is going on inside one's mind, and the outside of emotion map 400 represents behavior, so the further out you go on emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0082] Here, human emotions are based on various balances such as posture and blood sugar level, and when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and battery level, so that when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. The emotion map may be generated, for example, based on the emotion map of Dr. Mitsuyoshi (Research on speech emotion recognition and emotion brain physiological signal analysis system, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). On the left half of the emotion map, emotions belonging to an area called "reaction" where sensation is dominant are lined up. On the right half of the emotion map, emotions belonging to an area called "situation" where situation recognition is dominant are lined up.

[0083] The emotion map defines two emotions that promote learning. The first is the negative emotion around the middle of "repentance" or "remorse" on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the positive emotion around "desire" on the response side. In other words, this is when the robot has positive feelings such as "I want more" or "I want to know more."

[0084] The emotion identification model 59 inputs the user input to a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user. This neural network is pre-trained based on multiple learning data that are combinations of the user input and emotion values ​​indicating each emotion shown in the emotion map 400. In addition, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 6B. FIG. 6B shows an example in which multiple emotions, "relief," "calm," and "encouraging," have similar emotion values.

[0085] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers, including computer 22.

[0086] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable computer-readable non-transitory storage medium such as a Universal Serial Bus (USB) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0087] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 upon request from the data processing device 12.

[0088] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0089] As the hardware resource for executing the specific process, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the specific process by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing the specific process, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), or an Application Specific Integrated Circuit (ASIC). Each processor has a built-in or connected memory, and each processor executes the specific process by using the memory.

[0090] The hardware resource that executes the specific process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0091] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0092] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. The specific processes described above are merely examples. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed without departing from the spirit of the invention.

[0093] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0094] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0095] 10 Data Processing System 12 Data Processing Device 14 Smart Devices

Claims

1. a face acquisition unit for acquiring face data of an avatar realized by computer graphics; a skeleton acquisition unit for acquiring skeleton data corresponding to the face data; a musculoskeletal acquisition unit that acquires musculoskeletal model data representing a musculoskeletal structure corresponding to the skeletal data; a setting unit that sets an avatar model having as elements a face represented by the face data and a musculoskeletal component represented by the musculoskeletal model data; a feature generating unit that generates an acoustic feature representing a feature of a voice uttered via the avatar model based on at least one of the face data and the skeletal data; A conversion unit that converts text data into speech data having the acoustic features; an output unit that controls a face represented by the face data and a musculoskeletal system represented by the musculoskeletal model data to operate in accordance with the voice data and to output the voice data from a voice output unit; A data processing device comprising:

2. the feature generation unit generates the acoustic feature by inputting at least one of the face data and the skeletal data to an acoustic feature generation model, which is a trained model generated in advance and which outputs the acoustic feature when at least one of the face data and the skeletal data is input; 2. A data processing apparatus according to claim 1.

3. The skeleton acquisition unit generates the skeleton data by inputting the face data into a skeleton generation model, which is a trained model that is generated in advance and that outputs the skeleton data when the face data is input.

3. A data processing device according to claim 1 or 2.

4. the avatar model further includes a skin model representing skin and a fat model representing fat, The output unit controls skin and fat, which are elements of the avatar model, to move in accordance with the voice data.

3. A data processing device according to claim 1 or 2.

5. Acquire facial data of an avatar realized by computer graphics, acquiring skeletal data corresponding to the face data; obtaining musculoskeletal model data representing a musculoskeletal structure corresponding to the skeletal data; setting an avatar model having as elements a face represented by the face data and a musculoskeletal structure represented by the musculoskeletal model data; generating an acoustic feature representing a feature of a voice uttered via the avatar model based on at least one of the face data and the skeletal data; Converting text data into speech data having the acoustic features; controlling the face represented by the face data and the musculoskeletal system represented by the musculoskeletal model data to operate in accordance with the voice data, and to output the voice data from a voice output unit; A data processing method in which processing is performed by a computer.

6. Acquire facial data of an avatar realized by computer graphics, acquiring skeletal data corresponding to the face data; obtaining musculoskeletal model data representing a musculoskeletal structure corresponding to the skeletal data; setting an avatar model having as elements a face represented by the face data and a musculoskeletal structure represented by the musculoskeletal model data; generating an acoustic feature representing a feature of a voice uttered via the avatar model based on at least one of the face data and the skeletal data; Converting text data into speech data having the acoustic features; controlling the face represented by the face data and the musculoskeletal system represented by the musculoskeletal model data to operate in accordance with the voice data, and to output the voice data from a voice output unit; A program for causing a computer to execute a process including the above.

Images