Programs, methods, and information processing devices.

Abstract

This technology provides a way to make changes in the facial features of an avatar appear more natural. [Solution] A program executed by a computer equipped with a processor, the program causing the processor to perform the steps of: sensing the movement of one or more facial parts of a performer; changing the appearance of one or more facial parts of an avatar corresponding to the performer based on the sensed movement of one or more facial parts; and receiving a setting for the degree to which the appearance of one or more facial parts of the avatar should follow the sensed result, and in the step of changing, the program changes the appearance of one or more facial parts of the avatar according to the setting for the degree.

Description

Technical Field

[0001] The present disclosure relates to a program, a method, and an information processing apparatus.

Background Art

[0002] Techniques for real-time reflecting of a user's expression and the like on an avatar are known.

[0003] Patent Document 1 describes a technique for estimating which Japanese vowel (a, i, u, e, o) is being generated from the combination of the frequencies of the first formant and the second formant of a person's voice, and changing the mouth shape of an avatar to the mouth shape corresponding to each vowel.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The technique of Patent Document 1 discloses a technique for estimating which Japanese vowel is being generated based on the voice acquired from a microphone, and determining and changing the mouth shape and size of an avatar. However, in the technique of Patent Document 1, it only changes the mouth shape of an avatar based on voice. For example, when changing the aspect of a part of the avatar's face other than the mouth, the degree of change cannot be sufficiently reflected from the user's voice, and the avatar may move unnaturally, giving a sense of discomfort to viewers. Therefore, there is a need for a technique to make the change in the aspect of the avatar's face part look more natural.

Means for Solving the Problems

[0006] According to one embodiment, a program is provided which is executed by a computer equipped with a processor, and the program causes the processor to perform the steps of: sensing the movement of one or more facial parts of a performer; changing the appearance of one or more facial parts of an avatar corresponding to the performer based on the sensed movement of one or more facial parts; and receiving a setting for the degree to which the appearance of one or more facial parts of the avatar should follow the sensed result, and in the step of changing, the program changes the appearance of one or more facial parts of the avatar according to the setting for the degree. [Effects of the Invention]

[0007] According to this disclosure, it is possible to provide a technology that makes changes in the appearance of facial features of an avatar appear more natural. [Brief explanation of the drawing]

[0008] [Figure 1] Block diagram showing the overall configuration of System 1. [Figure 2] This diagram shows the functional configuration of terminal device 10. [Figure 3] This diagram shows the functional configuration of server 20. [Figure 4] This describes the data structure of the user information database (DB), avatar information DB, and wearable device information DB stored in the memory unit of server 20. [Figure 5] This flowchart illustrates a series of processes involved in acquiring the audio spectrum of a user's speech and, based on that acquired spectrum, changing the mouth shape of an avatar corresponding to the user in accordance with the performer's speech. [Figure 6] This is an example screen showing how a user registers their vowel speech spectrum with System 1. [Figure 7] This shows an example screen where a user can adjust the degree of change in the appearance of their avatar's mouth or other facial features. [Figure 8]This example screen shows how to estimate one or more possible emotions of the user from the user's utterances, and how to change the appearance of the avatar based on those estimated emotions. [Figure 9] This example shows a screen where a user can configure various settings for an avatar with attributes different from those of a human, based on factors such as the voice spectrum. [Figure 10] This flowchart represents a series of processes for sensing the movement of one or more facial features of a user, and then changing the appearance of one or more facial features of an avatar corresponding to the user based on the sensed movement of one or more facial features. [Figure 11] This shows an example screen for sensing the movement of one or more facial features of a user and changing the appearance of one or more facial features of a corresponding avatar based on the sensed movement of one or more facial features. [Figure 12] This example shows how the system estimates one or more emotions from the user and changes the appearance of one or more facial features of the corresponding avatar based on the emotion selected by the user. [Figure 13] This example screen shows how to set the degree of change in the avatar's appearance when sensing results cannot be obtained for at least one of the associated parts of one or more facial parts of the user. [Figure 14] This shows an example screen for correcting the degree of change in the avatar's appearance when the user is wearing wearable devices such as glasses. [Figure 15] This example shows how the avatar's mouth shape changes based on the degree of change in speech when the user's mouth movements cannot be sensed. [Figure 16] This is an example screen showing a notification to the user when the difference in the degree of settings between pre-associated facial parts of an avatar exceeds a predetermined threshold. [Figure 17]When presenting a predetermined notification to the user, a screen example is shown when presenting to the user the state in which at least one or a plurality of facial parts change when the degree of difference is set within a predetermined range.

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described while referring to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated.

[0010] <First Embodiment> <Overview> In the following embodiments, a technique for changing the mouth aspect of an avatar based on the voice spectrum of a user who is a performer operating the avatar will be described. Here, the devices and the like appropriately used when realizing the technology according to the present disclosure are not limited, and in some cases, they may be presented from a stationary PC (Personal Computer) as well as terminal devices such as smartphones and tablet terminals possessed by the user.

[0011] A technique for controlling the movement of the mouth of an avatar based on the voice of a user acquired via a sound collection device such as a microphone is known. However, in this system, the actual mouth movement of the user and the movement of the avatar may not be accurately synchronized, which may give a sense of discomfort to the viewer.

[0012] Therefore, in System 1, a technique for making the change in the mouth aspect of the avatar look more natural is provided.

[0013] System 1 can be used, for example, in live streaming broadcasts on video distribution sites, where an avatar that tracks the movements of the user (performer) is used. For example, System 1 tracks the user's movements via a camera (imaging device) installed in the user's terminal device (PC, etc.) and reflects this in the avatar's movements. System 1 also acquires the audio spectrum of the performer's speech via a microphone (sound collection device) also installed in the user's terminal device, and based on the acquired audio spectrum, changes the mouth shape of the avatar corresponding to the performer in accordance with the performer's speech. At this time, System 1 presents the viewer with an avatar corresponding to the performer and the performer's voice, and accepts the setting to change the degree to which the avatar's mouth shape changes in accordance with the performer's speech, to a degree lower than the change in the performer's speech. By executing this process, System 1 may change the avatar's mouth shape in accordance with the performer's speech. This makes the changes in the avatar's mouth shape appear more natural.

[0014] <1 System Configuration Diagram> Figure 1 shows the overall configuration of System 1 in the first embodiment.

[0015] As shown in Figure 1, System 1 includes multiple terminal devices (Figure 1 shows terminal devices 10A and 10B; hereinafter, these may be collectively referred to as "terminal device 10"; further terminal devices 10C, etc., may also be included in the configuration) and a server 20. The terminal devices 10 and the server 20 communicate with each other via a network 80.

[0016] Terminal device 10 is a device operated by each user. Terminal device 10 is implemented by a mobile terminal such as a smartphone or tablet that is compatible with a mobile communication system. Alternatively, terminal device 10 may be a stationary PC (Personal Computer), a laptop PC, etc. As shown as terminal device 10B in Figure 1, terminal device 10 includes a communication interface 12, an input device 13, an output device 14, a memory 15, a storage unit 16, and a processor 19. Server 20 includes a communication interface 22, an input / output interface 23, a memory 25, storage 26, and a processor 29.

[0017] Terminal device 10 is connected to server 20 via network 80 in a way that allows communication. Terminal device 10 is connected to network 80 by communicating with communication equipment such as a wireless base station 81 that supports communication standards such as 5G and LTE (Long Term Evolution), and a wireless LAN router 82 that supports wireless LAN (Local Area Network) standards such as IEEE (Institute of Electrical and Electronics Engineers) 802.11.

[0018] The communication interface 12 is an interface for inputting and outputting signals so that the terminal device 10 can communicate with external devices. The input device 13 is an input device (e.g., a touch panel, touchpad, mouse or other pointing device, keyboard, etc.) for receiving input operations from the user. The output device 14 is an output device (display, speaker, etc.) for presenting information to the user. The memory 15 is for temporarily storing programs and data processed by programs, etc., and is a volatile memory such as DRAM (Dynamic Random Access Memory). The storage unit 16 is a storage device for saving data, such as flash memory or an HDD (Hard Disk Drive). The processor 19 is hardware for executing the instruction set written in the program, and is composed of an arithmetic unit, registers, peripheral circuits, etc.

[0019] Server 20 manages information that users set when performing live streaming using avatars, etc. For example, Server 20 stores user information, avatar information, and information about the wearable device the user is wearing.

[0020] Communication IF22 is an interface for inputting and outputting signals so that the server 20 can communicate with external devices. Input / Output IF23 functions as an interface to an input device for receiving input operations from the user and an output device for presenting information to the user. Memory 25 is for temporarily storing programs and data processed by programs, etc., and is a volatile memory such as DRAM (Dynamic Random Access Memory). Storage 26 is a storage device for saving data, such as flash memory or HDD (Hard Disk Drive). Processor 29 is hardware for executing the instruction set written in the program, and is composed of an arithmetic unit, registers, peripheral circuits, etc.

[0021] In this embodiment, each device (terminal device, server, etc.) can also be considered as an information processing device. That is, the collection of each device can be considered as a single "information processing device," and System 1 may be formed as a collection of multiple devices. The way in which the multiple functions required to realize System 1 according to this embodiment are distributed to one or more hardware can be appropriately determined in view of the processing capacity of each hardware and / or the specifications required for System 1.

[0022] <1.1 Configuration of terminal device 10> Figure 2 is a block diagram of the terminal device 10 that constitutes System 1 of Embodiment 1. As shown in Figure 2, the terminal device 10 includes a plurality of antennas (antenna 111, antenna 112), wireless communication units corresponding to each antenna (first wireless communication unit 121, second wireless communication unit 122), an operation reception unit 130 (including a touch-sensitive device 1301 and a display 1302), an audio processing unit 140, a microphone 141, a speaker 142, a position information sensor 150, a camera 160, a motion sensor 170, a storage unit 180, and a control unit 190. The terminal device 10 also has functions and configurations not specifically shown in Figure 2 (for example, a battery for maintaining power, a power supply circuit for controlling the supply of power from the battery to each circuit, etc.). As shown in Figure 2, each block included in the terminal device 10 is electrically connected by a bus or the like.

[0023] Antenna 111 radiates signals emitted by terminal device 10 as radio waves. Antenna 111 also receives radio waves from space and provides the received signals to first wireless communication unit 121.

[0024] Antenna 112 radiates signals emitted by terminal device 10 as radio waves. Antenna 112 also receives radio waves from space and provides the received signals to second wireless communication unit 122.

[0025] The first wireless communication unit 121 performs modulation and demodulation processing, etc., for the terminal device 10 to transmit and receive signals via the antenna 111 in order to communicate with other wireless devices. The second wireless communication unit 122 performs modulation and demodulation processing, etc., for the terminal device 10 to transmit and receive signals via the antenna 112 in order to communicate with other wireless devices. The first wireless communication unit 121 and the second wireless communication unit 122 are a communication module that includes a tuner, an RSSI (Received Signal Strength Indicator) calculation circuit, a CRC (Cyclic Redundancy Check) calculation circuit, a high-frequency circuit, etc. The first wireless communication unit 121 and the second wireless communication unit 122 perform modulation and demodulation, and frequency conversion, of the wireless signals transmitted and received by the terminal device 10, and provide the received signal to the control unit 190.

[0026] The operation reception unit 130 has a mechanism for receiving user input operations. Specifically, the operation reception unit 130 is configured as a touchscreen and includes a touch-sensitive device 1301 and a display 1302. The touch-sensitive device 1301 receives user input operations of the terminal device 10. The touch-sensitive device 1301 detects the user's contact position with the touch panel, for example, by using a capacitive touch panel. The touch-sensitive device 1301 outputs a signal indicating the user's contact position detected by the touch panel to the control unit 190 as an input operation. The terminal device 10 may also be equipped with a physically usable keyboard (not shown) and may receive user input operations via the keyboard.

[0027] The display 1302 displays data such as images, videos, and text in accordance with the control of the control unit 190. The display 1302 is implemented by, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0028] The audio processing unit 140 modulates and demodulates the audio signal. The audio processing unit 140 modulates the signal received from the microphone 141 and provides the modulated signal to the control unit 190. The audio processing unit 140 also provides the audio signal to the speaker 142. The audio processing unit 140 is implemented, for example, by an audio processing processor. The microphone 141 receives an audio input and provides the audio signal corresponding to that audio input to the audio processing unit 140. The speaker 142 converts the audio signal received from the audio processing unit 140 into sound and outputs the sound to the outside of the terminal device 10.

[0029] The location information sensor 150 is a sensor that detects the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. A GPS module is a receiving device used in a satellite positioning system. In a satellite positioning system, signals are received from at least three or four satellites, and the current position of the terminal device 10, which is equipped with a GPS module, is detected based on the received signals. The location information sensor 150 may also be a transmitting and receiving device based on a communication standard used in short-range communication systems between information devices. Specifically, the location information sensor 150 uses the 2.4GHz band, such as a Bluetooth® module, to receive beacon signals from other information devices equipped with a Bluetooth® module.

[0030] Camera 160 is a device that receives light using a photodetector and outputs it as an image. Camera 160 is, for example, a depth camera that can detect the distance from camera 160 to the object being photographed. Furthermore, camera 160 acquires the body movements of the user using the terminal device 10. Specifically, for example, camera 160 acquires the user's mouth movements and the movements of various parts of the face (eyes, eyebrows, etc.). Any existing technology may be used to acquire these movements.

[0031] The motion sensor 170 is composed of a gyroscope, an accelerometer, and the like, and detects the tilt of the terminal device 10.

[0032] The storage unit 180 is composed of, for example, flash memory and stores data and programs used by the terminal device 10. In certain situations, the storage unit 180 stores user information 1801, avatar information 1802, wearable device information 1803, etc. This information is held in the storage unit 180 of the terminal device 10, or it may be stored as a database in the storage unit 202 of the server (described later) and retrieved via the network 80.

[0033] User information 1801 includes information such as an ID that identifies the user, a username, and information about the avatar corresponding to the user. Here, "user" refers to the performer who controls the avatar based on information obtained via microphone 141 or camera 160. Details of the information included in user information will be described later.

[0034] Avatar information 1802 contains various information about the avatar corresponding to the user. For example, avatar information 1802 holds information such as the corresponding user and the settings the user normally uses, and is referenced by the user to smoothly operate the avatar during live streaming and other broadcasts. Details of the information contained in the avatar information will be described later. The settings that users typically use include, for example, the basic settings for the degree of change in the avatar's appearance, the emotions displayed by default during regular broadcasts, and the user's sensing sensitivity. These are parameters and conditions that users can adjust when broadcasting using an avatar.

[0035] Wearable device information 1803 contains various information about the wearable device the user is wearing at the time of delivery. This information includes, for example, the following: Types of wearable devices • Size of wearable devices • Transmittance of wearable devices • Whether or not electronic information can be obtained. Wearable device information 1803 holds various information about the user's eyewear, such as eyeglasses, smart glasses, and head-mounted displays (HMDs). Details of the information included in wearable device information 1803 will be described later.

[0036] The control unit 190 controls the operation of the terminal device 10 by reading the program stored in the memory unit 180 and executing the instructions contained in the program. The control unit 190 is, for example, an application processor. By operating according to the program, the control unit 190 performs the functions of an input operation receiving unit 1901, a transmitting / receiving unit 1902, a data processing unit 1903, and a notification control unit 1904.

[0037] The input operation reception unit 1901 processes input operations from the user to an input device such as a touch-sensitive device 131. Based on the coordinate information of the touch-sensitive device 1301 where the user's finger or the like has made contact, the input operation reception unit 1901 determines the type of operation, such as whether the user's operation is a flick operation, a tap operation, or a drag (swipe) operation.

[0038] The transmitting / receiving unit 1902 performs processing to enable the terminal device 10 to send and receive data with an external device such as a server 20 in accordance with a communication protocol.

[0039] The data processing unit 1903 performs calculations on the data received as input by the terminal device 10 according to the program and outputs the calculation results to memory or the like.

[0040] The data processing unit 1903 receives the movements of the user's mouth and other objects acquired by the camera 160 and controls the processing to execute various operations. For example, the data processing unit 1903 executes a process to control the mouth movements of the user and the corresponding avatar based on the user's mouth movements acquired by the camera 160.

[0041] The notification control unit 1904 performs the following processes: displaying the image on the display 132, outputting sound to the speaker 142, and generating vibrations in the camera 160.

[0042] <1.2 Functional Configuration of Server 20> Figure 3 shows the functional configuration of server 20. As shown in Figure 3, server 20 functions as a communication unit 201, a storage unit 202, and a control unit 203.

[0043] The communications unit 201 performs processing to enable the server 20 to communicate with external devices.

[0044] The memory unit 202 stores data and programs used by the server 20. The memory unit 202 stores user information database 2021, avatar information database 2022, wearable device information database 2023, etc.

[0045] User Information Database 2021 is a database for holding various information about performers who control avatars. Details of each record held in this database will be described later.

[0046] The Avatar Information Database 2022 is a database for storing various information about avatars operated by users. Further details will be provided later.

[0047] The Wearable Device Information Database 2023 is a database for storing various information about eyewear worn by users who control avatars. Further details will be provided later.

[0048] The control unit 203 is composed of, for example, a processor 29, and by performing processing according to the program, this processor 29 performs functions such as the reception control module 2031, transmission control module 2032, user information acquisition module 2033, avatar information acquisition module 2034, voice spectrum acquisition module 2035, avatar change module 2036, avatar presentation module 2037, setting reception module 2038, wearable device information acquisition module 2039, and change correction module 2040, as various modules.

[0049] The receive control module 2031 controls the process by which the server 20 receives signals from external devices according to a communication protocol.

[0050] The transmission control module 2032 controls the process by which the server 20 transmits signals to external devices according to a communication protocol.

[0051] The user information acquisition module 2033 controls the process of acquiring various information about the user who is the performer controlling the avatar. This information includes, for example, the following: • User's name, identification ID • Information about the avatar corresponding to the user • Devices worn by the user (e.g., glasses) Specifically, for example, the user information acquisition module 2033 may acquire the information by referring to user information 1801 from the storage unit 180 of the terminal device 10 used by the user. Alternatively, the user information acquisition module 2033 may acquire the information by referring to the user information database 2021 held in the storage unit 202 of the server 20, which will be described later. In addition, the user information acquisition module 2033 may acquire the information by directly receiving various information about the user from the user.

[0052] The avatar information acquisition module 2034 acquires various information about the avatar operated by the user. This information includes, for example, the following: • Information about the avatar identification ID • Avatar attribute information (human, non-human, etc.) • Information on the mouth, face, or other body part characteristics that the corresponding user has set as their default. • Dedicated settings for the mouth, face, or other body parts, which are individually configured for each avatar. Specifically, for example, the avatar information acquisition module 2034 may acquire information about the avatar associated with each user by referring to the avatar information 1802 or the user information database 2021. Furthermore, in certain situations, the server 20 may receive input from the user regarding the degree of change in the appearance of the avatar's mouth, face, or other body parts, and if it receives an operation to reset the said setting as the default, it may perform a process to update the avatar information held in the avatar information 1802, etc. This allows the user to set frequently used settings among the degree of change in the avatar's appearance as the default, making it easier to operate the avatar.

[0053] Furthermore, each user does not necessarily have to have only one set of avatar information. For example, a user may have multiple avatars pre-associated with them, or additional avatars may be associated with them later.

[0054] Furthermore, in certain situations, the avatar ID may include the following information: • Information regarding the avatar's appearance (gender, eye color, hairstyle, mouth, facial features, or the size of other body parts, hair color, skin color, etc.) • Information regarding the appearance of the avatar's mouth, face, or other body parts (such as the number of types of appearances that change, the amount of change in those appearances, etc.) Here, server 20 may store the above avatar information in association with the type of content. Specifically, for example, server 20 may associate the type of content that the user provides in live streaming, etc. (chatting, singing, acting, etc.) with the settings for the degree of change in the appearance of the mouth, face, and other body parts, and when it receives an operation from the user to select which content to provide, it may reflect the settings corresponding to that content in the avatar.

[0055] This allows users to appropriately change the appearance of their avatar to match the content they are providing to the audience, thereby creating a more immersive experience for the viewers.

[0056] Furthermore, after the server 20 receives the user's selection of an avatar to use, it may not automatically reflect the setting of the degree of change in appearance, but instead accept the user's selection and simultaneously present the user with a predetermined notification (such as a dialog box). For example, after the user selects an avatar to use, the server 20 may present a notification such as "Is it okay to use the normal settings?" if the user selects a general-purpose setting that is normally used, rather than a dedicated setting set for each avatar.

[0057] This prevents users from live streaming with incorrect settings, thus preventing a loss of viewer immersion.

[0058] Furthermore, the server 20 may accept and reflect the setting of the degree of change in the above-mentioned manner during the live stream. Specifically, for example, if the server 20 receives a change in the setting of the degree of change in manner from a user during the live stream, after receiving the setting change, it may change the manner of the avatar based on the user's voice spectrum acquired after a predetermined time has elapsed, or the user's sensing results, by performing a process to change the manner of the avatar based on the changed setting. This allows users to change and reflect the avatar's appearance settings as needed during live streaming, ensuring that users can see the avatar's appearance changes seamlessly even when switching content during the live stream.

[0059] The voice spectrum acquisition module 2035 controls the process of acquiring the voice spectrum of the user's speech. Specifically, for example, the voice spectrum acquisition module 2035 controls the process of acquiring the voice spectrum from the voice spoken by the user acquired via the microphone 141. For example, the voice spectrum acquisition module 2035 acquires the user's voice via the microphone 141 and acquires the voice spectrum contained in that voice. For example, the voice spectrum acquisition module 2035 may perform a Fourier transform on the voice acquired from the microphone 141 and acquire information on the voice spectrum contained in the voice. In this case, the operation for acquiring the voice spectrum is not limited to the Fourier transform, but may be any existing method. Furthermore, in certain situations, the voice spectrum acquisition module 2035 may acquire vowel voice spectrum information from the user's voice. For example, the voice spectrum acquisition module 2035 may pre-accept vowel settings to be input by the user, and then receive the user's speech via the microphone 141, thereby associating and storing the accepted vowel settings with the acquired voice spectrum. Furthermore, in certain situations, the speech spectrum acquisition module 2035 may acquire speech information originating from consonants, such as sounds like "t," "c," "h," "k," "m," "r," "s," "n," and "w," and combine this with the stored vowel information to estimate the words spoken by the user. This allows System 1 to separately characterize and store the vowel-related parts of the user's voice spectrum, enabling it to more accurately change the movement of the avatar's mouth.

[0060] The avatar transformation module 2036 controls the process of changing the mouth shape of the avatar corresponding to the performer in accordance with the performer's speech, based on the acquired voice spectrum. Specifically, for example, the avatar transformation module 2036 estimates the words spoken by the user from the user's voice spectrum acquired by the voice spectrum acquisition module 2035, and changes the mouth shape of the avatar according to the estimated words. For example, the avatar transformation module 2036 estimates the vowel information spoken by the user from the user's voice spectrum and changes the mouth shape to match that vowel. For example, if the user's voice spectrum acquired by the voice spectrum acquisition module 2035 is "a", the avatar transformation module 2036 changes the mouth shape of the avatar to a shape corresponding to "a".

[0061] The avatar presentation module 2037 controls the process of presenting the avatar corresponding to the performer and the user's voice to the audience. Specifically, for example, the avatar presentation module 2037 transmits the video of the avatar corresponding to the user and the user's voice to the display 1302 and speaker 142 of the terminal device 10 used by the audience, and presents them to the audience. In this case, there is not necessarily only one audience, and the avatar and voice may be presented to the terminal devices 10 of multiple audiences.

[0062] The setting reception module 2038 controls the process of accepting settings that allow the degree to which the avatar's mouth shape changes in response to user speech to be lower than the degree of change in user speech. Specifically, for example, the setting reception module 2038 accepts from the user a setting for the time interval at which the user's speech is reflected in the avatar's mouth shape, as a setting for the degree to which the avatar's mouth shape changes. For example, the setting reception module 2038 may accept settings including the following: - Setting the time required for the avatar's mouth shape to change to a shape corresponding to the speech voice estimated from the user's voice spectrum. • Setting the frequency at which the avatar's behavior is changed and updated based on user speech within a certain period (e.g., number of updates per second). Here, we define changes in user speech. Changes in user speech include, for example, the rate of user speech, which may be calculated based on the following. • The time interval at which the vowel spoken by the user changes (for example, the time interval at which the vowel changes from "a" to "i"). In this case, the server 20 may simultaneously acquire sounds derived from consonants (such as c and k), and even if it acquires the same vowel consecutively, it may estimate the speech speed by assuming that different words are being spoken. • Number of vowels uttered within a specified period In this case, the setting reception module 2038 may accept the setting to a degree lower than the degree of avatar change estimated from the user's speech changes. For example, the setting reception module 2038 may accept the time required to change the mouth shape to correspond to the speech (vowel) estimated from the user's voice spectrum in advance. Based on the accepted time required, the server 20 obtains information on the time the user uttered a vowel from the user's voice spectrum, calculates the ratio with the pre-set time required, multiplies it by the amount of change in shape, and calculates the amount of change in the avatar's mouth shape. Based on the acquired speech time and the amount of change, the server 20 changes the mouth shape. For example, suppose the user inputs a setting where the avatar's mouth changes to the "a" shape in a time of "1 second". For example, let's say "100" represents the time when it becomes completely the "a" shape, and the setting is made so that it becomes "100" in "1 second". At this time, the server 20 may also receive information from the user regarding the degree to which the mouth shape changes per second (the amount and speed of change in the mouth). (That is, within the time that the mouth shape changes per second, a difference in the amount of change in the mouth shape may be set between the first 0.5 seconds and the remaining 0.5 seconds.) If the user utters the sound "ah" for 1 second, Server 20 will change the shape of the avatar's mouth to the shape of "ah" over 1 second, based on the change amount settings described above. However, if the user utters "ah" for only 0.5 seconds, Server 20 may change the amount of change in the avatar's mouth shape up to 50.

[0063] Furthermore, if the user speaks continuously (for example, "aiueo"), the server 20 may obtain the speech duration of each vowel and perform the above processing. In other words, the server 20 may calculate the amount of change in the mouth shape of the avatar corresponding to each vowel from the speech duration of each vowel and change the mouth shape of the avatar. For example, if the time required for the avatar's mouth shape to change to the shape corresponding to each vowel is "1 second", and the user speaks "a" for "0.2 seconds" and "i" for "0.3 seconds", the amount of change corresponding to "a" is "20", and the amount of change corresponding to "i" is "30". Also, if the user speaks for a longer time than the required time, the server 20 may maintain the avatar's mouth shape even after the required time has elapsed. This allows the user to change the avatar's mouth shape gradually according to the duration of their speech, rather than instantly changing it to an "a" shape when they utter the sound "a". Furthermore, by setting a time limit and multiplying the change in mouth shape by that limit if the speech is shorter than the set time, it is possible to prevent the avatar's mouth shape from changing drastically even when the user speaks lightly (for example, changing to a mouth shape of 100 even if the mouth opening is only about 30). This prevents the user from causing viewers to feel a sense of incongruity between the user's speech and the changes in the avatar's mouth shape, thus creating a more immersive experience for the viewers.

[0064] In certain situations, the server 20 may change the shape of the avatar's mouth based on information such as the magnitude and pitch of the audio spectrum obtained from the user. Specifically, for example, the server 20 may obtain information on the frequency (Hz) and sound pressure (dB) of the user's audio spectrum and change the shape of the avatar's mouth if this information exceeds a predetermined threshold. For example, suppose the server 20 accepts a setting to change the shape of the avatar's mouth in a time of "1 second," and the user's speaking time is "1 second." In this case, if the server 20 detects that the user has spoken at a sound pressure exceeding the threshold at "0.8 seconds," it may change the shape of the avatar's mouth more significantly than usual (to a shape where the mouth is wide open). In this case, the server 20 may also apply the same settings to parts of the face and body, not just the mouth. This allows users to reflect sudden shouting in the avatar's mouth movements, resulting in more natural-looking avatar movements for viewers.

[0065] In addition, the setting reception module 2038 may accept settings for the degree of change in the avatar's mouth shape, such that the update frequency of the avatar's movements is lower than the update frequency estimated from the speech rate estimated from the user's utterances. Subsequently, the server 20 transmits the information set by the setting reception module 2038 to the avatar change module 2036, changes the shape of the avatar's mouth according to the setting, and then the avatar presentation module 2037 presents the avatar and the user's voice to the viewer. This allows users to change the avatar's mouth shape more gradually than vowel changes, resulting in smoother changes in the avatar's mouth shape that better match their own speech. This prevents the avatar's mouth shape from moving too subtly and unnaturally, allowing for more natural mouth movements for the viewer and enhancing their immersion.

[0066] The wearable device information acquisition module 2039 controls the process of acquiring information about the wearable device being worn by the user. Specifically, for example, after acquiring user information, the wearable device information acquisition module 2039 refers to the wearable device information database 2023 (described later) and acquires information about the wearable device being worn by the user. The server 20 transmits the acquired wearable device information to the change correction module 2040.

[0067] The change correction module 2040 controls the process of correcting the setting of the degree of change in the appearance to be reflected in the avatar, based on the wearable device information acquired by the wearable device information acquisition module 2039. Specifically, for example, the change correction module 2040 corrects the setting of the degree of change in the appearance of parts of the user's face that are covered or obscured by the wearable device, based on the wearable device information acquired by the wearable device information acquisition module 2039. The server 20 acquires the sensing results of predetermined parts of the user's face (mouth, eyes, eyebrows, nose, etc.), and changes the appearance of the avatar by reflecting these sensing results and the settings received from the user (the degree to which the sensing results are reflected, parameter settings, etc.). In this case, for example, if the user is wearing glasses, the change correction module 2040, upon receiving information from the user regarding the degree of change in the appearance of the avatar's eyes corresponding to the user, corrects and reflects this based on a preset correction value. Correction refers to a process in which, for example, if the accuracy of sensing parts of the user's face decreases for each wearable device, the rate of decrease (or attenuation rate) is set in advance, and the degree to which the movement is reflected in the avatar during sensing and tracking is corrected based on this setting. This allows users to naturally present changes in their avatar's appearance to viewers, even when wearing glasses or other eyewear.

[0068] In addition, the change correction module 2040 may correct the degree of change in the avatar's appearance according to the avatar attributes acquired by the avatar information acquisition module 2034. Specifically, for example, the change correction module 2040 may acquire information on whether the avatar operated by the user is a human or a non-human entity whose manner of change differs from that of a human, and based on this information, it may perform a process to correct the degree of change in the avatar's manner. For example, if the attribute of the avatar operated by the user is "dragon," the movements of the eyes, mouth, etc., may behave differently from those of a human. In that case, the change correction module 2040 may correct the amount of change in the corners of the mouth, the amount of change in the eyeballs, etc., to conform to the avatar's characteristics, based on the "dragon" attribute. This allows users to present viewers with more natural movements based on their own speech and facial sensing results, even when controlling an avatar that is not human.

[0069] It should be noted that the above configuration is not essential in the embodiments described herein. That is, the terminal device 10 may assume the role of the server 20 and perform the same processing as the various modules that constitute the control unit 203 of the server 20. Furthermore, the terminal device 10 may perform the various functions disclosed in this invention based on information acquired via the microphone 141, camera 160, etc., provided on the terminal device 10, without going through the network 80.

[0070] <2 Data Structure> Figure 4 shows the data structures of the user information database 2021, avatar information database 2022, and wearable device information database 2023 stored by server 20.

[0071] As shown in Figure 4, the user information database 2021 includes items such as "ID", "Corresponding Avatar", "Device Used", "Dedicated Preset (Mouth)", "Dedicated Preset (Face)", "Basic Settings", "Frequently Used Emotions", and "Remarks".

[0072] The "ID" field is information that identifies each user who controls the avatar.

[0073] The item "Corresponding Avatar" is information that identifies each avatar corresponding to each user.

[0074] The item "Devices Used" is information that identifies each device worn by the user, such as the wearable devices the user is wearing.

[0075] The item "Dedicated Preset (Mouth)" is information that indicates the degree to which the mouth shape of the avatar changes when each user operates the avatar, with conditions pre-set for each user. Specifically, for example, it indicates various conditions individually set for the avatar when the avatar operated by the user is in a certain situation (for example, when the mouth shape changes significantly). The information included in the preset may include, for example, information such as the height of the corners of the mouth and the shape of the lips. Server 20 may accept the selection of the preset from the user, reflect the setting in the avatar, and present it to the viewer. This allows users to instantly reflect and present to viewers the changes in mouth shape specific to their avatar, resulting in a more natural-looking avatar movement.

[0076] The item "Dedicated Preset (Face)" is information that indicates the degree to which the appearance of the facial features of the avatar changes when each user operates the avatar, with conditions pre-set for each user. Specifically, for example, it indicates various conditions individually set for the avatar when the avatar operated by the user is in a certain situation (for example, when the avatar's facial expression changes significantly). The information included in the preset may include, for example, information such as eyebrow direction, eye shape, pupil size, cheek flushing, speech, or sensing of the user's facial expression. Server 20 may accept the selection of the preset from the user, reflect the setting in the avatar, and present it to the viewer. For example, suppose a user is using an avatar with non-human attributes (monster, inanimate object, robot, etc.). In that case, changes in the appearance of various parts of the avatar (mouth, face, body) may not perfectly match the user's voice spectrum and sensing results. Therefore, server 20 may accept the user to set a dedicated preset (mouth) or dedicated preset (face) as exemplified above. This allows the user to select such a preset during live streaming, etc., so that the user's voice spectrum and sensing results are reflected in the changes in the avatar's appearance without any sense of incongruity, regardless of the avatar used.

[0077] Furthermore, server 20 may also accept dedicated preset settings depending on the type of content provided by the user. For example, when a user streams a song, it may accept settings that significantly alter the avatar's mouth, facial features, and body parts compared to when chatting. This allows users to instantly reflect and present to viewers changes in facial features specific to their avatar, resulting in a more natural-looking avatar movement.

[0078] The "Basic Settings" item indicates the degree of change typically used by the user. Specifically, it indicates the conditions for the degree of change typically (generally) used by a user operating an avatar when changing the appearance of the mouth, face, or other body parts during regular broadcasts, live broadcasts, and live streaming. For example, the conditions may include the degree to which the avatar's movements are made to follow the user's sensing results. The degree to which the avatar's movements are made to follow the sensing results includes, for example, the degree of sensitivity, where 100 is the degree to which the sensing results are directly reflected in the changes in the avatar's appearance; the amount of change in the avatar's appearance compared to the amount of change in the user's face; and the degree to which the amount of change reflected in the avatar's movements is compared to the amount of change in the avatar per unit time estimated from the sensing results. This allows users to easily start broadcasting without having to set the degree of change each time they broadcast.

[0079] In this case, server 20 may accept a combination of basic settings and dedicated presets as content-specific settings. Specifically, for example, content-specific avatar settings are exemplified below. • ASMR (Autonomous Sensory Meridian Response) mode (whispering mode) Use a dedicated preset for the mouth (low sensitivity to create a soft, whispering sound), while using the basic settings for facial expressions. Alternatively, use dedicated facial expression settings in conjunction with the preset. • Action game streaming mode For the mouth, use a dedicated preset (with increased sensitivity to create an exaggerated reaction), and also increase the sensitivity for facial expressions. • Horror game streaming mode For the mouth, use a dedicated preset (low sensitivity and a low threshold for detection frequencies), and apply the same sensitivity settings to facial expressions. Alternatively, use a dedicated setting. • Casual chat mode (using default settings)

[0080] Furthermore, in certain situations, the server 20 may, in relation to switching between the above modes, present a switch button to the user and accept the user's press of the button, thereby reflecting the degree of change in the avatar's appearance based on the mode. In this case, the server 20 may present the toggle button to the user in a manner that is not visible to the viewer but is visible to the user. The server 20 may also change the placement of the toggle button based on user input. This allows users to choose from different presets depending on the content they are providing to their audience, enabling a wider range of expression.

[0081] Furthermore, server 20 may store associated information about the background in the virtual space and may change the background in response to a mode switch. In addition, server 20 may store associated predetermined objects, as exemplified below, and display those objects in the virtual space in response to a mode switch. • Equipment objects such as microphones and musical instruments used during live music streaming. • Objects related to gaming devices during game streaming • General-purpose objects (houseplants, room furniture, etc.) This allows server 20 to reduce the loading process when switching modes, preventing delays and other issues that might cause discomfort to viewers.

[0082] The above settings may be used in combination with basic settings, etc. The combination may be any setting accepted by the user, and may be stored in the memory as a user-specific combination. The server 20 may also acquire usage frequency information for multiple presets. Based on this usage frequency information, the server 20 may notify the user whether to save frequently used presets as "frequently used settings" or "basic settings". When the server 20 receives an instruction from the user to set a preset as "frequently used settings", etc., it may save that preset as "frequently used settings" in the memory.

[0083] The item "Frequently Used Emotions" indicates the settings for emotions that the user frequently uses when operating their avatar. Specifically, for example, if a user frequently uses the emotion "joy" during a live stream, the server 20 may store in the database in advance the conditions for changes in the avatar's appearance based on that emotion. In this case, the conditions for changes in appearance include the degree of change in the mouth's appearance, the amount of change in various parts of the face that move when expressing the emotion of "joy," the degree to which the avatar follows the sensing results, and so on. When server 20 receives a selection of the emotion setting from the user, it may change the appearance of the avatar based on that emotion and present it to the viewer. This allows users to instantly set changes in their avatar's appearance according to the emotions they normally use during live streams, making it easy to stream.

[0084] The "Remarks" field is used to store information if there are any special notes or other important details in the user's profile.

[0085] As shown in Figure 4, the avatar information database 2022 includes items such as "ID", "corresponding user", "attributes", "associated body part", "presence or absence of special body part", "special body part operation settings", "standard change rate", "frequently used emotions", and "remarks".

[0086] The "ID" field is information used in the broadcast to identify each avatar presented to viewers.

[0087] The "Corresponding User" field contains information that identifies the user to whom the avatar corresponds.

[0088] The "Attributes" field is information that identifies the attributes set for each avatar. Specifically, attributes indicate, for example, whether the avatar is human, or a non-human entity whose behavior differs from that of a human. Attributes include, for example, the following information: ·human • Living beings other than humans (animals, plants, etc.) • Imaginary creatures (dragons, angels, demons, etc.) ·machine • Formless beings (such as slimes and ghosts in fantasy) In certain situations, the record may hold information specific to the defined attribute as a sub-concept. Specifically, for example, if the attribute is "inorganic," it may hold a sub-concept such as "no eyes," and if the attribute is "virtual organism," it may hold information such as "multiple eyes." Server 20 may hold information to correct the degree of change in the avatar's appearance based on the attribute information. This allows users to appropriately change the mouth and facial features even when controlling avatars that are not human.

[0089] The item "Associated Parts" is information about associated parts among one or more facial parts of an avatar. Specifically, the associated parts may be stored when, for example, the "eyebrows" of the avatar's facial parts are associated with each other. Server 20 may accept settings for the degree of change of the same manner for the associated parts. This eliminates the need for users to individually set the degree of change in appearance for each associated part, thus reducing the effort required to set the degree of change in appearance.

[0090] The item "Presence or Absence of Special Parts" is information used to identify whether or not an avatar has special parts. Specifically, for example, server 20 may retain this information if the avatar's attributes are not human and if parts such as "horns" or "tail" exist. Here, special parts do not need to belong to the avatar's body; they may be objects floating around the avatar. Special parts are not limited to those mentioned above. For example, objects such as creatures other than the avatar may be placed around it.

[0091] The item "Special Part Operation Settings" contains information about the settings for operating the special parts of an avatar. Specifically, for example, if the avatar has special parts (e.g., "horns," "tail," etc.), the server 20 may store in this record information about the conditions under which those parts are triggered to operate. For example, if an avatar has the special part "horns" and the setting is "linked to the movement of the entire eye," the server 20 may reflect the user's setting for the degree of change in the appearance of the eye to the horns and change their appearance. Furthermore, in certain situations, the server 20 may accept settings from the user regarding the degree of change in appearance for each special part. For example, if the special part is not connected to the avatar's body but is an object floating around the avatar, the server 20 may accept setting input from the user for each of those objects. However, the server 20 may also change the appearance of those objects by reflecting the settings for the avatar's parts (mouth, face, etc.).

[0092] Furthermore, if the special part is an object such as a living organism that is different from the avatar and is located around the avatar, the server 20 may also accept settings for the degree to which the parts of the object (e.g., eyes, mouth, etc.) change their appearance based on the user's voice spectrum or sensing results. For example, the server 20 may set the amount of change of the object's eyes by multiplying the amount of change of the avatar by a predetermined percentage, or it may accept settings from the user for each part of the object. This allows users to control non-human avatars in a way that is appropriate to the characteristics of that avatar.

[0093] The "Remarks" field is used to store information if there are any special notes or details regarding the avatar.

[0094] As shown in Figure 4, the wearable device information database 2023 includes the items "ID", "Type", "Detection Accuracy", "Correction Amount", and "Remarks".

[0095] The "ID" field is information that identifies each wearable device worn by the user.

[0096] The "Type" field indicates the type of wearable device the user is wearing. The wearable device is not limited to eyeglasses or other eyewear, and may also include head-mounted displays (HMDs).

[0097] The item "Detection Accuracy" indicates the detection accuracy of sensing the user's eye or facial movements while the user is wearing a wearable device. Specifically, for example, server 20 may score the detection accuracy of sensing for each wearable device worn by the user and retain this information. For example, if the user is wearing glasses with high transmittance that are almost the same as seeing without glasses, the information may be retained with a detection accuracy of "○". In this case, the score retained by server 20 may not be a symbol such as "○", but may be a numerical value such as "100" based on transmittance, or it may be a notation such as "A" or "Good", and is not limited to these.

[0098] The item "Correction Amount" indicates the correction amount for the degree of avatar change, which is set for each wearable device. Specifically, for example, the server 20 sets the correction amount for the degree of change in the avatar's appearance based on the aforementioned detection accuracy value. For example, if the user is wearing glasses, the server may perform a process that multiplies the degree of change by a predetermined magnification based on the transmittance of the glasses, etc. In certain situations, if the server 20 can obtain sensing results of the user's eyes or face from a device such as an HMD, even if the detection accuracy is low, the server 20 may choose not to perform any correction processing. The wearable device information held by server 20 may also include information about masks, eye patches, etc. In that case, server 20 may reflect the movement of the part that is obscured by the mask, eye patch, etc., in the avatar not based on changes in appearance based on sensing results, but by reflecting the user's speech or the settings of other parts that are not obscured. This allows users to stream without worrying about how they look during the broadcast.

[0099] The "Remarks" field is used to store information about wearable devices, specifically any special notes or details.

[0100] <3 operations> The following describes the series of processes by which System 1 acquires the audio spectrum of the user's speech and, based on the acquired audio spectrum, changes the mouth shape of the avatar corresponding to the user in accordance with the performer's speech.

[0101] Figure 5 is a flowchart illustrating a series of processes for acquiring the audio spectrum of a user's speech and changing the mouth shape of an avatar corresponding to the user in accordance with the performer's speech based on the acquired audio spectrum. This flowchart discloses an example in which the control unit 190 of the terminal device 10 used by the user performs the series of processes, but is not limited to this. That is, the terminal device 10 may send some information to the server 20 and the server 20 may perform the process, or the server 20 may perform the entire series of processes.

[0102] In step S501, the control unit 190 of the terminal device 10 acquires the audio spectrum of the speech of the user, who is the performer operating the avatar. Specifically, for example, the control unit 190 of the terminal device 10 controls the process of acquiring the audio spectrum from the voice spoken by the user acquired via the microphone 141, similar to the audio spectrum acquisition module 2035 of the server 20. For example, the control unit 190 acquires the user's voice via the microphone 141 and acquires the audio spectrum contained in that voice. For example, the control unit 190 may perform a Fourier transform on the voice acquired from the microphone 141 and acquire information on the audio spectrum contained in the voice. In this case, the calculation for acquiring the audio spectrum is not limited to the Fourier transform, but may be any existing method. Furthermore, in certain situations, the control unit 190 may acquire vowel speech spectrum information from the user's voice. For example, the control unit 190 may pre-accept vowel settings to be input from the user, and then receive the user's speech via the microphone 141, thereby associating and storing the accepted vowel settings with the acquired speech spectrum. Furthermore, in certain situations, the speech spectrum acquisition module 2035 may acquire speech information originating from consonants, such as sounds like "t," "c," "h," "k," "m," "r," "s," "n," and "w," and combine this with the stored vowel information to estimate the words spoken by the user. This allows System 1 to separately characterize and store the vowel-related parts of the user's voice spectrum, enabling it to more accurately change the movement of the avatar's mouth.

[0103] In step S502, the control unit 190 of the terminal device 10 changes the mouth shape of the avatar corresponding to the user in accordance with the user's speech, based on the acquired voice spectrum. Specifically, for example, the control unit 190 of the terminal device 10, similar to the avatar change module 2036 of the server 20, estimates the word spoken by the user from the acquired voice spectrum of the user and changes the mouth shape of the avatar in accordance with the estimated word. For example, if the acquired voice spectrum of the user is "a", the control unit 190 changes the mouth shape of the avatar to a shape corresponding to "a".

[0104] In step S503, the control unit 190 of the terminal device 10 presents the avatar and voice of the user to the viewer. Specifically, for example, the control unit 190 of the terminal device 10, similar to the avatar presentation module 2037 of the server 20, transmits the video of the avatar corresponding to the user and the user's voice to the display 1302 and speaker 142 of the terminal device 10 used by the viewer, and presents them to the viewer. At this time, there is not necessarily only one viewer, and the avatar and voice may be presented to the terminal devices 10 of multiple viewers.

[0105] In step S504, the control unit 190 of the terminal device 10 accepts a setting to change the degree to which the avatar's mouth shape changes in accordance with the performer's speech, to a degree lower than the change in the user's speech. Specifically, for example, the control unit 190 of the terminal device 10 may accept a setting including the following, similar to the setting acceptance module 2038 of the server 20. • Setting the frequency at which the avatar's behavior is changed and updated based on user speech within a certain period (e.g., number of updates per second). Here, we define changes in user speech. Changes in user speech include, for example, the rate of user speech, which may be calculated based on the following. • The time interval at which the vowel spoken by the user changes (for example, the time interval at which the vowel changes from "a" to "i"). In this case, the control unit 190 may simultaneously acquire sounds derived from consonants (c, k, etc.) and estimate the speech speed as if different words were being spoken, even if the same vowel is acquired consecutively. • Number of vowels uttered within a specified period Furthermore, at this time, the control unit 190 may accept the setting to be lower than the degree of change in the avatar estimated from the change in the user's speech. For example, the control unit 190 may accept the time required to change the mouth shape to correspond to the speech (vowel) estimated in advance from the user's voice spectrum. Based on the accepted time required, the control unit 190 obtains information on the time the user uttered a vowel from the user's voice spectrum, calculates the ratio with the pre-set time required, multiplies it by the amount of change in shape, and calculates the amount of change in the avatar's mouth shape. Based on the acquired speech time and the amount of change, the control unit 190 changes the mouth shape. For example, suppose the user inputs a setting that the avatar's mouth changes to the "ah" shape in a time of "1 second". For example, let's say that when it becomes completely the "ah" shape, it is set to "100", and it is set to become "100" in "1 second". At this time, the control unit 190 may also receive information from the user regarding the degree to which the mouth shape changes in one second (amount and speed of mouth change). (That is, a difference in the amount of change in mouth shape may be set between the first 0.5 seconds and the remaining 0.5 seconds of the one-second change in mouth shape.) If the user utters the sound "ah" for 1 second, the control unit 190 changes the shape of the avatar's mouth to the shape of "ah" over 1 second, based on the above change amount setting, etc. However, if the user utters "ah" for only "0.5 seconds", the control unit 190 may perform a process to change the amount of change in the shape of the avatar's mouth up to "50".

[0106] Furthermore, if the user speaks continuously (for example, "aiueo"), the control unit 190 may obtain the speech duration of each vowel and perform the above processing. In other words, the control unit 190 may calculate the amount of change in the avatar's mouth shape corresponding to each vowel from the speech duration of each vowel and change the avatar's mouth shape accordingly. For example, if the time required for the avatar's mouth shape to change to the shape corresponding to each vowel is "1 second", and the user speaks "a" for "0.2 seconds" and "i" for "0.3 seconds", the amount of change corresponding to "a" is "20", and the amount of change corresponding to "i" is "30". Also, if the user speaks for a longer time than the required time, the control unit 190 may maintain the avatar's mouth shape even after the required time has elapsed. This allows the user to change the avatar's mouth shape gradually according to the duration of their speech, rather than instantly changing it to an "a" shape when they utter the sound "a". Furthermore, by setting a time limit and multiplying the change in mouth shape by that limit if the speech is shorter than the set time, it is possible to prevent the avatar's mouth shape from changing drastically even when the user speaks lightly (for example, changing to a mouth shape of 100 even if the mouth opening is only about 30). This prevents the user from causing viewers to feel a sense of incongruity between the user's speech and the changes in the avatar's mouth shape, thus creating a more immersive experience for the viewers.

[0107] In certain situations, the control unit 190 may change the shape of the avatar's mouth based on information such as the magnitude and pitch of the voice spectrum acquired from the user. Specifically, for example, the control unit 190 may acquire information on the frequency (Hz) and sound pressure (dB) of the user's voice spectrum and change the shape of the avatar's mouth if this information exceeds a predetermined threshold. For example, suppose the control unit 190 accepts a setting to change the shape of the avatar's mouth in a time of "1 second," and the user's speaking time is "1 second." In this case, if the control unit 190 detects that the user has spoken at a sound pressure exceeding the threshold at "0.8 seconds," it may change the shape of the avatar's mouth more significantly than usual (to a shape with the mouth wide open). In this case, the control unit 190 may also apply the same setting to parts of the face and body, not just the mouth. This allows users to reflect sudden shouting in the avatar's mouth movements, resulting in more natural-looking avatar movements for viewers.

[0108] In addition, the control unit 190 may accept a setting for the degree of change in the avatar's mouth shape, such that the update frequency of the avatar's movements is lower than the update frequency estimated from the speech rate estimated from the user's utterance. For example, the control unit 190 divides the user's speech into fixed time intervals and changes the avatar's mouth shape to correspond to the first and last vowels of each time interval. For example, if the speech changes to "aiueo" in one second, the control unit 190 may reflect the mouth shape at the beginning of "aiueo" ("a") and the mouth shape at the end of "o" in the avatar. In addition, when the control unit 190 holds the user's utterance as a buffer in temporary memory, it may change it from "a" to "o" over a certain time interval (for example, 1 second). Furthermore, the control unit 190 may accept a setting so that the shape of the avatar's mouth changes more slowly than the time that has elapsed when the user's vowel changes. For example, if the server 20 hears the user's vowel change from "a" to "u" and the change takes 1 second, it may take 1.5 seconds for the shape of the avatar's mouth to change from "a" to "u". Also, in this case, the server 20 may perform processing to compensate for the change in shape. That is, instead of immediately changing the shape of the avatar's mouth from "a" to "u", the server 20 may change the shape of the mouth by passing through a mouth shape that is intermediate between "a" and "u". This allows users to change the avatar's mouth in a way that closely resembles the movement of a real person's mouth, rather than having the avatar's mouth change instantly with each word, thus reducing the sense of unnaturalness viewers may experience when watching the avatar. In this case, the control unit 190 may accept from the user the setting of the degree to be lower than the user's speaking speed. Specifically, for example, the control unit 190 may calculate the user's speaking speed from the speech spectrum of the utterance received from the user. Then, the control unit 190 sets an upper limit on the amount of change per unit time of the avatar's appearance that can be accepted from the user based on the calculated user's speaking speed, thereby accepting from the user the setting of the degree to be lower than the user's speaking speed. This allows users to set the degree of avatar change to be slower than the changes in their own speech, resulting in smoother changes in the avatar's appearance.

[0109] In step S505, the control unit 190 of the terminal device 10 changes the shape of the avatar's mouth according to the settings. Specifically, for example, the control unit 190 of the terminal device 10 changes the shape of the avatar's mouth according to the settings based on the information set in step S604, and then presents the avatar and the user's voice to the viewer. This allows users to more smoothly change the mouth movements of their avatars to match their own speech, presenting viewers with more natural-looking mouth movements. In a certain situation, when changing the shape of the avatar's mouth, the control unit 190 of the terminal device 10 may change the shape of the mouth based on at least one of the group consisting of strength or pitch of the audio spectrum. Specifically, for example, the control unit 190 determines the intensity and pitch by analyzing the following parameters of the audio spectrum. • Parameter for the intensity of the sound spectrum: decibels (dB) • Parameters for high and low frequencies in the audio spectrum: Hertz (Hz) For example, when the control unit 190 acquires a sound spectrum that is larger than the decibel level of the reference sound spectrum, it may change the shape of the avatar's mouth more significantly than the change in the mouth shape at the reference time. This allows users to change the appearance of their avatar based on subtle changes in voice, reducing the likelihood of causing discomfort to viewers.

[0110] In a certain phase, the control unit 190 of the terminal device 10 may accept the setting of a frequency range for detecting the voice spectrum, and in response to detecting a voice spectrum within the set range, change the mouth shape of the avatar based on a first setting of degree. Specifically, for example, in step S604, the control unit 190 accepts from the user the setting of upper and lower limit values ​​for the frequency range for detecting the voice spectrum. The control unit 190 analyzes the voice spectrum of the user's speech acquired via the microphone 141 and determines whether the frequency of the voice spectrum is within the range. If the frequency is within the range, the control unit 190 may, in step S605, change the avatar's shape based on a first setting of degree, that is, a setting of the degree of change of the avatar's shape that has been set in advance by the user.

[0111] Furthermore, in certain situations, the control unit 190 of the terminal device 10 may, in response to detecting a voice spectrum outside the set range, change the appearance of the avatar's mouth based on a second setting, which is a predetermined setting and different from the first setting. Specifically, for example, when the control unit 190 detects a frequency outside the range of frequencies for detecting the voice spectrum, which is received from the user, it may change the appearance of the avatar based on a setting (second setting) that is different from the normal setting (first setting). For example, if the user speaks a voice outside the range of frequencies normally used (e.g., an extremely high-pitched scream), the voice spectrum will be outside the detection range. In that case, the control unit 190 may change the appearance of the avatar by applying a setting (second setting) that is only applicable outside the detection range, rather than the setting (first setting) of change received from the user. This allows users to perform unusual actions or utterances, and the avatar's appearance will change accordingly, providing viewers with a more immersive experience.

[0112] In certain situations, the control unit 190 may change the appearance of parts of the face other than the mouth or parts of the body in response to detecting a voice spectrum outside the set range. Specifically, for example, the control unit 190 may cause the avatar to perform the following actions in response to detecting a voice spectrum outside the set range. • Change the appearance of facial features (eyebrows, outer corners of eyes, inner corners of eyes, corners of the mouth, etc.) • Change the appearance of body parts (arms, hands, shoulders, etc.) In addition, the control unit 190 may, in response to detecting an audio spectrum outside the set range, display a predetermined object on the screen viewed by the viewer. As a result, the control unit 190 can better convey the user's emotions to the viewer, for example, when the user suddenly shouts or screams, by changing the appearance of facial features, body parts, or displaying objects.

[0113] In a given scenario, the control unit 190 of the terminal device 10 may estimate one or more candidate emotions of the user and present the estimated one or more candidate emotions of the user to the user. Subsequently, the control unit 190 may receive an input operation from the user to select one emotion from the one or more candidate emotions, and if an emotion selection is received from the user, it may perform a process to change the mouth shape of the avatar based on the selected emotion. Specifically, for example, the control unit 190 analyzes the voice spectrum obtained from the user and estimates candidate emotions when the user spoke. In this case, the candidates for emotion include, for example, the following: ・Anger, furious • Joy, pleasure Surprise, fear • Sorrow, lament Peace and tranquility Here, we illustrate the process of estimating emotion candidates from a voice spectrum. For example, the control unit 190 may receive information on voice spectra corresponding to emotions from the user in advance and store it in the memory unit 180, etc., thereby associating the user's voice spectrum with the user's emotions. Subsequently, when the control unit 190 obtains a voice spectrum from the user, it estimates emotion candidates associated with voice spectra that have similar waveforms to the obtained voice spectrum. Similar waveforms mean, for example, determining the degree of similarity between the waveforms of multiple voice spectra and indicating that the waveforms match by a predetermined percentage, or that the waveforms of multiple voice spectra deviate by a predetermined percentage (for example, matching within a range of ±10%). In certain situations, a trained model may be used as a method for estimating candidate emotions of a user from their voice spectrum. For example, the terminal device 10 may store in its memory unit 180 a trained model that associates the voice spectra of multiple users with the emotions corresponding to those users. Subsequently, when the control unit 190 of the terminal device 10 receives voice spectrum input from a user, it may estimate candidate emotions corresponding to that user's voice spectrum based on the trained model and present them to the user.

[0114] The control unit 190 may present the estimated emotion candidates to the user and accept the user's selection. The control unit 190 also accepts in advance settings for the degree of change in the mouth shape for each emotion, and when it receives an emotion selection from the user, it changes the mouth shape of the avatar based on the setting for the corresponding emotion. This allows users to change the appearance of their avatar based on emotions inferred from their speech.

[0115] In this case, if the control unit 190 cannot estimate the user's emotion, it may change the mouth shape based on conditions set in advance by the user. Specifically, for example, if the control unit 190 cannot estimate a candidate for the user's emotion from the voice spectrum acquired from the user, that is, if it cannot estimate a similar voice spectrum, it may change the mouth shape of the avatar based on conditions set in advance by the user. For example, if the control unit 190 cannot accurately obtain a voice spectrum from the user, or if it cannot estimate a candidate emotion similar to the obtained voice spectrum, and has received a setting from the user for a "calm" mouth gesture, it will change the mouth gesture of the avatar to one based on the "calm" emotion. This allows users to change their avatars to pre-set forms even when emotions cannot be estimated, thus reducing the sense of unease for viewers.

[0116] Furthermore, in certain situations, the control unit 190 may operate a body part other than the avatar's mouth based on the estimated emotion. Specifically, for example, the control unit 190 may operate a body part other than the avatar's mouth, such as the shoulders, arms, or hands. In addition, the control unit 190 may operate a special body part (for example, if the avatar is not human, such as wings, a tail, or objects floating around it) as a body part other than the avatar's mouth. For example, if the emotion estimated from the voice spectrum acquired from the user is "anger," the control unit 190 may perform an action of raising the avatar's arm. Furthermore, at this time, the control unit 190 may present the user with candidate emotions instead of the emotions estimated from the acquired audio spectrum, and operate a body part other than the avatar's mouth based on the emotion selected by the user.

[0117] In addition, the control unit 190 may present the user with one or more candidate body parts other than the avatar's mouth whose appearance is to be changed, based on the emotion estimated from the acquired voice spectrum, and may change the appearance of the body part in response to the user's selection of the body part whose appearance is to be changed. This allows users to control parts of their avatar other than its mouth based on emotions estimated from their own voice spectrum, creating a more immersive experience for the viewer.

[0118] In a certain situation, if the user's speech rate deviates by a predetermined rate from the speech rate estimated from the degree of change in the mouth shape set by the user, the control unit 190 of the terminal device 10 may change the mouth shape based on the speech rate rather than the degree set by the user. Specifically, for example, the control unit 190 calculates the speech rate from the user's speech. The method for calculating the speech rate may be, for example, defined by the control unit 190 when the user calculates the number of words per unit time from the speech spectrum acquired from the user, and this is used as the value of the speech rate. Alternatively, the control unit 190 may calculate the number of utterances per unit time from the degree of change in the mouth shape set by the user, and calculate the user's speech rate estimated from the degree of change in the avatar's mouth shape. Subsequently, if there is a predetermined speed difference between the speech rate calculated from the user's utterance and the speech rate estimated from the degree of change in the avatar's mouth shape, the control unit 190 may change the mouth shape based on the speech rate calculated from the user's utterance, rather than on a setting set by the user. This allows users to adjust the avatar's mouth shape based on their own speaking speed if it deviates too much from the speaking speed estimated from the degree of change in the avatar's mouth shape. This reduces the feeling of unnaturalness for the viewer.

[0119] In a given scenario, the control unit 190 of the terminal device 10 may receive avatar attributes from the user and correct the amount of change in the avatar's mouth shape based on those attributes. Specifically, for example, the control unit 190 may receive information as an avatar attribute that it is either human or a non-human whose mouth shape changes differently from that of a human, and correct the amount of change in the mouth shape based on those attributes. For example, similar to the change correction module 2040 of the server 20, the control unit 190 may acquire information on whether the avatar operated by the user is human or a non-human whose mouth shape changes differently from that of a human, and perform a process to correct the degree of change in the avatar's shape based on that information. For example, if the attribute of the avatar operated by the user is "dragon," the movement of the eyes, mouth, etc., may behave differently from that of a human. In that case, the control unit 190 may correct the amount of change in the corners of the mouth, the amount of change in the eyeballs, etc., to conform to the avatar based on the "dragon" attribute. This allows users to present viewers with more natural movements based on their own speech and facial sensing results, even when controlling an avatar that is not human.

[0120] <4 Screen Examples> Figures 6 to 9 are diagrams showing various screen examples when a user, who is an avatar operator, operates the avatar using System 1 disclosed in the present invention.

[0121] Figure 6 shows an example screen for when a user registers their vowel speech spectrum with System 1.

[0122] In Figure 6, the control unit 190 of the terminal device 10 displays the settings screen 601, avatar 602, etc., on the display 1302. The settings screen 601 is a settings screen displayed to the user when acquiring and associating information about the voice spectrum corresponding to each vowel from the user. For example, the control unit 190 of the terminal device 10 displays a settings screen on this screen for the six characters "A", "I", "U", "E", "O", and "N" as vowels to be associated with the user's voice spectrum. At that time, the control unit 190 may also display information about the vowels currently associated with the user's voice spectrum at the top of the screen. Furthermore, the control unit 190 may display information about the microphone 141 being used by the user at the bottom of the settings screen 601. If the frequency characteristics differ depending on the type of microphone 141 used by the user, the control unit 190 may associate the user's voice spectrum and vowel information for each microphone 141 being used.

[0123] Avatar 602 is an avatar whose mouth shape changes in response to user speech. The control unit 190 changes the mouth shape of avatar 602 in response to the speech spectrum acquired from the user. For example, when the user utters the vowel "a", the control unit 190 determines whether the utterance matches the speech spectrum held as the vowel "a". If the user's utterance is "a", the control unit 190 then changes the mouth shape of avatar 602 to the shape of "a". This allows users to precisely change the mouth shape of their avatar for each vowel.

[0124] Figure 7 shows an example screen where the user can adjust the degree of change in the appearance of the avatar's mouth or other facial features.

[0125] In Figure 7, the control unit 190 of the terminal device 10 displays an information display screen 701, user video 702, a settings screen 703, an avatar 704, etc., on the display 1302.

[0126] The information display screen 701 is a screen that displays the frequency of the voice spectrum acquired from the user, the range of the detectable voice spectrum, and the settings for the behavior when it falls outside the detection range. In addition, the terminal device 10 may also display the user's speech speed calculated from the user's utterance, the degree of change in behavior that the user can set, and the user's facial sensing results on the same screen, visually displaying various conditions that the user can set.

[0127] User video 702 is a screen that displays the user's own video captured via the camera 160 installed in the terminal device 10. When the user makes any sound in front of the terminal device 10, the control unit 190 of the terminal device 10 displays the user's own video and information such as the audio spectrum of the user's speech on the user video 702 and the information display screen 701, using the camera 160 and microphone 141 installed in the terminal device 10.

[0128] The settings screen 703 is a screen for the user to set the degree of change in the avatar's appearance. The control unit 190 of the terminal device 10 presents the user with the following settings and accepts input. • Speed ​​of switching between mouth movements The mouth switching speed refers to information about the time it takes to reach the maximum volume (100) in the speech spectrum acquired from the user. • Eye movement: Maximum upward movement • Eye movement: Maximum downward movement • Eye movement: Maximum horizontal movement • Eye movement: Sensitivity (device sensing sensitivity) Sensitivity refers to the degree to which the avatar's movement is reflected when the user's eyes are sensed. Specifically, for example, sensitivity is a parameter that sets how much the avatar's eyes are reflected in the actual movement of the eyes when the user moves their eyes left or right, with the coordinates of the user's eyes (when facing directly forward) set to "0". In this case, sensitivity is a proportional function when it is 100, and may be a downward-facing concave function as it approaches 0. In other words, when sensitivity is 100, the movement of the user's eyes and the movement of the avatar's eyes are perfectly synchronized. When sensitivity is 50, for example, if the user's eyes do not move much from the center, the movement of the avatar's eyes is reflected as less than the distance the user's eyes moved, and if the eyes move to the outer corners, the movement of the avatar's eyes is reflected as more than the distance the user's eyes moved. This prevents the avatar's eyes from moving erratically and immediately when the user's eyes do not move much. Furthermore, the above sensitivity settings are not limited to the eyes. Similar settings may be accepted for other parts of the face and body. In this case, the control unit 190 of the terminal device 10 may accept a setting for the degree of change that can be accepted from the user at a level lower than the change in the user's utterance. For example, the control unit 190 may accept the setting from the user at a level lower than the degree of change of the avatar (amount of change of the object, rate of change of the object) estimated from the user's utterance. In this case, if the user attempts to set a value that is outside the settable range, the control unit 190 may display a predetermined alert, or, if the setting screen is of the slider type, it may lock the value in advance so that it cannot be set to that value. This allows users to move their avatars more slowly than their own speech changes, resulting in a smoother perception of avatar changes for viewers and a more immersive experience.

[0129] Avatar 704 is an avatar that changes its appearance based on settings received from the user. When the control unit 190 of the terminal device 10 receives settings from the user on the settings screen 703, it may synchronize the user video 702 and the avatar 704 and display them to the user. This allows users to check for any inconsistencies or other issues before changing their avatar's appearance according to their own settings.

[0130] Figure 8 shows an example screen where the system estimates one or more possible emotions of the user from the user's utterances, and changes the appearance of the avatar based on the estimated emotions of the user.

[0131] In Figure 8, the control unit 190 of the terminal device 10 displays an information display screen 801, user video 802, avatar 803, etc., on the display 1302.

[0132] The information display screen 801, similar to the information display screen 701 in Figure 7, is a screen that displays the frequency of the voice spectrum obtained from the user. In Figure 8, it may also display one or more candidate emotions estimated from the voice spectrum, and candidate emotion settings that the user will reflect in the avatar's appearance. The control unit 190 may change the appearance of the avatar, for example, the appearance of the avatar's mouth or the appearance of other facial features, by accepting a selection from the user for the presented setting candidates.

[0133] User video 802, like user video 702 in Figure 7, is a screen that displays the user's own video captured via the camera 160 installed in the terminal device 10.

[0134] Avatar 803, like Avatar 704 in Figure 7, is an avatar that changes its appearance based on the emotion setting received from the user. The control unit 190 of the terminal device 10 changes the appearance of the avatar (e.g., the mouth) and displays it to the user based on the emotion setting received from the user. At this time, the control unit 190 may change or operate the appearance of other parts of the avatar, not just the appearance of the mouth. For example, if the emotion selected by the user is "anger", the control unit 190 may change the appearance of the avatar's mouth based on the emotion of "anger", and may also change the appearance of other parts of the avatar, such as the eyebrows and corners of the eyes. In addition, the control unit 190 may operate parts of the avatar's body (e.g., raising an arm) based on the emotion. Furthermore, the control unit 190 may display a predetermined object corresponding to the emotion on the screen where the avatar is displayed, based on the emotion. This allows users to make their avatars change and perform actions based on emotions inferred from their speech, thereby creating a more immersive experience for viewers.

[0135] Furthermore, in certain situations, the control unit 190 may refer to user information 1801 or user information database 2021 to obtain information on emotions frequently used by the user and present it as a candidate emotion to be reflected in the avatar. This allows users to easily change the appearance of their avatar, even when they are trying to change the avatar's appearance through effects or other means, regardless of their speech.

[0136] Figure 9 shows an example screen where a user can configure various settings for an avatar with attributes different from those of a human, based on the voice spectrum, etc.

[0137] In Figure 9, the control unit 190 of the terminal device 10 displays an information display screen 901, user video 902, a settings screen 903, an avatar 904, etc., on the display 1302.

[0138] The information display screen 901, like the information display screens 701 and 801 in Figures 7 and 8, displays the frequency of the audio spectrum acquired from the user, the range of the detectable audio spectrum, and settings for what to do if the spectrum is outside the detection range. At this time, the control unit 190 may also display information about the user and the attributes of the avatar corresponding to the user on the information display screen 901. For example, the control unit 190 may refer to user information 1801 or the user information database 2021 to acquire information about the user and the corresponding avatar, and then display information about the avatar's attributes on the screen.

[0139] User video 902, like user videos 702 and 802 in Figures 7 and 8, is a screen that displays the user's own video captured via the camera 160 installed in the terminal device 10.

[0140] The settings screen 903, like the settings screen 703 in Figure 7, is a screen for the user to set the degree of change in the avatar's appearance. In Figure 9, in addition to the screen presented to the user on the settings screen 703, the control unit 190 may also display suggestions for recommended settings based on the avatar's attributes. Specifically, for example, the control unit 190 may refer to the avatar information 1802 or the avatar information database 2022, etc., to obtain information on the correction amount for the degree of change in the avatar's appearance, and present the user with settings obtained by multiplying the basic settings for changing the appearance of a normal human avatar by the correction result. This allows users to configure their avatar's appearance to change in a way that doesn't feel unnatural, even if their avatar has attributes different from those of a human. Furthermore, in certain situations, if the avatar has special body parts, the control unit 190 may display a screen to the user for setting the degree of change in the appearance of those body parts. For example, if the control unit 190 is synchronizing with the settings of other body parts, it may reflect the change settings of those other avatar body parts, or it may present the user with a separate screen for setting the degree of change in appearance in detail. This allows users to freely adjust the degree of change in their avatar's appearance, even if it has special body parts, thereby providing viewers with a more immersive experience.

[0141] Avatar 904, like avatars 704 and 803 in Figures 7 and 8, is an avatar that changes its appearance based on the emotion settings received from the user. In Figure 9, the control unit 190 may simultaneously display special parts of the avatar on avatar 904. This allows users to broadcast to viewers while checking how their avatar's appearance changes, even if it has special body parts.

[0142] <Second Embodiment> Up to this point, we have described a series of processes that change the mouth shape of an avatar based on the speech spectrum of the user's utterance. In the invention according to the second embodiment, in addition to the voice spectrum of the user's speech, the appearance of the avatar, for example, the appearance of one or more facial features, can be changed based on the user's sensing results. The series of processes described below will be explained. Note that the description of parts that have a configuration common to the first embodiment (for example, the terminal device 10, the server 20, etc.) will be omitted, and only the configurations and processes specific to the second embodiment will be described.

[0143] <5 Operation in the second embodiment> The following describes a series of processes by which System 1 senses the movement of one or more facial parts of the user, and changes the appearance of one or more facial parts of the avatar corresponding to the user based on the sensed movement of one or more facial parts.

[0144] Figure 10 is a flowchart illustrating a series of processes for sensing the movement of one or more facial parts of a user's face and changing the appearance of one or more facial parts of an avatar corresponding to the user based on the sensed movement of one or more facial parts. In this flowchart, an example is disclosed in which the control unit 190 of the terminal device 10 used by the user performs the series of processes, but it is not limited to this. That is, the terminal device 10 may send some information to the server 20 and the server 20 may perform the process, or the server 20 may perform the entire series of processes.

[0145] In step S1001, the control unit 190 of the terminal device 10 senses the movement of one or more parts of the user's face. Specifically, for example, the control unit 190 of the terminal device 10 senses one or more parts of the user's face when the user moves their face in front of the camera 160 provided in the terminal device 10. At this time, the sensing method performed by the control unit 190 may be any existing technology. For example, the control unit 190 may sense the user's face parts by equipping the camera 160 with a sensing function, or it may sense the user's face parts using the motion sensor 170. At this time, the control unit 190 of the terminal device 10 senses at least one of the group consisting of the user's eyebrows, eyelids, inner corners of the eyes, outer corners of the eyes, eyeballs, pupils, and mouth as one or more parts of the user's face. However, this part is not limited to other parts of the face (cheeks, forehead, etc.).

[0146] In step S1002, the control unit 190 of the terminal device 10 changes the appearance of one or more facial parts of the avatar corresponding to the user based on the movement of one or more facial parts that it has sensed. Specifically, for example, the control unit 190 pre-associates one or more facial parts of the user with one or more facial parts of the avatar. Then, the control unit 190 changes the appearance of the avatar's facial parts that correspond to one or more facial parts of the user acquired by sensing, based on the sensing results. For example, if the user's eyes and the avatar's eyes have been associated, the control unit 190 changes the appearance of the avatar's eyes based on the sensing results of the user's eyes.

[0147] In step S1003, the control unit 190 of the terminal device 10 accepts a setting for the degree to which the appearance of one or more facial parts of the avatar should follow the sensed results, and changes the appearance of one or more facial parts of the avatar according to the setting. Specifically, for example, the control unit 190 accepts a setting for conditions including the following as the degree to which the appearance should follow the user's sensing results. • The amount of change in the avatar's appearance (for example, the amount of change in the opening and closing of the eyes, etc.) This allows users to finely adjust the degree to which their sensing results are followed by changes in the avatar's appearance, preventing the movement from appearing unnatural to viewers.

[0148] In the second embodiment, the control unit 190 may accept user settings for the degree of change in the appearance of the avatar's face and other body parts, similar to the setting of the degree of change in the appearance of the avatar's mouth in the first embodiment. That is, the control unit 190 may have previously obtained from the user, through sensing, the mouth appearances corresponding to various vowels, various faces, and body parts used by the user. The control unit 190 may identify the difference between the user's sensing results and the changes in the user's mouth, face, and body parts that were previously obtained, calculate the ratio with the previously obtained sensing results, and multiply this ratio by the amount of change in appearance to calculate the amount of change in the appearance of the avatar's mouth, face, and body parts. Based on the calculated amount of change, the control unit 190 may change the appearance of the avatar's mouth, face, and body parts. For example, if the user only moves their mouth and eyebrows partially (for instance, if a position of 100 is set in advance, and the sensing results indicate that the user only moves their mouth, eyebrows, etc. up to position 50), the avatar's mouth, eyebrows, etc. may also only move up to position 50. This allows users to gradually change the appearance of their avatar in accordance with their own sensing results, presenting natural movements to viewers. This prevents viewers from feeling any incongruity between the user's actions and the changes in the avatar's appearance, thus creating a more immersive experience for the audience.

[0149] In a given situation, the control unit 190 may accept the same settings for a set of predetermined associated parts among one or more facial parts of the avatar. Specifically, the control unit 190 may accept from the user a setting to associate, for example, the following parts among one or more facial parts of the avatar, and accept the same settings regarding the degree of those parts. • Paired facial features such as eyebrows and eyes • Parts that move in conjunction with each other, like the eyebrows and eyes. • Parts of the face and parts of the body other than the face (shoulders, arms, legs, neck, etc.) In addition, the control unit 190 may accept settings that associate facial features with special features other than the face, based on the avatar attributes described later. This allows users to easily change the appearance of their avatar and distribute it without having to individually set the degree of changes for paired or interconnected facial features.

[0150] In a given scenario, the control unit 190 of the terminal device 10 may present the user with one or more options for setting the degree to which the system should follow the sensed results, and accept the user's selection of one or more options. Subsequently, the control unit 190 may change the appearance of one or more facial features of the avatar based on the selected degree setting. Specifically, for example, when the control unit 190 acquires the user's sensing results, instead of accepting the user's setting for the degree to which the system should follow the sensed results, it may present one or more options (presets) for the degree of following. In this case, as a method for presenting the options, the control unit 190 may have previously received information from the user about one or more options for the degree of following to be used, and present the options based on that information. This allows users to change the appearance of parts of their avatar's face based on sensing results without having to set the degree of tracking each time, making streaming easier.

[0151] Furthermore, in certain situations, the control unit 190 of the terminal device 10 may receive avatar attributes from the user and correct the degree based on those attributes. Here, the control unit 190 may receive information as attributes that the avatar is human, or a non-human being whose facial features change in a manner different from that of a human, and correct the degree based on those attributes. For example, similar to the change correction module 2040 of the server 20, the control unit 190 may acquire information that the avatar operated by the user is human, or a non-human being whose facial features change in a manner different from that of a human, and perform a process to correct the degree of change in the avatar's appearance based on that information. For example, if the attribute of the avatar operated by the user is "dragon," the movements of the eyes, mouth, etc., may behave differently from those of a human. In that case, the control unit 190 may correct the amount of change in the corners of the mouth, the amount of change in the eyeballs, etc., to conform to the avatar based on the "dragon" attribute. This allows users to present viewers with more natural movements based on their own speech and facial sensing results, even when controlling an avatar that is not human.

[0152] In addition, in certain situations, the control unit 190 of the terminal device 10 may acquire the user's voice spectrum and obtain information on the degree of change in the user's speech from the acquired voice spectrum. Subsequently, the control unit 190 may accept setting the degree within a range associated with the degree of change in the user's speech, and may change the appearance of one or more facial parts of the avatar according to the setting of the degree. Specifically, for example, the control unit 190 may acquire a voice spectrum from the user's speech via the microphone 141, etc., and obtain the following information as the degree of change in the user's speech. • The number of words a user utters per unit of time (speech rate) • Changes in the volume of the user's voice • Changes in the pitch of the user's voice For example, the control unit 190 performs the following processing to change the avatar's appearance to a degree lower than the degree of change estimated from the user's speech changes. Regardless of vowel changes in the speech spectrum obtained from the user, the mouth movements are reflected in the avatar at regular time intervals. The control unit 190 identifies a settable range for the degree to which the system follows the sensing result, based on the acquired information on the degree of change in the utterance. For example, the control unit 190 may accept a setting from the user for the degree to which the acquired degree of change in the utterance does not exceed the degree of change in the utterance. This allows users to change the appearance of their avatar's face based not only on sensing results but also on audio spectrum information, enabling them to present the avatar to viewers with more natural movements.

[0153] At this time, the control unit 190 may accept a setting for the frequency range to be detected in the voice spectrum, and in response to detecting a voice spectrum within the set range, may change the appearance of one or more facial parts of the avatar based on a first setting of degree. Specifically, for example, when acquiring a voice spectrum from the user's speech, the control unit 190 may accept a setting for the detectable range from the user. If the voice spectrum acquired from the user is within the frequency range, the control unit 190 may change the appearance of the avatar's face based on the degree setting accepted from the user as described above. Furthermore, the control unit 190 may, in response to detecting an audio spectrum outside the set range, change the appearance of one or more facial features of the avatar based on a predetermined degree setting, which is different from the first degree setting, and is a second degree setting. In this case, the second degree setting may reflect a predetermined degree (second degree) corresponding to the frequency in question, such as when the user emits an extremely high-frequency sound (e.g., a shriek), rather than the degree setting (first degree) received from the user, and change the appearance of the avatar's face accordingly. This allows users to change the appearance of their avatar's face even when they make sounds at frequencies they wouldn't normally produce, thus providing viewers with a more immersive experience.

[0154] Furthermore, in certain situations, if the user's mouth movements cannot be sensed, the control unit 190 of the terminal device 10 may change the shape of the avatar's mouth based on the degree of change in the user's speech. Specifically, for example, in the following cases, the control unit 190 may change the shape of the avatar's mouth not from the user's sensing results, but from the audio spectrum of the user's speech, as described above. • If the user is wearing a mask or similar item over their mouth and mouth movements cannot be sensed. • If mouth movements cannot be sensed due to an error in the sensing function of terminal device 10. • When mouth movements cannot be sensed due to external environmental factors. This allows users to change the mouth shape of their avatar to match their speech, for example, when they have to wear a mask while streaming.

[0155] In a given scenario, the control unit 190 of the terminal device 10 may estimate one or more candidate emotions of the user and present the estimated one or more candidate emotions of the user to the user. Subsequently, the control unit 190 may accept an input operation from the user to select one emotion from the one or more candidate emotions, and based on the selected emotion, may change the appearance of one or more facial features of the avatar corresponding to the user. Specifically, for example, the control unit 190 may obtain and associate sensing results of facial features corresponding to the user's emotions in advance from the user. Subsequently, the control unit 190 senses the user's face via the camera 160 or the like and determines whether all or part of the facial sensing results included in the associated emotions match. Subsequently, based on the determination result, the control unit 190 may present candidate emotions of the user, accept a selection from the user, and change the appearance of the avatar's face based on the selected emotion. Furthermore, if the user's emotions cannot be estimated at this time, the control unit 190 may change the appearance of one or more facial features based on settings previously set by the user. For example, if the control unit 190 cannot accurately sense parts of the user's face, or if it cannot estimate a candidate emotion similar to the sensing result, and has received a setting from the user for a "calm" mouth, it will change the mouth shape of the avatar to one based on the "calm" emotion. This allows users to reflect their emotions in changes to their avatar's behavior by selecting emotion candidates, even when accurate sensing is not possible.

[0156] Furthermore, in certain situations, if the control unit 190 of the terminal device 10 is unable to obtain a sensing result for at least one of the associated parts of the user's face (one or more parts), it may apply the degree of change of the part for which a sensing result was obtained to the associated part. Specifically, for example, if the user is wearing an eye patch or the like, and sensing one eye is difficult or impossible, the control unit 190 may reflect the degree of change of the other eye for which a sensing result was obtained. This allows the avatar corresponding to the user to change its appearance without being affected, even if the user is wearing an eye patch or the like.

[0157] Furthermore, in certain situations, the control unit 190 of the terminal device 10 may acquire information about the wearable device worn by the user and correct the degree setting based on the acquired wearable device information. Also, when correcting the degree setting, the control unit 190 may accept input operations from the user to adjust the degree of correction. Specifically, for example, the control unit 190 may refer to the wearable device information 1803 or the wearable device information database 2023 to acquire information about the wearable device worn by the user. After that, the control unit 190 may perform the same processing as the change correction module 2040 in the server 20 described above and correct the degree setting.

[0158] In a certain scenario, the control unit 190 of the terminal device 10 may present a predetermined notification to the user if the difference in the degree settings between pre-associated parts of one or more facial parts of the avatar exceeds a predetermined threshold. Specifically, the control unit 190 associates paired parts of one or more facial parts of the avatar, such as eyebrows, and sets the control unit 190 to accept numerical values ​​for the degree of change between these parts so that the difference in degree of change does not exceed a predetermined difference. Subsequently, when the control unit 190 receives input from the user regarding the degree of change of the said part, it may present a notification such as an alert to the user if it receives input that exceeds the threshold. This prevents users from changing the appearance of parts that are associated with each other in a way that results in extreme differences in the degree of change. Furthermore, the control unit 190 may also set the settings for areas that change in conjunction with each other, such as cheeks and eyebrows (including special areas, etc.), in addition to the paired areas.

[0159] In this case, when the control unit 190 presents a predetermined notification to the user, it may present the part in which the difference in degree exceeds a predetermined threshold in a different manner along with the numerical value. Specifically, for example, when the control unit 190 receives input from the user regarding the degree of change in the appearance of the eyes, if the degree of change in both eyes is too great (for example, the change in one eye is too great), it may present the eyes in a different manner (for example, in a different color) along with the notification to the user. In this case, the different manner presented by the control unit 190 is not limited to changing the color, a pop-up notification, or the shape of the relevant part. Furthermore, when the control unit 190 presents a predetermined notification to the user, it may also present to the user an image showing how at least one or more facial features change when the degree of difference is set within a predetermined range. For example, the control unit 190 may display, on a screen different from the screen that displays the notification described above, an image showing how the avatar's appearance changes when the degree of difference is within an appropriate range (a range that does not cause discomfort to the viewer). This allows users to see what the change in behavior would look like if the degree of change they set exceeded a predetermined threshold, along with the results of setting an appropriate value.

[0160] Furthermore, in certain situations, the control unit 190 of the terminal device 10 may set the degree of one or more facial parts and related parts for which the degree setting has been received to a predetermined value. The control unit 190 may also accept degree settings for one or more parts of the avatar within a predetermined range. Specifically, for example, the control unit 190 may associate the following parts with one or more facial parts of the avatar and accept degree settings from the user. • Special body parts such as horns, tails, and wings, which are present when the avatar is not human. • Body parts other than the avatar's face (arms, shoulders, legs, etc.) This allows users to change their avatar's appearance in accordance with their own sensing results, even if the avatar is not human or is an inanimate object.

[0161] <6. Screen example in the second embodiment> Figures 11 to 17 are diagrams showing various screen examples when the appearance of the avatar is changed based on the user's sensing results, as disclosed in the second embodiment.

[0162] Figure 11 shows an example screen where the movement of one or more facial parts of the user is sensed, and the appearance of one or more facial parts of the corresponding avatar is changed based on the sensed movement of one or more facial parts.

[0163] In Figure 11, the control unit 190 of the terminal device 10 displays an information display screen 1101, a user video 1102, a settings screen 1103, an avatar 1104, etc., on the display 1302.

[0164] The information display screen 1101 is a screen that displays the sensing results of the user's facial parts, the associated facial parts, and a list of preset degrees of change in the facial features. At this time, the control unit 190 of the terminal device 10 may accept the following selections from the user. • Selection of the part of the user's face to be sensed. • Selection of the associated area from among the sensed areas. • Selection of candidate degrees of change This allows users to reduce the number of sensing points in some cases, thereby reducing the load during distribution.

[0165] User video 1102 is a screen that displays the user's own video captured via the camera 160 installed in the terminal device 10. The control unit 190 of the terminal device 10 displays the user's own video on user video 1102 using the camera 160 installed in the terminal device 10.

[0166] The settings screen 1103 is a screen for the user to set the degree of change in the avatar's appearance. The control unit 190 of the terminal device 10 presents the user with the following settings and accepts input. • Speed ​​of switching between mouth movements • Eye movement: Maximum upward movement • Eye movement: Maximum downward movement • Eye movement: Maximum horizontal movement • Eye movements: Sensitivity At this time, the control unit 190 of the terminal device 10 may accept a setting for the degree to which the appearance of one or more facial parts of the avatar follows the sensed results, within a range associated with the degree of change in the performer's speech. For example, the control unit 190 may accept this setting from the user so that it is lower than the degree of change in the avatar (amount of change in the object, rate of change in the object) estimated from the user's speech. At this time, if the user attempts to set a value that is outside the settable range, the control unit 190 may display a predetermined alert, or if the setting screen is a slider type, it may lock the value in advance so that it cannot be set to that value. This allows users to move their avatars more slowly than their own speech changes, resulting in a smoother perception of avatar changes for viewers and a more immersive experience.

[0167] Avatar 1104 is an avatar that changes its appearance based on settings received from the user. When the control unit 190 of the terminal device 10 receives settings from the user on the settings screen 1103, it may synchronize the user video 1102 and the avatar 1104 and display them to the user. This allows users to check for any inconsistencies or other issues before changing their avatar's appearance according to their own settings.

[0168] Figure 12 shows an example screen where the system estimates one or more of the user's emotions and changes the appearance of one or more facial features of the corresponding avatar based on the emotion selected by the user.

[0169] In Figure 12, the control unit 190 of the terminal device 10 displays an information display screen 1201, a user video 1202, a settings screen 1203, an avatar 1204, etc., on the display 1302.

[0170] The information display screen 1201, similar to the information display screen in Figure 11, displays the sensing results of the user's facial features, the associated facial features, and a list of preset degrees of change in the user's facial features. In addition, the control unit 190 may display information on the screen regarding one or more candidate emotions of the user identified from the sensing results. When the control unit 190 receives a selection of emotion candidates from the user, it reflects the degree of change in the avatar's appearance corresponding to that emotion. For example, the control unit 190 may acquire and associate sensing results of facial parts corresponding to the user's emotions in advance from the user. Then, the control unit 190 senses the user's face via the camera 160 or the like and determines whether all or part of the facial sensing results included in the associated emotions match. After that, the control unit 190 may present candidate emotions for the user based on the determination result, accept a selection from the user, and change the appearance of the avatar's face based on the selected emotion. Furthermore, if the user's emotions cannot be estimated at this time, the control unit 190 may change the appearance of one or more facial features based on settings previously set by the user. This allows users to reflect their emotions in changes to their avatar's behavior by selecting emotion candidates, even when accurate sensing is not possible.

[0171] User video 1202, like user video 1102 in Figure 11, is a screen that displays the user's own video captured via the camera 160 installed in the terminal device 10.

[0172] The settings screen 1203, like the settings screen 1103 in Figure 11, is a screen for the user to set the degree to which the avatar's appearance changes.

[0173] Avatar 1204, like Avatar 1104 in Figure 11, is an avatar that changes its appearance based on settings received from the user.

[0174] Figure 13 shows an example screen for setting the degree of change in the avatar's appearance when sensing results cannot be obtained for at least one of the associated parts of the user's face (one or more parts).

[0175] In Figure 13, the control unit 190 of the terminal device 10 displays an information display screen 1351, a user video 1352, a settings screen 1353, an avatar 1354, etc., on the display 1302.

[0176] The information display screen 1351, similar to the information display screen 1201 in Figure 12, displays the sensing results of the user's facial features, the associated facial features, and a list of preset degrees of change in the user's facial features. In addition, the control unit 190 may display information on the screen about equipment, devices, etc., that the user is wearing and that obscure a part of the user's face. For example, if the user is wearing an eye patch or the like, and sensing with one eye is difficult or impossible, the control unit 190 may reflect the degree of change in the other eye from which sensing results have been obtained. This allows the avatar corresponding to the user to change its appearance without being affected by the user wearing an eye patch or the like.

[0177] User video 1352, like user video 1202 in Figure 12, is a screen that displays the user's own video captured via the camera 160 installed in the terminal device 10.

[0178] The settings screen 1353, like the settings screen 1203 in Figure 12, is a screen for the user to set the degree to which the avatar's appearance changes.

[0179] Avatar 1354, like Avatar 1204 in Figure 12, is an avatar that changes its appearance based on settings received from the user.

[0180] Figure 14 shows an example screen for correcting the degree of change in the avatar's appearance when the user is wearing a wearable device such as glasses.

[0181] In Figure 14, the control unit 190 of the terminal device 10 displays an information display screen 1401, a user video 1402, a settings screen 1403, an avatar 1404, etc., on the display 1302.

[0182] The information display screen 1401, similar to the information display screen 1351 in Figure 13, displays the sensing results of the user's facial features, the associated facial features, and a list of preset degrees of change in the facial features. In addition, the control unit 190 may display information about the wearable device the user is wearing, information about the correction amount for the degree of change for each wearable device, and other information on this screen. For example, the control unit 190 may refer to the wearable device information 1803 or the wearable device information database 2023 to obtain information about the wearable device being worn by the user. Subsequently, the control unit 190 may perform the same processing as the change correction module 2040 in the server 20 described above to correct the degree setting.

[0183] User video 1402, like user video 1352 in Figure 13, is a screen that displays the user's own video captured via the camera 160 installed in the terminal device 10.

[0184] The settings screen 1403, like the settings screen 1353 in Figure 13, is a screen for the user to set the degree to which the avatar's appearance changes.

[0185] Avatar 1404, like Avatar 1354 in Figure 13, is an avatar that changes its appearance based on settings received from the user.

[0186] Figure 15 shows an example screen where the avatar's mouth shape is changed based on the degree of change in speech when the user's mouth movements cannot be sensed.

[0187] In Figure 15, the control unit 190 of the terminal device 10 displays an information display screen 1501, a user video 1502, a settings screen 1503, an avatar 1504, etc., on the display 1302.

[0188] The information display screen 1501, similar to the information display screen 1401 in Figure 14, displays the sensing results of the user's facial features, the associated facial features, and a list of preset degrees of change in the facial features. In addition, the control unit 190 may display information such as the mask the user is wearing and the speech spectrum information obtained from the user's speech on this screen. For example, if the user is wearing a mask or the like and the mouth movements cannot be sensed, the control unit 190 may change the shape of the avatar's mouth based on the speech spectrum of the user's utterance, rather than the user's sensing results, as described above. This allows users to change the mouth shape of their avatar to match their speech, for example, when they have to wear a mask while streaming.

[0189] User video 1502, like user video 1402 in Figure 14, is a screen that displays the user's own video captured via the camera 160 installed in the terminal device 10.

[0190] The settings screen 1503, like the settings screen 1403 in Figure 14, is a screen for the user to set the degree to which the avatar's appearance changes.

[0191] Avatar 1504, like Avatar 1404 in Figure 14, is an avatar that changes its appearance based on settings received from the user.

[0192] Figure 16 shows an example screen for displaying a notification to the user when the difference in the degree of settings between pre-associated facial parts of an avatar exceeds a predetermined threshold.

[0193] In Figure 16, the control unit 190 of the terminal device 10 displays an information display screen 1601, a user video 1602, a settings screen 1603, an avatar 1604, etc., on the display 1302.

[0194] The information display screen 1601, similar to the information display screen 1501 in Figure 15, displays the sensing results of the user's facial features, the associated facial features, and pre-set candidate degrees of change in the facial features.

[0195] User video 1602, like user video 1502 in Figure 15, is a screen that displays the user's own video captured via the camera 160 installed in the terminal device 10.

[0196] The settings screen 1603, like the settings screen 1503 in Figure 15, is a screen for the user to set the degree of change in the appearance of the avatar. At this time, when the control unit 190 receives input from the user regarding the degree of change in the appearance of a part of the face, if the degree of change of a pair of related parts (such as both eyes) is too large (for example, the change in one eye is too large), it may display that the setting value for that part is abnormal and suggest recommended settings.

[0197] Avatar 1604, like Avatar 1504 in Figure 15, is an avatar that changes its appearance based on settings received from the user. When the control unit 190 receives input from the user regarding the degree of change in the appearance of the face on the settings screen 1503 described above, if the degree of change of a paired or related part is too large, it may present that part to the user in a different appearance (for example, in a different color) along with a notification. In this case, the different appearance presented by the control unit 190 is not limited to changing the color, pop-up notification, or the shape of the relevant part.

[0198] This allows users to visually identify abnormal values ​​when setting up changes to the appearance of their avatar's facial features, preventing them from causing discomfort to viewers.

[0199] Figure 17 shows an example screen for presenting a predetermined notification to a user, where the degree of difference is set within a predetermined range, and the user is shown how at least one or more facial features change.

[0200] In Figure 17, the control unit 190 of the terminal device 10 displays the settings screen 1701, avatar 1702, settings preview screen 1703, avatar preview screen 1704, etc., on the display 1302.

[0201] The settings screen 1603, like the settings screen 1503 in Figure 15, is a screen for the user to set the degree of change in the appearance of the avatar. At this time, when the control unit 190 receives input from the user regarding the degree of change in the appearance of a part of the face, if the degree of change of a pair of related parts (such as both eyes) is too large (for example, the change in one eye is too large), it may display that the setting value for that part is abnormal and suggest recommended settings.

[0202] The settings preview screen 1703 displays recommended settings when there are abnormal values ​​in the degree of change between paired, related parts of the avatar's face, etc., as seen in the settings screen 1701. The control unit 190 of the terminal device 10 displays the degree of change of settings that differ from the settings entered in the settings screen 1701 on the settings preview screen 1703. At this time, the control unit 190 may display numerical values ​​or objects in a manner different from the manner displayed in the settings screen 1701. (For example, different colors, sizes, shapes, etc.)

[0203] The avatar preview screen 1704 is a screen that displays an avatar that reflects the recommended settings from the settings preview screen 1703. For example, the control unit 190 of the terminal device 10 may display, on a screen different from the screen that displays the notification described above, the changes in the avatar's appearance when the degree of difference is within an appropriate range (a range that does not cause discomfort to the viewer). This allows users to see what the change in behavior would look like if the degree of change they set exceeded a predetermined threshold, along with the results of setting an appropriate value.

[0204] <7 Variations> Modifications of this embodiment will now be described. That is, the following embodiments may be adopted. (1) An information processing device, which may have this program pre-installed, or may install it afterward, or may store such a program on an external non-temporary storage medium, or may run on cloud computing. (2) A method in which a computer is made to function as an information processing device, and the program may be pre-installed on the information processing device or installed thereafter, or such program may be stored on an external non-temporary storage medium or run on cloud computing.

[0205] <6. Addendum> The matters described in each of the above embodiments are appended below.

[0206] (Appendix 1) A program executed by a computer 20 including a processor 29, the program causing the processor 29 to perform steps of: sensing, as parts of the performer's face, movements of one or more parts of the face (S1001); changing, based on the sensed movements of the one or more parts of the face, the aspect of one or more parts of the avatar corresponding to the performer (S1002); and receiving a setting of a degree of following the sensed result for the aspect of one or more parts of the avatar (S1003), and in the changing step (S1002), changing the aspect of one or more parts of the avatar according to the setting of the degree.

[0207] (Appendix 2) The program according to Appendix 1, wherein in the sensing step (S1001), at least one of the group consisting of the performer's eyebrows, eyelids, inner corners of the eyes, outer corners of the eyes, eyeballs, pupils, and mouth is sensed as the one or more parts of the face.

[0208] (Appendix 3) The program according to Appendix 1 or 2, wherein in the receiving setting step (S1003), the same setting is received for predetermined associated parts among the one or more parts of the avatar.

[0209] (Appendix 4) The program causes the processor 29 to further perform steps of presenting one or more candidates for the setting of the degree of following the sensed result to the performer, and receiving a selection of one or more candidates for the setting of the degree from the performer, and in the changing step, changing the aspect of one or more parts of the avatar based on the selected setting of the degree. The program is according to any one of Appendices 1 to 3.

[0210] (Appendix 5) The program causes the processor 29 to further execute steps of obtaining the voice spectrum of the speaker and obtaining information on the degree of change in the speech of the speaker from the obtained voice spectrum, and in the step of receiving the setting of the degree, receives the setting of the degree so that it can be set within a range associated with the degree of change in the speech of the speaker, and in the step of changing, changes the aspect of one or more facial parts of the avatar according to the setting of the degree, the program according to any one of Appendices 1 to 4.

[0211] (Appendix 6) In the step of receiving the setting (S1003), the setting of the frequency range for detecting the voice spectrum is received, In the step of changing (S1002), in response to detecting the voice spectrum within the set range, the aspect of one or more facial parts of the avatar is changed based on the first setting of the degree, the program according to Appendix 5.

[0212] (Appendix 7) In the step of changing (S1002), in response to detecting the voice spectrum outside the set range, based on a preset setting of the degree, which is a second setting of the degree different from the first setting of the degree, the aspect of one or more facial parts of the avatar is changed, the program according to Appendix 6.

[0213] (Appendix 8) In the step of sensing (S1001), if the movement of the speaker's mouth cannot be sensed, in the step of changing (S1002), the aspect of the avatar's mouth is changed based on the degree of change in the speech, the program according to Appendix 5.

[0214] (Appendix 9) In the step of receiving the setting (S1003), the attributes of the avatar are received from the speaker, and the degree is corrected based on the attributes, the program according to any one of Appendices 1 to 8.

[0215] (Appendix 10) The program described in Appendix 9, which, in the step of accepting settings (S1003), accepts information as an attribute that is either human, or a non-human whose facial features differ from those of a human in how the appearance of one or more facial features changes, and corrects the degree based on said attribute.

[0216] (Note 11) The program is as described in any of Appendix 1 to 10, and in step (S1002), the program causes the processor 29 to perform the steps of: estimating one or more candidate emotions for the performer; presenting the estimated one or more candidate emotions for the performer to the performer; and receiving an input operation from the performer to select one emotion from the one or more candidate emotions for the performer, and in which the program changes the appearance of one or more facial parts of the avatar corresponding to the performer based on the emotion selected by the performer.

[0217] (Note 12) In the estimation step, if the performer's emotions cannot be estimated, there is a step to change them (S The program described in Appendix 11, in 1002), which changes the appearance of one or more facial features based on settings predetermined by the performer.

[0218] (Note 13) The program is one of the programs described in any of Appendix 1 to 12, which further causes the processor 29 to perform a step of applying the degree of the parts for which a sensing result was obtained to the associated parts if a sensing result cannot be obtained for at least one of the associated parts among the one or more facial parts of the performer.

[0219] (Note 14) The program is one of the programs described in any of Appendix 1 to 13, which causes the processor 29 to perform the steps of acquiring information about the wearable device worn by the performer, and correcting the degree setting based on the acquired information about the wearable device.

[0220] (Appendix 15) The program according to Appendix 14, which in the step of correcting the degree setting, receives an input operation from the performer to adjust the degree of correction.

[0221] (Appendix 16) The program according to any one of Appendices 1 to 15, which further causes the processor to execute a step of presenting a predetermined notification to the performer when the difference in the degree setting between the pre-associated parts among one or more face parts of the avatar exceeds a predetermined threshold.

[0222] (Appendix 17) The program according to any one of Appendices 1 to 16, which in the step of receiving the setting (S1003), sets the degree of the parts related to the one or more face parts that have received the degree setting to a predetermined value.

[0223] (Appendix 18) The program according to Appendix 16, which in the step of presenting a predetermined notification to the performer, presents to the performer the parts whose degree difference exceeds the predetermined threshold together with the numerical value in a different manner.

[0224] (Appendix 19) The program according to Appendix 18, which in the step of presenting a predetermined notification to the performer, presents to the performer the state of change of at least one or more face parts when the degree difference is set within a predetermined range.

[0225] (Appendix 20) The program according to any one of Appendices 1 to 19, which in the step of receiving the setting (S1003), receives the degree setting within a predetermined range for each of one or more parts of the avatar.

[0226] (Appendix 21) A method executed by a computer 20 including a processor 29, the method comprising a step (S in which the processor 29 senses the movement of one or more face parts, which are the face parts of the performer A method for performing the steps of 1001) changing the appearance of one or more facial parts of an avatar corresponding to a performer based on the movement of one or more facial parts that have been sensed (S1002), and receiving a setting for the degree to which the appearance of one or more facial parts of the avatar should follow the sensed result (S1003), and changing the appearance of one or more facial parts of the avatar according to the setting for the degree in step (S1002).

[0227] (Note 22) Information processing device 20 comprising a control unit 203, wherein the control unit 203 performs the following steps: S1001: sensing the movement of one or more facial parts of a performer; S1002: changing the appearance of one or more facial parts of an avatar corresponding to the performer based on the sensed movement of one or more facial parts; S1003: receiving a setting for the degree to which the appearance of one or more facial parts of the avatar should follow the sensed result; and in step S1002, changing the appearance of one or more facial parts of the avatar according to the setting for the degree. [Explanation of Symbols]

[0228] 10 Terminal device, 12 Communication interface, 13 Input device, 14 Output device, 15 Memory, 16 Storage unit, 19 Processor, 20 Server, 22 Communication interface, 23 Input / Output interface, 25 Memory, 26 Storage, 29 Processor, 80 Network, 1801 User information, 1802 Avatar information, 1803 Wearable device information, 1901 Input operation reception unit, 1902 Transmit / receive unit, 1903 Data processing unit, 1904 Notification control unit, 1302 Display, 140 Audio processing unit, 141 Microphone, 142 Speaker, 150 Location information sensor, 160 Camera, 170 Motion sensor, 2021 User information database, 2022 Avatar information database, 2023 Wearable device information database, 2031 Receiving control module, 2032 Transmitting control module, 2033 User information acquisition module, 2034 Avatar information acquisition module, 2035; voice spectrum acquisition module, 2036; avatar change module, 2037; avatar presentation module, 2038; setting acceptance module, 2039; wearable device information acquisition module, 2040; change correction module.

Claims

1. A program that is executed by a computer having a processor, wherein the program is executed by the processor, A step of sensing the movement of one or more facial parts of the performer, The steps include changing the appearance of one or more facial parts of the avatar corresponding to the performer based on the movement of the one or more facial parts that have been sensed, The process involves receiving a setting for the degree to which the appearance of one or more facial parts of the avatar should follow the sensed result, and then executing the following steps: A program that, in the step of changing, changes the appearance of one or more facial features of the avatar according to the setting of the degree.

2. The program according to claim 1, wherein in the sensing step, at least one of the group consisting of the performer's eyebrows, eyelids, inner corner of the eye, outer corner of the eye, eyeball, pupil, and mouth is sensed as the one or more facial parts.

3. The program according to claim 1, wherein in the step of accepting the above settings, the program accepts the same settings for associated predetermined parts among the one or more facial parts of the avatar.

4. The program further includes the step of presenting the processor with one or more candidate settings for the degree to which the performer should follow the sensed results, The performer is given the step of selecting one or more candidates for setting the degree, The program according to claim 1, wherein in the step of changing, the appearance of one or more facial features of the avatar is changed based on the setting of the degree to which the selection was accepted.

5. The program further includes the step of obtaining the performer's voice spectrum from the processor, The procedure involves performing the step of obtaining information on the degree of change in the performer's speech from the acquired speech spectrum, In the step of accepting the setting of the degree, the setting of the degree is accepted in a manner that allows it to be set within a range associated with the degree of change in the performer's speech. The program according to claim 1, wherein in the step of changing, the appearance of one or more facial parts of the avatar is changed according to the setting of the degree.

6. In the step of accepting the above settings, the setting of the frequency range for detecting the audio spectrum is accepted. The program according to claim 5, wherein in the step of changing, in response to detecting an audio spectrum within the set range, the appearance of one or more facial features of the avatar is changed based on the first setting of the degree.

7. The program according to claim 6, wherein in the step of changing, in response to the detection of an audio spectrum outside the set range, the program changes the appearance of one or more facial parts of the avatar based on a predetermined degree setting, which is a second degree setting different from the first degree setting.

8. The program according to claim 5, wherein, in the sensing step, if the mouth movements of the performer cannot be sensed, in the changing step, the mouth shape of the avatar is changed based on the degree of change in the speech.

9. In the step of accepting the above settings, The program according to claim 1, which receives the attributes of the avatar from the performer and corrects the degree based on those attributes.

10. The program according to claim 9, wherein in the step of receiving the above setting, the program receives information as the attribute of either a human or a non-human whose facial features change in a manner different from that of a human, and corrects the degree based on the attribute.

11. The program further includes the step of the processor estimating one or more candidate emotions of the performer, The steps include: presenting the performer with one or more candidate emotions of the performer that have been estimated; and receiving an input operation from the performer to select one emotion from the one or more candidate emotions of the performer. The program according to claim 1, wherein in the step of changing, the appearance of one or more facial features of an avatar corresponding to the performer is changed based on the emotion selected by the performer.

12. If, in the estimation step, the performer's emotions cannot be estimated, the program according to claim 11, wherein in the altering step, the appearance of one or more facial features is altered based on settings predetermined by the performer.

13. The program according to claim 1, further causing the processor to perform the step of applying the degree of the part for which a sensing result was obtained to the associated part if a sensing result cannot be obtained for at least one of the associated parts of the performer's face.

14. The program further includes the step of obtaining information about the wearable device worn by the performer from the processor, The program according to claim 1, which causes the program to perform the step of correcting the setting of the degree based on the information of the wearable device acquired.

15. The program according to claim 14, wherein in the step of correcting the setting of the degree, the program receives an input operation from the performer to adjust the degree of correction.

16. The program according to claim 1, further comprising the step of causing the processor to perform a predetermined notification to the performer when the difference in the degree settings between pre-associated parts of one or more facial parts of the avatar exceeds a predetermined threshold.

17. The program according to claim 1, wherein in the step of receiving the aforementioned settings, the degree of the one or more facial parts and related parts for which the degree settings have been received is set to a predetermined value.

18. The program according to claim 16, wherein in the step of presenting the predetermined notification to the performer, the program presents to the performer, in a different manner, the parts in which the difference in degree exceeds the predetermined threshold, along with the numerical value.

19. The program according to claim 18, wherein in the step of presenting the predetermined notification to the performer, the program presents to the performer how the at least one or more facial features change when the difference in degree is set within the predetermined range.

20. The program according to claim 1, wherein in the step of receiving the above settings, the program receives the setting of the degree within a predetermined range for one or more parts of the avatar.

21. A method performed by a computer having a processor, wherein the processor A step of sensing the movement of one or more facial parts of the performer, The steps include changing the appearance of one or more facial parts of the avatar corresponding to the performer based on the movement of the one or more facial parts that have been sensed, The process includes receiving a setting for the degree to which the appearance of one or more facial parts of the avatar should follow the sensed result, A method for changing the appearance of one or more facial features of the avatar in accordance with the setting of the degree, in the step of changing.

22. An information processing apparatus comprising a control unit, wherein the control unit is A step of sensing the movement of one or more facial parts of the performer, The steps include changing the appearance of one or more facial parts of the avatar corresponding to the performer based on the movement of the one or more facial parts that have been sensed, The process includes receiving a setting for the degree to which the appearance of one or more facial parts of the avatar should follow the sensed result, An information processing device that, in the step of changing, changes the appearance of one or more facial parts of the avatar according to the setting of the degree.

Images