Information processing device, control program, and control method

Abstract

To provide an information processing device, a control program, and a control method that make it easy to talk with a user who is a dialog partner.SOLUTION: In an information processing system in which a user-side terminal communicably connects to an operator-side terminal and a server, the operator-side terminal includes a CPU. The operator looks at an avatar image 210 which is shown to a display device and which corresponds to a user who is a dialog partner, and talks with a user who uses the communicably connected user-side terminal. When the operator is not speaking, the operator-side terminal recognizes the expression of the user and, when the recognized expression is a smiling or a displeased expression, an avatar image having been represented by favorably changing the same expression is displayed. Meanwhile, when the operator is speaking, the operator-side terminal recognizes the expression of the operator and, when the recognized expression is a smiling or a sad expression, an avatar image in which the same expression is represented with a preset level of expression is displayed.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control program, and a control method, and more particularly, for example, to an information processing apparatus, a control program, and a control method used by an operator who interacts with the user of a communicably connected terminal by voice.

Background Art

[0002] An example of this type of conventional information processing apparatus is disclosed in Patent Document 1. In the information processing system disclosed in Patent Document 1, when a user wants to consult about a product or how to use a shopping site, the user can call an operator for consultation by pressing a call button. When the user terminal and the operator terminal are connected, the website displayed on the user terminal is displayed on the operator terminal in the current display mode. Also, on the user terminal, an image of the operator or an avatar image synchronized with it is displayed on the website. Therefore, the operator serves the user while using body language.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above-mentioned Patent Document 1, the operator terminal does not display any screens that the user did not intend, such as website screens other than the shopping site screen or other personal information. In other words, the operator terminal does not display the user's image or an avatar image synchronized with it, and the operator cannot see the face of the user or the synchronized avatar, making interaction difficult. For this reason, it is conceivable to display the user's image on the operator terminal, but this may infringe on the user's privacy. It is also conceivable to display an avatar image synchronized with the user's image, but there is room for improvement in order to make interaction easier for the operator.

[0005] Therefore, the primary objective of this invention is to provide a novel information processing device, control program, and control method.

[0006] Another object of this invention is to provide an information processing device, a control program, and a control method that can facilitate interaction with another party. [Means for solving the problem]

[0007] The first invention is an information processing device comprising: a voice detection means for detecting operator voice, which is the voice spoken by an operator interacting with a user; a shooting means for capturing an image of the operator's face; a first recognition means for recognizing the operator's facial expression based on the image of the operator's face captured by the shooting means when the voice detection means has detected operator voice; an avatar display means for displaying an image of an avatar corresponding to the user, who is the person the operator is interacting with, on a display device, which expresses the same facial expression as the predetermined first facial expression when the facial expression of the operator recognized by the first recognition means is a predetermined first facial expression; and a transmission means for transmitting the operator voice detected by the voice detection means to a user-side terminal used by the user.

[0008] The second invention is subordinate to the first invention, and the given first facial expressions are a smiling expression and a sad expression.

[0009] The third invention is dependent on the first or second invention and further comprises a timing determination means for determining whether it is the right time to nod based on the operator's voice when the operator's voice is detected by the voice detection means, and an avatar control means for causing the avatar to perform a nodding motion when the timing determination means determines that it is the right time to nod.

[0010] The fourth invention is dependent on any of the first to third inventions and further comprises a receiving means for receiving a user's face image transmitted from a user-side terminal, and a second recognition means for recognizing the user's facial expression based on the user's face image received by the receiving means when the operator's voice has not been detected by the voice detection means. The avatar display means, when the user's facial expression recognized by the second recognition means is a predetermined second facial expression, causes the avatar to display the same facial expression as the predetermined second facial expression, but modified to be favorable to the operator.

[0011] The fifth invention is subordinate to the fourth invention, wherein a predetermined second facial expression is a smiling expression and a grumpy expression, the second recognition means further recognizes the degree of the user's smiling expression and grumpy expression, and the avatar display means, when the user's expression is a smiling expression, emphasizes the degree of the smile and displays it on the avatar, and when the user's expression is a grumpy expression, mitigates the degree of grumpiness and displays it on the avatar.

[0012] The sixth invention is dependent on any of the first to third inventions and further comprises a receiving means for receiving the user's facial expression transmitted from the user's terminal, wherein the avatar display means, when the voice detection means has not detected the operator's voice, and the user's facial expression received by the receiving means is a predetermined second facial expression, causes the avatar to display the same facial expression as the predetermined second facial expression, but modified to be favorable to the operator.

[0013] The seventh invention is subordinate to the sixth invention, wherein a predetermined second facial expression is a smiling expression and a grumpy expression, the receiving means further receives the degree of the user's smiling expression and grumpy expression, and the avatar display means, when the user's expression is a smiling expression, causes the degree of smiling to be emphasized and displayed on the avatar, and when the user's expression is a grumpy expression, causes the degree of grumpiness to be mitigated and displayed on the avatar.

[0014] The eighth invention is the In the 4th or 5th field invention The system further includes a subordinate gaze detection means that detects the user's gaze based on the user's facial image received by the receiving means. The avatar display means sets the avatar's gaze to match the user's gaze detected by the gaze detection means, and if the voice detection means does not detect the operator's voice, and the user's gaze detected by the gaze detection means has been facing forward for a first predetermined time, the avatar's gaze is averted for a second predetermined time, regardless of the user's gaze.

[0015] The ninth invention is dependent on the sixth or seventh invention, the receiving means further receives the user's gaze, the avatar display means sets the avatar's gaze to match the received user's gaze, and when the voice detection means does not detect the operator's voice, Received by receiving means If the user's gaze remains directed forward for a specified period of time (1), the avatar's gaze will be averted for a specified period of time (2), regardless of the user's gaze.

[0016] The tenth invention is a control program executed in an information processing device, which causes the processor of the information processing device to execute a voice detection step for detecting operator voice, which is the voice spoken by an operator interacting with a user; a shooting step for capturing an image of the operator's face; a recognition step for recognizing the operator's facial expression based on the image of the operator's face captured in the shooting step, if operator voice is detected in the voice detection step; an avatar display step for displaying an image of an avatar corresponding to the user, who is the person the operator is interacting with, on a display device, which expresses the same facial expression as the predetermined facial expression, if the facial expression of the operator recognized in the recognition step is a predetermined facial expression; and a transmission step for transmitting the operator voice detected in the voice detection step to a user-side terminal used by the user.

[0017] The eleventh invention is a control method for an information processing device, comprising the steps of (a) detecting operator voice, which is the voice spoken by an operator interacting with a user; (b) capturing an image of the operator's face; and (c) when operator voice is detected in step (a) ,vinegar Step (b) The control method includes the steps of: (d) recognizing the operator's facial expression based on the operator's face image captured in step (c); (c) if the operator's facial expression recognized in step (c) is a predetermined facial expression, displaying an image of an avatar corresponding to the user, who is the conversation partner, expressing the same facial expression as the predetermined facial expression on a display device; and (e) transmitting the operator's voice detected in step (a) to the user's terminal used by the user. [Effects of the Invention]

[0018] This invention makes it easier to communicate with the user, who is the other party to the conversation.

[0019] The above-mentioned objectives, other objectives, features, and advantages of this invention will become even clearer from the following detailed description of embodiments with reference to the drawings. [Brief explanation of the drawing]

[0020] [Figure 1]FIG. 1 is a diagram showing an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing the electrical configuration of the user-side terminal shown in FIG. 1. [Figure 3] FIG. 3 is a block diagram showing the electrical configuration of the operator-side terminal shown in FIG. 1. [Figure 4] FIG. 4 is a diagram showing an example of a screen displayed on the display device of the user-side terminal. [Figure 5] FIG. 5 is a diagram showing an example of an avatar image displayed on the display device of the operator-side terminal. [Figure 6] FIG. 6(A) is a diagram showing an example of a smiling expression of avatars with different degrees, FIG. 6(B) is a diagram showing an example of an angry expression of avatars with different degrees, and FIG. 6(C) is a diagram showing an example of a sad expression of avatars with different degrees. [Figure 7] FIG. 7 is a diagram showing an example of a memory map of the RAM of the operator-side terminal. [Figure 8] FIG. 8 is a diagram showing an example of the specific content of the data storage area shown in FIG. 7. [Figure 9] FIG. 9 is a flowchart showing a part of an example of the control process of the CPU of the operator-side terminal shown in FIG. 3. [Figure 10] FIG. 10 is another part of an example of the control process of the CPU of the operator-side terminal shown in FIG. 3, and is a flowchart following FIG. 9. [Figure 11] FIG. 11 is another part of an example of the control process of the CPU of the operator-side terminal shown in FIG. 3, and is a flowchart following FIG. 9. [Figure 12] FIG. 12 is still another part of an example of the control process of the CPU of the operator-side terminal shown in FIG. 3, and is a flowchart following FIGS. 10 and 11. [Figure 13] FIG. 13 is a flowchart showing an example of the transmission / reception process of the CPU of the operator-side terminal shown in FIG. 3. [Figure 14] FIG. 14 is a flowchart showing an example of the transmission / reception process of the CPU of the user-side terminal shown in FIG. 2. [Figure 15]Figure 15 is a flowchart showing a portion of the CPU control processing of the operator-side terminal in the second embodiment. [Figure 16] Figure 16 is a flowchart showing the transmission and reception processing of the CPU of the user-side terminal in the second embodiment. [Modes for carrying out the invention]

[0021] <First Example> Referring to Figure 1, the information processing system 10 of this first embodiment includes a user-side terminal 12, which is communicated via a network 14 to an operator-side terminal 16 and a server 18.

[0022] In this first embodiment, one user-side terminal 12 and one operator-side terminal 16 are shown. However, in reality, multiple user-side terminals 12 and multiple operator-side terminals 16 are provided, and as will be described later, chat or talk processing takes place between one user-side terminal 12 and one operator-side terminal 16 selected by the server 18 in response to a request from this user-side terminal 12.

[0023] The user terminal 12 is used by users who utilize a predetermined service provided by the server 18, and the operator terminal 16 is used by operators who respond to users.

[0024] The user terminal 12 is an information processing device, and one example is a general-purpose smartphone equipped with a browser function. In other examples, the user terminal 12 can also be a general-purpose device such as a tablet PC, notebook PC, or desktop PC.

[0025] Network 14 consists of an IP network (or IP network) including the Internet, and an access network (or access network) for accessing this IP network. The access network can include public telephone networks, mobile phone networks, wired LANs, wireless LANs, CATV (Cable Television), etc.

[0026] The operator-side terminal 16 is an information processing device different from the user-side terminal 12. For example, it may be a general-purpose notebook PC or desktop PC, but in other examples, other general-purpose terminals such as smartphones or tablet PCs may also be used.

[0027] Server 18 is an information processing device different from the user terminal 12 and the operator terminal 16, and a general-purpose server can be used. Therefore, server 18 includes a CPU 18a and a storage unit (including HDD, ROM, and RAM) 18b, as well as components such as a communication interface and an input / output interface. In the first embodiment, server 18 operates a site that provides a predetermined service.

[0028] Figure 2 is a block diagram showing the electrical configuration of the user terminal 12 shown in Figure 1. As shown in Figure 2, the user terminal 12 includes a CPU 20, which is connected to a storage unit 22, a communication interface (hereinafter referred to as "communication I / F") 24, and an input / output interface (hereinafter referred to as "input / output I / F") 26 via an internal bus.

[0029] The CPU 20 is responsible for the overall control of the user terminal 12. However, instead of the CPU 20, a System-on-a-chip (SoC) that includes multiple functions such as CPU function and GPU (Graphics Processing Unit) function may be provided. The storage unit 22 includes an HDD, ROM, and RAM. However, non-volatile memory such as an SSD may be used instead of the HDD, or in addition to the HDD, ROM, and RAM.

[0030] The communication interface 24 has a wired interface for sending and receiving control signals and data between the operator terminal 16 and external computers such as the server 18 via the network 14 under the control of the CPU 20. However, a wireless interface such as wireless LAN or Bluetooth® can also be used as the communication interface 24.

[0031] The input / output interface 26 is connected to an input device 28, a display device 30, a microphone 32, and a speaker 34. The input device 28 consists of a touch panel and hardware buttons. The touch panel is a general-purpose touch panel, and any type of touch panel can be used, such as a capacitive, electromagnetic induction, resistive, or infrared touch panel. The same applies to the operator terminal 16, which will be described later.

[0032] However, if a notebook PC or desktop PC is used as the user terminal 12, a keyboard and computer mouse are used as the input devices 28.

[0033] Furthermore, the display device 30 is an LCD or an organic EL display device. The touch panel may be provided on the display surface of the display device 30, or a touch panel display in which the touch panel is integrally formed with the display device 30 may be provided. The same applies to the operator-side terminal 16, which will be described later.

[0034] The input / output interface 26 outputs operation data (or operation information) received from the input device 28 to the CPU 20, and also outputs image data generated by the CPU 20 to the display device 30, causing the display device 30 to display a screen or image corresponding to the image data. However, image data received from an external computer (for example, the operator terminal 16 or server 18) may also be output by the CPU 20.

[0035] Furthermore, the input / output interface 26 converts the user's voice detected by the microphone 32 into digital audio data and outputs it to the CPU 20, and also converts the audio data output by the CPU 20 into an analog audio signal and outputs it from the speaker 34. However, in the first embodiment, the audio data output from the CPU 20 is the audio data received from the operator terminal 16.

[0036] Furthermore, the user terminal 12 is equipped with a sensor interface (sensor I / F) 36 and a camera 38. The CPU 20 is connected to the camera 38 via the bus and the sensor I / F 36. The camera 38 is a camera using an image sensor such as a CCD or CMOS.

[0037] Note that the electrical configuration of the user terminal 12 shown in Figure 2 is just one example and is not limited to that configuration.

[0038] Furthermore, if the user terminal 12 is a smartphone, it is equipped with a call circuit for making calls via the mobile phone network, or the mobile phone network and the public telephone network. However, in the first embodiment, such calls are not made, so they are omitted from the illustration. The same applies when the operator terminal 16, which will be described later, is a smartphone.

[0039] Figure 3 is a block diagram showing the electrical configuration of the operator-side terminal 16 shown in Figure 1. As shown in Figure 3, the operator-side terminal 16 includes a CPU 50, which is connected to the storage unit 52, communication interface 54, and input / output interface 56 via an internal bus.

[0040] The CPU 50 is responsible for the overall control of the operator terminal 16. However, instead of the CPU 50, an SoC (System on a Chip) including multiple functions such as CPU and GPU functions may be provided. The storage unit 52 includes an HDD, ROM, and RAM. However, instead of the HDD, or in addition to the HDD, ROM, and RAM, non-volatile memory such as an SSD may be used.

[0041] The communication interface 54 has a wired interface for sending and receiving control signals and data between the user terminal 12 and external computers such as the server 18 via the network 14 under the control of the CPU 50. However, a wireless interface such as wireless LAN or Bluetooth® can also be used as the communication interface 54.

[0042] The input / output interface 56 is connected to an input device 58, a display device 60, a microphone 62, and a speaker 64. The microphone 62 and speaker 64 constitute a headset with a microphone used by the operator to communicate with the user.

[0043] Furthermore, a keyboard and a computer mouse are used as input devices 58. However, if a smartphone or tablet PC is used as the operator terminal 16, a touch panel and hardware buttons are provided as input devices 58. The display device 60 is an LCD or an organic EL display device.

[0044] The input / output interface 56 outputs operation data (or operation information) received from the input device 58 to the CPU 50, and also outputs image data generated by the CPU 50 to the display device 60, causing the display device 60 to display a screen corresponding to the image data.

[0045] Furthermore, the input / output interface 56 converts the operator's voice detected by the microphone 62 into digital audio data and outputs it to the CPU 50, and also converts the audio data output by the CPU 50 into an analog audio signal and outputs it from the speaker 64. However, in the first embodiment, the audio data output from the CPU 50 is the audio data received from the user terminal 12.

[0046] Furthermore, the operator-side terminal 16 is equipped with a sensor I / F 66 and a camera 68. The CPU 50 is connected to the camera 68 via the bus and the sensor I / F 66. The camera 68 is a camera using an image sensor such as a CCD or CMOS.

[0047] In this information processing system 10, users use a user-side terminal 12 to view a web screen 100 of a predetermined service provided by the server 18 and perform tasks such as shopping. Buttons 110 and 112 for communicating with the operator via chat or talk are displayed on the front of the web screen 100.

[0048] However, the web screen 100 is displayed on the display device 30 by launching a web browser and entering a predetermined URL. The web screen 100 is the screen of the website (or web page) of a predetermined service. Figure 4 shows an example of a web screen 100 for an online shopping service. As an example, the predetermined service is online shopping, but it is any online service that can respond to user inquiries via chat or talk.

[0049] Furthermore, buttons 110 and 112 may be displayed when certain conditions are met. These conditions include: the user instructing buttons 110 and 112 to be displayed; the user not operating for an extended period (e.g., 30 seconds to several minutes); the user repeatedly or continuously instructing the same or a similar location (a nearby location) on the web screen 100 displayed on the display device 30; and the user returning to the same web screen 100 multiple times (e.g., three times) in a predetermined service.

[0050] When the user turns on button 110, a chat service is provided to the user, and a chat takes place between the user and an operator selected by the server 18. In other words, text messages are sent and received between the user terminal 12 and the operator terminal 16. Chat services are already well known and are not essential to the present invention, so a description is omitted. As an example, the chat service disclosed in Japanese Patent Application Publication No. 2020-86677 can be used. However, a chatbot can also respond instead of an operator.

[0051] Furthermore, when the user turns on button 112, the user can voice-inquire about how to use the online shopping site and about products to an operator selected by server 18.

[0052] In this case, an image of an avatar corresponding to the operator is displayed on the display device 30 of the user terminal 12. For example, the image of the avatar corresponding to the operator is a partial or full-body image of a character resembling a human, including its face, and is displayed in front of the web screen 100. Furthermore, the image of the avatar corresponding to the operator moves in accordance with the operator's voice output. The avatar moves its mouth, moves its head, blinks, and changes its facial expression. However, if the avatar image is an image of the upper body or full body of a character resembling a human, the avatar will also perform gestures and movements.

[0053] Furthermore, the display and control of the avatar image corresponding to the operator are not essential to the present invention and can be handled using prior art; therefore, a description is omitted.

[0054] However, instead of displaying an image of an avatar corresponding to the operator, an image of the operator may be displayed.

[0055] In the information processing system 10 described above, the operator typically interacts with the user either while viewing the user's video feed or without viewing the user's video feed.

[0056] In the former case, the operator can see the user's video feed and even recognize their facial expressions, making it easier to interact with the user. However, because the user's video feed is displayed, there is room for improvement in protecting the user's privacy.

[0057] Furthermore, in the latter case, while it is believed that user privacy is protected, there is a problem in that the operator cannot see the user's video, making interaction difficult.

[0058] Therefore, in the first embodiment, in order to protect the user's privacy and facilitate interaction with the user, an image of an avatar corresponding to the user is displayed on the display device 60 of the operator-side terminal 16, and the avatar's facial expression (in the first embodiment, facial expression), gaze, and movements are controlled.

[0059] Figure 5 shows an example of an avatar image 210 corresponding to the user, displayed on the display device 60 of the operator terminal 16. As shown in Figure 5, the avatar image 210 is displayed within a rectangular display frame 200. The avatar image 210 corresponding to the user is a partial image of a human-like character, including its face. More specifically, the avatar image 210 is an image of a human head (including the face), neck, and part of the shoulders.

[0060] However, if the avatar image 210 is an image of the upper body or full body of a character modeled after a human, the avatar will also perform gestures. While a detailed explanation is omitted here, in this case, as will be described later, if the avatar's facial expressions are controlled, the gestures will also be controlled.

[0061] Alternatively, the same web screen 100 displayed on the display device 30 of the user terminal 12 can be displayed on the display device 60, and a display frame 200 and an image 210 of an avatar corresponding to the user can be displayed in front of this web screen 100.

[0062] In this case, display information including the URL of the website displayed on the display device 30 of the user terminal 12, the window size, and the cursor position coordinates is sent to the server 18 when button 112 is turned on on the user terminal 12, and further sent from the server 18 to the operator terminal 16 when the responding operator, i.e., the operator terminal 16, is selected. However, the display information may also be sent directly from the user terminal 12 to the operator terminal 16 between the start of communication and the start of the talk.

[0063] The avatar image 210 is an image of an avatar pre-selected by the user or an avatar pre-selected by the operator. When an image of an avatar pre-selected by the user is displayed, the type of avatar is notified to the operator's terminal 16 either from the user's terminal 12 via the server 18 or directly from the user's terminal 12.

[0064] Furthermore, in the first embodiment, the operator terminal 16 causes the avatar corresponding to the user to express a favorable or empathetic expression towards the operator. The operator terminal 16 also causes the avatar corresponding to the user to operate in a way that avoids situations where the operator finds it difficult to speak.

[0065] Specifically, when the operator is not speaking, the operator's terminal 16 controls the facial expressions, gaze, and head (neck) movements of the avatar corresponding to the user based on the user's facial expressions or the user's gaze and head movements. However, some of the user's facial expressions are reflected in the avatar corresponding to the user, either emphasized or softened. In addition, the gaze of the avatar corresponding to the user may be controlled independently of the user's gaze.

[0066] In the first embodiment, if the user's facial expression is a smile, the degree of the smile is increased, causing the avatar corresponding to the user to display a smiling expression. In other words, an image 210 of the avatar with the user's smile emphasized is displayed. Conversely, if the user's facial expression is an unhappy expression, the degree of unhappiness is reduced, causing the avatar corresponding to the user to display an unhappy expression. In other words, an image 210 of the avatar with the user's unhappiness mitigated is displayed. In this way, by controlling the facial expression of the avatar corresponding to the user, the avatar can be made to display the user's facial expression to a degree that is favorably altered.

[0067] The parameters for the degree to which a smile is emphasized (hereinafter referred to as "emphasis level") and the degree to which displeasure is mitigated (hereinafter referred to as "mitigation level") (hereinafter referred to as "setting parameters") are pre-set by the operator.

[0068] However, the settings parameters for emphasis and mitigation may be set automatically. In this case, for example, if the user's smile is low, the emphasis will be set high, and if the user's smile is high, the emphasis will be set low. Also, for example, if the user's displeasure is low, the mitigation will be set low, and if the user's displeasure is high, the mitigation will be set high.

[0069] Furthermore, in the first embodiment, the gaze of the avatar corresponding to the user is controlled according to the user's gaze. If the user's gaze is directed forward and this continues for a first predetermined time (for example, several seconds to about 10 seconds), the avatar corresponding to the user averts its gaze for a second predetermined time (for example, a predetermined percentage of the first predetermined time), regardless of the user's gaze. In other words, by making the avatar corresponding to the user act (i.e., controlling it) to avert its gaze, the avatar acts in a way that does not create a sense of intimidation towards the operator, thus avoiding a situation where the operator finds it difficult to speak. The predetermined percentage parameter is also included in the above setting parameters and is set in advance by the operator.

[0070] However, the user's gaze during interaction with the operator is detected based on the user's facial image. Since the method for detecting gaze from facial images is already well known, we will omit its explanation.

[0071] Furthermore, the user's head movements are detected (estimated) as follows: The user's face orientation is calculated based on the current face image, using the orientation of the user's face image facing the camera 38 as a reference. Based on the current face orientation, the user's head movements are detected (or estimated). However, face orientation can also be detected by the movement of multiple facial feature points extracted from the face image. The same applies when calculating the operator's face orientation and detecting head movements, although this explanation is omitted.

[0072] As mentioned above, the user's gaze and head movements are reflected in the gaze and head (neck) movements of the avatar corresponding to the user. This is also true when the user is speaking.

[0073] Furthermore, if the operator is not speaking, the user may speak. When the user is speaking, the lips of the avatar corresponding to the user move in accordance with the user's voice output. Hereafter, the movement of the avatar's lips in accordance with the user's voice output may be referred to as the "speech action."

[0074] Furthermore, when the operator is speaking, the operator terminal 16 may control the facial expressions of the avatar that responds to the user based on the operator's facial expressions, and may also control the actions of the avatar that responds to the user based on the voice spoken by the operator.

[0075] In the first embodiment, the operator terminal 16 recognizes the operator's facial expression and causes the avatar to express the same facial expression as the operator. Specifically, if the operator's facial expression is a smile, the avatar corresponding to the user will express a smile. If the operator's facial expression is a sad expression, the avatar corresponding to the user will express a sad expression. In other words, when the operator is speaking, the avatar will empathize with the operator.

[0076] In this first embodiment, the parameters for the degree of smile and the degree of sadness are pre-set by the operator and are included in the above-mentioned setting parameters. However, the parameters for the degree of smile and the degree of sadness can also be set according to the operator's own degree of smile and sadness.

[0077] As described above, the facial expressions of the user and the operator are recognized based on their respective facial images, and the recognized facial expressions are represented in the avatar image 210. The facial expressions of the user and the operator can be recognized based on their respective facial images during the conversation.

[0078] Since methods for recognizing human facial expressions, such as those of users and operators, using facial images are already publicly known, a description of these methods will be omitted. As an example, publicly known techniques disclosed in "Hiroshi Kobayashi, Fumio Hara: Basic Human Facial Expression Recognition using Neural Networks, Transactions of the Society of Instrument and Control Engineers, Vol. 29, No. 1, 112 / 118 (1993)", "Yosuke Kotani, Tsuneyasu Honma, Masao Sakai, Kenichi Abe: Facial Expression Recognition using Neural Networks, Bulletin of the Faculty of Medicine and Health Sciences, Tohoku University 13(1): 23~32, 2004", and "Daiki Nishime, Satoshi Endo, Yoshiaki Toma, Koji Yamada, Yuhei Akamine: Acquisition of Facial Expression Representation and Analysis of Facial Features using Convolutional Neural Networks, Transactions of the Japanese Society for Artificial Intelligence, Vol. 32, No. 5, FZ (2017)" can be used.

[0079] Furthermore, in other known technologies, the technique disclosed in Japanese Patent Publication No. 2020-163660 can be used as a method for recognizing human facial expressions based on feature points extracted from facial images.

[0080] However, the circuit components and data necessary for recognizing facial expressions based on human facial images are provided on the operator's terminal 16 as appropriate. Alternatively, a device for recognizing facial expressions based on facial images (hereinafter referred to as the "recognition device") may be provided on the cloud, and facial images may be sent to the recognition device, with the facial recognition results being received from the estimation device.

[0081] Furthermore, by training the neural network with multiple facial expressions of varying degrees (in this first embodiment, a smiling expression, a grumpy expression, and a sad expression), it is possible to recognize (or estimate) not only facial expressions but also their degree. It is also possible to recognize the degree of an expression based on the difference in the neural network's output when an expression is recognized. For example, the degree of an expression is recognized based on the difference between the output for a neutral face image and the output for a recognized expression. As an example, the degree of an expression is recognized on a scale of 0-100%.

[0082] Furthermore, when recognizing human facial expressions using the method described in Japanese Patent Publication No. 2020-163660, the degree of expression can also be recognized (or estimated) based on the difference (distance) between feature points extracted from the face image. For example, the distance between each feature point extracted from the user's face image used for facial expression recognition and each feature point extracted from the face image with no expression is calculated, and the degree of expression is determined based on the calculated distances. Since the distance is calculated for each feature point, the degree of expression is determined, for example, based on the average, maximum, or variance of the multiple calculated distances.

[0083] Figure 6(A) illustrates the differences in the degree of a smiling expression in an avatar, Figure 6(B) illustrates the differences in the degree of a grumpy expression in an avatar, and Figure 6(C) illustrates the differences in the degree of a sad expression in an avatar.

[0084] Smiling, grumpy, and sad expressions can each be set to a degree (or magnitude) in multiple stages (for example, 20 stages) from minimum (0%) to maximum (100%). The facial expression of an avatar is determined for each expression and its degree by parameters (hereinafter referred to as "expression parameters") for each part of the face, including eyebrows (position, shape), frown lines (degree of furrowing), eyes (size of pupils, degree of upward or downward tilt of the outer corners of the eyes), eyelids (degree of opening), nasolabial folds (degree of furrowing), and mouth (position, shape, and degree of opening).

[0085] As described above, the degree of human facial expressions, such as those of users and operators, is recognized on a scale of 0-100%, and the avatar's facial expressions are represented on a scale of 0-100%. Therefore, in this first embodiment, the facial expression parameters are determined according to the degree of human facial expressions. Consequently, the avatar image 210, which represents the same facial expression as the human facial expression recognized from the face image, will have the same facial expression as that human.

[0086] In the examples shown in Figures 6(A)-6(C), the degree of facial expression increases from the left end (i.e., expressionless) to the right end, and decreases as you approach the left end. Figures 6(A)-6(C) show avatar images 210 for each facial expression, both when the degree of expression is medium (normal) and when the degree of expression is at its maximum.

[0087] However, the above image 210 of the expressionless avatar represents the lowest degree of expression for each of the following: smiling, grumpy, and sad.

[0088] Although not shown in the diagram, as mentioned above, the degree of each emotion is set to multiple levels, making it possible to display images 210 of avatars with expressions for each level.

[0089] Furthermore, since each facial expression can be set in 20 steps, the parameters for emphasis, relaxation, smile, and sadness can be set in 5% increments.

[0090] Furthermore, expressionless avatars have no wrinkles and their faces are set to be symmetrical. This design makes it difficult to determine the gender, resulting in avatars that are accepted by users with a preference for either male or female. Also, by creating a featureless face that is wrinkle-free and symmetrical, it becomes easy to express expressions such as smiles or frowns by simply adding a few wrinkles, and the intensity of those expressions can be easily controlled.

[0091] Furthermore, in the first embodiment, when the operator is speaking, the operator's terminal 16 causes the avatar corresponding to the user to nod at an appropriate time. In other words, situations where it is difficult for the operator to speak are avoided.

[0092] Specifically, the system causes the avatar corresponding to the user to nod when the operator's voice is interrupted or when the operator speaks something that seeks the user's consent. An interruption in the operator's voice is determined when the volume of the operator's voice remains below a predetermined level for a third predetermined time (for example, several milliseconds). Furthermore, whether the operator has spoken something that seeks the user's consent is determined by recognizing the operator's voice and checking whether the operator has spoken a predetermined phrase that seeks consent. Examples of such phrases include "Isn't that right?" and "Is that alright?".

[0093] However, if the avatar corresponding to the user does not nod and the operator's facial expression is not reflected in the avatar, the user's facial expression, gaze, and head movements will be reflected in the avatar. Furthermore, if the avatar corresponding to the user does not nod and the operator's facial expression is reflected in the avatar, the user's gaze and head movements will be reflected in the avatar. In addition, if the avatar corresponding to the user does nod, the user's facial expression will be reflected in the avatar.

[0094] Furthermore, even when the operator is speaking, the user may also speak. In this case as well, the avatar corresponding to the user will perform speech actions in accordance with the user's voice output.

[0095] Figure 7 shows an example of the memory map 300 of the memory unit (here, RAM) 52 built into the operator terminal 16. The RAM is used as the work area and buffer area for the CPU 50. As shown in Figure 7, the RAM includes a program storage area 302 and a data storage area 304. The program storage area 302 stores a control program for the overall processing performed by the operator terminal 16 in this first embodiment.

[0096] The control program includes an operation detection program 302a, a shooting program 302b, a sound detection program 302c, a speech recognition program 302d, a communication program 302e, an image generation program 302f, an image output program 302g, an avatar control program 302h, an expression recognition program 302i, a gaze and head movement detection program 302j, and a sound output program 302k, among others.

[0097] The operation detection program 302a is a program for detecting operation data 304a input from the input device 58 according to the operator's operation and storing it in the data storage area 304. The shooting program 302b is a program for taking an image with the camera 68, storing the captured image data as transmission data 304b in the data storage area 304, and also storing the captured image data 304i in the data storage area 304.

[0098] The sound detection program 302c is a program that detects sound input from the microphone 62, stores the corresponding sound data as transmission data 304b in the data storage area 304, and also stores the sound data 304k in the data storage area 304. The speech recognition program 302d is a program that performs speech recognition on the sound detected according to the sound detection program 302c. The dictionary data necessary for speech recognition is not shown in the diagram, but it is stored in the storage unit (in this case, HDD or ROM) 52 of the operator terminal 16.

[0099] The communication program 302e is a program for communicating (sending and receiving data) with external devices, in this first embodiment, the user terminal 12 and the server 18, via wired or wireless means.

[0100] The image generation program 302f is a program for generating image data (including the avatar image 210) corresponding to all or part of various screens to be displayed on the display device 60, using the image generation data 304d. The image output program 302g is a program for outputting the image data generated according to the image generation program 302f to the display device 60.

[0101] The avatar control program 302h is a program for controlling an avatar that corresponds to a user. In this first embodiment, the CPU 50 changes the avatar's facial expression, makes the avatar move (speak and move its head (neck)), and moves the avatar's gaze according to the avatar control program 302h.

[0102] The facial expression recognition program 302i is a program for recognizing the facial expressions of the user and the operator. As described above, the user's facial expression and its degree are recognized based on the captured image data received from the user terminal 12. In addition, the operator's facial expression is recognized based on the captured image data 304i captured by the camera 68 of the operator terminal 16.

[0103] The gaze and head movement detection program 302j is a program for detecting the user's gaze and head movements based on captured image data received from the user terminal 12. The sound output program 302k is a program for outputting the user's voice data received from the user terminal 12 to the speaker 64.

[0104] Although not shown in the diagram, the program storage area 302 also stores middleware such as the operating system of the operator terminal 16, a program for executing browser functions, and other programs such as various application programs.

[0105] Figure 8 shows an example of the specific contents of the RAM data storage area 304 shown in Figure 7. As shown in Figure 8, the data storage area 304 stores operation data 304a, transmission data 304b, reception data 304c, image generation data 304d, facial expression parameter data 304e, setting parameter data 304f, user facial expression data 304g, user gaze and head movement data 304h, captured image data 304i, operator facial expression data 304j, and audio data 304k.

[0106] Operation data 304a is operation data detected according to the operation detection program 302a. Transmission data 304b is data to be sent to the user terminal 12, and consists of text data about the operator's response in chat and audio data about the operator's response in talk.

[0107] The received data 304c is data transmitted from and received by the user terminal 12, and includes text data about the user's questions in the chat, audio data about the user's questions in the talk, and image data captured by the camera 38 of the user terminal 12. The received data 304c also includes connection information data for the user terminal 12 transmitted from the server 18.

[0108] The image generation data 304d is data for generating various screens displayed on the display device 60 of the operator terminal 16, and includes data for generating the avatar image 210. The data for generating the avatar image 210 includes image data of the avatar in a static state and data about neck movements. Neck movements include neck movements when speaking and neck movements when nodding. However, since multiple types of avatars are provided, image data of the avatar in a static state is stored for each avatar, and the image data of the selected avatar is used.

[0109] The facial expression parameter data 304e contains data on the facial expression parameters for each part of the face, allowing for multiple stages of change in the degree of expression, from minimum to maximum, for each of the following expressions: smiling, unhappy, and sad. However, since multiple avatars are provided, the facial expression parameter data is stored for each avatar, and the facial expression parameters for the selected avatar are used.

[0110] The setting parameter data 304f contains data for each setting parameter: degree of emphasis, degree of relaxation, degree of smile, degree of sadness, and predetermined percentages.

[0111] User facial expression data 304g is data indicating the user's facial expression and its degree, as recognized from the captured image data received from the user terminal 12. User gaze and head movement data 304h is data indicating the user's gaze and head movement, calculated from the captured image data received from the user terminal 12.

[0112] The captured image data 304i is image data captured by camera 68. The operator facial expression data 304j is data showing the operator's facial expression recognized from the captured image data 304i. The audio data 304k is data about the operator's voice detected according to the sound detection program 302c, and is used to recognize the operator's voice.

[0113] Although not shown in the diagram, the data storage area 304 stores other data necessary for executing control processing, and also contains timers (counters) and flags.

[0114] Although not shown in the diagram, the user terminal 12 engages in chat or talk with the operator terminal 16. Therefore, the memory unit (in this case, RAM) 22 of the user terminal 12 stores the same programs and data as those stored in the RAM of the operator terminal 16 that are necessary for chat or talk.

[0115] Specifically, the program storage area of ​​the RAM of the user terminal 12 stores programs such as an operation detection program, a shooting program, a sound detection program, a communication program, an image generation program, an image output program, and a sound output program.

[0116] The operation detection program is a program that detects operation data input from the input device 28 according to the user's operation and stores it in the data storage area of ​​the storage unit 22. The shooting program is a program that takes an image with the camera 38 and stores the captured image data as transmission data in the data storage area. The sound detection program is a program that detects sound input from the microphone 32 and stores the sound data of the detected sound as transmission data in the data storage area.

[0117] The communication program is a program for communicating with external devices, in this first embodiment, the operator terminal 16 and the server 18, by wired or wireless connection. The image generation program is a program for generating image data corresponding to various screens to be displayed on the display device 30, using image generation data. The image output program is a program for outputting the image data generated according to the image generation program to the display device 30. The sound output program is a program for outputting the received voice data of the operator.

[0118] Furthermore, the data storage area of ​​the storage unit 22 stores operation data, transmission data, reception data, and image generation data, among other things.

[0119] Operation data is operation data detected according to the operation detection program. Transmission data is data sent to the operator terminal 16 and the server 18. The data sent to the operator terminal 16 is text data about the user's questions in the chat, audio data about the user's questions in the talk, and image data captured by the camera 38. The data sent to the server 18 is browser operation data related to online shopping (including operation data for buttons 110 and 112).

[0120] The received data is data transmitted from and received by either the operator terminal 16 or the server 18. The data received from the operator terminal 16 consists of text data about the operator's responses in chat and audio data about the operator's responses in talk. However, it may also receive image data captured by the operator terminal 16. The data received from the server 18 consists of data to be displayed in the browser and connection information data for the operator terminal 16 selected by the server 18.

[0121] The image generation data is data used to generate various screens that are displayed on the display device 30 of the user terminal 12.

[0122] In addition, the memory unit 52 stores middleware such as the operating system of the user terminal 12, programs for executing browser functions, and other programs and data necessary for chatting or talking with the user.

[0123] Figures 9-12 are flowcharts showing the control processing of the CPU 50 of the operator-side terminal 16. Figure 13 is a flowchart showing the transmission and reception processing of the CPU 50 of the operator-side terminal 16. Figure 14 is a flowchart showing the transmission and reception processing of the CPU 20 of the user-side terminal 12.

[0124] Although not shown in the diagram, the CPU 50 performs the following processes in parallel with the control processing and transmission / reception processing: detecting the operator's actions, capturing an image of the operator, and detecting the operator's voice.

[0125] As shown in Figure 9, when the CPU 50 of the operator terminal 16 starts control processing, in step S1, it displays an image 210 of the avatar corresponding to the user. In other words, the CPU 50 generates image data of a display frame 200 that includes the image 210 of the avatar corresponding to the user, and outputs the generated image data to the display device 60.

[0126] In the next step, S3, the user's facial expression is recognized. Here, the CPU 50 acquires the captured image data contained in the received data 304c, recognizes the facial expression and its degree from the user's face image contained in the acquired captured image data, and stores (updates) the corresponding user facial expression data 304g.

[0127] In the next step, S5, the user's gaze and head movements are detected. Here, the CPU 50 detects the user's gaze and head movements from the user's face image included in the captured image data acquired in step S3, and stores (updates) the corresponding user gaze and head movement data 304h.

[0128] However, if the acquired image data does not include the user's face, facial expressions will not be recognized, nor will the user's gaze or head movements be detected.

[0129] Next, in step S7, it is determined whether the operator is speaking. Here, the CPU 50 determines whether it has detected sound from the microphone 62. If the answer in step S7 is "YES," that is, if the operator is speaking, the process proceeds to step S25 shown in Figure 11.

[0130] On the other hand, if the answer in step S7 is "NO," that is, if the operator did not speak, then in step S9, as shown in Figure 10, it is determined whether the user smiled. Here, the CPU 50 determines whether the user's facial expression recognized in step S3 is a smile.

[0131] If the answer in step S9 is "YES," that is, if it is determined that the user has smiled, then in step S11, the avatar image 210 is aligned with the user's gaze detected in step S5 and displayed with an exaggerated smiling expression, and the process proceeds to step S49 shown in Figure 12. In step S11, the CPU 50 generates an expression for the avatar that is smiling to a greater degree than the user's smile. However, the degree to which it is increased is determined by a pre-set exaggeration parameter.

[0132] On the other hand, if the answer in step S9 is "NO," that is, if it is determined that the user is not smiling, then in step S13, it is determined whether the user has an unhappy expression. Here, the CPU 50 determines whether the user's facial expression recognized in step S3 is an unhappy expression.

[0133] If the answer in step S13 is "YES," that is, if it is determined that the user has a displeased expression, then in step S15, the avatar image 210 is aligned with the user's gaze detected in step S5 and displayed with a softened displeased expression, and the process proceeds to step S49. In step S15, the CPU 50 generates a displeased expression for the avatar at a lower degree than the user's displeased expression. However, the degree to which it is softened is determined by a pre-set parameter for the degree of softening.

[0134] On the other hand, if the answer in step S13 is "NO," that is, if it is determined that the user does not have an unhappy expression, then in step S17, the user's facial expression recognized in step S3 is reflected in the avatar. In other words, the avatar image 210 is displayed with the user's facial expression.

[0135] In the following step S19, it is determined whether to avert the avatar's gaze. Here, the CPU 50 determines whether the user has been facing forward for a predetermined time (for example, a few seconds to about 10 seconds) or longer. Although not shown in the diagram, if the user's gaze is directed forward, that time is counted, and if it is not, it is not counted. Also, if the user looks away from the front and then turns to face forward again, the time spent facing forward from the beginning is counted.

[0136] If the answer in step S19 is "YES," that is, if it is decided to avert the gaze, then in step S21 the avatar's gaze is averted from the front and the process proceeds to step S49. In step S21, the CPU 50 directs the avatar's gaze away from the front for a second predetermined time. The second predetermined time is set to a predetermined percentage of the first predetermined time (for example, about 10%). The predetermined percentage is determined by a predetermined percentage parameter that has been set in advance. The direction in which the avatar averts its gaze, i.e., a direction other than the front, is randomly selected or randomly drawn from a plurality of predetermined directions.

[0137] On the other hand, if the answer in step S19 is "NO," that is, if the user does not decide to look away, then in step S23 the avatar image 210 is aligned with the user's gaze, and the process proceeds to step S49.

[0138] As described above, if the operator is speaking, step S7 is "YES", and in step S25 shown in Figure 11, the image captured by camera 68 is acquired, and in step S27, the operator's facial expression is recognized. However, as described above, the process of capturing the operator is executed in parallel with the control process, and in step S25, the CPU 50 acquires the current captured image data 304i. Also, in step S27, the CPU 50 recognizes the facial expression and its degree from the operator's face image contained in the captured image data 304i, and stores (updates) the corresponding operator facial expression data 304j.

[0139] Next, in step S29, it is determined whether it is time to nod. Here, the CPU 50 determines whether it is the moment when the operator's voice is interrupted or when the operator has spoken something that seeks the user's consent.

[0140] Although not shown in the diagram, when the CPU 50 executes the process in step S29, it also performs a process to recognize the operator's voice.

[0141] If the answer in step S29 is "YES," that is, if it is time to nod, then in step S31, the avatar is made to perform a nodding motion. In step S33, the avatar corresponding to the user is aligned with the user's gaze, the user's facial expression is reflected in the avatar, and the process proceeds to step S49. In other words, the avatar image 210 is aligned with the user's gaze and displayed with the user's facial expression.

[0142] On the other hand, if the answer in step S29 is "NO," that is, if it is not the right time to nod, then in step S35, the CPU determines whether the operator smiled. Here, the CPU 50 determines whether the facial expression of the operator recognized in step S27 is a smile.

[0143] If the answer in step S35 is "YES," that is, if the system determines that the operator has smiled, then in step S37, the system causes the avatar to display a smiling expression with a predetermined degree of smile, and then proceeds to step S45. In step S37, the CPU 50 generates the avatar's facial expression to smile with a predetermined degree of smile. However, the degree of smile is determined by the smile degree parameter included in the setting parameter data 304f.

[0144] On the other hand, if the answer in step S35 is "NO," that is, if it is determined that the operator is not smiling, then in step S39, it is determined whether the operator has a sad face. Here, the CPU 50 determines whether the facial expression of the operator recognized in step S27 is sad.

[0145] If the answer in step S39 is "NO," that is, if it is determined that the operator does not have a sad expression, then in step S41, the user's facial expression is reflected in the avatar corresponding to the user, and the process proceeds to step S45.

[0146] On the other hand, if the answer in step S39 is "YES," that is, if it is determined that the operator has made a sad face, then in step S43, the avatar is made to express a sad expression according to a predetermined degree of sadness, and the process proceeds to step S45. In step S43, the CPU 50 generates an expression for the avatar to show sadness according to a predetermined degree of sadness. However, the degree of sadness is determined by the sadness degree parameter included in the setting parameter data 304f.

[0147] In step S45, the gaze of the avatar corresponding to the user is aligned with the user's gaze. In the following step S47, the user's head movements are reflected in the avatar corresponding to the user, and the process proceeds to step S49 shown in Figure 12.

[0148] As shown in Figure 12, in step S49, it is determined whether or not the user is speaking. Here, the CPU 50 determines whether or not the received data 304c contains the user's voice data.

[0149] If the answer in step S49 is "YES," that is, if the user is speaking, then in step S51, the user's voice is output, and the avatar corresponding to the user performs a speaking action, before proceeding to step S55. In step S51, the CPU 50 outputs the received user's voice data from the speaker 64, and moves the lips and head (neck) of the avatar image 210 in accordance with this voice data. In other words, it is made to appear as if the avatar corresponding to the user is actually speaking.

[0150] On the other hand, if the answer in step S49 is "NO," that is, if the user has not spoken, then in step S53, the user's head movements are reflected in the avatar corresponding to the user, and the process proceeds to step S55.

[0151] In step S55, it is determined whether the process has ended. Here, the CPU 50 determines whether the operator has instructed the system to terminate the control process or whether the user has ended the interaction.

[0152] If the answer in step S55 is "NO," that is, if the process is not finished, the process returns to step S1 as shown in Figure 9. Consequently, the avatar image 210 is updated. On the other hand, if the answer in step S55 is "YES," that is, if the process is finished, the control process is terminated.

[0153] As shown in Figure 13, when the CPU 50 starts the transmission and reception process, in step S71 it starts communicating with the user terminal 12. In the following step S73 it determines whether or not the operator's voice has been detected. If the answer in step S73 is "NO", that is, if the operator's voice has not been detected, the process proceeds to step S77.

[0154] On the other hand, if the answer in step S73 is "YES," that is, if the operator's voice has been detected, then in step S75, the operator's voice is sent to the user terminal 12, and the process proceeds to step S77.

[0155] In step S77, it is determined whether or not data has been received from the user terminal 12. If the answer in step S77 is "NO," that is, if no data has been received from the user terminal 12, the process proceeds to step S81.

[0156] On the other hand, if the answer in step S77 is "YES," that is, if data is being received from the user terminal 12, then in step S79 the received data is stored and the process proceeds to step S81.

[0157] In step S81, it is determined whether the process has ended. Here, the CPU 50 determines whether the operator has instructed the system to terminate the send / receive process, or whether the user has ended the interaction.

[0158] If the answer in step S81 is "NO," that is, if the process is not finished, the process returns to step S73. On the other hand, if the answer in step S81 is "YES," that is, if the process is finished, the sending and receiving process with the user terminal 12 is terminated.

[0159] As shown in Figure 14, when the CPU 20 of the user terminal 12 starts the transmission and reception process, in step S101 it starts communication with the operator terminal 16. In the next step S103 it determines whether or not the user's voice has been detected.

[0160] If the answer in step S103 is "NO," that is, if the user's voice is not detected, the process proceeds to step S107. On the other hand, if the answer in step S103 is "YES," that is, if the user's voice is detected, the detected voice is sent to the operator's terminal 16 in step S105, and the process proceeds to step S107.

[0161] In step S107, the captured image is acquired. In the next step S109, the captured image is sent to the operator terminal 16. Subsequently, in step S111, it is determined whether or not the operator's voice has been received.

[0162] If the answer in step S111 is "NO," that is, if the operator's voice has not been received, the system proceeds to step S115. On the other hand, if the answer in step S111 is "YES," that is, if the operator's voice has been received, the system outputs the operator's voice in step S113 and proceeds to step S115.

[0163] In step S115, it is determined whether the process has ended. Here, the CPU 20 determines whether the user has instructed the system to terminate the send / receive process, or whether the operator has ended the interaction.

[0164] If the answer in step S115 is "NO," that is, if the process is not finished, the process returns to step S103. On the other hand, if the answer in step S115 is "YES," that is, if the process is finished, the transmission and reception process with the operator terminal 16 is terminated.

[0165] According to the first embodiment, by displaying an avatar corresponding to the user on the operator's terminal display device, the user's privacy is protected. Furthermore, when the operator is speaking, the avatar responds to the user with the same facial expression as the operator. In other words, the avatar empathizes with the operator, making it easier for the operator to converse with the user. This makes it easier to respond to users.

[0166] Furthermore, according to the first embodiment, when the operator is not speaking, the avatar is made to express the user's facial expression in a way that is favorable to the operator, thus creating a situation where the operator can easily speak. Therefore, it becomes easier to respond to the user.

[0167] Furthermore, according to the first embodiment, when the operator is not speaking, the avatar's gaze is appropriately averted, causing the avatar to perform actions that do not create a sense of intimidation, thus creating a situation where the operator can speak more easily. Therefore, it becomes easier to respond to the user.

[0168] In the first embodiment, when the operator was not speaking, the system performed processes such as changing the avatar's facial expression to match the user's and making it look more favorably towards the operator, or having it avert its gaze as appropriate. When the operator was speaking, the system performed processes such as having the avatar display the same facial expression as the operator and having it nod as appropriate. However, it is not necessary for all of these processes to be executed. Even if only one or more of these processes are executed, the operator can still easily interact with the user. The operator sets whether or not to execute each process, and processes that are not executed are skipped in the control processes shown in Figures 9-12.

[0169] For example, if the operator is not speaking, and the process of changing the avatar's facial expression to one that is favorable to the operator and matches the user's facial expression is not performed, then if the answer to step S7 is "NO", steps S9 and S13 shown in Figure 10 are skipped, and the process proceeds to step S17. Therefore, steps S11 and S15 are not executed. In this case, it is not necessary to send captured image data from the user terminal 12.

[0170] Furthermore, if the operator is not speaking and does not perform the appropriate action of averting their gaze, then when the process in step S17 is performed, the process in step S19 is skipped and the process proceeds to step S23.

[0171] Furthermore, if the operator is speaking and the process of making the avatar express the same facial expression as the operator is not performed, then if the answer in step S29 is "NO", the processes in steps S35 and S39 are skipped, and the process proceeds to step S41. Therefore, steps S37 and S43 are not executed.

[0172] Furthermore, if the operator is speaking and the process to prompt a nodding motion is not performed as appropriate, then when the process in step S27 is executed, the process in step S29 is skipped and the process proceeds to step S35. Therefore, the processes in steps S31 and S33 are not executed.

[0173] I will omit the explanation, but if two or more processes are not performed, the corresponding processes will be skipped as described above.

[0174] In the first embodiment, the facial expressions of the user and operator were recognized from facial images, but the system is not limited to this. The facial expressions of the user and operator can also be recognized from their voices, respectively. Publicly known techniques can be used to estimate human facial expressions from voice. For example, techniques disclosed in Japanese Patent Application Publication No. 2021-12285 and "Daiki Mori: Understanding Emotions and Attitudes from Voice, IEICE Journal Vol. 101, No. 9, 2018" can be used.

[0175] Furthermore, in the first embodiment, when the operator was speaking, the operator's facial expression was recognized, and the avatar corresponding to the user was made to display a smiling expression based on a pre-set degree of smile, or a sad expression based on a pre-set degree of sadness. However, the embodiment is not limited to this. Since it is also possible to recognize the degree of the operator's facial expression from the operator's face image, the avatar may be made to display a smiling or sad expression based on the recognized degree of the operator's facial expression.

[0176] Furthermore, in the first embodiment, the avatar corresponding to the user was made to nod when the operator's voice was interrupted or when the operator spoke something seeking the user's consent, but it is not limited to this. If there are too many or too few nods, the operator may find it difficult to speak, so it is also possible to set the frequency of nodding. The frequency of nodding is stored as a setting parameter. If the frequency of nodding is set to be low, the nodding will not be performed even when the operator's voice is interrupted or when the operator speaks something seeking the user's consent, in proportion to the reduction. On the other hand, if the frequency of nodding is set to be low, the nodding will be performed not only when the operator's voice is interrupted or when the operator speaks something seeking the user's consent, but also when a fourth predetermined time (for example, from 0.5 seconds to several seconds) has elapsed since the previous nodding, in proportion to the increase.

[0177] Furthermore, in the first embodiment, the avatar that responds to the user is made to change the user's facial expression to a degree that is favorable towards the operator, thereby creating a situation that makes it easier for the operator to interact with the user. However, the system is not limited to this. If the user has a stern or grumpy-looking face, and an angry or grumpy expression is recognized, the system may reduce the degree of anger or grumpiness or convert it to a happy expression to create a situation that makes it easier for the operator to interact with the user. The operator may be able to set or disable the reduction of the degree of anger or grumpiness or the conversion to a happy expression. However, the degree of anger or grumpiness to be reduced is set in the setting parameters. In this way, the avatar that responds to the user is made to change the user's facial expression to a degree that is favorable towards the operator, or the avatar that responds to the user is made to change the user's facial expression to a favorable expression towards the operator.

[0178] <Second Example> In the second embodiment, the user terminal 12 recognizes the user's facial expression, detects the user's gaze and head movements, and transmits the recognized user's facial expression and the detected user's gaze and head movements to the operator terminal 16. Except for these differences, the second embodiment is the same as the first embodiment. Therefore, we will explain the differences and omit explanations of redundant information.

[0179] Therefore, in the second embodiment, the facial expression recognition program 302i is also stored in the user terminal 12. In addition, the gaze and head movement detection program 302j is stored in the user terminal 12 and deleted from the operator terminal 16.

[0180] Specifically, as shown in Figure 13, a portion of the control processing of the CPU 50 of the operator-side terminal 16 is modified, and as shown in Figure 14, a portion of the transmission and reception processing of the CPU 20 of the user-side terminal 12 is modified.

[0181] In the second embodiment, the process of recognizing the user's facial expression in step S5 and the process of detecting the user's gaze in step S7 are removed from the control processing of the CPU 50.

[0182] Furthermore, in the second embodiment, the process of transmitting the captured image in step S109 to the operator terminal 16 is removed from the CPU 20's transmission and reception processing, and the processes of recognizing the user's facial expression in step S121, detecting the user's gaze and head movements in step S123, and transmitting the user's facial expression, gaze and head movements in step S125 to the operator terminal 16 are added in that order between steps S107 and S111.

[0183] In the second embodiment, as in the first embodiment, it is possible to facilitate dialogue with the user, who is the other party to the conversation, and to respond to the user more easily.

[0184] In the embodiments described above, the user terminal and the operator terminal communicate via a network, but the user terminal and the operator terminal may also communicate via a network and a server. In this case, when the server receives image data transmitted from the user terminal to the operator terminal, it may recognize the user's facial expression based on the received image data, detect the user's gaze and head movements, and transmit data about the user's facial expression, gaze, and head movements to the operator terminal in place of the received image data.

[0185] Furthermore, while the above embodiments describe a scenario where a user of a shopping site interacts with an operator who interacts with that user, the invention is not limited to this. When two or more participants are making a video call or participating in a web conference, images of avatars corresponding to the person being spoken to or other participants in the conference may be displayed on the display device of the terminal used by each participant, and each avatar may be controlled individually in the manner described in the above embodiments.

[0186] Furthermore, the order in which each step in the flowchart shown in the above-described embodiment is processed can be changed if the same result can be obtained.

[0187] Furthermore, the various screens and specific numerical values ​​mentioned in each of the above-mentioned embodiments are merely examples and can be modified as needed. [Explanation of symbols]

[0188] 10. Information Processing Systems 12 ... User-side terminal 14 ... Network 16 ... Operator-side terminal 18 ... Server 18a, 20, 50 ...CPU 18b, 22, 52...Storage section 24, 54 ... Communication I / F 26, 56 ... Input / Output Interface 28, 58 ... Input device 30, 60...display device 32, 62... Mike 34, 64... speakers 36, 66 ... Sensor I / F 38, 68... Camera

Claims

1. A voice detection means for detecting operator voice, which is the voice spoken by an operator interacting with a user. A means for capturing an image of the operator's face, When the voice detection means detects the operator's voice, the first recognition means recognizes the operator's facial expression based on the operator's facial image captured by the shooting means. An avatar display means that, when the operator's facial expression recognized by the first recognition means is a predetermined first facial expression, displays on a display device an image of an avatar corresponding to the user, who is the conversation partner, that expresses the same facial expression as the predetermined first facial expression, and An information processing device comprising a transmission means for transmitting the operator's voice detected by the voice detection means to a user-side terminal used by the user.

2. The information processing apparatus according to claim 1, wherein the predetermined first facial expression is a smiling expression and a sad expression.

3. When the operator's voice is detected by the voice detection means, a timing determination means for determining whether it is the timing for a nod based on the operator's voice, and The information processing apparatus according to claim 1 or 2, further comprising an avatar control means for causing the avatar to perform a nodding motion when the timing determination means determines that it is the timing for nodding.

4. Receiving means for receiving the user's facial image transmitted from the user's terminal, and If the voice detection means does not detect the operator's voice, the system further includes a second recognition means that recognizes the user's facial expression based on the user's facial image received by the receiving means. The information processing apparatus according to any one of claims 1 to 3, wherein the avatar display means, when the user's facial expression recognized by the second recognition means is a predetermined second facial expression, causes the avatar to display the same facial expression as the predetermined second facial expression, but in a way that is favorable to the operator.

5. The aforementioned second predetermined facial expression is a smiling expression and a displeased expression. The second recognition means further recognizes the degree of the user's smiling and unhappy expressions, The information processing apparatus according to claim 4, wherein the avatar display means expresses the degree of the smile in the avatar when the user's facial expression is a smiling expression, and expresses the degree of displeasure in the avatar when the user's facial expression is a displeased expression.

6. The system further comprises a receiving means for receiving the user's facial expressions transmitted from the user's terminal, The information processing apparatus according to any one of claims 1 to 3, wherein the avatar display means, when the voice detection means has not detected the operator's voice, and the user's facial expression received by the receiving means is a predetermined second facial expression, causes the avatar to display the same facial expression as the predetermined second facial expression, but in a favorable manner towards the operator.

7. The aforementioned second predetermined facial expression is a smiling expression and a displeased expression. The receiving means further receives the degree of the user's smiling and unhappy expressions, The information processing apparatus according to claim 6, wherein the avatar display means expresses the degree of the smile in the avatar when the user's facial expression is a smiling expression, and expresses the degree of displeasure in the avatar when the user's facial expression is a displeased expression.

8. The system further comprises gaze detection means for detecting the user's gaze based on the user's facial image received by the receiving means, The information processing apparatus according to claim 4 or 5, wherein the avatar display means sets the gaze of the avatar to match the gaze of the user detected by the gaze detection means, and when the voice detection means does not detect the operator's voice, if the time during which the user's gaze detected by the gaze detection means is directed forward elapses, the avatar's gaze is averted for a second predetermined time regardless of the user's gaze.

9. The receiving means further receives the user's gaze, The information processing apparatus according to claim 6 or 7, wherein the avatar display means sets the gaze of the avatar to match the gaze of the user that has been received, and if the voice detection means has not detected the operator's voice, and the time during which the user's gaze received by the receiving means has been facing forward has elapsed to a first predetermined time, the avatar's gaze is averted for a second predetermined time regardless of the user's gaze.

10. A control program executed on an information processing device, The processor of the aforementioned information processing device, A voice detection step to detect operator voice, which is the voice spoken by the operator interacting with the user. A shooting step in which an image of the operator's face is captured, If the operator's voice is detected in the voice detection step, a recognition step is performed to recognize the operator's facial expression based on the operator's facial image captured in the shooting step. If the facial expression of the operator recognized in the recognition step is a predetermined facial expression, an avatar display step is performed, in which an image of an avatar corresponding to the user, who is the other party in the conversation, expressing the same facial expression as the predetermined facial expression is displayed on the display device, and A control program that causes the program to execute a transmission step, which transmits the operator's voice detected in the voice detection step to a user-side terminal used by the user.

11. A method for controlling an information processing device, (a) A step of detecting operator voice, which is the voice spoken by the operator interacting with the user, (b) A step of taking a facial image of the operator, (c) If the operator's voice is detected in step (a), the step of recognizing the operator's facial expression based on the operator's facial image captured in step (b), (d) If the facial expression of the operator recognized in step (c) is a predetermined facial expression, the step of displaying on the display device an image of an avatar corresponding to the user who is the other party to the conversation, which expresses the same facial expression as the predetermined facial expression, and (e) A control method comprising the step of transmitting the operator voice detected in step (a) to a user-side terminal used by the user.

Images