Information processing system, control program, and control method

Abstract

To provide an information processing system, a control program, and a control method that allow a speech of an avatar to be easily accepted by a user.SOLUTION: An information processing system 10 displays, on a display device 18, an image of an avatar for an operator responding to a user, and causes the avatar to speak to the user for executing a recreation. An animation display program stored in a program storage area of a memory in an operator terminal displays an animation prior to the execution of the recreation, a position control specifying program specifies a display position and a posture of an animation character in a final frame of the animation, and an avatar display program displays the avatar in the display position and the posture specified by the position control specifying program in a first frame at which the recreation begins.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to an information processing system, a control program, and a control method, and particularly relates to, for example, an information processing system, a control program, and a control method in which an avatar is displayed on a display for a child (such as an infant) and the avatar speaks to the child.

Background Art

[0002] Patent Document 1 discloses a technique of displaying a favorite anime character video of a child on a liquid crystal display unit in order to direct the child's attention to the liquid crystal display unit when photographing the child with a video camera.

[0003] Patent Document 2 discloses a technique of displaying a character, which is a virtual assistant imitating a person in a specific occupation such as a caregiver, a nurse, a doctor, or an animal or a robot, on a display, and the character automatically conversing with a user using a chatbot function.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] In technologies such as Patent Documents 1 and 2, when an avatar is displayed on a display for response, improvement is needed. For example, when making a user such as a child (an infant or a toddler) hear the speech of the avatar, a device for making the user accept the speech of the avatar is required.

[0006] Therefore, the main object of the present disclosure is to provide a novel information processing system, control program, and control method.

[0007] Another object of the present disclosure is to provide an information processing system, control program, and control method in which the utterance of an avatar is easily accepted by a user.

Means for Solving the Problems

[0008] In order to solve the above problems, the present disclosure adopts the following configuration. The reference numerals and supplementary explanations in parentheses show the correspondence with the embodiments described to assist in understanding the present disclosure, and do not limit the present disclosure in any way.

[0009] A first embodiment is an information processing system that displays an avatar of an operator who responds to a user on a display device and causes the avatar to speak to the user, the information processing system including an animation display unit that displays an animation on the display device before the operator starts responding to the user, a specifying unit that specifies the display position of the animation character in the final frame of the animation, and an avatar display unit that displays the avatar at a display position that is the same as or similar to the display position specified by the specifying unit in the first frame when the operator starts responding to the user.

[0010] In the first embodiment, the information processing system (10: a reference numeral that exemplifies the corresponding part in the embodiment but is not intended to be limiting. The same applies hereinafter) displays an avatar of an operator who responds to a user on a display device and causes the avatar to speak to the user. The animation display unit (40, 70e) displays an animation in which an animation character (62) appears on the display device (18) before the operator starts responding to the user. The specifying unit (40, 70f) specifies the display position of the animation character in the final frame of the animation. The avatar display unit (70g) displays the avatar (64) at a display position that is the same as or similar to the display position specified by the specifying unit in the first frame when the operator starts responding to the user.

[0011] According to the first embodiment, an animation is displayed as an introduction part before the operator corresponds to the user, and the display position of the animation character in the final frame of the animation is taken over by the avatar that appears when the operator corresponds to the user. Therefore, the familiarity of the user who is accustomed to the animation with respect to the avatar is enhanced, the utterance of the avatar is easily accepted by the user, and for example, it can be expected that the conversation (dialogue) between the user and the avatar proceeds smoothly.

[0012] The second embodiment is an information processing system subordinate to the first embodiment, wherein the specifying unit further specifies the posture of the animation character in the final frame of the animation, and the avatar display unit displays the avatar in the same or similar posture as the posture specified by the specifying unit.

[0013] According to the second embodiment, since the avatar is made to appear in the operator's correspondence with the user in the same or similar posture as the posture of the animation character, the familiarity of the user with respect to the avatar is further enhanced, and it can be expected that the conversation (dialogue) between the user and the avatar proceeds more smoothly.

[0014] The third embodiment is an information processing system subordinate to the first or second embodiment, wherein the avatar display unit displays the avatar in the same or similar appearance as the appearance of the animation character.

[0015] According to the third embodiment, since the avatar having the same or similar appearance as the appearance of the animation character is made to appear in the operator's correspondence with the user, the familiarity of the user with respect to the avatar is further enhanced, and it can be expected that the conversation (dialogue) between the user and the avatar proceeds more smoothly.

[0016] The fourth embodiment is an information processing system subordinate to the first embodiment, wherein the operator is a program using a human or artificial intelligence.

[0017] The fifth embodiment is a computer control program that displays an operator's avatar for dealing with a user on a display device of a user terminal and causes the avatar to speak to the user. The processor of the computer is caused to execute an animation display step of displaying an animation on the display device before the operator starts dealing with the user, a specifying step of specifying the display position of the animation character in the final frame of the animation, and an avatar display step of displaying the avatar at a display position identical or similar to the display position specified in the specifying step in the first frame when the operator starts dealing with the user.

[0018] The sixth embodiment is a computer control method that displays an operator's avatar for dealing with a user on a display device of a user terminal and causes the avatar to speak to the user. The processor of the computer displays an animation on the display device before the operator starts dealing with the user, specifies the display position of the animation character in the final frame of the animation, and displays the avatar at a display position identical or similar to the display position specified in the specifying step in the first frame when the operator starts dealing with the user.

[0019] Also according to the fifth and sixth embodiments, the same effects as those of the first embodiment can be expected.

Advantages of the Invention

[0020] According to the present disclosure, since the user can easily become familiar with the avatar, the speech of the avatar is more likely to be accepted by the user.

[0021] The above objects, other objects, features, and advantages of the present disclosure will become more apparent from the following detailed description of the embodiments with reference to the drawings.

Brief Description of the Drawings

[0022]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

DETAILED DESCRIPTION OF THE INVENTION

[0023] Referring to FIG. 1, an information processing system 10 of an embodiment of the present disclosure includes a user terminal 12, and the user terminal 12 is communicably connected to an operator terminal 16 via a network 14.

[0024] As an example, by the technology of Web Real-Time Communication (WebRTC), the user terminal 12 and the operator terminal 16 can perform P2P (Peer to Peer) communication and transmit and receive video and / or audio in real time via a web browser.

[0025] Although illustration is omitted, WebRTC can be used in combination with multiple servers. Specifically, multiple servers for using WebRTC include a signaling server, a STUN (Simple Traversal of UDP through NATs) server, a TURN (Traversal Using Relay NAT) server, and the like. Although detailed description is omitted, the signaling server is a server for acquiring information about a communication partner by WebRTC. Also, the STUN server and the TURN server are servers for performing so-called NAT (Network Address Translation) traversal when the communication partner exists in a different network.

[0026] In the embodiment of FIG. 1, one user terminal 12 and one operator terminal 16 are shown. Actually, a plurality of user terminals 12 and a plurality of operator terminals 16 are connected to the network 14, and one user terminal 12 and one operator terminal 16 that are set to perform a video call or a Web conference among them are communicably connected.

[0027] Note that if the user terminal 12 and the operator terminal 16 are predetermined and the user terminal 12 and the operator terminal 16 have each other's connection information, the user terminal 12 and the operator terminal 16 can be communicably connected without using the above-described servers.

[0028] The user terminal 12 is used, for example, by a child (user) who receives the utterance of an avatar during recreation in a kindergarten or a nursery school, and the operator terminal 16 is used, for example, by an operator (not shown) such as a teacher who speaks as an avatar. However, basically, the child does not operate the user terminal 12, but views the screen of the display device 18 (FIG. 2) of the user terminal 12 or listens to the sound output from the speaker 28 (FIG. 2) of the user terminal 12. Also, the voice of the child is detected by the microphone 26 (FIG. 2) of the user terminal 12, and the child and / or the background are photographed by the camera 24 (FIG. 2) of the user terminal 12. In this embodiment, an image of an avatar corresponding to the operator's avatar is displayed on the display device 18 of the user terminal 12, and the user 20 (FIG. 2) and the operator can interact through the avatar.

[0029] However, the recreation that can be provided by the information processing system 10 of the embodiment is mainly targeted at nursery school children and kindergarten children as users, but the users are not necessarily limited to such persons.

[0030] Referring to FIG. 2, for example, a table 22 is provided as an example on an interactive location PLC set in a nursery school or the like for interaction with the user 20, and a display device 18 is installed on the table 22. The user 20 faces the display device 18 in front of the table 22.

[0031] For example, a camera 24 is installed on the upper side of the display device 18 on the table 22 of the interactive location PLC of the embodiment. The camera 24 captures the user 20 and the background thereof at the interactive location PLC and sends the video signal to the operator terminal 16.

[0032] A microphone 26 and a speaker 28 are installed in front of the display device 18, that is, in front of the user 20. The microphone 26 mainly captures the speech of the user 20 and sends the voice signal to the operator terminal 16. The speaker 28 outputs the speech from the operator of the operator terminal 16. However, the voice of the operator or the voice converted from the voice of the operator to the voice set for the avatar is output.

[0033] The user terminal 12 is an information processing device, and a general-purpose notebook PC, desktop PC, tablet terminal, or smartphone can be used.

[0034] The network 14 is composed of an IP network (or IP network) including the Internet and an access network (or access network) for accessing this IP network. As the access network, a public telephone network, a mobile phone network, a wired LAN, a wireless LAN, a CATV (Cable Television), etc. can be used.

[0035] The operator terminal 16 is another information processing device different from the user terminal 12. As an example, it is a general-purpose notebook PC or desktop PC, but other general-purpose terminals such as smartphones or tablet PCs can also be used.

[0036] FIG. 3 is a block diagram showing an example of the electrical configuration of the user terminal 12 shown in FIG. 1. As shown in FIG. 3, the user terminal 12 includes a CPU 30, and the CPU 30 is connected to a memory 32, a communication interface (hereinafter referred to as "communication I / F") 34, and an input / output interface (hereinafter referred to as "input / output I / F") 36 via an internal bus.

[0037] The CPU 30 controls the overall operation of the user terminal 12. However, instead of the CPU 30, a SoC (System-on-a-Chip) including a plurality of functions such as a CPU function and a GPU (Graphics Processing Unit) function may be provided.

[0038] The memory 32 is a main storage device, and the RAM among them is used as a work area or buffer area of the CPU 30. Although not shown, the user terminal 12 is provided with an HDD and a ROM as auxiliary storage devices. However, instead of or in addition to the HDD, a non-volatile memory such as an SSD may be used.

[0039] The communication I / F 34 functioning as a communication unit is a wired interface for transmitting and receiving control signals and data between the CPU 30 and an external computer such as the operator terminal 16 via the network 14 under the control of the CPU 30. However, as the communication I / F 34, a wireless interface such as a wireless LAN or Bluetooth (registered trademark) can also be used.

[0040] An input device 38, a display device 18, a camera 24, a microphone 26, and a speaker 28 are connected to the input / output I / F 36. The input device 38 is a keyboard and a computer mouse. Further, a touch panel may be provided on the display device 18.

[0041] The display device 18 is, for example, a liquid crystal display device. The microphone 26 is a general-purpose sound collection microphone. The speaker 28 is a general-purpose speaker. The camera 24 is a general-purpose CCD camera.

[0042] Also, the input / output I / F 36 outputs the operation data (or operation information) input from the input device 38 to the CPU 30, and outputs the image data generated by the CPU 30 to the display device 18 to display a screen or image corresponding to the image data on the display device 18. However, there may be a case where the image data received from an external computer (for example, the operator terminal 16) is output by the CPU 30. For example, animations and avatar images described later are transmitted from the operator terminal 16 and displayed on the display device 18.

[0043] Also, the input / output I / F 36 converts the user's voice detected by the microphone 26 into digital voice data (user voice data) and outputs it to the CPU 30, or converts the voice data output by the CPU 30 into an analog voice signal and outputs it from the speaker 28, or outputs the video data (user video data) of the video including the user photographed (detected) by the camera 24 to the CPU 30.

[0044] However, in this embodiment, the voice data output from the CPU 30 is the operator's voice data received from the operator terminal 16 or the operator's voice data obtained by converting the operator's voice into the voice of the avatar.

[0045] Also, the video including the user displayed on the operator terminal 16 during the interaction with the operator is, for example, a moving image.

[0046] Note that the electrical configuration of the user terminal 12 shown in FIG. 3 is an example and need not be limited.

[0047] When the user terminal 12 is a smartphone, it is provided with a call circuit for making calls via a mobile phone communication network or a mobile phone network and a public telephone network. However, in this embodiment, since such calls are not made, the illustration is omitted.

[0048] FIG. 4 is a block diagram showing the electrical configuration of the operator terminal 16 shown in FIG. 1. As shown in FIG. 4, the operator terminal 16 includes a CPU 40, and the CPU 40 is connected to a memory 42, a communication I / F 44, and an input / output I / F 46 via an internal bus.

[0049] The CPU 40 controls the overall operation of the operator terminal 16. However, instead of the CPU 40, an SoC including a plurality of functions such as a CPU function and a GPU function may be provided.

[0050] The memory 42 is a main storage device, and the RAM therein is used as a work area or a buffer area for the CPU 40. Although not shown, the operator terminal 16 is provided with an HDD and a ROM as auxiliary storage devices. However, instead of or in addition to the HDD, a non-volatile memory such as an SSD may be used.

[0051] The communication I / F 44 functioning as a communication unit is a wired interface for transmitting and receiving control signals and data to and from an external computer such as the user terminal 16 via the network 14 under the control of the CPU 40. However, as the communication I / F 44, a wireless interface such as a wireless LAN or Bluetooth (registered trademark) can also be used.

[0052] An input device 48, a display device 50, a microphone 52, a speaker 54, and a camera 56 are connected to the input / output I / F 46. The input device 48 is a keyboard and a computer mouse. Further, a touch panel may be provided on the display device 50. The touch panel is a general-purpose touch panel, and any type such as a capacitance type, an electromagnetic induction type, a resistive film type, or an infrared type can be used.

[0053] The display device 50 is, for example, a liquid crystal display device. The microphone 52 is a general-purpose sound pickup microphone. The speaker 54 is a general-purpose speaker. The camera 56 is a general-purpose CCD camera.

[0054] However, when a smartphone or a tablet PC is used as the operator terminal 16, the input device 48 outputs the operation data (or operation information) input from the input device 48 to the CPU 40, and also outputs the image data generated by the CPU 40 to the display device 50 to cause the display device 50 to display a screen corresponding to the image data. However, there may be a case where the image data received from an external computer (for example, the user terminal 12) is output by the CPU 40. The image data received from the user terminal 12 is user video data.

[0055] Also, the input / output I / F 46 converts the voice of the operator detected by the microphone 52 into digital voice data and outputs it to the CPU 40, or converts the voice data output by the CPU 40 into an analog voice signal and outputs it from the speaker 54, or outputs the video data of the video of the operator captured (detected) by the camera 56 (hereinafter referred to as "operator video data") to the CPU 40.

[0056] However, in this embodiment, the voice data output from the CPU 40 is the user voice data received from the user terminal 12. Also, the video of the operator is a moving image.

[0057] Note that the electrical configuration of the operator terminal 16 shown in FIG. 4 is an example and is not necessarily limited.

[0058] Also, when the operator terminal 16 is a smartphone, it includes a call circuit for making a call via a mobile phone communication network, or a mobile phone network and a public telephone network. However, in this embodiment, since such a call is not made, the illustration is omitted.

[0059] Here, referring to FIGS. 5-6, the operation of the information processing system 10 of this embodiment will be outlined.

[0060] Assuming that this information processing system 10 is used, for example, in recreation at a nursery school, in order to have the children as users accept the utterances of the avatar, it is important to make the avatar likable.

[0061] In order to realize such a requirement, in the information processing system 10 of the embodiment, before the recreation executed through the display device 18 of the user terminal 12, on the screen 60 of the display device 18, an animation, preferably a short animation video, is displayed as an introduction part to the execution of the recreation, and then the recreation using a CG (Computer Graphics) avatar is started.

[0062] In the animation, the avatar appearing in the recreation is used as an animation character, and the animation character 62 shown in FIG. 5(A) appears in the final frame of the animation, and the display position (coordinates) and posture of the animation character 62 at that time are detected (specified).

[0063] Although not shown in the drawings, characters different from the avatar such as animation characters and / or game characters popular among infants and children may appear in the animation to attract the attention of the user.

[0064] The detected or specified display position and posture of the animation character 62 are reflected in the avatar display (avatar display program), and the animation video is switched to the video by the avatar display to execute the recreation.

[0065] Therefore, in the first frame when starting the recreation, the avatar 64 shown in FIG. 5(B) appears with the same appearance, the same display position, and the same posture as the animation character 62 in the final frame of the animation.

[0066] Note that the display position of the avatar 64 is preferably the same as the display position of the animation character 62, but the display positions of both do not necessarily have to be exactly the same.

[0067] For example, if the animation character 62 is displayed at the display position 63 shown by the dotted line in Fig. 5(A) and the avatar 64 is displayed at the display position 65 shown by the dotted line in Fig. 5(B), the avatar 64 may be displayed at a position where a part of the display position 63 and the display position 65 overlap or at an adjacent position. That is, even in a case where the avatar 64 is displayed such that at least a part of the display position 63 and the display position 65 overlap or are adjacent, the display position of the avatar 64 is considered to be similar to the display position of the animation character 62. The display position of the avatar 64 being similar to the display position of the animation character 62 means that when the screen 60 displaying the animation character 62 and the screen 60 displaying the avatar 64 are overlaid, the animation character 62 and the avatar 64 are positioned such that they partially overlap, are adjacent, or are in the vicinity.

[0068] In subsequent recreations, as shown in Figs. 6(A) - 6(C), the posture or movement of the avatar 64 is controlled based on the motion capture of the operation of an operator (not shown) who operates the operator terminal 16. However, in the examples shown in Figs. 6(A) - 6(C), the image of the avatar 64 is gradually enlarged. In this embodiment, since the image of the avatar is generated on the operator terminal 16 side, on the operator terminal 16, in the virtual space, the position of the avatar and the position of the virtual camera are brought closer, or the virtual camera is zoomed in.

[0069] In this way, since the appearance (look), display position, and pose of the animation character 62 that appears in the last frame of the animation video are inherited by the avatar 64 that appears in the first frame of the recreation, the person being recreated, that is, the user, falls into the illusion of conversing (dialoguing) with the avatar 64 that appeared in the animation, and the user experiences an interaction as if they had entered the animation. Therefore, the fear of the avatar 64 is significantly alleviated, and the utterances of the avatar are more likely to be accepted by the user. As a result, it can be expected that the dialogue (conversation) between the avatar and the user will proceed smoothly.

[0070] Note that, for example, the appearance (look) of the avatar 64 in the first frame of the recreation does not have to be exactly the same as the appearance (look) of the animation character 62 in the final frame of the animation. As long as both have the same or similar appearance (look). For example, even if in some parts of the appearance such as the face or clothes, the animation character 62 and the avatar 64 look the same, their appearances are considered similar.

[0071] The same applies to the pose. For example, the pose of the avatar 64 in the first frame of the recreation does not have to be exactly the same as that of the animation character 62 in the final frame of the animation. For example, in Fig. 5(A), the pose of the animation character 62 and the avatar 64 in Fig. 5(B) are both standing in the same direction. In this case, it can be said that their poses are the same. For example, in Fig. 5(A) and Fig. 5(B), if both are in the same standing pose but facing different directions, and there is no significant difference in their silhouettes, it can be said that their poses are similar. That is, regarding the pose, when there is a commonality in the overall silhouettes of both, it can be said that the pose of the avatar 64 is similar to that of the animation character 62.

[0072] In addition, the reason for displaying an animation as an introduction to the execution of the recreation is that since the animation character moves within the animation, even if an avatar identical or similar to the animation character moves during the execution of the recreation, it is less likely to make the user feel uncomfortable. Conversely, if a still image of a character is displayed as an introduction to the execution of the recreation, even if the avatar has the same or similar appearance (appearance) as the character, the user may feel uncomfortable when the avatar moves during the execution of the recreation.

[0073] The provision of such animation videos and the recreation using the avatar 64 are, as an example, the responsibility of the operator terminal 16 and the operator of the operator terminal 16.

[0074] That is, in the user terminal 12, the image of the avatar 64 is displayed on the display device 18 according to the avatar image data from the operator terminal 16. Also, in the recreation, although the details of the content are omitted, the user listens to the operator's speech or the user speaks to the avatar. Therefore, in the operator terminal 16, the operator's voice is detected, and the operator voice data corresponding to the detected operator's voice or the voice obtained by converting the voice to the voice set for the avatar is transmitted to the user terminal 12 and output from the speaker 28 of the user terminal 12. In this way, the user terminal 12 controls the movement and speech of the avatar 64 according to the avatar image data and the operator voice data from the operator terminal 16.

[0075] On one hand, an image including the user is captured by the camera 24, and the video data corresponding to the video captured by the camera 24, that is, the user video data, is transmitted from the user terminal 12 to the operator terminal 16. Also, the user's voice is detected by the microphone 26, and the voice data corresponding to the voice detected by the microphone 26, that is, the user voice data, is transmitted from the user terminal 12 to the operator terminal 16. The user video data is output to the display device 50 of the operator terminal 16, and the user voice data is output to the speaker 54 of the operator terminal 16. Therefore, the operator can control the avatar 64, that is, respond to the user, while seeing and hearing the user's situation.

[0076] FIG. 7 is an illustrative diagram showing an example of the memory map of the RAM of the memory 42 of the operator terminal 16 shown in FIG. 4. As shown in FIG. 7, the RAM includes a program storage area 66 and a data storage area 68. The control program of the operator terminal 16 is stored in the program storage area 66.

[0077] The control program executed by the operator terminal 16 includes a main processing program 70a, an operation detection program 70b, a communication program 70c, a user image display program 70d, an animation display program 70e, a position and orientation identification program 70f, an avatar display program 70g, a voice detection program 70h, a voice output program 70i, and the like.

[0078] The main processing program 70a is a program for executing the main routine of the control process of the operator terminal 16 in this embodiment.

[0079] The operation detection program 70b is a program for detecting operation data input from the input device 48 according to the operator's operation and storing it in the operation data storage area 72a of the data storage area 68.

[0080] The communication program 70c is a program for communicating with external devices, such as the user terminal 12 and the like.

[0081] The user image display program 70d is a program that outputs, to the display device 50, image data for displaying the video of the user 22 being photographed by the camera 24 (Fig. 2) sent from the user terminal 12.

[0082] The animation display program 70e is a program for displaying, on the display device 18 of the user terminal 12, an animation video as shown in Fig. 5(A) according to the animation data pre-stored in the animation data storage area 72c of the data storage area 68. That is, the animation display program 70e transmits a video signal obtained by playing back the animation data to the user terminal 12. This animation display program 70e functions as an animation display unit.

[0083] The position and pose identification program 70f is a program for identifying the display position and the human pose of the animation character 62 in the last frame of the animation shown in Fig. 5(A). For example, it uses a bone estimation technique such as Vision Pose (trade name) to identify the pose. However, the display position is identified as the position in the three-dimensional coordinates at the feet of the animation character 62. This position and pose identification program 70f functions as an identification unit.

[0084] The data of the display seat position and the human pose identified by this position and pose identification program 70f is stored in the identification result data storage area 72d.

[0085] The avatar display program 70g functions as an avatar display, and is a program for displaying, on the display device 18 of the user terminal 12 shown in Fig. 5(B) and Fig. 6 etc., the image of the avatar 64 generated using the image generation data 72b of the data storage area 68. That is, the avatar display program 70g functions as an avatar display, and transmits data (avatar image data) corresponding to the generated image of the avatar 64 to the user terminal 12 as a video signal. However, the avatar 64 operates according to the operation of the operator.

[0086] Specifically, the motion of the operator captured by the camera 56 (Fig. 4) is acquired according to the motion capture technique, and according to the motion of the operator, the positions (coordinate data) of each part of the face of the avatar 64, namely the eyebrows, eyes (upper eyelids, lower eyelids), nose and mouth (upper lip, lower lip), and the positions (coordinate data) of the head, hands, arms, shoulders, waist, knees and ankles, and the orientations of the face (neck), wrists and ankles are controlled. However, as shown in Fig. 6(C), when the lower part from the waist of the avatar 64 is not displayed, it is not necessary to control the positions and orientations of the knees and ankles.

[0087] Such an avatar display program 70f can control the avatar motion using an image processing library such as MediaPipe Holistic, for example. In other examples, the method disclosed in JP-A-2021-56940 can also be used.

[0088] However, the lips of the avatar 64 are controlled according to the voice of the operator during recreation when the voice of the operator is output at the user terminal 12 using a technique called Lip Sync.

[0089] The voice detection program 70h is a program for detecting the voice of the operator input from the microphone 52 and sending the operator voice data corresponding to the detected voice to the user terminal 12. However, the voice of the operator may be converted into the voice set for the avatar 64. Note that the voice conversion may be performed on the user terminal 12 side.

[0090] The voice output program 70i is a program for outputting the user voice data received from the user terminal 12 to the speaker 54. Therefore, the voice corresponding to the user voice data comes out from the speaker 54.

[0091] Although illustration is omitted, in the program storage area 70, in addition to the operating system and middleware of the operator terminal 16, a browser and other application programs other than the control program of the present disclosure are also stored. However, since the other application programs are programs different from the control program of this embodiment, that is, the information processing program, they are not included in the information processing program.

[0092] In the data storage area 68, an operation data storage area 72a, an image generation data storage area 72b, an animation data storage area 72c, a specific result storage area 72d, avatar image data 72e, etc. are stored.

[0093] The operation data storage area 72a temporarily stores the data of the operator's operations detected according to the operation detection program 70b. When the operation data is used for the processing of the CPU 40 or transmitted to the user terminal 12 by the CPU 40, it is deleted from the operation data storage area 72a.

[0094] The image generation data storage area 72b stores the image generation data used to generate a display screen such as a user image displayed on the display device 50 of the operator terminal 16.

[0095] The animation data storage area 72c stores the animation data used in the above-mentioned animation display program 70e to display an animation on the display device 18 of the user terminal 12.

[0096] The avatar image data storage area 72d stores the avatar image data generated according to the above-mentioned avatar display program 70g.

[0097] Next, with reference to FIG. 8, the information processing in the information processing system 10 of the embodiment will be described.

[0098] In the first step S1, the CPU 40 of the operator terminal 16 establishes a connection state with the user terminal 12.

[0099] In the subsequent step S3, the CPU 40 displays an animation as an introduction to the execution of the recreation by the animation display program 70e. The display of the animation is executed until it is detected in step S5 that the final frame of the animation has been reached.

[0100] When it is determined in step S5 that it is the final frame, the CPU 40, in the subsequent step S7, detects the image area of the animation character 62 in the final frame and detects the display position coordinates of the animation character 62 according to the position and orientation identification program 70f. These display position coordinates are stored in the specific result data storage area 72d and used as the display position coordinates of the avatar 64 in step S11.

[0101] In the next step S9, the CPU 40 also identifies the posture of the person (animation character 62) from the image area of the animation character 62 according to the position and orientation identification program 70f. The identified person posture is stored in the specific result data storage area 72d and used as the avatar posture in step S11.

[0102] In this way, in step S11, the avatar display coordinates data and the avatar posture data are reflected in the avatar display, that is, the avatar display program 70g.

[0103] However, the avatar display coordinates data may be set to coordinates data that makes the display position of the avatar 64 the same as or similar to the display position of the animation character 62 as described above.

[0104] Similarly, the avatar posture data may be set to posture data that makes the posture of the avatar 64 the same as or similar to the posture of the animation character 62 as described above.

[0105] Thereafter, in step S13, the CPU 40 switches the display of the display device 18 of the user terminal 12 from the screen of the final frame of the animation to the avatar video by the avatar display program 70g.

[0106] In the embodiment, since both the animation display and the avatar display are the responsibility of the operator terminal 16, this video switching only needs to switch the video signal for the display device 18 transmitted from the operator terminal 16 to the user terminal 12 from the reproduced animation data to the avatar video image data.

[0107] In the next step S15, the posture control of the avatar on the avatar display is switched to the control by the motion capture technology described above.

[0108] Thereafter, in step S17, the recreation process is executed. As described above, in the first frame when the recreation is started, an image of the avatar 64 is displayed on the display device 18 of the user terminal 12 with the same appearance as the animation character 62 in the final frame of the animation, at the same display position and in the same posture. In the subsequent recreation, until the end process in step S19, the avatar image data generated based on the operator's operation and the operator voice data based on the operator's voice are transmitted to the user terminal 12, and the user video data and user voice data transmitted from the user terminal 12 are received. Therefore, an image of the avatar 64 that operates based on the operator's operation is displayed on the display device 18 of the user terminal 12, and the voice based on the operator's voice is output from the speaker 28 of the user terminal 12. Also, the user's video is displayed on the display device 50 of the operator terminal 16, and the user's voice is output from the speaker 54 of the operator terminal 16. However, since the content of the recreation is not specifically related to the present disclosure, the description thereof is omitted.

[0109] Thus, according to the embodiment, an animation is displayed as an introduction before the execution of the recreation, and the display position and posture of the animation character in the final frame thereof are taken over by the avatar appearing in the recreation. Therefore, even if the target person of the recreation is, for example, an infant or a child, a user who is familiar with the animation will find it easier to get close to the avatar, and the user can easily accept the speech of the avatar.

[0110] In particular, in the embodiment, an animation with the avatar appearing in the recreation as the protagonist is displayed. Since the protagonist of the animation and the avatar appearing in the recreation are the same, the user's familiarity with the avatar is further enhanced, and it can be expected that the user can more easily accept the speech of the avatar.

[0111] Furthermore, in the above-described embodiment, the case where the avatar of one operator speaks to one user (target person) has been described. However, the embodiment of the present disclosure is also applicable to the case where one or a plurality of operators speak through a common avatar, and is also applicable to the case where the avatar speaks to a number of users. Even in such a case, if the animation character and the avatar have the same appearance, it can be expected that a number of users will also accept the speech of the avatar without resistance.

[0112] Note that, in the above-described embodiment, the animation character 62 appearing in the introduction animation and the avatar 64 appearing in the recreation have the same appearance. However, as shown in FIG. 9, the appearance of the animation character 62 and the appearance of the avatar 64 may be different.

[0113] Furthermore, in the above-described embodiments, animation data is sent from the operator terminal 16 to the user terminal 12 in order to display the animation video on the display device 18 of the user terminal 12. However, the user terminal 12 may be configured to display such an animation on its own display device 18. In this case, a program and animation data equivalent to the animation display program 70e and the animation data stored in the animation data storage area 72c set in the memory of the operator terminal 16 may be set in the memory 22 of the user terminal 12.

[0114] Also, the user terminal 12 may generate an avatar image in the recreation and display it on its own display device 18. In that case, the operator terminal 16 performs motion capture of the operator's movement, and only the data of the operator's movement detected by the motion capture is sent from the operator terminal 16 to the user terminal 12. Then, the CPU 30 of the user terminal 12 controls the operation of the avatar displayed on the display device 18 based on the movement data and generates the avatar image.

[0115] However, in all of the above-described embodiments, the avatar display is set as a program in the memory of each of the operator terminal 16 or the user terminal 12, but it may be incorporated into the operator terminal 16 or the user terminal 12 as a dedicated circuit.

[0116] Also, in the recreation, the avatar may be automatically controlled using artificial intelligence. That is, in this specification, the avatar includes both those that are operated by remote control by a human and those that are operated by automatic control using artificial intelligence. In this case, in the information processing system 10, the operator of the operator terminal 16 is a program using artificial intelligence for controlling the avatar (hereinafter referred to as the "automatic recreation program"). Also, since the avatar is automatically controlled, on the operator terminal 16, the user's video is not displayed on the display device 50, and the user's voice is not output from the speaker 54.

[0117] Further, in the operator terminal 16, an execution unit for the avatar 64 to automatically execute recreation is connected to circuit components such as the CPU 40 via an internal bus. The execution unit automatically controls the avatar 64 according to an automatic recreation program using artificial intelligence (AI) such as a chatbot. Therefore, the microphone 52 and the camera 56 may not be provided. In automatic control, the avatar 64 speaks according to a talk script or operates according to pre-created automatic motion data. As an example, the talk script and the automatic motion data of the avatar 64 are generated by machine learning the operator's speech and motion when performing recreation. However, the text data of the speech content of the avatar 64 according to the talk script is converted into voice data of the avatar and transmitted to the user terminal 12. Note that the text data of the speech content of the avatar 64 may be transmitted to the user terminal 12, and the user terminal 12 may convert the text data into voice data.

[0118] In addition, when using artificial intelligence, an execution unit may be provided in the user terminal 12, and in the user terminal 12, the animation data may be reproduced and the avatar 64 may be controlled according to an automatic recreation program. That is, the user terminal 12 may perform the operations shown in FIG. 8. However, step S1 is deleted.

[0119] Furthermore, in the above-described embodiment, the posture of the animation character 62 is specified in step S9, and the same or similar posture is set as the avatar posture in step S11. However, the setting of the posture in step S11 can be omitted. That is, the posture of the avatar 64 appearing in the first frame of the recreation may be different from the posture of the animation character 62 in the last frame of the animation.

[0120] Also, in the above-described embodiments, the information processing system 10 was used, for example, for recreation for kindergarten children. However, the information processing system 10 of the present disclosure is applicable to all embodiments (uses) in which an avatar speaks to a user, in addition to such recreational uses.

Explanation of Signs

[0121] 10 … Information processing system 12 … User terminal 16 … Operator terminal 30, 40 … CPU 18, 50 … Display device 24, 56 … Camera 62 … Animation character 64 … Avatar 70e … Animation display program 70f … Position and orientation identification program 70g … Avatar display program

Claims

1. An information processing system that displays an avatar of an operator who responds to a user on a display device and causes the avatar to speak to the user, an animation display unit that displays an animation on the display device before the operator starts responding to the user, a specifying unit that specifies the display position of an animation character in the final frame of the animation, and an avatar display unit that displays the avatar at a display position that is the same as or similar to the display position specified by the specifying unit in the first frame when the operator starts responding to the user. The information processing system.

2. The specifying unit further specifies the posture of the animation character in the final frame of the animation, The avatar display unit displays the avatar in a posture that is the same as or similar to the posture specified by the specifying unit. The information processing system according to claim 1.

3. The avatar display unit displays the avatar with an appearance that is the same as or similar to the appearance of the animation character. The information processing system according to claim 1 or 2.

4. The operator is a program using a human or artificial intelligence. The information processing system according to claim 1 or 2.

5. A control program for a computer that displays an avatar of an operator who responds to a user on a display device of a user terminal and causes the avatar to speak to the user, in the processor of the computer, an animation display step of displaying an animation on the display device before the operator starts responding to the user, a specifying step of specifying the display position of an animation character in the final frame of the animation, and an avatar display step of causing the avatar to be displayed at a display position that is the same as or similar to the display position specified in the specifying step in the first frame when the operator starts responding to the user. The control program.

6. A control method for a computer that displays an avatar of an operator who responds to a user on a display device of a user terminal and causes the avatar to speak to the user, the processor of the computer, displays an animation on the display device before the operator starts responding to the user, specifies the display position of the animation character in the final frame of the animation, and then A control method for displaying the avatar at a display position that is the same as or similar to the display position identified in the specific step in the first frame in which the operator starts to respond to the user.

Images