Information processing device, information processing system, control program, and control method

Abstract

To provide an information processing device, an information processing system, a control program, and a control method that are closely associated with the avatar of a dialog partner or an operator of the avatar.SOLUTION: A user terminal in an information processing system displays a dialog screen 100 when talking to an operator of an avatar image 102. The dialog screen includes the avatar image 102, a window part image 104, and a frame image 106. The frame image is generated by superimposing the window part image in front of a rear video in which the rear of a display device 30 is imaged by a rear camera. The avatar image is displayed in front of the window part image. In a regular size, the avatar image fits into a range of the window part image. The avatar image is enlarged in accordance with instructions of the operator and is displayed overlapping a front face of the frame image of the same color or pattern as a rear landscape of the display device. Thus, the avatar pops out from a scene displayed in the window part image and the user has an illusion that the avatar exists in the same space as the user.SELECTED DRAWING: Figure 6

Description

Technical Field

[0005] , ,

[0001] This invention relates to an information processing apparatus, an information processing system, a control program, and a control method, and more particularly, for example, to an information processing apparatus, an information processing system, a control program, and an information processing method for displaying an avatar image or an image of an interaction partner and interacting with a user.

Background Art

[0002] An example of this type of information processing apparatus is disclosed in Patent Document 1. In the communication system disclosed in this Patent Document 1, in a state where the tele-immersion mode is set, the operator terminal converts the operator's expression and gesture into coordinate data and transmits it to the response terminal together with the response voice data. The response terminal generates avatar character response information in which the operator's expression and gesture are reflected in the avatar's expression and gesture based on the coordinate data sent from the operator terminal, and displays it to the user.

[0003] Other examples of this type of information processing apparatus are disclosed in Non-Patent Document 1 and Non-Patent Document 2. In the Web conference system or online conference system disclosed in Non-Patent Document 1 and Non-Patent Document 2, multiple people can display the video of other participants in the conference on the display device of their own terminal and talk to each other.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

Non-Patent Document 2

[0006] In the above-mentioned Patent Document 1, the display screen of the response terminal is two-dimensional, and even if the avatar image is generated using three-dimensional CG image data, the avatar is displayed in a flat plane. Therefore, even if an avatar image based on character response information is displayed on the display unit, the user cannot feel as if they are in the same space as the avatar, and it is difficult to interact closely with the avatar.

[0007] Furthermore, while Non-Patent Documents 1 and 2 display video of meeting participants, similar to Patent Document 1, the screens of the terminals used by each participant are two-dimensional, and the video of other participants is displayed in a flat plane. Therefore, even when interacting one-on-one, participants cannot feel as if they are in the same space as the other participants, making it difficult to interact intimately with them.

[0008] Therefore, the primary objective of this invention is to provide a novel information processing device, information processing system, control program, and control method.

[0009] Another object of this invention is to provide an information processing device, an information processing system, a control program, and a control method that enable intimate interaction with an avatar or the operator of an avatar who is the conversation partner. [Means for solving the problem]

[0010] The first invention is an information processing device that is communicatively connected to the terminal of a conversation partner, comprising: a display device; a camera that photographs the back of the display device; a frame image display means that displays a frame image on the display device using the image of the back of the display device captured by the camera; a conversation partner display means that displays an image of an avatar or an image of the conversation partner on the display device in front of the frame image; and an image of an avatar or an image of the conversation partner. , part of the avatar's image or part of the conversation partner's image This is an information processing device equipped with a means for enlarging an image to a size that overlaps with the frame image.

[0011] The second invention is subordinate to the first invention, and includes a receiving means for receiving instructions or notifications from the conversation partner's terminal, and an enlargement means which, in response to the receiving means receiving an enlargement instruction, displays the avatar image or the conversation partner's image. Expand It's great.

[0012] The third invention is subordinate to the second invention and further comprises a restoration means that restores the avatar image or the image of the conversation partner to its original size in response to receiving an instruction to return by the receiving means.

[0013] The fourth invention is subordinate to the second invention and comprises an enlargement request means for requesting the terminal of the conversation partner to enlarge the image of the avatar or the conversation partner, the enlargement means, in response to the request by the enlargement request means, enlarges the image of the avatar or the conversation partner if it receives an acceptance notification by the receiving means, and does not enlarge the image of the avatar or the conversation partner if it receives a rejection notification by the receiving means.

[0014] The fifth invention is subordinate to the third invention and comprises a restoration request means that requests the conversation partner's terminal to restore the avatar image or the conversation partner's image to its original size, and the restoration means, in response to the request by the restoration request means, restores the avatar image or the conversation partner's image to its original size when it receives an acceptance notification by the receiving means, and does not restore the avatar image or the conversation partner's image to its original size when it receives a rejection notification by the receiving means.

[0015] The sixth invention is an information processing system comprising an information processing device that is communicatively connected to the terminal of a conversation partner, the information processing device comprising a display device, a camera that photographs the back of the display device, a frame image display means that displays a frame image on the display device using the image of the back of the display device captured by the camera, a conversation partner display means that displays an image of an avatar or an image of the conversation partner on the display device in front of the frame image, a receiving means that receives an enlargement instruction from the terminal of the conversation partner, and, in response to receiving an enlargement instruction by the receiving means, the image of the avatar or the conversation partner , part of the avatar's image or part of the conversation partner's image This information processing system includes an enlargement means that enlarges the frame image to a size that overlaps it, and the conversation partner's terminal includes an operation detection means that detects the conversation partner's actions, and a transmission means that transmits an enlargement instruction to the information processing device when the operation detection means detects an enlargement instruction to enlarge the avatar image.

[0016] The seventh invention is a control program executed in an information processing device that includes a display device and a camera that captures the area behind the display device, and is communicatively connected to the terminal of a conversation partner, wherein the processor of the information processing device is instructed to perform a frame image display step of displaying a frame image on the display device using the image of the area behind the display device captured by the camera, a conversation partner display step of displaying an avatar image or an image of the conversation partner in front of the frame image on the display device, and an image of the avatar image or the image of the conversation partner , part of the avatar's image or part of the conversation partner's image This is a control program that executes a scaling step to enlarge the image to a size that overlaps the frame image.

[0017] The eighth invention is a control method for an information processing device comprising a display device and a camera that captures the area behind the display device, and which is communicatively connected to the terminal of a conversation partner, comprising the steps of: displaying a frame image on the display device using the image of the area behind the display device captured by the camera; displaying an avatar image or an image of the conversation partner on the display device in front of the frame image; and displaying the avatar image or the image of the conversation partner , part of the avatar's image or part of the conversation partner's image This control method includes a step of enlarging the image to a size that overlaps the frame image. [Effects of the Invention]

[0018] According to this invention, it is possible to closely interact with an avatar as the interaction partner or the operator of the avatar.

[0019] The above object, other objects, features, and advantages of this invention will become more apparent from the following detailed description of the embodiments with reference to the drawings.

Brief Description of the Drawings

[0020] [Figure 1] FIG. 1 is a diagram showing an information processing system according to an embodiment of this invention. [Figure 2] FIG. 2 is a block diagram showing an example of the electrical configuration of the user terminal shown in FIG. 1. [Figure 3] FIG. 3 is a block diagram showing an example of the electrical configuration of the operator terminal shown in FIG. 1. [[ID=NO]] [Figure 4] FIG. 4 is a diagram showing an example of an interaction screen displayed on the display device of the user terminal shown in FIG. 1 and an example of the state in which the display device is arranged. [Figure 5] FIG. 5 is a diagram showing an example of the state of the user terminal shown in FIG. 1 when viewed from the side. [Figure 6] FIG. 6 is a diagram showing another example of the interaction screen shown in FIG. 4. [Figure 7] FIG. 7 is a diagram showing an example of an operator-side screen displayed on the display device of the operator terminal shown in FIG. 1. [Figure 8] FIG. 8 is a diagram showing an example of the memory map of the RAM of the user terminal shown in FIG. 2. [Figure 9] FIG. 9 is a diagram showing an example of the memory map of the RAM of the operator terminal shown in FIG. 3. [Figure 10] FIG. 10 is a flowchart showing a part of an example of the control process of the CPU of the user terminal shown in FIG. 2. [Figure 11] FIG. 11 is another part of an example of the control process of the CPU of the user terminal shown in FIG. 2, which is a flowchart following FIG. 10. [Figure 12] FIG. 12 is a flowchart showing a part of an example of the control process of the CPU of the operator terminal shown in FIG. 3. [Figure 13] Figure 13 is another part of an example of the CPU control process of the operator terminal shown in Figure 3, and is a flowchart that follows Figure 12. [Figure 14] Figure 14 shows an example of a dialogue screen displayed on the user terminal display device of the second embodiment, and an example of the display device being positioned. [Figure 15] Figure 15 shows an example of an enlarged acceptance / rejection screen displayed on the operator terminal display device of the second embodiment. [Figure 16] Figure 16 shows another example of the dialogue screen shown in Figure 14. [Figure 17] Figure 17 shows an example of a hidden confirmation / rejection screen displayed on the display device of the operator terminal in the second embodiment. [Figure 18] Figure 18 is a flowchart showing a part of an example of the CPU control process of a user terminal in the second embodiment. [Figure 19] Figure 19 is a flowchart that follows Figure 18, showing another part of an example of the CPU control process of a user terminal in the second embodiment. [Figure 20] Figure 20 is a flowchart showing a part of an example of the CPU control process of the operator terminal in the second embodiment. [Figure 21] Figure 21 is a flowchart that follows Figure 20, showing another part of an example of the CPU control process of the operator terminal in the second embodiment. [Figure 22] Figure 22 is a flowchart showing a part of an example of the CPU control process of a user terminal in the third embodiment. [Figure 23] Figure 23 is a flowchart that follows Figure 22, showing another part of an example of the CPU control process of a user terminal in the third embodiment. [Figure 24] Figure 24 is a flowchart showing a part of an example of the CPU control process of the operator terminal in the third embodiment. [Modes for carrying out the invention]

[0021] <First Example> Referring to Figure 1, the information processing system 10 of this first embodiment includes a user terminal 12, which is communicated to an operator terminal 16 via a network 14.

[0022] As an example, using Web Real-Time Communication (WebRTC) technology, the user terminal 12 and the operator terminal 16 can communicate via P2P (Peer to Peer) and send and receive video and / or audio in real time via a web browser.

[0023] Although not shown in the diagram, WebRTC can be used in combination with multiple servers. Specifically, the multiple servers used for WebRTC include a signaling server, a STUN server, and a TURN server. Although not explained in detail, the signaling server is a server that obtains information about the communication partner via WebRTC. The STUN server and TURN server are servers that perform so-called NAT (Network Address Translation) traversal when the communication partner is on a different network.

[0024] In this first embodiment, one user terminal 12 and one operator terminal 16 are shown, but in reality, multiple user terminals 12 and multiple operator terminals 16 are connected to the network 14, and one user terminal 12 and one operator terminal 16 configured to conduct video calls or web conferences are connected in a communication-enabled manner.

[0025] User terminal 12 is used by the user, and operator terminal 16 is used by the operator. In this first embodiment, an avatar image corresponding to the operator's avatar is displayed on the display device 30 of user terminal 12, and the user and operator interact through the avatar. The operator interacts with the user by manipulating the avatar. In other words, the user is a user of the avatar-based conversation service provided by the operator on operator terminal 16.

[0026] The user terminal 12 is an information processing device and can be a general-purpose notebook PC, desktop PC, tablet device, or smartphone.

[0027] Network 14 consists of an IP network (or IP network) including the Internet, and an access network (or access network) for accessing this IP network. The access network can include public telephone networks, mobile phone networks, wired LANs, wireless LANs, CATV (Cable Television), etc.

[0028] The operator terminal 16 is an information processing device different from the user terminal 12. For example, it may be a general-purpose notebook PC or desktop PC, but other general-purpose terminals such as smartphones or tablet PCs can also be used.

[0029] Figure 2 is a block diagram showing an example of the electrical configuration of the user terminal 12 shown in Figure 1. As shown in Figure 2, the user terminal 12 includes a CPU 20, which is connected to RAM 22, a communication interface (hereinafter referred to as "communication I / F") 24, and an input / output interface (hereinafter referred to as "input / output I / F") 26 via an internal bus.

[0030] The CPU 20 is responsible for the overall control of the user terminal 12. However, instead of the CPU 20, a System-on-a-chip (SoC) that includes multiple functions such as CPU functionality and GPU (Graphics Processing Unit) functionality may be provided.

[0031] RAM22 is the main memory and is used as the work area or buffer area of ​​the CPU20. Although not shown in the diagram, the user terminal 12 is equipped with an HDD and ROM as auxiliary storage devices. However, non-volatile memory such as an SSD may be used instead of the HDD, or in addition to the HDD.

[0032] The communication interface 24 is a wired interface for sending and receiving control signals and data to and from an external computer, such as an operator terminal 16, via the network 14 under the control of the CPU 20. However, a wireless interface such as Wi-Fi or Bluetooth® can also be used as the communication interface 24.

[0033] The input / output interface 26 is connected to an input device 28, a display device 30, a microphone 32, a speaker 34, a front camera 36, ​​and a rear camera 38. The input device 28 is a keyboard and a computer mouse. In addition, a touch panel may be provided. The touch panel is a general-purpose touch panel, and any type of touch panel can be used, such as a capacitive touch panel, an electromagnetic induction touch panel, a resistive touch panel, or an infrared touch panel.

[0034] Display device 30 is, for example, a liquid crystal display device. Microphone 32 is a general-purpose sound-gathering microphone. Speaker 34 is a general-purpose stereo speaker. Front camera 36 and rear camera 38 are general-purpose CCD cameras, respectively.

[0035] Furthermore, the input / output interface 26 outputs operation data (or operation information) received from the input device 28 to the CPU 20, and outputs image data generated by the CPU 20 to the display device 30, causing the display device 30 to display a screen or image corresponding to the image data. However, image data received from an external computer (for example, the operator terminal 16) may also be output by the CPU 20.

[0036] Furthermore, the input / output I / F 26 converts the user's voice detected by the microphone 32 into digital audio data (hereinafter referred to as "user audio data") and outputs it to the CPU 20, converts the audio data output by the CPU 20 into an analog audio signal and outputs it from the speaker 34, outputs video data of the image including the user captured (detected) by the front camera 36 (hereinafter referred to as "user video data") to the CPU 20, and outputs video data of the image behind the display device 30 captured (detected) by the rear camera 38 (hereinafter referred to as "background video data") to the CPU 20.

[0037] However, in this first embodiment, the audio data output from the CPU 20 is either the operator's audio data received from the operator terminal 16 or audio data obtained by converting the operator's audio into the avatar's audio. Hereafter, these audio data will be referred to as operator audio data.

[0038] Furthermore, the image including the user is, for example, a moving image. The background image can be a moving image or a still image. As will be described later, in this first embodiment, the background image is used for the frame image 106 displayed on the display device 30. Therefore, this frame image 106 has the same color or pattern as the background scenery of the display device 30 and blends in with the background scenery. When a stationary display device 30 is used, the background scenery basically does not change, so the background image can be a still image. When a portable (handheld) display device 30 (user terminal 12) is used, the background scenery may change, so it is considered better to use a moving image for the background image.

[0039] Note that the electrical configuration of the user terminal 12 shown in Figure 2 is just one example and is not limited to that configuration.

[0040] Furthermore, if the user terminal 12 is a smartphone, it is equipped with a call circuit for making calls via a mobile phone network, or a mobile phone network and a public telephone network. However, in this first embodiment, such calls are not made, so they are not shown in the illustration.

[0041] Figure 3 is a block diagram showing the electrical configuration of the operator terminal 16 shown in Figure 1. As shown in Figure 3, the operator terminal 16 includes a CPU 50, which is connected to the RAM 52, communication I / F 54, and input / output I / F 56 via an internal bus.

[0042] The CPU 50 is responsible for the overall control of the operator terminal 16. However, instead of the CPU 50, an SoC (System on a Chip) that includes multiple functions such as CPU and GPU functions may be provided.

[0043] RAM 52 is the main memory and is used as the work area or buffer area of ​​the CPU 50. Although not shown in the diagram, the operator terminal 16 is equipped with an HDD and ROM as auxiliary storage devices. However, non-volatile memory such as an SSD may be used instead of the HDD, or in addition to the HDD.

[0044] Communication I / F 54 is a wired interface for sending and receiving control signals and data to and from an external computer, such as an operator terminal 16, via the network 14 under the control of the CPU 50. However, a wireless interface such as Wi-Fi or Bluetooth® can also be used as Communication I / F 54.

[0045] The input / output interface 56 is connected to an input device 58, a display device 60, a microphone 62, a speaker 64, and a camera 66. The input device 58 is a keyboard and a computer mouse. In addition, a touch panel may be provided. The touch panel is a general-purpose touch panel, and any type of touch panel can be used, such as a capacitive touch panel, an electromagnetic induction touch panel, a resistive touch panel, or an infrared touch panel.

[0046] Display device 60 is, for example, a liquid crystal display device. Microphone 62 is a general-purpose sound-gathering microphone. Speaker 64 is a general-purpose stereo speaker. Camera 66 is a general-purpose CCD camera.

[0047] However, if a smartphone or tablet PC is used as the operator terminal 16, the input device 58 is a touch panel and hardware buttons.

[0048] Furthermore, the input / output interface 56 outputs operation data (or operation information) received from the input device 58 to the CPU 50, and also outputs image data generated by the CPU 50 to the display device 60, causing the display device 60 to display a screen corresponding to the image data. However, image data received from an external computer (for example, the user terminal 12) may also be output by the CPU 50. The image data received from the user terminal 12 is user video data.

[0049] Furthermore, the input / output interface 56 converts the operator's voice detected by the microphone 62 into digital audio data and outputs it to the CPU 50, converts the audio data output by the CPU 50 into an analog audio signal and outputs it from the speaker 64, and outputs video data of the operator's video captured (detected) by the camera 66 (hereinafter referred to as "operator video data") to the CPU 50. However, in this first embodiment, the audio data output from the CPU 50 is user audio data received from the user terminal 12. Also, the operator's video is a moving image.

[0050] Note that the electrical configuration of the operator terminal 16 shown in Figure 3 is just one example and is not limited to this configuration.

[0051] Furthermore, if the operator terminal 16 is a smartphone, it is equipped with a call circuit for making calls via a mobile phone network, or a mobile phone network and a public telephone network. However, in this first embodiment, such calls are not made, so they are not shown in the illustration.

[0052] The operation of the information processing system 10 in this first embodiment will now be described. In this information processing system 10, the user of the user terminal 12 and the operator of the operator terminal 16 conduct video calls or web conferences using the user terminal 12 and the operator terminal 16.

[0053] Figure 4 shows an example of the dialogue screen 100 displayed on the display device 30 of the user terminal 12 when conducting a video call or web conference (or online meeting), and an example of the display device 30 in its configuration as viewed from the front. Figure 5 shows an example of the user terminal 12 (display device 30) in its configuration as viewed from the side of the display device 30. However, in Figures 4 and 5, a portion of the wall and desk are shown.

[0054] As shown in Figures 4 and 5, as an example, the display device 30 of the user terminal 12 is placed on a desk in the user's room, with the wall of the user's room behind the display device 30. For example, the wall is covered with vertically striped wallpaper. A front camera 36 with a built-in microphone 32 or microphone 32 attached is positioned at the upper center of the display device 30. The rear camera 38 is positioned at or near the center of the back side of the display unit of the display device 30, slightly above the support unit that supports the display unit. Figure 5 shows a part of the user, but as an example, the user sits facing the display device 30 and interacts with the avatar or the operator of the avatar displayed on the display device 30.

[0055] Note that in Figures 4 and 5, the main unit of the user terminal 12, the input device 28, and the speaker 34 are omitted, but these are also located near or around the display device 30. However, the main unit of the user terminal 12 is a device that houses the CPU 20, RAM 22, communication I / F 24, and input / output I / F 26, etc., in a casing.

[0056] Furthermore, the placement of the front camera 36 and rear camera 38 shown in Figure 5 is merely an example and is not limited to any particular location. The front camera 36 may be placed in any other location as long as it can capture the user's face. Similarly, the rear camera 38 may be placed in any other location as long as it can capture the scene behind the display device 30.

[0057] As shown in Figure 4, the dialogue screen 100 includes an avatar image 102, a window image 104, and a frame image 106. The frame image 106 is displayed by superimposing a window image 104 of a predetermined size and shape onto the front of a video (moving image or still image) captured by the rear camera 38 behind the display device 30, aligning their centers. For example, the video of the area behind the display device 30 (hereinafter referred to as "background video") is displayed across the entire area of ​​the horizontally elongated rectangular display section (or display surface) of the display device 30. Also, for example, the window image 104 is a horizontally elongated rectangle and is two-thirds the size of the display surface of the display device 30. In this embodiment, the window image 104 is shown in white, but in reality, a scene such as a virtual room is displayed as the window image 104.

[0058] The avatar image 102 is displayed in front of the window image 104. In the dialogue screen 100 shown in Figure 4, the avatar image 102 is displayed at a size that fits within the range of the window image 104 (hereinafter referred to as "normal size"). As shown in Figure 4, the avatar image 102 is contained within the range of the window image 104 from its head to its waist. As will be described later, the normal size is set so that when the avatar image 102 is enlarged by a predetermined magnification, the avatar image 102 extends beyond the window image 104. For example, the normal size is set so that the vertical length of the avatar image 102 is 95% of the vertical length of the window image 104. Therefore, the avatar appears to exist in a scene such as a virtual room displayed in the window image 104.

[0059] In this first embodiment, the movements and voice of the avatar displayed on the display device 30 are controlled based on the movements and voice of the operator on the operator terminal 16. The movements include gestures, head orientation, and facial expressions. The operator's movements are detected from the operator's video footage captured by the camera 66 on the operator terminal 16. The operator's voice is detected by the microphone 62.

[0060] On the operator terminal 16, motion data is generated to control the avatar's movements based on the operator's actions, and audio data is generated to control the avatar's speech (speech output) based on the operator's voice. The motion data is information (data) about the position and orientation of each part of the avatar to control its gestures, facial orientation, and facial expressions. Specifically, the motion data includes information about the position (coordinate data) of each part of the face, namely the eyebrows, eyes (upper eyelids, lower eyelids), nose, and mouth (upper lip, lower lip), as well as the orientation of the face, hands, arms, shoulders, waist, knees, and ankles, and the orientation of the wrists and ankles. However, if the avatar's lower body is not displayed, information about the position and orientation of the knees and ankles does not need to be included in the motion data. The audio data is audio data obtained by converting the operator's voice into the avatar's voice. However, the operator's voice data can also be used as is.

[0061] Furthermore, motion data for controlling the avatar's movements based on the operator's actions can be generated, for example, using an image processing library such as MediaPipe Holistic. In other examples, the method disclosed in Japanese Patent Application Publication No. 2021-56940, which is the background technology, can also be used.

[0062] The user terminal 12 changes the avatar image 102 according to the operation data. Therefore, the avatar is operated. The user terminal 12 also outputs the received operator voice data to the speaker 34. In other words, the user terminal 12 controls the avatar image 102 (avatar) displayed on the display device 30 according to instructions from the operator terminal 16. However, the instructions from the operator terminal 16 may consist of operation data only, or operation data and operator voice data.

[0063] Meanwhile, video footage including the user is captured by the front camera 36, ​​and the video data corresponding to the footage captured by the front camera 36, ​​i.e., the user video data, is transmitted from the user terminal 12 to the operator terminal 16. In addition, the user's voice is detected by the microphone 32, and the audio data corresponding to the voice detected by the microphone 32, i.e., the user audio data, is transmitted from the user terminal 12 to the operator terminal 16. The user video data is output to the display device 60 of the operator terminal 16, and the user audio data is output from the speaker 64 of the operator terminal 16. Therefore, the operator can control the avatar while seeing and hearing the user's actions.

[0064] Furthermore, during a conversation between the user and the avatar (or operator), the avatar can be made to "jump out" to the user's side at the user's request (or call) from either the user or the avatar. When the operator accepts the user's call, or when the user accepts the operator's call, the operator instructs the avatar to jump out. In other words, an instruction to jump out the avatar (hereinafter referred to as the "jump-out instruction") is sent from the operator terminal 16 to the user terminal 12.

[0065] When the user terminal 12 receives an instruction to display the avatar, it enlarges and displays the avatar image 102 at a predetermined magnification. Figure 6 shows an example of a dialogue screen 100 including the enlarged avatar image 102. In this first embodiment, the avatar image 102 is enlarged to 1.5 times its normal size. The predetermined magnification is set to 1.5 times because the size of the window image 104 is set to two-thirds of the size of the background image, and the avatar image 102, which is normally sized to fit within the range of the window image 104, is made to extend beyond the window image 104. Therefore, the predetermined magnification is set appropriately according to the size of the avatar image 102 and / or the size of the window image 104.

[0066] As can be seen in Figure 6, the enlarged avatar image 102 extends beyond the window image 104 and is displayed in front of the frame image 106. Therefore, although the display surface of the display device 30 is two-dimensional, the avatar (avatar image 102) appears to leap out of the scene (or the display device 30) displayed in the window image 104, giving the user the illusion that the avatar exists in the user's room, i.e., in the same space as the user. In this first embodiment, the frame image 106 is displayed using video footage taken from behind the display device 30, and the frame image 106 portion of the display device 30 is the same color or pattern as the wall. Therefore, at least the frame image 106 portion of the dialogue screen 100 blends into the scene behind the display device 30, allowing the user to feel more strongly that the avatar exists in the same space as themselves. Consequently, the user's intimacy with the avatar is increased. In other words, the user can interact intimately with the avatar or operator who is their conversation partner.

[0067] Figures 4 and 6 show an example where striped wallpaper is pasted onto a wall, and the frame image 106 is displayed so that its stripes perfectly match those of the wallpaper. However, even if there is a slight misalignment, the wallpaper and frame image 106 have the same pattern, so the user will have the illusion that the avatar exists in the same space as the user. Furthermore, since walls are generally a single color, it is thought that this makes it easier to create the illusion that the avatar exists in the same space as the user.

[0068] Furthermore, when using a stationary display device 30, the distance between the wall and the display device 30 and / or the field of view of the rear camera 38 can be adjusted to ensure a perfect match between the wallpaper and the frame image 106. Also, when the stationary display device 30 is placed against a wall, the distance between the rear camera 38 and the wall is usually short, so a wide-angle camera is used to correct distortion in the captured image. In addition, when shooting in a dark place, it is possible to use a high-sensitivity camera as the rear camera 38 or to use lighting.

[0069] Furthermore, while the avatar is visible, the user can request (or initiate) the avatar to retract during a conversation between the user and the avatar (or operator). When the operator responds to the user's request, or the user responds to the operator's request, the operator instructs the avatar to retract. In other words, an instruction to retract the avatar (hereinafter referred to as the "retraction instruction") is sent from the operator terminal 16 to the user terminal 12.

[0070] When the user terminal 12 receives a command to hide the avatar, it displays the avatar image 102 at its normal size. Therefore, as shown in Figure 4, the avatar image 102 is displayed at its normal size, fitting within the area of ​​the window image 104.

[0071] Figure 7 shows an example of the operator-side screen 200 displayed on the display device 60 of the operator terminal 16. As shown in Figure 7, the operator-side screen 200 includes a display area 202 slightly above the center of the screen. In addition, the operator-side screen 200 has a button image 210 in the upper right part of the screen, and button images 212 and 214 are provided side by side at the bottom edge of the screen.

[0072] Display area 202 is the area for outputting user video data. Button image 210 is an icon for ending the interaction with the user. Button image 212 is an icon for sending a command to the user terminal 12 to make the avatar pop out. Button image 214 is an icon for sending a command to the user terminal 12 to make the avatar retract.

[0073] Note that button image 212 may be disabled if the avatar is visible. Similarly, button image 214 may be disabled if the avatar is not visible. Disabling button images 212 and 214 includes hiding them.

[0074] Figure 8 shows an example of a memory map 300 of the RAM 22 built into the user terminal 12. As shown in Figure 8, the RAM 22 includes a program storage area 302 and a data storage area 304. The program storage area 302 stores the control program that is executed in the user terminal 12 of this first embodiment.

[0075] The control program executed on the user terminal 12 includes a main processing program 302a, an operation detection program 302b, a communication program 302c, an image generation program 302d, an image output program 302e, a shooting program 302f, an avatar control program 302g, a voice detection program 302h, and a voice output program 302i, among others.

[0076] The main processing program 302a is a program for executing the main routine of the control processing of the user terminal 12 in this first embodiment.

[0077] The operation detection program 302b is a program for detecting operation data 304a input from the input device 28 in accordance with user operations and storing it in the data storage area 304.

[0078] The communication program 302c is a program for communicating (sending and receiving data, etc.) with an external device, in this first embodiment, the operator terminal 16.

[0079] The image generation program 302d is a program for generating image data corresponding to all or part of a screen (such as 100) to be displayed on the display device 30, using the image generation data 304b.

[0080] The image output program 302e is a program for outputting image data generated according to the image generation program 302d to the display device 30. Therefore, a screen corresponding to the image data is displayed on the display device 30.

[0081] The shooting program 302f is a program that causes the front camera 36 to perform shooting processing and stores the user video data 304c input from the front camera 36 in the data storage area 304. The shooting program 302f is also a program that causes the rear camera 38 to perform shooting processing and stores the rear video data 304d input from the rear camera 38 in the data storage area 304.

[0082] The avatar control program 302g is a program that controls the avatar using motion data 304f or reaction data 504g.

[0083] The voice detection program 302h is a program that detects the user's voice input from the microphone 32 and stores the user voice data 304e corresponding to the detected voice in the data storage area 304.

[0084] The audio output program 302i is a program for outputting operator voice data 304g received from the operator terminal 16 to the speaker 34. Therefore, the audio corresponding to the operator voice data 304g is output from the speaker 34.

[0085] Although not shown in the diagram, the program storage area 302 stores the operating system and middleware of the user terminal 12, as well as a browser and other application programs other than the control program of this invention.

[0086] Furthermore, the data storage area 304 stores operation data 304a, image generation data 304b, user video data 304c, background video data 304d, user voice data 304e, motion data 304f, and operator voice data 304g.

[0087] Operation data 304a is data of user operations detected according to the operation detection program 302b. Once the operation data 304a is used for processing by the CPU 20, it is erased from the data storage area 304.

[0088] The image generation data 304b is data for generating a screen to be displayed on the display device 30 of the user terminal 12, and includes data for generating an avatar image 102.

[0089] User video data 304c is video data captured by the front camera 36. This user video data 304c basically includes video data that includes the user, but if the user leaves their seat or moves out of the shooting range of the front camera 36, ​​the video may not include the user. When the user video data 304c is transmitted to the operator terminal 16 by the CPU 20, it is erased from the data storage area 304.

[0090] The rear view data 304d is the rear view data of the display device 30 captured by the rear camera 38. When the rear view data 304d is used for processing by the CPU 20, it is erased from the data storage area 304.

[0091] User voice data 304e is the user's voice data detected by microphone 32. When user voice data 304e is transmitted to the operator terminal 16 by CPU 20, it is erased from data storage area 304.

[0092] Operation data 304f is operation data received from the operator terminal 16. Once operation data 304f is used for processing by the CPU 20, it is erased from the data storage area 304. Operator voice data 304g is operator voice data received from the operator terminal 16. Once operator voice data 304g is output to the speaker 34 by the CPU 20, it is erased from the data storage area 304.

[0093] Additionally, a jump-out flag 304h is provided in the data storage area 304. The jump-out flag 304h is a flag used to determine whether or not to make the avatar jump out. If the avatar is to jump out, the jump-out flag 304h is turned on, and if the avatar is not to jump out or if a jumped-out avatar is to be retracted, the jump-out flag 304h is turned off.

[0094] Although not shown in the diagram, the data storage area 304 stores other data necessary for executing control processing, and also contains timers (counters) and other flags necessary for executing control processing.

[0095] Figure 9 shows an example of the memory map 400 of the RAM 52 built into the operator terminal 16. As shown in Figure 9, the RAM 52 includes a program storage area 402 and a data storage area 404. The program storage area 402 stores the control program that is executed in the operator terminal 16 of this first embodiment.

[0096] The control program executed on the operator terminal 16 includes a main processing program 402a, an operation detection program 402b, a communication program 402c, an image generation program 402d, an image output program 402e, a shooting program 402f, an operation data generation program 402g, an audio detection program 402h, and an audio output program 402i, among others.

[0097] The main processing program 402a is a program for executing the main routine of the control processing of the operator terminal 16 in this first embodiment.

[0098] The operation detection program 402b is a program for detecting operation data 404a input from the input device 58 in accordance with the operator's actions and storing it in the data storage area 404.

[0099] The communication program 402c is a program for communicating with an external device, in this first embodiment, the user terminal 12.

[0100] The image generation program 402d is a program for generating image data corresponding to a screen to be displayed on the display device 60, using the image generation data 404b. However, when generating image data corresponding to the operator-side screen 200 as shown in Figure 7, user video data 404f is also used.

[0101] The image output program 402e is a program for outputting image data generated according to the image generation program 402d to the display device 60. Therefore, a screen corresponding to the image data is displayed on the display device 60.

[0102] The shooting program 402f is a program that causes the camera 66 to perform shooting processing and stores the operator video data 404c input from the camera 66 in the data storage area 404.

[0103] The motion data generation program 402g is a program that detects the operator's movements from the operator video data 404c and generates motion data 404e for controlling the avatar's movements.

[0104] The voice detection program 402h is a program that detects the operator's voice input from the microphone 62 and stores the operator's voice data 404d corresponding to the detected voice in the data storage area 404.

[0105] The audio output program 402i is a program for outputting user voice data 404g received from the user terminal 12 to the speaker 64. Therefore, the audio corresponding to the user voice data 404g is output from the speaker 64.

[0106] Although not shown in the diagram, the program storage area 402 stores the operating system and middleware of the operator terminal 16, as well as a browser and other application programs other than the control program of this invention.

[0107] Furthermore, the data storage area 404 stores operation data 404a, image generation data 404b, operator video data 404c, operator voice data 404d, motion data 404e, user video data 404f, and user voice data 404g.

[0108] Operation data 404a is data of the operator's operations detected according to the operation detection program 402b. This operation data 404a also includes the pop-out and retraction instructions sent to the user terminal 12. When the operation data 404a is used for processing by the CPU 50 or sent to the user terminal 12 by the CPU 50, it is erased from the data storage area 404.

[0109] Image generation data 404b is data used to generate a screen that is displayed on the display device 60 of the operator terminal 16.

[0110] Operator video data 404c is video data captured by camera 66. This operator video data 404c basically includes video data that includes the operator, but if the operator leaves their seat or moves out of the camera's field of view, the operator may not be included in the video. When the operator video data 404c is used for processing by the CPU 50, it is erased from the data storage area 404.

[0111] Operator voice data 404d is the operator's voice data detected by microphone 62. When operator voice data 404d is transmitted to the user terminal 12 by CPU 50, it is erased from data storage area 404.

[0112] The motion data 404e is data for controlling the avatar's movements, and in this first embodiment, it is information (data) about the position and orientation of each part for controlling the avatar's gestures, facial orientation, and facial expressions. Specifically, as described above, the motion data 404e is information about the position of each part of the face, namely the eyebrows, eyes (upper eyelids, lower eyelids), nose, and mouth (upper lip, lower lip), the facial orientation, the position of the hands, arms, shoulders, waist, knees, and ankles, and the orientation of the wrists and ankles. When the motion data 404e is used for processing by the CPU 50, it is erased from the data storage area 404.

[0113] User video data 404f is user video data received from user terminal 12. When user video data 404f is used for processing by CPU 50, it is erased from data storage area 404. User audio data 404g is user audio data received from user terminal 12. When user audio data 404g is output to speaker 64 by CPU 50, it is erased from data storage area 404.

[0114] Although not shown in the diagram, the data storage area 404 stores other data necessary for executing control processing, and also contains timers (counters) and flags necessary for executing control processing.

[0115] Figures 10 and 11 are flowcharts showing the control processing of the CPU 20 of the user terminal 12. Although not shown in the diagram, the CPU 20, in parallel with the control processing, performs operations such as detecting operation data and storing it, having the front camera 36 perform a shooting process and storing user video data, having the rear camera 38 perform a shooting process and storing rear video data, detecting sound and storing user voice data, and receiving and storing data transmitted from the operator terminal 16. The control processing of the CPU 20 will be explained below with reference to Figures 10 and 11, but overlapping content will be briefly explained.

[0116] As shown in Figure 10, when the CPU 20 of the user terminal 12 starts control processing, in step S1 it establishes a connection with the operator terminal 16, and in step S3 it displays the dialogue screen 100 shown in Figure 4 on the display device 30. At the beginning of the control processing, since no operation data has been received, an upright avatar image 102 is displayed as an example. At this time, the avatar may also perform an action (and speak) to greet the user.

[0117] In the next step, S5, it is determined whether the control process has ended. Here, the CPU 20 determines whether a command to terminate the control process has been entered by the user on the user terminal 12.

[0118] If the answer in step S5 is "YES," that is, if the control process is finished, the control process is terminated. On the other hand, if the answer in step S5 is "NO," that is, if the control process is not finished, then in step S7, it is determined whether or not data has been received from the operator terminal 16. Here, the CPU 20 determines whether or not operator video data, or operator video data and operator audio data, has been received.

[0119] If the answer in step S7 is "NO," that is, if no data has been received from the operator terminal 16, the process proceeds to step S19 as shown in Figure 11. On the other hand, if the answer in step S7 is "YES," that is, if data has been received from the operator terminal 16, the process proceeds to step S9 to determine whether the jump-out flag 304h is on.

[0120] If the answer in step S9 is "YES," that is, if the sudden appearance flag 304h is on, the process proceeds to step S19 as shown in Figure 11. On the other hand, if the answer in step S9 is "NO," that is, if the sudden appearance flag 304h is off, the process proceeds to step S11. Determine whether operator voice data 304g exists.

[0121] If the answer in step S11 is "YES," that is, if operator voice data 304g is available, then in step S13, the operator voice data 304g is output to the speaker 34. In step S15, in accordance with the output of the operator voice data 304g, image data of the avatar that performs actions according to the action data is generated in normal size and output to the display device 30, and the process proceeds to step S19. Consequently, the avatar image 102 is updated, and the normal-sized avatar displayed on the dialogue screen 100 speaks to the user while making gestures and changing the direction of its face. At this time, the avatar image 102 is lip-synced to the sound output from the speaker 34. The same applies to the case where the avatar is operated in accordance with the voice data. As an example, when the avatar image 102 is updated, the background video data is also updated, and as a result, the frame image 106 is also updated. The same applies to steps S17, S23, and S25.

[0122] On the other hand, if the answer in step S11 is "NO," that is, if there is no operator voice data 304g, then in step S17, image data of an avatar performing actions according to the action data is generated in normal size, output to the display device 30, and the process proceeds to step S19. Consequently, the avatar image 102 is updated, and the normal-sized avatar displayed on the dialogue screen 100 performs gestures and changes the direction of its face.

[0123] As shown in Figure 11, in step S19, it is determined whether or not operator voice data 304g exists. If the answer in step S19 is "YES", then in step S21, the operator voice data 304g is output to the speaker 34, and in step S23, image data of an avatar that performs actions according to the action data in accordance with the output of the operator voice data 304g is generated at an enlarged size and output to the display device 30, and the process proceeds to step S27. Consequently, the avatar image 102 is updated, and the enlarged avatar displayed on the dialogue screen 100 speaks to the user while making gestures and changing the direction of its face.

[0124] On the other hand, if the answer in step S19 is "NO", then in step S25, an enlarged image data of the avatar performing actions according to the action data is generated and output to the display device 30, and the process proceeds to step S27. Consequently, the avatar image 102 is updated, and the enlarged avatar displayed on the dialogue screen 100 performs gestures and changes the direction of its face.

[0125] In step S27, it is determined whether or not there is audio input. If the answer in step S27 is "YES," that is, if there is audio input, then in step S29, the user video data 304c and user audio data 304e are sent to the operator terminal 16, and the process proceeds to step S33. On the other hand, if the answer in step S27 is "NO," that is, if there is no audio input, then in step S31, the user video data 304c is sent to the operator terminal 16, and the process proceeds to step S33.

[0126] In step S33, it is determined whether the operator has given a command to jump out. If the answer in step S33 is "YES," that is, if the operator has given a command to jump out, in step S35, the jump-out flag 304h is turned on and the process returns to step S5. On the other hand, if the answer in step S33 is "NO," that is, if the operator has not given a command to jump out, in step S37, it is determined whether the operator has given a command to retract.

[0127] If the answer in step S37 is "YES," that is, if the operator gives a command to retract, then in step S39, the pop-out flag 304h is turned off and the process returns to step S5. On the other hand, if the answer in step S37 is "NO," that is, if the operator does not give a command to retract, the process returns to step S5.

[0128] Figures 12 and 13 are flowcharts showing the control processing of the CPU 50 of the operator terminal 16. Although not shown in the diagrams, the CPU 50, in parallel with the control processing, performs operations such as detecting operation data and storing the operation data, having the camera 66 perform shooting and storing operator video data, detecting sound and storing operator voice data, and receiving and storing data transmitted from the user terminal 12. The control processing of the CPU 50 of the operator terminal 16 will be explained below, with only a brief explanation given for any overlapping content.

[0129] As shown in Figure 12, when the CPU 50 starts control processing, in step S101 it establishes a connection with the user terminal 12, and in step S103 it displays the operator-side screen 200 shown in Figure 7 on the display device 60. At the beginning of the control processing, since user video data has not been received, the user's video is not displayed in the display area 202.

[0130] In the next step, S105, it is determined whether the control process has ended. Here, the CPU 50 determines whether the operator has instructed it to end the control process. If the answer in step S105 is "YES," that is, if the control process has ended, the control process is terminated. On the other hand, if the answer in step S105 is "NO," that is, if the control process has not ended, then in step S107, operation data 404e is generated from the operator video data 404c, and in step S109, it is determined whether there is audio input.

[0131] If the answer in step S109 is "NO," that is, if there is no voice input, then in step S111, the operation data 404e generated in step S107 is sent to the user terminal 12, and the process proceeds to step S115 shown in Figure 13. On the other hand, if the answer in step S109 is "YES," that is, if there is voice input, then in step S113, the operation data 404e generated in step S107 and the operator voice data 404d are sent to the user terminal 12, and the process proceeds to step S115.

[0132] In step S115 shown in Figure 13, it is determined whether or not data has been received from the user terminal 12. Here, the CPU 50 determines whether or not user video data, or user video data and user audio data, has been received.

[0133] If the answer in step S115 is "NO," that is, if no data has been received from the user terminal 12, the process proceeds to step S123. On the other hand, if the answer in step S115 is "YES," that is, if data has been received from the user terminal 12, the process proceeds to step S117 to determine whether or not user voice data 404g exists.

[0134] If the answer in step S117 is "YES," that is, if user voice data 404g exists, then in step S119, user video data 404f and user voice data 404g are output, and the process proceeds to step S123. Consequently, the user's video is displayed on the display device 60, and the user's voice is output from the speaker 64.

[0135] On the other hand, if the answer in step S117 is "NO," that is, if there is no user voice data 404g, then in step S121, user video data 404f is output, and the process proceeds to step S123. Consequently, the user's video is displayed on the display device 60.

[0136] In step S123, it is determined whether or not there is an input for a pop-out command. Here, the CPU 50 determines whether or not the button image 212 is turned on. If the answer in step S123 is "YES", that is, if there is an input for a pop-out command, then in step S125, the pop-out command is sent to the user terminal 12 and the process returns to step S105.

[0137] On the other hand, if the answer in step S123 is "NO," that is, if there is no instruction to pop out, then in step S127, it is determined whether or not there is an input for a retraction instruction. Here, the CPU 50 determines whether or not the button image 214 is turned on.

[0138] If the answer in step S127 is "YES," that is, if a retraction instruction is received, then in step S129, a retraction instruction is sent to the user terminal 12, and the process returns to step S105. On the other hand, if the answer in step S127 is "NO," that is, if no retraction instruction is received, the process returns to step S105.

[0139] According to the first embodiment, the frame image is a video taken of the area behind the display device, and at least a portion of the frame image blends into the scenery behind the display device, allowing the user to feel more strongly that the avatar exists in the same space as themselves. Consequently, the user's sense of intimacy with the avatar is increased. In other words, the user can interact intimately with the avatar or operator who is their conversation partner.

[0140] In the first embodiment, the operator terminal generates the avatar's motion data, sends the motion data to the user terminal, and the user terminal generates the avatar image. However, the operator terminal may generate the avatar image and send the avatar image to the user terminal.

[0141] Furthermore, in this first embodiment, the avatar's movements were controlled using motion data generated from video data of the operator, but it is not limited to this. Motion data corresponding to a series of actions to be performed by the avatar may be prepared in advance, and the avatar's movements may be controlled by sending motion data corresponding to a series of actions instructed by the operator via button operation or other means to the user terminal. Examples of a series of actions include bowing and returning to the original posture (for example, standing upright), turning the head from side to side several times as if looking around and returning to the original posture, waving the right hand and / or the left hand in front of the torso and then returning to the original posture, and raising the right hand and / or the left hand and then returning to the original posture.

[0142] Furthermore, in this first embodiment, the avatar was made to speak based on the operator's voice, but it is also possible to prepare voice data for the content to be spoken in advance, and send the voice data corresponding to the content to be spoken, instructed by the operator via button operation or other means, to the user terminal to control the avatar's speech.

[0143] However, it is also possible to set up a single button to contain both the action data corresponding to a series of avatar movements and the audio data for what the avatar will say, and then send the action data and audio data to the user's terminal at the same time.

[0144] Furthermore, in this first embodiment, user video data is output to the operator terminal's display device, but the operator terminal may also display an avatar. In such a case, the operator terminal's display device may also be equipped with a rear camera to capture a video of its rear, and motion data may be generated from the user video data captured by the user terminal's front camera and transmitted to the operator terminal, so that the operator's display device displays a frame image, a window image, and an avatar image alongside the dialogue screen. In this case, the dialogue screen displayed on the user terminal's display device is also provided with buttons for pop-out and retraction instructions, and a pop-out or retraction instruction is sent to the operator terminal by the user's operation, and in response, the avatar image on the operator terminal is enlarged or returned to its normal size.

[0145] Furthermore, in this first embodiment, the operator made the avatar pop out or retract by turning on a button image, but it is not limited to button operation. The time spent interacting between the user and the operator can also be counted, and the avatar can pop out or retract at predetermined intervals. However, the time spent with the avatar popped out and the time spent with it retracted can be of different lengths. In addition, the avatar can pop out or retract in response to the operator or user uttering predetermined words. In this case, the predetermined words are pre-registered on the operator's terminal, and the operator's and user's voices are recognized by speech recognition on the operator's terminal to determine whether the operator or user has uttered the predetermined words. For example, if the operator says "I'm coming over there" or the user says "Come over here," the avatar pops out (i.e., it enlarges). Also, for example, if the operator says "I'm going back to the room" or the user says "Get out," the avatar retracts (i.e., it returns to its original size). These are just examples, and the avatar may pop out or retract under other conditions. However, instead of a button press, one of the following conditions may be used: the passage of a predetermined time, the utterance of a predetermined word, or any two or more of the following conditions may be used in combination. For example, the avatar can be made to pop out or retract in response to a button press or the passage of a predetermined time.

[0146] <Second Example> The information processing system 10 of the second embodiment is the same as the first embodiment, except that it controls the appearance and retraction of the avatar by requesting the appearance and retraction of the avatar through user operation and accepting or rejecting the request through the operator's operation. Therefore, we will explain the differences and omit any redundant explanations.

[0147] Figure 14 shows an example of the dialogue screen 100 of the second embodiment and an example of the display device 30 as seen from the front of the display device 30. In the dialogue screen 100 shown in Figure 14, a button image 110 is further provided in the lower left. The button image 110 is provided in front of the frame image 106. However, in the second embodiment, the background of the button image 110 is colorless and transparent. Therefore, the button image 110 is the same color or pattern as the background of the display device 30. As a result, the button image 110 blends into the frame image 106.

[0148] Furthermore, button image 110 is an icon for requesting or asking the operator to make the avatar pop out (i.e., "enlarge the size of avatar image 102"). When button image 110 is turned on, a pop-out request is sent from user terminal 12 to operator terminal 16.

[0149] When the operator terminal 16 receives a pop-up request from the user terminal 12, a screen (enlargement consent / rejection screen) is displayed on the display device 60 as the operator-side screen 200 to instruct the user whether to consent to enlarging the avatar.

[0150] An example of the enlarged acceptance / rejection screen is shown in Figure 15. In the enlarged acceptance / rejection screen, button images 222 and 224 are provided instead of button images 212 and 214. In addition, a message 220 is displayed between the display area 202 and button images 222 and 224 to notify that a pop-up request has been made and whether to accept the pop-up.

[0151] Button image 222 is an icon used to notify the user that they consent to the avatar popping out (hereinafter referred to as the "pop-out consent notification"). When button image 222 is turned on, the pop-out consent notification is sent to the user terminal 12. Button image 224 is an icon used to notify the user that they refuse the avatar popping out (hereinafter referred to as the "pop-out rejection notification"). When button image 224 is turned on, the pop-out rejection notification is sent to the user terminal 12.

[0152] When user terminal 12 receives a notification of acceptance of popping out, it makes the avatar pop out. Therefore, the avatar, which is normally sized, is enlarged. Conversely, when user terminal 12 receives a notification of rejection of popping out, it does not make the avatar pop out. Therefore, the avatar is displayed as is, at its normal size. At this time, a message indicating that the popping out request has been rejected is displayed on the dialogue screen 100. In other words, the user is notified that the popping out has been rejected. However, since the user can understand that the popping out request has been rejected because the size of the avatar does not change, it is not necessary for operator terminal 16 to send a notification of rejection of popping out. In this case, the user is not notified that the popping out has been rejected.

[0153] Figure 16 shows an example of a dialogue screen 100 including an enlarged avatar image 102 in the second embodiment. In the dialogue screen 100 shown in Figure 16, a button image 112 is provided instead of a button image 110. The button image 112 is provided in front of the frame image 106. However, in the second embodiment, the background of the button image 112 is colorless and transparent. Therefore, the button image 112 is the same color or pattern as the background of the display device 30. As a result, the button image 112 blends into the frame image 106.

[0154] Furthermore, button image 112 is an icon for requesting or demanding the operator to retract the avatar (i.e., to return the size of avatar image 102 to its original normal size). When button image 112 is turned on, a retraction request is sent from user terminal 12 to operator terminal 16.

[0155] When the operator terminal 16 receives a hide request from the user terminal 12, a screen (hide-out confirmation screen) is displayed on the display device 60 as the operator-side screen 200 to instruct the user whether to consent to hide the avatar.

[0156] An example of a retraction acceptance / rejection screen is shown in Figure 17. On the retraction acceptance / rejection screen, instead of message 220 in the enlargement acceptance / rejection screen, message 230 is displayed between the display area 202 and button images 222 and 224 to notify that a retraction request has been made and whether to accept or reject the retraction.

[0157] On the avatar retreat acceptance / rejection screen, button image 222 is an icon for notifying the user that they accept the retreat of their avatar (hereinafter referred to as the "retreat acceptance notification"). When button image 222 is turned on, the retreat acceptance notification is sent to the user terminal 12. Button image 224 is an icon for notifying the user that they reject the retreat of their avatar (hereinafter referred to as the "retreat rejection notification"). When button image 224 is turned on, the retreat rejection notification is sent to the user terminal 12.

[0158] When user terminal 12 receives a retraction acceptance notification, it retracts the avatar. As a result, the enlarged avatar returns to its normal size. Also, when user terminal 12 receives a retraction rejection notification, it does not retract the avatar. As a result, the enlarged avatar remains displayed. At this time, a message indicating that the retraction request has been rejected is displayed on the dialogue screen 100. In other words, the user is notified that the retraction has been rejected. However, since the user can understand that the retraction request has been rejected because the size of the avatar does not change, it is not necessary for operator terminal 16 to send a retraction rejection notification. In this case, the user is not notified that the retraction has been rejected.

[0159] Figures 18 and 19 are flowcharts showing a part of the control process of the user terminal 12 in the second embodiment. In the control process of the user terminal 12 in the second embodiment, steps S51-S73 are executed instead of steps S33-S39 of the control process of the user terminal 12 in the first embodiment.

[0160] As shown in Figure 18, after the CPU 20 has completed the processing in step S29 or S31, in step S51 it determines whether there is a pop-up request. Here, the CPU 20 determines whether the button image 110 has been turned on.

[0161] If the answer in step S51 is "YES," that is, if there is a request to jump out, then in step S53, the request to jump out is sent to the operator terminal 16, and the process proceeds to step S59 shown in Figure 19.

[0162] On the other hand, if the answer in step S51 is "NO," that is, if there is no request for the button to pop out, then in step S55, it is determined whether there is a request for the button to retract. Here, the CPU 20 determines whether the button image 120 has been turned on.

[0163] If the answer in step S55 is "YES," that is, if there is a withdrawal request, the withdrawal request is sent to the operator terminal 16 in step S57, and the process proceeds to step S59. On the other hand, if the answer in step S55 is "NO," that is, if there is no withdrawal request, the process proceeds to step S59.

[0164] As shown in Figure 19, in step S59, it is determined whether or not there is a notification of consent to suddenly appear. If the answer in step S59 is "YES", that is, if there is a notification of consent to suddenly appear, then in step S61, the sudden appearance flag 304h is turned on and the process proceeds to step S67.

[0165] On the other hand, if the answer in step S59 is "NO," that is, if there is no notification of consent to the sudden appearance, then in step S63, it is determined whether there is a notification of rejection of the sudden appearance. If the answer in step S63 is "YES," that is, if there is a notification of rejection of the sudden appearance, then in step S65, the user is notified of the rejection and the process proceeds to step S67. On the other hand, if the answer in step S63 is "NO," that is, if there is no notification of rejection of the sudden appearance, the process proceeds to step S67.

[0166] In step S67, it is determined whether or not there is a withdrawal acceptance notification. If the answer in step S67 is "YES", that is, if there is a withdrawal acceptance notification, then in step S69, the jump-out flag 304h is turned off, and the process returns to step S5 as shown in Figure 10.

[0167] On the other hand, if the answer in step S67 is "NO," that is, if there is no withdrawal acceptance notification, then in step S71, it is determined whether there is a withdrawal rejection notification. If the answer in step S71 is "YES," that is, if there is a withdrawal rejection notification, then in step S73, the user is notified of the withdrawal rejection and the process returns to step S5. On the other hand, if the answer in step S71 is "NO," that is, if there is no withdrawal rejection notification, the process returns to step S5.

[0168] Figures 20 and 21 are flowcharts showing a part of the control process of the CPU 50 of the operator terminal 16 in the second embodiment. In the control process of the CPU 50 of the operator terminal 16 in the second embodiment, steps S151-S173 are executed instead of steps S123-S129 of the control process of the operator terminal 16 in the first embodiment.

[0169] As shown in Figure 20, after the CPU 50 has executed the process in step S129 or S131, in step S151 it determines whether there is an outbound request. Here, the CPU 50 determines whether there is an outbound request from the user terminal 12.

[0170] If the answer in step S151 is "NO," that is, if there is no request for a sudden movement, the process proceeds to step S163 as shown in Figure 21. On the other hand, if the answer in step S151 is "YES," that is, if there is a request for a sudden movement, in step S153, a sudden movement acceptance / rejection screen as shown in Figure 15 is displayed on the display device 60, and the process proceeds to step S155.

[0171] In step S155, it is determined whether the user has accepted the pop-up notification. Here, the CPU 50 determines whether the button image 222 has been turned on. If the answer in step S155 is "YES", that is, if the user has accepted the pop-up notification, then in step S157, a pop-up notification acceptance notice is sent to the user terminal 12, and the process returns to step S105 as shown in Figure 12.

[0172] On the other hand, if the answer in step S155 is "NO," that is, if the jump is not accepted, then in step S159, it is determined whether the jump is rejected. Here, the CPU 50 determines whether the button image 224 is turned on. If the answer in step S159 is "NO," that is, if the jump is not rejected, the process returns to step S155.

[0173] On the other hand, if the answer in step S159 is "YES," that is, if the sudden entry is rejected, then in step S161, a sudden entry rejection notification is sent to the user terminal 12, and the process returns to step S105.

[0174] As shown in Figure 21, in step S163, it is determined whether or not there is a withdrawal request. Here, the CPU 50 determines whether or not there is a withdrawal request from the user terminal 12.

[0175] If the answer in step S163 is "NO," that is, if there is no request to withdraw, the process proceeds to step S105, as shown in Figure 12. On the other hand, if the answer in step S163 is "YES," that is, if there is a request to withdraw, the process proceeds to step S165, where a withdrawal acceptance / rejection screen as shown in Figure 17 is displayed on the display device 60, and the process proceeds to step S167.

[0176] In step S167, it is determined whether the user has accepted the button but has not pressed it. Here, the CPU 50 determines whether the button image 222 has been turned on. If the answer in step S167 is "YES", that is, if the user has accepted the button but has not pressed it, then in step S169, a notification of acceptance is sent to the user terminal 12, and the process returns to step S105.

[0177] On the other hand, if the answer in step S167 is "NO," that is, if it is not an acceptance of the retreat, then in step S171, it is determined whether it is a refusal to retreat. Here, the CPU 50 determines whether the button image 224 has been turned on. If the answer in step S171 is "NO," that is, if it is not a refusal to retreat, the process returns to step S167.

[0178] On the other hand, if the answer in step S171 is "YES," that is, if the withdrawal is rejected, then in step S173, a withdrawal rejection notification is sent to the user terminal 12, and the process returns to step S105.

[0179] In the second embodiment, as in the first embodiment, the user can interact intimately with the avatar or operator who is their conversation partner.

[0180] Furthermore, according to the second embodiment, since the avatar can be requested to pop out and retract based on user input, the user can request to make the avatar pop out or return to its original position according to their preference.

[0181] Furthermore, according to the second embodiment, even if the user requests the avatar to pop out or retract, the operator decides whether to approve or reject the request, so the avatar's popping out and retraction can be controlled by the operator.

[0182] In the second embodiment, instead of the operator sending an extension or retraction instruction as described in the first embodiment, the user sends an extension or retraction request, and the operator notifies the user of whether to approve or reject the extension or retraction in response. However, both of these actions may be performed. In such cases, in addition to the control processing shown in Figures 10 and 11, the CPU 20 of the user terminal 12 executes the processing in steps S51-S73 following the processing in steps S35 and S39. Similarly, in addition to the control processing shown in Figures 12 and 13, the CPU 50 of the operator terminal 16 executes the processing in steps S151-S173 following the processing in steps S125 and S129.

[0183] <Third Example> The information processing system 10 of the third embodiment is the same as the first embodiment, except that it displays an image of the operator instead of an avatar image 102. Therefore, we will explain the differences and omit any redundant explanations.

[0184] In the third embodiment, instead of the avatar image 102, the operator's image is displayed, and operator video data is transmitted from the operator terminal 16 to the user terminal 12. Therefore, it is not necessary for the operator terminal 16 to generate motion data from the operator video data.

[0185] Furthermore, when the user terminal 12 receives operator video data, it extracts an image of a person (in this embodiment, the operator) from the operator's video and displays the extracted image of the person (i.e., the operator) at its normal size or enlarged size. Whether to display the operator's image at its normal size or enlarged size is determined according to the pop-out flag 304h, similar to when displaying the avatar image 102. In the third embodiment, the remaining image after extracting the person's image from the operator video is displayed as the window image 104. However, as in the first embodiment, a scene such as a virtual room may be displayed as the window image 104.

[0186] Figures 22 and 23 are flowcharts of the control process of the user terminal 12 in the third embodiment. In Figures 22 and 23, in the control process shown in Figures 10 and 11, step S201 is executed between steps S7 and S9, step S203 is executed instead of step S15, and step S205 is executed instead of step S17. Also, step S207 is executed instead of step S23, and step S209 is executed instead of step S25.

[0187] Specifically, if the answer in step S7 is "YES", then in step S201, the image data of the person is extracted (or cut out) from the operator's video data. Also, if the answer in step S11 is "YES", then in step S13, the operator's audio data is output, and in step S203, the image data of the person is output at the normal size, and the process proceeds to step S27. On the other hand, if the answer in step S11 is "NO", then in step S205, the image data of the person is output at the normal size, and the process proceeds to step S27.

[0188] Furthermore, as shown in Figure 23, if the answer in step S19 is "YES", then in step S21, the operator's voice data is output, and in step S207, the person's image data is output at an enlarged size, and the process proceeds to step S27. On the other hand, if the answer in step S19 is "NO", then in step S209, the person's image data is output at an enlarged size, and the process proceeds to step S27.

[0189] Figure 24 is a flowchart showing a part of the control process of the CPU 50 of the operator terminal 16 in the third embodiment. In the control process of the CPU 50 of the operator terminal 16 in the third embodiment, step S301 is executed instead of step S111 of the control process of the operator terminal 16 in the first embodiment, and step S303 is executed instead of S113.

[0190] As shown in Figure 24, if the answer in step S109 is "NO", then in step S301, the operator video data is sent to the user terminal 12, and the process proceeds to step S115 as shown in Figure 13. On the other hand, if the answer in step S109 is "YES", then in step S303, the operator video data and operator audio data 404d are sent to the user terminal 12, and the process proceeds to step S115.

[0191] In the third embodiment, as in the first embodiment, the user can interact closely with the operator who is their conversation partner.

[0192] Furthermore, the modifications in the third embodiment can also be applied to the second embodiment. In addition, the modifications in the third embodiment can also be applied to an example in which, in addition to the operator sending an extension or retraction instruction (first embodiment), the user sends an extension or retraction request, and the operator notifies the user in response whether to approve or reject the extension or retraction (second embodiment).

[0193] In the embodiments described above, the user terminal and the operator terminal communicate via P2P, but communication may also be performed via a server located on the network.

[0194] Furthermore, if the same result can be obtained from each step in the flowchart shown in the above-described embodiment, the order in which the processes are performed can be changed.

[0195] Furthermore, the various screens and specific numerical values ​​mentioned in each of the above-described embodiments are merely examples and can be modified as needed. [Explanation of Symbols]

[0196] 10. Information Processing Systems 12 ... User terminal 14 ... Network 16 ... Operator terminal 20, 50 ...CPU 22, 52... RAM 24, 54 ... Communication I / F 26, 56 ... Input / Output Interface 28, 58 ... Input device 30, 60...display device 32, 62... Mike 34, 64... speakers 36, 38, 66... ​​Camera

Claims

1. An information processing device that is connected to the terminal of the person with whom it is communicating in a communication manner, display device, A camera that photographs the back of the aforementioned display device, A frame image display means that displays a frame image on the display device using the image of the background of the display device captured by the camera. A means for displaying an avatar image or an image of the conversation partner on the display device in front of the frame image, and An information processing device comprising an enlargement means for enlarging the image of the avatar or the image of the person with whom the conversation is taking place to such an extent that a portion of the image of the avatar or a portion of the image of the person with whom the conversation is taking place overlaps the frame image.

2. Receiving means for receiving instructions or notifications from the terminal of the aforementioned conversation partner, The information processing apparatus according to claim 1, wherein the magnification means magnifies the image of the avatar or the image of the conversation partner in response to receiving a magnification instruction by the receiving means.

3. The information processing apparatus according to claim 2, further comprising a restoration means that restores the image of the avatar or the image of the conversation partner to its original size in response to receiving an instruction to return by the receiving means.

4. Enlargement request means that requests the terminal of the conversation partner to enlarge the image of the avatar or the image of the conversation partner, The information processing apparatus according to claim 2, wherein the enlargement means enlarges the image of the avatar or the image of the conversation partner when the receiving means receives an acceptance notification in response to a request by the enlargement request means, and does not enlarge the image of the avatar or the image of the conversation partner when the receiving means receives a rejection notification.

5. A restoration request means that requests the terminal of the conversation partner to restore the avatar image or the image of the conversation partner to its original size. The information processing apparatus according to claim 3, wherein the restoration means, in response to a request by the restoration request means, restores the image of the avatar or the image of the conversation partner to its original size when the receiving means receives an acceptance notification, and does not restore the image of the avatar or the image of the conversation partner to its original size when the receiving means receives a rejection notification.

6. An information processing system comprising an information processing device that is capable of communicating with the terminal of a conversation partner, The aforementioned information processing device is display device, A camera that photographs the back of the aforementioned display device, A frame image display means that displays a frame image on the display device using the image of the background of the display device captured by the camera. A means for displaying an avatar image or an image of the conversation partner on the display device in front of the frame image, Receiving means for receiving an enlargement instruction from the terminal of the conversation partner, and The system includes an enlargement means that, upon receiving an enlargement instruction via the receiving means, enlarges the image of the avatar or the image of the conversation partner to a size such that a portion of the avatar image or a portion of the conversation partner's image overlaps the frame image. The terminal of the aforementioned conversation partner is, Operation detection means for detecting the actions of the conversation partner, and An information processing system comprising a transmission means for transmitting an enlargement instruction to the information processing device when the operation detection means detects an enlargement instruction to enlarge the image of the avatar.

7. A control program executed on an information processing device that includes a display device and a camera for taking pictures of the back of the display device, and is connected to a terminal of an interlocutor in a manner that enables communication with the terminal of the interlocutor, The processor of the aforementioned information processing device, A frame image display step in which a frame image is displayed on the display device using the image of the background of the display device captured by the camera, A dialogue partner display step in which an image of an avatar or an image of the dialogue partner is displayed on the display device in front of the frame image, and A control program that causes the program to perform an enlargement step in which the image of the avatar or the image of the person it is interacting with is enlarged to a size in which a part of the image of the avatar or a part of the image of the person it is interacting with overlaps the frame image.

8. A control method for an information processing device comprising a display device and a camera for photographing the rear of the display device, and which is connected to a terminal of a conversation partner in a communicative manner, A step of displaying a frame image on the display device using the image of the background of the display device captured by the camera, The steps of displaying an avatar image or an image of the person being interacted with on the display device in front of the frame image, and A control method comprising the step of enlarging the image of the avatar or the image of the person interacting with the other party to a size such that a part of the image of the avatar or a part of the image of the person interacting with the other party overlaps the frame image.

Images