Information processing system and information processing method

Abstract

To provide an information processing system and an information processing method that can appropriately respond to a customer.SOLUTION: In an information processing system in which a store terminal is communicatively connected to an operator terminal via a network, the store terminal is installed in a store that is provided with an electronic device such as a check terminal, and captures a video of a customer operating the check terminal and detects a voice of the customer, and also captures a video of an operation screen of the check terminal with a first camera 36 and a second camera 38. The video and the voice of the customer and the video of the operation screen are transmitted to the operator terminal, and an operator watches the video of the customer and listens to the voice of the customer and watches the video of the operation screen, and controls an avatar displayed on a display device 30 of the store terminal to respond to the customer. In other words, the operator not only watches the video and listens to the voice of the customer but also watches the operation screen of the check terminal, and responds to the customer face to face at the store.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an information processing system and an information processing method, and more particularly, to an information processing system and an information processing method for displaying an image of an avatar of an operator who serves a customer and interacting with the customer.

Background Art

[0002] An example of this type of information processing system is disclosed in Patent Document 1. The communication system disclosed in this Patent Document 1 provides a remote customer service.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, for the purpose of both reducing labor costs and securing personnel, as well as for crime prevention, etc., in the retail, service, and financial industries, etc., the number of personnel working in stores is being reduced, and for some services, there is a shift towards remote customer service. However, in remote customer service, there are cases where customers cannot be appropriately served compared to the case of in-person service in a store.

[0005] Therefore, the main object of the present invention is to provide a novel information processing system and information processing method.

[0006] Another object of the present invention is to provide an information processing system and an information processing method that can appropriately serve customers.

Means for Solving the Problems

[0007] The first invention is an information processing system including a user terminal that is arranged at a predetermined location and used by a user together with a predetermined electronic device, and an operator terminal that is communicably connected to the user terminal and used by an operator who responds to the user. The user terminal transmits the user's video, the user's voice, and the video of the operation unit of the electronic device to the operator terminal, and controls an avatar corresponding to the operator according to the operation data and voice data from the operator terminal. The operator terminal receives and outputs the user's video, the user's voice, and the video of the operation unit of the electronic device, and transmits the operation data and voice data of the avatar according to the operator's operation to the user terminal.

[0008] The second invention is dependent on the first invention. The user terminal includes a first display device, a first microphone provided to be able to detect the user's voice, a first voice transmission means for transmitting voice data corresponding to the voice detected by the first microphone to the operator terminal, a first camera provided to be able to photograph the user, a first video transmission means for transmitting first video data corresponding to the first video photographed by the first camera to the operator terminal, a second camera provided to be able to photograph the operation unit of the electronic device, a second video transmission means for transmitting second video data corresponding to the second video photographed by the second camera to the operator terminal, an avatar display means for displaying an image of an avatar corresponding to the operator on the first display device, and a first speaker for outputting the voice of the avatar to the user. The operator terminal includes a second display device, a second microphone provided to be able to detect the operator's voice, a second voice transmission means for transmitting voice data based on the voice detected by the second microphone to the user terminal, a third camera provided to be able to photograph the operator, an operation data generation means for generating operation data of the avatar based on the third video data corresponding to the third video photographed by the third camera, an operation data transmission means for transmitting the operation data generated by the operation data generation means to the user terminal, an avatar display means for displaying an image of an avatar corresponding to the operator on the first display device, a first voice reception means for receiving the voice data transmitted by the first voice transmission means, a speaker for outputting the voice data received by the first voice reception means, a first video reception means for receiving the first video data transmitted by the first video transmission means, a first video display means for displaying a second video corresponding to the first video data received by the first video reception means on the second display device, a second video reception means for receiving the second video data received by the second video transmission means, and a second video display means for displaying a second video corresponding to the second video data received by the second video reception means on the second display means.

[0009] The third invention is dependent on the second invention. Before the operator starts responding to the user, the second microphone and the third camera are turned off, and in response to the operator instructing to start responding to the user, the second microphone and the third camera are turned on.

[0010] The fourth invention is a method for an information processing system, comprising a user terminal disposed at a predetermined location and used by a user together with a predetermined electronic device, and an operator terminal communicably connected to the user terminal and used by an operator who responds to the user. The user terminal transmits the user's video and voice and the video of the operation unit of the electronic device to the operator terminal, controls an avatar corresponding to the operator according to the operation data and voice data from the operator terminal, and the operator terminal receives and outputs the user's video and voice and the video of the operation unit of the electronic device, and transmits the operation data and voice data of the avatar according to the operator's operation to the user terminal.

Advantages of the Invention

[0011] According to this invention, it is possible to appropriately respond to customers.

[0012] The above object, other objects, features and advantages of this invention will become more apparent from the following detailed description of the embodiments with reference to the drawings.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

DETAILED DESCRIPTION OF THE INVENTION

[0014] Referring to FIG. 1, the information processing system 10 of this embodiment includes a store terminal 12, and the store terminal 12 is communicably connected to an operator terminal 16 via a network 14.

[0015] As an example, by the technology of Web Real-Time Communication (WebRTC), the store terminal 12 and the operator terminal 16 can perform P2P (Peer to Peer) communication and transmit and receive video and / or audio in real time via a web browser.

[0016] Although illustration is omitted, WebRTC can be used in combination with a plurality of servers. Specifically, the plurality of servers for using WebRTC include a signaling server, a STUN (Simple Traversal of UDP through NATs) server, a TURN (Traversal Using Relay NAT) server, and the like. Although detailed description is omitted, the signaling server is a server for acquiring information about a communication partner by WebRTC. Also, the STUN server and the TURN server are servers for performing so-called NAT (Network Address Translation) traversal when the communication partner exists in a different network.

[0017] In this embodiment, one store terminal 12 and one operator terminal 16 are shown. Actually, a plurality of store terminals 12 and a plurality of operator terminals 16 are connected to the network 14, and the connection state of one store terminal 12 and one operator terminal 16 that are set to perform a video call or a web conference among them is established.

[0018] Note that when the operator terminal 16 connected to the store terminal 12 is predetermined, the store terminal 12 has the connection information of the operator terminal 16, and the operator terminal 16 has the connection information of the store terminal 12, the connection state between the store terminal 12 and the operator terminal 16 can be established without using the above servers.

[0019] The store terminal 12 is basically installed in a store or facility that applies the information processing system 10 and is used by customers or store employees of this store (for example, hotel employees). The operator terminal 16 is used by an operator. In this embodiment, an avatar image corresponding to the operator's avatar (hereinafter, "avatar image") is displayed on the display device 30 of the store terminal 12, and the customer and the operator interact through the avatar. The operator operates the avatar, that is, controls the actions and speech of the avatar to interact with the customer. That is, the customer is a person who uses the dialogue service using the avatar provided by the operator of the operator terminal 16.

[0020] The store terminal 12 is an information processing device and is a general-purpose notebook PC or desktop PC, but other general-purpose terminals such as a smartphone or tablet PC can also be used. However, when using a notebook PC, smartphone, or tablet PC, a relatively large-sized display device for displaying the avatar image may be prepared separately and connected to the notebook PC, smartphone, or tablet PC.

[0021] The network 14 is composed of an IP network (or IP network) including the Internet and an access network (or access network) for accessing this IP network. As the access network, a public telephone network, a mobile phone network, a wired LAN, a wireless LAN, CATV (Cable Television), etc. can be used.

[0022] The operator terminal 16 is another information processing device different from the store terminal 12. As an example, it is a general-purpose notebook PC or desktop PC, but other general-purpose terminals such as a smartphone or tablet PC can also be used.

[0023] FIG. 2 is a block diagram showing an example of the electrical configuration of the store terminal 12 shown in FIG. 1. As shown in FIG. 2, the store terminal 12 includes a CPU 20, and the CPU 20 is connected via an internal bus to a RAM 22, a communication interface (hereinafter referred to as "communication I / F") 24, and an input / output interface (hereinafter referred to as "input / output I / F") 26.

[0024] The CPU 20 controls the overall operation of the store terminal 12. However, instead of the CPU 20, a SoC (System-on-a-chip) including a plurality of functions such as a CPU function and a GPU (Graphics Processing Unit) function may be provided.

[0025] The RAM 22 is a main memory device and is used as a work area or buffer area for the CPU 20. Although not shown, the store terminal 12 is provided with an HDD and a ROM as auxiliary storage devices. However, instead of or in addition to the HDD, a non-volatile memory such as an SSD may be used.

[0026] The communication I / F 24 is a wired interface for transmitting and receiving control signals and data between the CPU 20 and an external computer such as the operator terminal 16 via the network 14 under the control of the CPU 20. However, as the communication I / F 24, a wireless interface such as a wireless LAN or Bluetooth (registered trademark) may also be used.

[0027] An input device 28, a display device 30, a microphone 32, a speaker 34, a first camera 36, and a second camera 38 are connected to the input / output I / F 26.

[0028] The input device 28 includes a keyboard, a computer mouse, and a touch panel. However, the touch panel is provided on the display surface of the display device 30.

[0029] The display device 30 is, for example, a liquid crystal display device. The microphone 32 is a general-purpose sound pickup microphone. The speaker 34 is a general-purpose stereo speaker. The first camera 36 and the second camera 38 are each general-purpose CCD cameras.

[0030] Also, the input / output I / F 26 outputs the operation data (or operation information) input from the input device 28 to the CPU 20, and outputs the image data generated by the CPU 20 to the display device 30 to display a screen or an image corresponding to the image data on the display device 30. However, there may be a case where the image data received from an external computer (for example, the operator terminal 16) is output by the CPU 20.

[0031] Also, the input / output I / F 26 converts the voice of the customer detected by the microphone 32 into digital voice data (hereinafter referred to as "customer voice data") and outputs it to the CPU 20, or converts the voice data output by the CPU 20 into an analog voice signal and outputs it from the speaker 34, or outputs the video data of the video including the customer photographed (detected) by the first camera 36 (hereinafter referred to as "customer video data") to the CPU 20, or outputs the video data of the video of the operation unit or operation screen on the customer side of the electronic device (check terminal described later) photographed (detected) by the second camera 38 (hereinafter referred to as "screen video data") to the CPU 20. The customer video data and the screen video data are, for example, moving images.

[0032] However, in this embodiment, the voice data output from the CPU 20 is the voice data of the operator received from the operator terminal 16 or the voice data obtained by converting the voice of the operator into the voice of the avatar. Hereinafter, these voice data will be referred to as operator voice data.

[0033] Note that the electrical configuration of the store terminal 12 shown in FIG. 2 is an example and need not be limited.

[0034] When the store terminal 12 is a smartphone, it is provided with a call circuit for making calls via a mobile phone communication network or a mobile phone network and a public telephone network. However, in this embodiment, since such calls are not made, the illustration is omitted.

[0035] Figure 3 is a block diagram showing the electrical configuration of the operator terminal 16 shown in Figure 1. As shown in Figure 3, the operator terminal 16 includes a CPU 50, and the CPU 50 is connected to a RAM 52, a communication I / F 54, and an input / output I / F 56 via an internal bus.

[0036] The CPU 50 controls the overall operation of the operator terminal 16. However, instead of the CPU 50, an SoC including a plurality of functions such as a CPU function and a GPU function may be provided.

[0037] The RAM 52 is a main memory device and is used as a work area or a buffer area for the CPU 50. Although not shown, the operator terminal 16 is provided with an HDD and a ROM as auxiliary storage devices. However, instead of or in addition to the HDD, a non-volatile memory such as an SSD may be used.

[0038] The communication I / F 54 is a wired interface for transmitting and receiving control signals and data between the CPU 50 and an external computer such as the operator terminal 16 via the network 14 under the control of the CPU 50. However, as the communication I / F 54, a wireless interface such as a wireless LAN or Bluetooth (registered trademark) may also be used.

[0039] An input device 58, a display device 60, a microphone 62, a speaker 64, and a camera 66 are connected to the input / output I / F 56.

[0040] The input device 58 is a keyboard and a computer mouse. Further, a touch panel may be provided. However, when a smartphone or a tablet PC is used as the operator terminal 16, the input device 58 is a touch panel and hardware buttons.

[0041] The display device 60 is, for example, a liquid crystal display device. The microphone 62 is a general-purpose sound pickup microphone. The speaker 64 is a general-purpose stereo speaker. The camera 66 is a general-purpose CCD camera.

[0042] Also, the input / output I / F 56 outputs the operation data (or operation information) input from the input device 58 to the CPU 50, and also outputs the image data generated by the CPU 50 to the display device 60 to cause the display device 60 to display a screen corresponding to the image data. However, there may be a case where the image data received from an external computer (for example, the store terminal 12) is output by the CPU 50. The image data received from the store terminal 12 is customer video data and screen video data.

[0043] Also, the input / output I / F 56 converts the voice of the operator detected by the microphone 62 into digital voice data and outputs it to the CPU 50, or converts the voice data output by the CPU 50 into an analog voice signal and outputs it from the speaker 64, or outputs the video data of the video of the operator captured (detected) by the camera 66 (hereinafter referred to as "operator video data") to the CPU 50. However, in this embodiment, the voice data output from the CPU 50 is the customer voice data received from the store terminal 12. Also, the video of the operator is a moving image.

[0044] Note that the electrical configuration of the operator terminal 16 shown in FIG. 3 is an example and does not need to be limited.

[0045] Also, when the operator terminal 16 is a smartphone, it is provided with a call circuit for making a call via a mobile phone communication network, or a mobile phone network and a public telephone network. However, in this embodiment, since such a call is not made, the illustration is omitted.

[0046] The operation of the information processing system 10 of this embodiment will be described below. In such an information processing system 10, a customer facing the display device 30 of the store terminal 12 and an operator of the operator terminal 16 conduct a video call or a web conference using the store terminal 12 and the operator terminal 16.

[0047] FIG. 4 shows an example of how the information processing system 10 of this embodiment is applied to a hotel check-in counter. As shown in FIG. 4, as an example, the display device 30 of the store terminal 12 is on the check-in counter and is arranged beside an electronic device or terminal (hereinafter referred to as a "check terminal") used for self-check-in (and also for check-out). The microphone 32, the speaker 34, and the first camera 36 are arranged on the upper side of the display device 30, and the second camera 38 is fixed to the ceiling or the other end of a support member with one end fixed to the check-in counter.

[0048] Note that around the check terminal, there are provided terminals for electronic settlement, a key return port, etc., but since these are not the essential content of the present invention, the description thereof is omitted.

[0049] The microphone 32 is arranged so as to be able to detect the voice of a customer in front of the display device 30. The speaker 34 is arranged in a direction to deliver voice to the customer in front of the display device 30. The first camera 36 is arranged so as to be able to photograph the face of the customer in front of the display device 30, that is, the customer facing the display device 30. The second camera 38 is arranged in a direction to be able to photograph the display surface of the customer-side display device (hereinafter referred to as the "customer-side display device") of the check terminal operated by the customer in front of the display device 30, that is, the customer-side operation screen (hereinafter referred to as the "customer-side operation screen").

[0050] However, in FIG. 4, a part of the check-in counter, the check terminal, the store terminal 12, and the support member of the second camera 38 is shown, and the illustration of the other part is omitted. Also, the arrangement shown in FIG. 4 is an example and does not need to be limited.

[0051] With such a configuration, the display device 30 displays an avatar image corresponding to the operator's avatar, outputs the voice of the avatar from the speaker 34, and the operator who responds to the customer views the video captured by the first camera 36 and the second camera 38 and listens to the voice of the customer detected by the microphone 32. That is, the operator responds to the customer by viewing the (face) video of the customer, listening to the voice of the customer, and viewing the video of the customer-side operation screen through the avatar. For example, while checking the customer's condition and the progress of the customer-side operation screen, the operator responds to the customer. However, when the operator is not responding to the customer, a predetermined character video or still image imitating a store clerk is displayed on the display device 30.

[0052] When the customer is not present in front of the check terminal or the display device 30 (hereinafter referred to as the "standby state"), a standby screen is displayed on the display device 30. As the standby screen, a video (which may be a still image) using a predetermined character is displayed. However, even in the standby state, a video call or a web conference is being executed between the store terminal 12 and the operator terminal 16, but the microphone 62 and the camera 66 of the operator terminal 16 are turned off.

[0053] Also, the first camera 36 executes a shooting process even in the standby state, performs a well-known face recognition process on the captured image captured by the first camera 36, and determines whether a human, that is, a customer, exists in front of the display device 30.

[0054] Note that for the face recognition process, an image processing library such as Face API (a face recognition application provided by Microsoft Corporation) or MediaPipe Face Mesh can be used.

[0055] Also, in this embodiment, the first camera 36 is used to determine whether a customer exists in front of the display device 30, but instead of the first camera 36, a human presence sensor may be used.

[0056] In the standby state, when the customer's face is recognized and the first predetermined time (for example, 5 seconds) has elapsed, it is determined that the customer is staying in front of the display device 30. When it is determined that the customer is staying in front of the display device 30, this is notified to the operator. That is, a notification that the customer is staying in front of the display device 30 (hereinafter referred to as "stay notification") is transmitted from the store terminal 12 to the operator terminal 16. Also, instead of the standby screen, the stay screen 100 is displayed on the display device 30.

[0057] FIG. 5 shows an example of the stay screen 100 displayed on the display device 30 of the store terminal 12. The stay screen 100 includes an image 102 during stay, and a button image 104 is provided in front of the image 102 during stay. The image 102 during stay includes an image 102a of a predetermined character and an image 102b of a message. As an example, the character image 102a is a moving image, and the predetermined character performs actions such as bowing and waving hands from an upright state and returning to the upright state. The predetermined character repeats such actions periodically. The message image 102b is a character string of a message explaining how to call the operator. However, the message does not necessarily have to be limited to one type, and other messages explaining that the operator can respond when the operator is called may also be displayed. When two or more types of messages are displayed, each message is repeatedly displayed in order.

[0058] When the button image 104 is turned on, the operator is called. That is, when the button image 104 is turned on, a notification to call the operator (hereinafter referred to as "call notification") is transmitted from the store terminal 12 to the operator terminal 16. Along with this, a cancellation notification of the stay notification is transmitted from the store terminal 12 to the operator terminal 16. Also, if the operator is not called even after the second predetermined time (for example, 10 seconds) has elapsed since the call notification was transmitted, that is, if the button image 104 is not turned on, it is determined that the customer has left in front of the display device 30 or that the customer does not need the operator's response, and a cancellation notification of the stay notification is transmitted from the store terminal 12 to the operator terminal 16.

[0059] Also, when the button image 104 is turned on, the calling screen 150 is displayed on the display device 30 instead of the stay screen 100. FIG. 6 shows an example of the calling screen 150 displayed on the display device 30 of the store terminal 12.

[0060] The calling screen 150 includes a calling image 152. The image 152 includes an image 152a of a predetermined character and an image 152b of a message. As an example, the image 152a of the predetermined character is a moving image, and the predetermined character performs actions such as bowing, waving a hand, or swaying the upper body from an upright state and then returning to the upright state. The predetermined character repeats such actions periodically. The image 152b of the message is a character string of a message explaining that the operator is being called. However, the message does not necessarily have to be limited to one type, and other messages explaining the profile of the operator being called, etc., may also be displayed. When two or more types of messages are displayed, each message is repeatedly displayed in order.

[0061] When the operator decides (gives an instruction) to respond to the customer, a notification of responding to the customer (hereinafter referred to as "response notification") is transmitted from the operator terminal 16 to the store terminal 12. However, as will be described later, since the operator can view the video of the customer and listen to the voice of the customer, when the operator thinks that it is better to respond to the customer, such as when confirming that the customer is in trouble, the operator can give an instruction to respond to the customer even if not called by the customer. Then, on the store terminal 12, an avatar image of an avatar in the same posture as the predetermined character is displayed on the display device 30. Also, the store terminal 12 transmits a cancellation notification of all notifications to the operator terminal 16. All notifications are stay notifications or call notifications.

[0062] Also, when it is determined that the operator will respond to the customer, the microphone 62 and the camera 66 of the operator terminal 16 are turned on, and the avatar is controlled in terms of movement and speech by the operator.

[0063] FIG. 7 shows an example of an avatar screen 200 displayed on the display device 30 of the store terminal 12. The avatar screen 200 includes an avatar image 202. In FIG. 7, the background of the avatar screen 200 is omitted, but a single color or other background may be displayed.

[0064] As described above, in this embodiment, the operations and voices of the avatar corresponding to the avatar image 202 displayed on the display device 30 are controlled based on the operations and voices of the operator of the operator terminal 16. The operations include gestures, face orientation, and facial expressions. The operations of the operator are detected from the video of the operator captured by the camera 66 of the operator terminal 16. The voice of the operator is detected by the microphone 62.

[0065] In the operator terminal 16, motion data for controlling the operations of the avatar is generated based on the operations of the operator, and voice data for controlling the utterance (voice output) of the avatar is generated based on the voice of the operator. The motion data is information (data) on the positions and orientations of each part for controlling the gestures, face orientation, and facial expressions of the avatar. Specifically, the motion data is the position (coordinate data) of each part of the face, namely the eyebrows, eyes (upper eyelid, lower eyelid), nose, and mouth (upper lip, lower lip), the positions (coordinate data) of the head, hands, arms, shoulders, waist, knees, and ankles, and information about the orientations of the neck (or face), wrists, and ankles. However, when the lower part from the thighs of the avatar is not displayed, the information on the positions and orientations of the knees and ankles may not be included in the motion data. Also, the voice data is the data of the voice of the operator or the voice obtained by converting the voice of the operator into the voice of the avatar.

[0066] Note that, as an example, the motion data for controlling the operations of the avatar based on the operations of the operator can be generated using an image processing library such as MediaPipe Holistic. In other examples, the method disclosed in Japanese Patent Application Laid-Open No. 2021-56940 of the background art can also be used.

[0067] At the store terminal 12, the avatar image 202 is changed according to the operation data. Therefore, the avatar is operated. Also, the store terminal 12 outputs the received operator voice data to the speaker 34. That is, the store terminal 12 controls the avatar image 202 (avatar) displayed on the display device 30 according to the instruction from the operator terminal 16. However, the instruction from the operator terminal 16 is only the operation data, or the operation data and the operator voice data.

[0068] On the other hand, the video including the customer is captured by the first camera 36, and the video data corresponding to the video captured by the first camera 36, that is, the customer video data, is transmitted from the store terminal 12 to the operator terminal 16. Also, the voice of the customer is detected by the microphone 32, and the voice data corresponding to the voice detected by the microphone 32, that is, the customer voice data, is transmitted from the store terminal 12 to the operator terminal 16. Further, the video of the customer operation screen is captured by the second camera 38, and the video data corresponding to the video captured by the second camera 38, that is, the screen video data, is transmitted from the store terminal 12 to the operator terminal 16. The customer video data and the screen video data are output to the display device 60 of the operator terminal 16, and the customer voice data is output from the speaker 64 of the operator terminal 16. Therefore, the operator can control the avatar and respond to the customer while watching and listening to the customer's situation and confirming the progress of the customer operation screen.

[0069] FIG. 8 shows an example of the operator-side response screen 250 displayed on the display device 60 of the operator terminal 16. As shown in FIG. 8, on the operator-side response screen 250, a display area 252 is provided at the upper end of the screen, and below the display area 252 and in the central part of the screen, a display area 254 and a display area 256 are provided side by side. A button image 260 is provided below the display area 254, and a button image 262 is provided below the display area 256.

[0070] The display area 252 is an area for notifying (or displaying) the content of the notification from the store terminal 12 to the operator terminal 16 (or the operator) as a character string. When there is a stay notification, a message indicating that the customer is staying in front of the display device 30 or the check terminal is displayed as a character string. When there is a call notification, a message indicating that the customer has called the operator is displayed as a character string. When there are both notifications, each message is displayed alternately or in two lines. When there is a cancellation notification for the stay notification, the stay notification is cancelled. That is, the message indicating that the customer is staying in front of the display device 30 or the check terminal is made invisible. Also, when there is a cancellation notification for all notifications, all notifications are cancelled. That is, all messages in the display area 252 are made invisible.

[0071] The display area 254 is an area for outputting screen video data. That is, in the store where the operator is dealing with a customer, the video of the customer-side operation screen of the check terminal in front of that customer is displayed. In FIG. 8, the image of the customer-side operation screen is omitted.

[0072] The display area 256 is an area for outputting customer video data. The video of the customer with whom the operator is dealing is displayed. In FIG. 8, the image of the customer is simply shown.

[0073] The button image 260 is an icon for starting the handling of a customer (or the interaction with the customer) according to the operator's intention. When the button image 260 is turned on, a start notification for the handling is sent to the store terminal 12, and at the same time, the microphone 62 and the camera 66 are turned on.

[0074] The button image 262 is an icon for ending the handling of a customer. When the button image 262 is turned on, an end notification for the handling is sent to the store terminal 12, and at the same time, the microphone 62 and the camera 66 are turned off.

[0075] FIG. 9 shows an example of a memory map 300 of a RAM 22 incorporated in the store terminal 12. As shown in FIG. 9, the RAM 22 includes a program storage area 302 and a data storage area 304. In the program storage area 302, an information processing program executed by the store terminal 12 of this embodiment is stored.

[0076] The information processing program executed by the store terminal 12 includes a main processing program 302a, an operation detection program 302b, a communication program 302c, an image generation program 302d, an image output program 302e, a photographing program 302f, a face detection program 302g, a notification program 302h, an avatar control program 302i, a voice detection program 302j, and a voice output program 302k, etc.

[0077] The main processing program 302a is a program for executing the main routine of the information processing of the store terminal 12 of this embodiment.

[0078] The operation detection program 302b is a program for detecting operation data 304a input from the input device 28 according to the operation of a customer or a store clerk and storing it in the data storage area 304.

[0079] The communication program 302c is a program for communicating (transmitting and receiving data, etc.) with an external device, in this embodiment, the operator terminal 16.

[0080] The image generation program 302d is a program for generating image data corresponding to all or part of a screen (such as 100, 150, 200, etc.) to be displayed on the display device 30 using the image generation data 304b.

[0081] The image output program 302e is a program for outputting the image data generated according to the image generation program 302d to the display device 30. Therefore, the screen corresponding to the image data is displayed on the display device 30.

[0082] The shooting program 302f is a program for causing the first camera 36 to execute a shooting process and temporarily storing the customer video data 304d input from the first camera 36 in the data storage area 304. Further, the shooting program 302f is a program for causing the second camera 38 to execute a shooting process and temporarily storing the screen video data 304f input from the second camera 38 in the data storage area 304.

[0083] The face detection program 302g is a program for performing known face recognition processing on the customer video data 304d shot according to the shooting program 302f to detect the face of the customer.

[0084] The notification program 302h is a program for sending various notifications to the operator terminal 16. However, when a notification is sent to the operator terminal 16 according to the notification program 302h, the communication program 302c is also executed.

[0085] The avatar control program 302i is a program for operating the avatar using the operation data 304h.

[0086] The voice detection program 302j is a program for detecting the voice of the customer input from the microphone 32 and temporarily storing the customer voice data 304e corresponding to the detected voice in the data storage area 304.

[0087] The voice output program 302k is a program for outputting the operator voice data 304i received from the operator terminal 16 to the speaker 34. Therefore, the voice corresponding to the operator voice data 304i or the voice obtained by converting this voice into the voice of the avatar comes out from the speaker 34.

[0088] Although illustration is omitted, in addition to the operating system and middleware of the store terminal 12, a browser and other application programs other than the information processing program of the present application are also stored in the program storage area 302.

[0089] FIG. 10 shows an example of the specific content of the data storage area 304 shown in FIG. 9. The data storage area 304 stores operation data 304a, image generation data 304b, video data 304c, customer video data 304d, customer voice data 304e, screen video data 304f, response notification data 304g, motion data 304h, operator voice data 304i, and the like.

[0090] The operation data 304a is data on operations of customers or store clerks detected according to the operation detection program 302b. When the operation data 304a is used for the processing of the CPU 20, it is deleted from the data storage area 304.

[0091] The image generation data 304b is data for generating a screen to be displayed on the display device 30 of the store terminal 12, and includes data for generating the avatar image 202.

[0092] The video data 304c is data on various videos displayed on the display device 30 of the store terminal 12 in the standby state. In this embodiment, the video includes an image of a predetermined character having the same figure as the avatar.

[0093] The customer video data 304d is data on a video captured by the first camera 36. This customer video data 304d is basically data on a video including a customer, but when the customer leaves or goes out of the shooting range of the first camera 36, the video may not include an image of the customer. When the customer video data 304d is transmitted by the CPU 20 to the operator terminal 16, it is deleted from the data storage area 304.

[0094] The customer voice data 304e is data on the voice of a customer detected by the microphone 32. This customer voice data 304e is basically data on the voice of a customer in front of the display device 30, but may include the voices of other customers and / or sounds in the store (such as the sound of in-store broadcasting), and may not include the voice of the customer when the customer is not speaking. When the customer voice data 304e is transmitted by the CPU 20 to the operator terminal 16, it is deleted from the data storage area 304.

[0095] The screen image data 304f is data of an image of the customer-side operation screen of the check terminal photographed by the second camera 38. When the screen image data 304f is transmitted to the operator terminal 16 by the CPU 20, it is deleted from the data storage area 304.

[0096] The response notification data 304g is data regarding the response notification received from the operator terminal 16.

[0097] The operation data 304h is operation data received from the operator terminal 16. When the operation data 304h is used for the processing of the CPU 20, it is deleted from the data storage area 304.

[0098] The operator voice data 304i is operator voice data received from the operator terminal 16. When the operator voice data 304i is output to the speaker 34 by the CPU 20, it is deleted from the data storage area 304.

[0099] Also, a face detection flag 304j and a response flag 304k are provided in the data storage area 304.

[0100] The face detection flag 304j is a flag for determining whether the face of the customer is detected. When the face of the customer is detected, the face detection flag 304j is turned on, and when the face of the customer is not detected, the face detection flag 304j is turned off.

[0101] The response flag 304k is a flag for determining whether the operator is responding to the customer. When the operator is responding to the customer, the response flag 304k is turned on, and when the operator is not responding to the customer, the response flag 304k is turned off.

[0102] Although illustration is omitted, other data necessary for executing information processing is stored in the data storage area 304, or a timer (counter) and other flags necessary for executing information processing are provided.

[0103] FIG. 11 shows an example of a memory map 400 of a RAM 52 incorporated in the operator terminal 16. As shown in FIG. 11, the RAM 52 includes a program storage area 402 and a data storage area 404. A control program executed by the operator terminal 16 in this embodiment is stored in the program storage area 402.

[0104] The control program executed by the operator terminal 16 includes a main processing program 402a, an operation detection program 402b, a communication program 402c, an image generation program 402d, an image output program 402e, a photographing program 402f, an operation data generation program 402g, a voice detection program 402h, a voice output program 402i, and the like.

[0105] The main processing program 402a is a program for executing the main routine of the control process of the operator terminal 16 in this embodiment.

[0106] The operation detection program 402b is a program for detecting operation data 404a input from the input device 58 according to an operation of the operator and storing it in the data storage area 404.

[0107] The communication program 402c is a program for communicating with an external device, in this embodiment, the store terminal 12.

[0108] The image generation program 402d is a program for generating image data corresponding to a screen to be displayed on the display device 60 using the image generation data 404b. However, when generating image data corresponding to the operator-side response screen 250 as shown in FIG. 8, the customer video data 404g and the screen video data 404i are also used.

[0109] The image output program 402e is a program for outputting the image data generated according to the image generation program 402d to the display device 60. Therefore, a screen corresponding to the image data is displayed on the display device 60.

[0110] The shooting program 402f is a program for causing the camera 66 to execute shooting processing and temporarily storing the operator video data 404d input from the camera 66 in the data storage area 404.

[0111] The motion data generation program 402g is a program for detecting the operator's motion from the operator video data 404d and generating motion data 404f for controlling the motion of the avatar.

[0112] The voice detection program 402h is a program for detecting the operator's voice input from the microphone 62 and temporarily storing the operator voice data 404e corresponding to the detected voice in the data storage area 404.

[0113] The voice output program 402i is a program for outputting the customer voice data 404h received from the store terminal 12 to the speaker 64. Therefore, the voice corresponding to the customer voice data 404h comes out from the speaker 64.

[0114] Although illustration is omitted, in addition to the operating system and middleware of the operator terminal 16, a browser and other application programs other than the control program of the present application are also stored in the program storage area 402.

[0115] FIG. 12 shows an example of the specific content of the data storage area 404 shown in FIG. 11. The data storage area 404 stores operation data 404a, image generation data 404b, notification data 404c, operator video data 404d, operator voice data 404e, motion data 404f, customer video data 404g, customer voice data 404h, screen video data 404i, and the like.

[0116] The operation data 404a is data of the operator's operation detected according to the operation detection program 402b. When the operation data 404a is used for the processing of the CPU 50, it is deleted from the data storage area 404.

[0117] The image generation data 404b is data used to generate the screen displayed on the display device 60 of the operator terminal 16.

[0118] The notification data 404c is notification data transmitted from the store terminal 12. Specifically, it is data regarding a stay notification, a call notification, a cancellation notification of the stay notification, and a cancellation notification of all notifications. Also, when the data of the cancellation notification of the stay notification is stored, the data of the stay notification is deleted, and then the data of the cancellation notification of the stay notification is also deleted. Further, when the data of the cancellation notification of all notifications is stored, the data of the stay notification or the data of the call notification is deleted, and then the data of the cancellation notification of all notifications is also deleted.

[0119] The operator video data 404d is data of the video captured by the camera 66. This operator video data 404d is basically data of a video including the operator, but when the operator leaves the seat and goes out of the shooting range of the camera 66, the operator may not be included in the video. When the operator video data 404d is used for the processing by the CPU 50, it is deleted from the data storage area 404.

[0120] The operator voice data 404e is data of the voice of the operator detected by the microphone 62. When the operator voice data 404e is transmitted to the store terminal 12 by the CPU 50, it is deleted from the data storage area 404.

[0121] The motion data 404f is data for controlling the motion of the avatar. In this embodiment, it is information (data) on the positions and orientations of the respective parts for controlling the gestures, facial orientations, and facial expressions of the avatar. Specifically, as described above, the motion data 404f is information on the positions of the respective parts of the face, namely the eyebrows, eyes (upper eyelids, lower eyelids), nose, and mouth (upper lip, lower lip), the positions of the head, hands, arms, shoulders, waist, knees, and ankles, and the orientations of the neck (or face), wrists, and ankles. When the motion data 404f is transmitted to the store terminal 12 by the CPU 50, it is deleted from the data storage area 404.

[0122] The customer video data 404g is the customer video data received from the store terminal 12. When the customer video data 404g is used for the processing by the CPU 50, it is deleted from the data storage area 404.

[0123] The customer audio data 404h is the customer audio data received from the store terminal 12. When the customer audio data 404h is output to the speaker 64 by the CPU 50, it is deleted from the data storage area 404.

[0124] The screen video data 404i is the screen video data received from the store terminal 12. When the screen video data 404i is used for the processing by the CPU 50, it is deleted from the data storage area 404.

[0125] Although illustration is omitted, other data necessary for executing control processing is stored in the data storage area 404, or a timer (counter) and a flag necessary for executing control processing are provided.

[0126] Figures 13 - 15 are flowcharts showing the information processing of the CPU 20 of the store terminal 12. Although illustration is omitted, the CPU 20 executes a process of detecting operation data and storing the operation data in parallel with the information processing, a process of causing the first camera 36 to execute a shooting process and storing the customer video data, a process of causing the second camera 38 to execute a shooting process and storing the operation screen video data, a process of executing a voice detection process and storing the customer audio data, and a process of receiving and storing data etc. transmitted from the operator terminal 16. However, the second camera 38 starts the shooting process when the response by the avatar, that is, the operator, is started. Hereinafter, the information processing of the CPU 20 will be described with reference to Figures 13 - 15, but the overlapping content will be briefly described.

[0127] As shown in FIG. 13, when the CPU 20 of the store terminal 12 starts information processing, in step S1, it establishes a connection state with the operator terminal 16, and in step S3, it displays a standby screen for the standby video as described above on the display device 30. Although detailed description is omitted, the operator terminal 16 that establishes the connection state is the operator terminal 16 of the operator in charge of this store terminal 12 and is an operator terminal 16 that is communicably connected to the network 14. In the next step S5, face authentication processing is executed, and in step S7, it is determined whether the face of the customer has been detected.

[0128] If it is "NO" in step S7, that is, if the face of the customer is not detected, in step S9, it is determined whether the face detection flag 304j is on. If it is "NO" in step S9, that is, if the face detection flag 304j is off, the process returns to step S5.

[0129] On the other hand, if it is "YES" in step S9, that is, if the face detection flag 304j is on, in step S11, the face detection flag 304j is turned off and the process returns to step S5.

[0130] Also, if it is "YES" in step S7, that is, if the face of the customer is detected, in step S13, it is determined whether the face detection flag 304j is on. If it is "NO" in step S13, in step S15, the face detection flag 304j is turned on, and in step S17, the count of the stay time is started and the process returns to step S5. In step S17, the CPU 20 resets and starts a timer for counting the stay time.

[0131] On the other hand, if it is "YES" in step S13, that is, if the face detection flag 304j is on, in step S19, it is determined whether the customer has stayed in front of the display device 30 of the store terminal 12 for a first predetermined time (in this embodiment, 5 seconds) or more.

[0132] If the answer is "NO" in step S19, that is, if the customer has not stayed in front of the display device 30 of the store terminal 12 for more than the first predetermined time, the process returns to step S5. On the other hand, if the answer is "YES" in step S19, that is, if the customer has stayed in front of the display device 30 of the store terminal 12 for more than the first predetermined time, in step S21, a stay screen 100 as shown in FIG. 5 is displayed on the display device 30, and in step S23, a stay notification is sent to the operator terminal 16, and the process proceeds to step S25 shown in FIG. 14. At this time, the CPU 20 starts the shooting process of the second camera 38.

[0133] In step S25 shown in FIG. 14, the counting of the cancellation time of the stay notification is started. In step S25, the CPU 20 resets and starts a timer for counting the cancellation time of the stay notification. In the next step S27, it is determined whether there is an audio input.

[0134] If the answer is "YES" in step S27, that is, if there is an audio input, in step S29, the customer video data 304d, the customer audio data 304e, and the screen video data 304f are sent to the operator terminal 16, and the process proceeds to step S33. On the other hand, if the answer is "NO" in step S27, that is, if there is no audio input, in step S31, the customer video data 304d is sent to the operator terminal 16, and the process proceeds to step S33.

[0135] In step S33, it is determined whether the button image 104 is on. If the answer is "YES" in step S33, that is, if the button image 104 is on, in step S35, a calling screen 150 as shown in FIG. 6 is displayed on the display device 30, in step S37, a call notification is sent to the operator terminal 16, and in step S39, a cancellation notification of the stay notification is sent to the operator terminal 16, and the process proceeds to step S41.

[0136] In step S41, an avatar screen 200 as shown in FIG. 7 is displayed on the display device 30, and in step S43, a cancellation notification of all notifications is sent to the operator terminal 16, and the process proceeds to step S53 shown in FIG. 15.

[0137] Also, if it is "NO" in step S33, that is, if the button image 104 is not on, it is determined in step S45 whether there is a response notification from the operator terminal 16.

[0138] If it is "YES" in step S45, that is, if there is a response notification from the operator terminal 16, the process proceeds to step S41. On the other hand, if it is "NO" in step S45, that is, if there is no response notification from the operator terminal 16, it is determined in step S47 whether a second predetermined time has elapsed since the stay notification was sent. Here, the CPU 20 determines whether the count value of the release time has elapsed the second predetermined time.

[0139] If it is "NO" in step S47, that is, if the second predetermined time has not elapsed since the stay notification was sent, the process returns to step S33. On the other hand, if it is "YES" in step S47, that is, if the second predetermined time has elapsed since the stay notification was sent, it is determined that the customer has left in front of the display device 30 or the customer does not require a response from the operator, and in step S49, the face detection flag 304j is turned off, and in step S51, a cancellation notification of the stay notification is sent to the operator terminal 16, and the process proceeds to step S55 shown in FIG. 15.

[0140] As shown in FIG. 15, in step S53, it is determined whether the conversation has ended. Here, the CPU 20 determines whether there is a notification of the end of the conversation from the operator terminal 16.

[0141] If it is "YES" in step S53, that is, if the conversation has ended, it is determined in step S55 whether to end the information processing. Here, the CPU 20 determines whether an instruction to end the information processing has been input by a store clerk of the store where the store terminal 12 is arranged.

[0142] If it is "YES" in step S53, that is, if the information processing is to end, the information processing is ended. On the other hand, if it is "NO" in step S53, that is, if the information processing is not to end, the process returns to step S3.

[0143] If the result in step S53 is "NO", that is, if the conversation has not ended, then in step S57, it is determined whether data has been received from the operator terminal 16. Here, the CPU 20 determines whether it has received the operation data 304h, or the operation data 304h and the operator voice data 304i.

[0144] If the result in step S57 is "NO", that is, if data has not been received from the operator terminal 16, the process proceeds to step S67. On the other hand, if the result in step S57 is "YES", that is, if data has been received from the operator terminal 16, then in step S59, it is determined whether there is operator voice data 304i.

[0145] If the result in step S59 is "YES", that is, if there is operator voice data 304i, then in step S61, the operator voice data 304i is output to the speaker 34, and in step S63, in accordance with the output of the operator voice data 304i, image data of an avatar that performs an operation according to the operation data 304h is generated and output to the display device 30, and then the process proceeds to step S67. Therefore, the avatar image 202 is updated, and the avatar displayed on the avatar screen 200 talks to the customer while performing gestures or changing the direction of the face. At this time, the avatar image 202 is lip-synced to the voice output from the speaker 34. Hereinafter, the same applies to the case where the avatar is operated in accordance with the voice data.

[0146] On the other hand, if the result in step S59 is "NO", that is, if there is no operator voice data 304i, then in step S65, image data of an avatar that performs an operation according to the operation data 304h is generated and output to the display device 30, and then the process proceeds to step S67. Therefore, the avatar image 202 is updated, and the avatar displayed on the avatar screen 200 performs gestures or changes the direction of the face.

[0147] In step S67, it is determined whether there is voice input. If it is "YES" in step S67, then in step S69, the customer video data 304d, the customer voice data 304e, and the screen video data 304f are transmitted to the operator terminal 16, and the process returns to step S53. On the other hand, if it is "NO" in step S67, that is, if there is no voice input, then in step S69, the customer video data 304d and the screen video data 304f are transmitted to the operator terminal 16, and the process returns to step S53.

[0148] Figures 16 - 18 are flowcharts showing the control processes of the CPU 50 of the operator terminal 16. Although the illustration is omitted, the CPU 50, in parallel with the control process, executes a process of detecting operation data and storing the operation data, executes a process of causing the camera 66 to perform a shooting process and storing the operator video data, executes a process of detecting voice and storing the operator voice data, and executes a process of receiving and storing data transmitted from the store terminal 12, etc. Hereinafter, the control process of the CPU 50 of the operator terminal 16 will be described, but duplicate contents will be briefly described. However, before starting the response, the shooting process of the camera 66 and the voice detection process are not executed.

[0149] As shown in Figure 16, when the CPU 50 starts the control process, in step S101, it establishes a connection state with the store terminal 12, and in step S103, it displays the operator-side response screen 250 as shown in Figure 8 on the display device 60. However, the store terminal 12 for establishing the connection state is the store terminal 12 in charge of the operator and is the store terminal 12 communicably connected to the network 14. Also, at the beginning of starting the control process, since the customer video data has not been received, the customer's video is not displayed in the display area 256.

[0150] In step S105, it is determined whether there is a stay notification. If it is "YES" in step S105, that is, if there is a stay notification, then in step S107, it is notified that the customer is staying in front of the display device 30, and the process proceeds to step S121 shown in FIG. 17. In step S107, the CPU 50 displays a message notifying that the customer is staying in front of the display device 30 in the display area 252 of the response screen 250. However, instead of or together with the display of the message, a notification voice may also be output. These are the same when other notifications are notified.

[0151] If it is "NO" in step S105, that is, if there is no stay notification, then in step S109, it is determined whether there is a call notification. If it is "YES" in step S109, that is, if there is a call notification, then in step S111, the call is notified, and the process proceeds to step S121.

[0152] If it is "NO" in step S109, that is, if there is no call notification, then in step S113, it is determined whether there is a cancellation notification of the stay notification. If it is "YES" in step S113, that is, if there is a cancellation notification of the stay notification, then in step S115, the notification that the customer is staying in front of the display device 30 is terminated, and the process proceeds to step S121. In step S115, the CPU 50 makes the message to be notified non-displayed. Also, the CPU 50 stops the output of the notification voice. These are the same for step S119 described later.

[0153] If it is "NO" in step S113, that is, if there is no cancellation notification of the stay notification, then in step S117, it is determined whether there is a cancellation notification of all notifications. If it is "YES" in step S117, that is, if there is a cancellation notification of all notifications, then in step S119, the notification of all notifications is terminated, and the process proceeds to step S121.

[0154] If the answer is "NO" in step S117, that is, if there is no cancellation notice for all notifications, then in step S121, it is determined whether there is a response instruction. Here, the CPU 50 determines whether the button image 260 is turned on.

[0155] If the answer is "YES" in step S121, that is, if there is a response instruction, the process proceeds to step S131 shown in FIG. 18. On the other hand, if the answer is "NO" in step S121, that is, if there is no response instruction, then in step S123, it is determined whether data has been received from the store terminal 12. Here, the CPU 50 determines whether the customer video data 404g, the screen video data 404i, or the customer video data 404g, the customer audio data 404h, and the screen video data 404i have been received.

[0156] If the answer is "NO" in step S123, that is, if data has not been received from the store terminal 12, the process returns to step S105. On the other hand, if the answer is "YES" in step S123, that is, if data has been received from the store terminal 12, then in step S125, it is determined whether there is customer audio data 404h.

[0157] If the answer is "YES" in step S125, that is, if there is customer audio data 404h, then in step S127, the customer video data 404g, the customer audio data 404h, and the screen video data 404i are output, and the process returns to step S5. Accordingly, the customer's video and the video of the customer-side operation screen of the check terminal are displayed on the display device 60, and the customer's voice is output from the speaker 64.

[0158] On the other hand, if the answer is "NO" in step S125, that is, if there is no customer audio data 404h, then in step S129, the customer video data 404g and the screen video data 404i are output, and the process returns to step S5. Accordingly, the customer's video and the customer-side operation screen of the check terminal are displayed on the display device 60.

[0159] As shown in FIG. 18, in step S131, it is determined whether the response has ended. Here, the CPU 50 determines whether the button image 262 has been turned on. If it is "YES" in step S131, that is, if the response has ended, the process returns to step S5. At this time, the CPU 50 sends a notification of the end of the response to the store terminal 12. On the other hand, if it is "NO" in step S131, that is, if the response has not ended, in step S133, operation data 404f is generated from the operator video data 404d, and in step S135, it is determined whether there is voice input.

[0160] If it is "YES" in step S135, that is, if there is voice input, in step S137, the operation data 404f generated in step S133 and the operator voice data 404e are sent to the store terminal 12, and the process proceeds to step S141. On the other hand, if it is "NO" in step S135, that is, if there is no voice input, in step S139, the operation data 404f generated in step S133 is sent to the store terminal 12, and the process proceeds to step S141.

[0161] In step S141, it is determined whether data has been received from the store terminal 12. If it is "NO" in step S141, the process returns to step S131. On the other hand, if it is "YES" in step S141, in step S143, it is determined whether there is customer voice data 404h.

[0162] If it is "YES" in step S143, in step S145, the customer video data 404g, the customer voice data 404h, and the screen video data 404i are output, and the process returns to step S131. On the other hand, if it is "NO" in step S143, in step S145, the customer video data 404g and the screen video data 404i are output, and the process returns to step S131.

[0163] According to this embodiment, an avatar image is displayed on the display device of the store terminal, and screen video data obtained by photographing the operation screen of the electronic device arranged near the display device of the store terminal is transmitted to the operator terminal. Therefore, while observing the customer's situation and the operation screen of the electronic device, the operator can operate the avatar to respond to the customer. That is, even when receiving customers remotely, the operator can appropriately respond to the customers as if dealing with them face to face.

[0164] In the embodiment, the operation data of the avatar is generated by the operator terminal, and the operation data is transmitted to the store terminal to generate the avatar image by the store terminal. However, the avatar image may be generated by the operator terminal and transmitted to the store terminal.

[0165] Also, in this embodiment, the operation of the avatar is controlled using the operation data generated from the video data of the operator. However, it is not necessary to be limited to this. Operation data corresponding to a series of operations to be executed by the avatar may be prepared in advance, and the operation data corresponding to the series of operations instructed by the operator through button operations or the like may be transmitted to the store terminal to control the operation of the avatar. Examples of a series of operations include an operation of bowing and returning to the original posture (for example, an upright state), an operation of turning the face left and right several times to look around and then returning to the original posture, an operation of waving the right hand or / and the left hand in front of the body and then returning to the original posture, an operation of raising the right hand or / and the left hand and then returning to the original posture, and the like.

[0166] In this embodiment, the avatar is made to speak based on the voice of the operator. However, voice data of the content to be spoken may be prepared in advance, and the voice data corresponding to the content to be spoken instructed by the operator through button operations or the like may be transmitted to the store terminal to control the speech of the avatar.

[0167] However, the operation data corresponding to a series of operations of the avatar and the voice data of the content to be spoken may be set to one button, and the operation data and the voice data may be transmitted to the store terminal at once.

[0168] Furthermore, in this embodiment, the video of the customer and the video of the customer-side operation screen are displayed on the display device of the operator terminal, but an avatar image may also be displayed on the operator terminal.

[0169] Furthermore, in this embodiment, a check terminal is cited as an example of the electronic device, but it is not necessary to be limited thereto. In other examples, an information processing system can also be configured to respond to a customer (user) who operates an electronic device such as a multi-copy machine, a kiosk terminal, and a bank ATM. In such a case, a second camera is installed so as to be able to photograph the operation screen (or the operation unit) of the multi-copy machine, the kiosk terminal, or the bank ATM. However, it is not necessary to be limited to the electronic device that displays the operation screen. An information processing system can also be configured to respond to a customer (user) who operates an electronic device such as a ticket vending machine (various railways) having an operation unit provided with hardware buttons or the like. In such a case, a second camera is installed so as to be able to photograph the operation surface (or the operation unit) of the ticket vending machine.

[0170] Note that, in the above-described embodiment, the store terminal and the operator terminal perform P2P communication, but they may communicate via a server provided on the network.

[0171] Also, in the above-described embodiment, each step of the flowchart shown can be changed in the processing order if the same result can be obtained.

[0172] Furthermore, all of the various screens and specific numerical values cited in the above-described embodiment are merely examples and can be appropriately changed as needed.

Explanation of Signs

[0173] 10... Information processing system 12... Store terminal 14... Network 16... Operator terminal 20, 50... CPU 22, 52... RAM 24, 54... Communication I / F 26, 56... Input / Output I / F 28, 58... Input device 30, 60... Display device 32, 62... Microphone 34, 64... Speaker 36, 38, 66... Camera

Claims

1. A user terminal arranged at a predetermined location and used by a user together with a predetermined electronic device, An information processing system including an operator terminal communicably connected to the user terminal and used by an operator who responds to the user, The user terminal, Transmits an image of the user, the voice of the user, and an image of the operation unit of the electronic device to the operator terminal, Controls an avatar corresponding to the operator according to operation data and voice data from the operator terminal, The operator terminal, Receives and outputs an image of the user, the voice of the user, and an image of the operation unit of the electronic device, An information processing system that transmits the operation data and voice data of the avatar according to the operation of the operator to the user terminal.

2. The user terminal, A first display device, A first microphone provided so as to be able to detect the voice of the user, A first voice transmission means for transmitting voice data corresponding to the voice detected by the first microphone to the operator terminal, A first camera provided so as to be able to photograph the user, A first video transmission means for transmitting first video data corresponding to the first video photographed by the first camera to the operator terminal, A second camera provided so as to be able to photograph the operation unit of the electronic device, A second video transmission means for transmitting second video data corresponding to the second video photographed by the second camera to the operator terminal, An avatar display means for displaying an image of an avatar corresponding to the operator on the first display device, and A first speaker for outputting the voice of the avatar to the user, The operator terminal, A second display device, A second microphone provided so as to be able to detect the voice of the operator, A second voice transmission means for transmitting the voice data based on the voice detected by the second microphone to the user terminal, A third camera provided so as to be able to photograph the operator, An operation data generation means for generating the operation data of the avatar based on third video data corresponding to the third video photographed by the third camera, An operation data transmission means for transmitting the operation data generated by the operation data generation means to the user terminal, An avatar display means for displaying an image of an avatar corresponding to the operator on the first display device, A first voice reception means for receiving the voice data transmitted by the first voice transmission means, A speaker for outputting the voice data received by the first voice reception means, A first video receiving means for receiving the first video data transmitted by the first video transmitting means, A first video display means for displaying a second video corresponding to the first video data received by the first video receiving means on the second display device, A second video receiving means for receiving the second video data received by the second video transmitting means, and The information processing system according to claim 1, further comprising a second video display means for displaying a second video corresponding to the second video data received by the second video receiving means on the second display means.

3. Before the operator starts responding to the user, the second microphone and the third camera are turned off, The information processing system according to claim 2, wherein the second microphone and the third camera are turned on in response to the operator instructing to start responding to the user.

4. A user terminal that is placed at a predetermined location and used by a user together with a predetermined electronic device, An information processing system method, comprising an operator terminal that is communicably connected to the user terminal and used by an operator who responds to the user, The user terminal, Transmits the video of the user, the voice of the user, and the video of the operation unit of the electronic device to the operator terminal, Controls an avatar corresponding to the operator according to the motion data and voice data from the operator terminal, The operator terminal, Receives and outputs the video of the user, the voice of the user, and the video of the operation unit of the electronic device, An information processing method for transmitting the motion data and the voice data of the avatar according to the operation of the operator to the user terminal.

Images