Information processing device, information processing method, and information processing program

Abstract

When a first user converses with a second user online, the second user can evaluate the first user with high reliability, and information representing the second user's evaluation of the first user is presented to the first user in real time. [Solution] The online response system 10 according to the present invention is equipped with a real-time evaluation function. According to this real-time evaluation function, when a response staff member, acting as a first user, is providing online support to a user, acting as a second user, the response user's evaluation of the response staff member is estimated based on user images, including images of the user captured by a camera (314), and user voice, including the voice of the user detected by a microphone (310). The estimated evaluation result is then presented to the response staff member in real time by an evaluation indicator 706.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program, and more particularly to an information processing apparatus, an information processing method, and an information processing program used by a first user for the first user to have an online conversation with a second user.

Background Art

[0002] An example of this type of technology is disclosed in Patent Document 1. According to the technology disclosed in this Patent Document 1, in a virtual store on a virtual space, customer service is provided by a store clerk as a first user to a user (customer) as a second user. Strictly speaking, customer service is provided by a virtual store clerk reflecting the store clerk to an avatar reflecting the user. In addition, an evaluation by the user of the store clerk's customer service is received, and this evaluation by the user is stored in a state associated with the store clerk. Then, when matching a store clerk to serve the user after the next time, the store clerk with the highest evaluation among the store clerks evaluated by the user in the past is selected.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] As mentioned above, according to the technology disclosed in Patent Document 1, user evaluations of store clerk service are accepted and used to match the user with a store clerk to serve them on subsequent visits. However, Patent Document 1 does not disclose how the user evaluations are specifically conducted. Possible methods for user evaluations include questionnaires and online reviews, where the user consciously makes the evaluation. However, with these methods, the user's intentions and expectations may be involved, which could result in unreliable evaluations. Furthermore, if highly reliable evaluations are conducted, presenting these evaluations to store clerks, especially in real time, would allow clerks to experience objective evaluations of themselves, which could lead to various effects such as improvements in the clerks' customer service skills.

[0005] Therefore, the present invention aims to provide a novel information processing device, information processing method, and information processing program that enable a second user to evaluate the first user with high reliability when a first user converses with a second user online, such as when a store clerk interacts with a customer online, and that allows the first user to experience an objective evaluation of themselves. [Means for solving the problem]

[0006] To achieve this objective, the present invention includes a first invention relating to an information processing device, a second invention relating to an information processing method, and a third invention relating to an information processing program.

[0007] The first invention relating to an information processing device is a device used by a first user for the first user to converse online with a second user, and comprises an estimation means and a first presentation means. The estimation means estimates the second user's evaluation of the first user based on at least one of an image captured by a camera and a sound detected by a microphone while a conversation is taking place between the first user and the second user. Here, the image captured by the camera includes an image of the second user captured by the camera, and the sound detected by the microphone includes the voice of the second user detected by the microphone. The first presentation means then presents (outputs) first evaluation information representing the second user's evaluation of the first user estimated by the estimation means to the first user while a conversation is taking place between the first user and the second user, that is, in real time.

[0008] Furthermore, when the estimation means estimates the evaluation of the second user by the first user based on images captured by the camera, it may estimate the evaluation of the second user by the first user based on at least one of the following: the facial expression of the second user recognized from the captured image, and the actions (gestures) of the second user that are separate from the facial expression. Here, the actions of the second user include actions that are expressed according to the emotions of the second user, such as the movement of the second user's neck, the direction of their face, bowing, and waving. Also, when the estimation means estimates the evaluation of the first user by the second user based on sounds detected by the microphone, it may estimate the evaluation of the first user by the second user by at least one of the following: the tone of the second user's voice recognized from the detected sound, the words contained in the second user's voice, and the sounds of actions caused by the second user's actions, such as noises or clapping.

[0009] In this first invention, a display means may be provided. This display means displays an image captured by a camera. In this case, the first presentation means may display the first evaluation information on the display means in conjunction with the display of the image captured by the camera, thereby presenting the first evaluation information to the first user.

[0010] Furthermore, in this first invention, a receiving means and a second presentation means may be provided. The receiving means receives input of second evaluation information, preferably when a conversation is taking place between a first user and a second user. The second evaluation information is information representing the evaluation of the first user by a third user who is listening to the conversation between the first and second users. The second presentation means presents the second evaluation information received by the receiving means to the first user, preferably when a conversation is taking place between the first and second users. In other words, it is preferable that the second evaluation information is also presented to the first user in real time.

[0011] The information processing method according to the second invention of the present invention is a method for a first user to have an online conversation with a second user, and includes an estimation step and a first presentation step. In the estimation step, while a conversation is taking place between the first user and the second user, the evaluation of the second user by the second user is estimated based on at least one of an image captured by a camera and a sound detected by a microphone. Here, the image captured by the camera includes an image of the second user captured by the camera, and the sound detected by the microphone includes the voice of the second user detected by the microphone. Then, in the first presentation step, first evaluation information representing the evaluation of the first user by the second user estimated in the estimation step is presented to the first user while the conversation is taking place between the first user and the second user, that is, in real time.

[0012] The information processing program according to the third invention of the present invention is a program for a first user to have an online conversation with a second user, and causes a computer to execute an estimation procedure and a first presentation procedure. In the estimation procedure, while a conversation is taking place between the first user and the second user, the evaluation of the second user by the second user is estimated based on at least one of an image captured by a camera and a sound detected by a microphone. Here, the image captured by the camera includes an image of the second user taken by the camera, and the sound detected by the microphone includes the voice of the second user detected by the microphone. Then, in the first presentation procedure, first evaluation information representing the evaluation of the first user by the second user estimated by the estimation procedure is presented to the first user while the conversation is taking place between the first user and the second user, that is, in real time. [Effects of the Invention]

[0013] According to the present invention, when a first user converses with a second user online, the second user can evaluate the first user with high reliability. Furthermore, since the first evaluation information representing the second user's evaluation of the first user is presented to the first user in real time, the first user can experience an objective evaluation of themselves. As a result, various effects can be expected, such as an improvement in the first user's conversational skills. [Brief explanation of the drawing]

[0014] [Figure 1] Figure 1 shows the overall configuration of an online response system according to the first embodiment of the present invention. [Figure 2] Figure 2 is a block diagram showing the electrical configuration of the operator terminal in the first embodiment of the present invention. [Figure 3] Figure 3 is a block diagram showing the electrical configuration of a user terminal in the first embodiment of the present invention. [Figure 4]FIG. 4 is a diagram showing a state in which a user terminal in the first embodiment of the present invention is installed, and a user-side standby screen is displayed on the display device of the user terminal. [Figure 5] FIG. 5 is a block diagram showing the electrical configuration of the management server in the first embodiment of the present invention. [Figure 6] FIG. 6 is a diagram showing an example of an operator-side standby screen in the first embodiment of the present invention. [Figure 7] FIG. 7 is a diagram showing an example of another state of the operator-side standby screen in the first embodiment of the present invention. [Figure 8] FIG. 8 is a diagram showing an example of an operator-side response screen in the first embodiment of the present invention. [Figure 9] FIG. 15 is a diagram showing an example of a state in which a user-side response screen is displayed on the display device of the user terminal, where the user terminal in the first embodiment of the present invention is installed. [Figure 10] FIG. 10 is a diagram showing an example of a response evaluation screen in the first embodiment of the present invention. [Figure 11] FIG. 11 is a diagram conceptually showing the configuration of an operator terminal management table in the first embodiment of the present invention. [Figure 12] FIG. 12 is a diagram conceptually showing the configuration of a user terminal management table in the first embodiment of the present invention. [Figure 13] FIG. 13 is a flowchart showing the flow of a user evaluation task in the first embodiment of the present invention. [Figure 14] FIG. 14 is a flowchart showing the flow of a user evaluation control task in the first embodiment of the present invention. [Figure 15] FIG. 15 is a flowchart showing the flow of a user evaluation presentation task in the first embodiment of the present invention. [Figure 16] FIG. 16 is a diagram showing another example of an operator-side response screen in the first embodiment of the present invention. [Figure 17] FIG. 17 is a diagram showing an example of an operator-side standby screen in the second embodiment of the present invention. [Figure 18]FIG. 18 is a diagram showing an example of a centralized monitoring screen in the second embodiment of the present invention. [Figure 19] FIG. 19 is a diagram showing an example of an operator-side response screen in the second embodiment of the present invention. [Figure 20] FIG. 20 is a diagram showing an example of another state of the operator-side response screen in the second embodiment of the present invention. [Figure 21] FIG. 21 is a flowchart showing a part of the flow of an inter-operator evaluation task in the second embodiment of the present invention. [Figure 22] FIG. 22 is a flowchart showing the remaining part of the flow of an inter-operator evaluation task in the second embodiment of the present invention. [Figure 23] FIG. 23 is a flowchart showing the flow of an inter-operator evaluation control task in the second embodiment of the present invention. [Figure 24] FIG. 24 is a flowchart showing the flow of an inter-operator evaluation presentation task in the second embodiment of the present invention.

MODE FOR CARRYING OUT THE INVENTION

[0015] [First Embodiment] The first embodiment of the present invention will be described by taking the online response system 10 shown in FIG. 1 as an example.

[0016] The online response system 10 according to this first embodiment is applied to, for example, a home improvement store and comprises multiple operator terminals 20, 20, ..., multiple user terminals 30, 30, ..., and one management server 40. Each operator terminal 20 is an example of an information processing device according to the present invention and is used by an operator, who is an example of a first user, such as a staff member of the home improvement store. It may be installed, for example, in an office within the home improvement store, or in a contact center separate from the store, or even in the operator's home. Each user terminal 30 is a counterparty device that can communicate with each operator terminal 20, and is installed in an appropriate location such as each sales area within the home improvement store. It is used by an unspecified number of users, who are an example of a second user, such as a customer of the home improvement store. The management server 40 is, so to speak, a management device for managing each operator terminal 20,20,... and each user terminal 30,30,... It may be installed, for example, in a management room within a home improvement store, or in a management center separate from the store, and used by an authorized person with predetermined privileges, such as a system administrator. Note that the management server 40 may consist of multiple devices (servers) rather than just one.

[0017] Each operator terminal 20, 20, ..., each user terminal 30, 30, ..., and the management server 40 are connected to a common network 50. Network 50 is, for example, a LAN within a home improvement store, but it may also be a WAN that includes the LAN, or it may include the internet. Each operator terminal 20, 20, ..., each user terminal 30, 30, ..., and the management server 40 are assigned individual IP addresses. Furthermore, each operator terminal 20, 20, ... is assigned a terminal ID as individual identification information. Similarly, each user terminal 30, 30, ... is also assigned an individual terminal ID.

[0018] According to this online response system 10, operators using each operator terminal 20 can provide online (remote) support (customer service) to users using each user terminal 30.

[0019] Now, focusing on each operator terminal 20, each operator terminal 20 is, for example, a personal computer and, as shown in Figure 2, has a control unit 202, an input / output interface (I / F) 204, an input device 206, a display device 208, a microphone 210, a speaker 212, a camera 214, an auxiliary storage unit 216, a communication unit 218, and so on. Note that the operator terminal 20 also has various other elements, but in Figure 2, elements not directly related to the present invention are omitted from the illustration.

[0020] The control unit 202 is a control means responsible for controlling the entire operator terminal 20. This control unit 202 has a computer, such as a CPU 202a, as a control execution means. In addition, the control unit 202 has a main memory unit 202b as a main memory means that the CPU 202a can directly access. The main memory unit 202b includes, for example, ROM and RAM. The ROM stores firmware, including the BIOS. The RAM constitutes a work area and buffer area when the CPU 202a executes processing based on various programs contained in the firmware and various software.

[0021] The input / output interface 204 is an interface means that acts as a bridge between the control unit 202, particularly the CPU 202a, and various elements such as the input device 206, and includes, for example, a chipset. Therefore, the control unit 202 is connected to the input / output interface 204, as are various elements such as the input device 206, specifically the input device 206, display device 208, microphone 210, speaker 212, camera 214, auxiliary storage unit 216, communication unit 218, etc.

[0022] The input device 206 is an operation reception means that accepts user operations, specifically operations by an operator who is a user of the operator terminal 20, and includes, for example, a keyboard and a mouse. Note that if the display device 208, described below, is, for example, a touch panel, the display device 208 can also function as the input device 206.

[0023] The display device 208 is an example of a display means that displays various information, such as the operator-side standby screen 500, which will be described in more detail later. This display device 208 is, for example, a flat panel display such as a liquid crystal display or an organic EL display, but is not limited to these. The display device 208 may also be, for example, a touch panel, in which case it can also function as an input device 206 as described above.

[0024] The microphone 210 is a means for detecting (collecting) the voice emitted by the operator, in other words, the operator's voice. Therefore, the microphone 210 is naturally provided in a way that allows for the detection of the operator's voice. Note that a headset integrated with the speaker 212, which will be described next, may be used as the microphone 210.

[0025] Speaker 212 is, so to speak, a user voice output means, primarily for outputting sound corresponding to the user's voice, as described later. This speaker 212 is installed so that the operator can hear the sound output from it. In addition, a headset integrated with the aforementioned microphone 210 may be used as speaker 212.

[0026] Camera 214 is a means of photographing the operator, specifically to obtain moving images of the operator. Therefore, it is natural that camera 214 is installed in a way that allows it to photograph the operator.

[0027] The auxiliary storage unit 216 is an auxiliary storage means, and for example, it has a hard disk drive. The auxiliary storage unit 216 stores the operating system and various software, and in particular the operator-side response software 216a, which is application software for realizing online responses as described later. In addition, the auxiliary storage unit 216 stores various data necessary for online responses, including the operator's voice detected by the microphone 210 and the operator's image captured by the camera 214, and is strictly speaking stored temporarily. The auxiliary storage unit 216 may have rewritable non-volatile memory such as flash memory instead of, or in addition to, the hard disk drive.

[0028] The communication unit 218 is a communication connection means responsible for connecting to the network 50. The connection to the network 50 may be made by wire or by wireless connection.

[0029] Such an operator terminal 20 may be a desktop personal computer or a notebook personal computer. Alternatively, a portable computer such as a tablet or smartphone, specifically a portable computer equipped with a microphone 210, speaker 212, and camera 214, may be used as the operator terminal 20. In this case, the touch panel of the portable computer also functions as an input device 206 and a display device 208. Furthermore, the operator terminal 20 may be an embedded system dedicated to that operator terminal 20.

[0030] Next, focusing on each user terminal 30, the user terminal 30 is, for example, a personal computer, and more particularly a desktop personal computer, and as shown in Figure 3, has a control unit 302, an input / output interface (I / F) 304, an input device 306, a display device 308, a microphone 310, a speaker 312, a camera 314, an auxiliary storage unit 316, a communication unit 318, and so on. Note that the user terminal 30 has various other elements, but in Figure 3, elements not directly related to the present invention are omitted from the illustration.

[0031] The control unit 302 is a control means responsible for controlling the entire user terminal 30. This control unit 302 has a computer, such as a CPU 302a, as a control execution means. In addition, the control unit 302 has a main memory unit 302b as a main memory means that the CPU 302a can directly access. The main memory unit 302b includes, for example, ROM and RAM. The ROM stores firmware, including the BIOS. The RAM constitutes a work area and buffer area when the CPU 302a executes processing based on various programs contained in the firmware and various software.

[0032] The input / output interface 304 is an interface means that acts as a bridge between the control unit 302, particularly the CPU 302a, and various elements such as the input device 306, and includes, for example, a chipset. Therefore, the control unit 302 is connected to the input / output interface 304, as are various elements such as the input device 306, specifically the input device 306, display device 308, microphone 310, speaker 312, camera 314, auxiliary storage unit 316, communication unit 318, etc.

[0033] The input device 306 is an operation reception means that accepts user operations, specifically operations by a specific user such as a manager responsible for managing the user terminal 30, and includes, for example, a keyboard and mouse. The display device 308, which will be described next, also functions as an input device 306, or more precisely, functions as a part of the input device 306. Except for the display device 308 which functions as a part of the input device 306, the input device 306, especially the input device 306 including the keyboard and mouse, is used only at predetermined times such as when the user terminal 30 is started up, when it is powered off, or during maintenance, and is not used at all times.

[0034] The display device 308 is a display means that displays various information, including various screens such as the user-side standby screen 600, which will be described in more detail later. This display device 308 is a relatively large touch panel, and more specifically, it is a component in which a transparent touch sheet is attached to the display surface of a flat panel display such as a liquid crystal display or an organic EL display. Furthermore, as mentioned above, the display device 308 functions as part of the input device 306. This display device 308, which functions as part of the input device 306, accepts operations not only from specific users such as administrators, but also from an unspecified number of users.

[0035] Microphone 310 is a user voice detection means for detecting the voice emitted by the user of the user terminal 30, in other words, the user's voice. Therefore, naturally, microphone 310 is provided in a manner that allows for the detection of user voice, and is provided, for example, on top of the display device 308 as described later. More precisely, microphone 310 detects not only the user's voice (i.e., including the user's voice) but also the sound of the space in which the user is present (the room described later), and more precisely, it detects sounds within a range corresponding to the sound pickup capability of microphone 310.

[0036] Speaker 312 is an operator voice output means for outputting the voice of the operator acting as the operator in charge of online support, or more precisely, for outputting voice corresponding to the operator voice mentioned above. This speaker 312 is installed so that the sound output from the speaker 312 can be heard by the user, and is installed, for example, on top of the display device 308 as described later. Note that the sound output from speaker 312 may be voice processed by a voice changer.

[0037] Camera 314 is a means of capturing images of the user, more specifically, a means of capturing images of the user. Therefore, naturally, camera 314 is installed in a way that allows it to capture images of the user, for example, by being installed on top of the display device 308 as described later. More precisely, camera 314 captures not only the user (i.e., including the user) but also the space in which the user is located, and more precisely, it captures the range corresponding to the field of view of camera 316.

[0038] The auxiliary storage unit 316 is an auxiliary storage means, and for example, it has a hard disk drive. The operating system is stored in this auxiliary storage unit 316, as well as various software, and in particular, the user-side response software 316a, which is application software for realizing online responses as described later, is stored in it. In addition, the auxiliary storage unit 316 stores various data necessary for online responses, including user voice detected by the microphone 310 and user images captured by the camera 314, and is strictly speaking stored temporarily. The auxiliary storage unit 316 may have rewritable non-volatile memory such as flash memory instead of, or in addition to, the hard disk drive.

[0039] The communication unit 318 is a communication connection means responsible for connecting to the network 50. The connection to the network 50 may be made by wire or by wireless connection.

[0040] The user terminal 30, and in particular the display device 308, are installed on a suitable mounting base 350, as shown in Figure 4, and are installed vertically, for example, so that the longitudinal direction of the display surface of the display device 308 is vertical. A microphone 310, a speaker 312, and a camera 314 are provided at a suitable position on the top of the display device 308. In the configuration shown in Figure 4, the microphone 310 is built into the camera 314, but these two may be provided separately (independently). Also, the display device 308 may be installed horizontally, that is, so that the longitudinal direction of its display surface is horizontal, rather than vertically. The main body of the user terminal 30, including the control unit 302, input / output interface 304, auxiliary storage unit 316, and communication unit 318, is housed inside the mounting base 350. In addition, the input device 306, excluding the display device 308 which functions as part of the input device 306, and in particular the keyboard and mouse, are also housed inside the mounting base 350. Incidentally, Figure 4 shows the state in which the user-side standby screen 600 is displayed on the display surface of the display device 308, but the user-side standby screen 600 will be explained in detail later.

[0041] As mentioned above, the user terminal 30 is, for example, a desktop personal computer, but it may also be a computer other than a desktop personal computer, such as a notebook personal computer. Furthermore, the user terminal 30 may be an embedded system dedicated to that user terminal 30.

[0042] Furthermore, focusing on the management server 40, as shown in Figure 5, the management server 40 includes a control unit 402, an input / output interface (I / F) 404, an input device 406, a display device 408, an auxiliary storage unit 410, a communication unit 412, and the like. Note that the management server 40 also has various other elements, but in Figure 5, elements not directly related to the present invention are omitted from the illustration.

[0043] The control unit 402 is a control means responsible for controlling the entire management server 40. This control unit 402 has a computer, such as a CPU 402a, as a control execution means. In addition, the control unit 402 has a main memory unit 402b as a main memory means that the CPU 402a can directly access. Detailed illustrations are omitted, but the main memory unit 402b includes, for example, ROM and RAM. The ROM stores firmware, including the BIOS. The RAM constitutes a work area and buffer area when the CPU 402a executes processing based on various programs contained in the firmware and various software.

[0044] The input / output interface 404 is an interface means that acts as a bridge between the control unit 402, particularly the CPU 402a, and various elements such as the input device 406, and includes, for example, a chipset. Therefore, the control unit 402 is connected to the input / output interface 404, as are various elements such as the input device 406, specifically the input device 406, the display device 408, the auxiliary storage unit 410, the communication unit 412, etc.

[0045] The input device 406 is an operation reception means that accepts user operations, specifically operations by authorized persons with predetermined privileges, such as the system administrator described above, and includes, for example, a keyboard and a mouse. This input device 406 is used only at predetermined times, such as when the management server 40, which includes the input device 406, is installed or maintained, and is not used at all times.

[0046] The display device 408 is a display means for displaying various information, and is, for example, a flat panel display such as a liquid crystal display or an organic EL display, but is not limited to these. Like the input device 406, this display device 408 is also used only at predetermined times, such as when the management server 40 is installed or maintained, and is not used at all times.

[0047] The auxiliary storage unit 410 is an auxiliary storage means, and for example, it has a hard disk drive. The operating system is stored in this auxiliary storage unit 410, as well as various software, and in particular, the response control software 410a, which is application software for realizing online responses as described later, is stored in it. In addition, the auxiliary storage unit 410 stores the response management database 410b, which is a database for realizing online responses. This response management database 410b includes various tables, such as the operator terminal management table 1000 and the user terminal management table 1100, which will be described later.

[0048] The communication unit 412 is a communication connection means responsible for connecting to the network 50. The connection to the network 50 may be made by wire, for example, but it may also be made by wireless means.

[0049] As described above, according to the online response system 10 of this first embodiment, operators using each operator terminal 20 can provide online support to users using each user terminal 30. To do this, operator-side response software 216a is started on each operator terminal 20. Simultaneously, user-side response software 316a is started on each user terminal 30. In addition, response control software 410a is started on the management server 40. This completes the preparation for providing online support.

[0050] Here, the space, such as a sales area, in which each user terminal 30 is installed is referred to as a "room." In this first embodiment, there are M (M: a natural number greater than or equal to 2) rooms, meaning that M user terminals 30, 30, ... are installed, and the value of M is, for example, 10 or greater.

[0051] Furthermore, focusing on each operator terminal 20, and especially on the operator terminals 20 that are not being used for online support and are in a standby state, the display device 208 of these terminals displays an operator-side standby screen 500 as shown in Figure 6. This operator-side standby screen 500 contains N (N; a natural number less than or equal to M) room display areas 502, 502, ... which are 10 rectangular areas corresponding to rooms. Although detailed illustrations are omitted, each room display area 502 displays an image of the user of the room corresponding to that room display area 502. In addition, a string 504 representing the location of the room corresponding to each room display area 502 is placed near each room display area 502, for example, slightly to the left and below the room display area 502.

[0052] Each room display area 502, 502, ... is arranged in a so-called 2x5 configuration, for example, with 2 vertically and 5 horizontally. The order of these room display areas 502, 502, ... is determined, for example, by the individual room number assigned to each room corresponding to each room display area 502, 502, ..., and more specifically, by the order in which the room numbers increase according to the Z-shaped pattern of eye-tracking guidance. Furthermore, if the total number of rooms is greater than 10, that is, if the value of M mentioned above is greater than the value of N, the operator-side standby screen 500 can be scrolled, for example, in the left-right direction, although this is not shown in the illustration. This allows all room display areas 502, 502, ... corresponding to all rooms, in other words, all user images for all rooms, to be displayed on the operator-side standby screen 500. Note that the scrolling direction of the operator-side standby screen 500 may be up and down instead of left and right.

[0053] Upon seeing this operator-side standby screen 500, the operator of the operator terminal 20 in standby mode can intuitively recognize (understand) the status of each room, including whether or not there are users in that room, and can, in a sense, monitor it. In other words, the operator-side standby screen 500 functions as a monitoring screen for monitoring each room.

[0054] When this operator-side standby screen 500 is displayed, the standby operator terminal 20 acquires user images from each room in a time-division multiplexing manner, or more precisely, acquires user images in a time-division multiplexing manner from each user terminal 30, 30, ... installed in each respective room. The standby operator terminal 20 then displays the user images acquired from each room (each user terminal 30, 30, ...) in the corresponding room display areas 502, 502, ..., or more precisely, in the room display areas 502, 502, ... that are displayed on the operator-side standby screen 500.

[0055] Note that the configuration including the design of the operator-side standby screen 500 shown in Figure 6 is just one example and is not limited to the configuration shown in Figure 6. This also applies to other screens displayed on the display device 208 of the operator terminal 20, including the operator-side response screen 700 which will be described later.

[0056] In contrast, focusing on each user terminal 30, and especially on user terminals 30 that are in a standby state and not being used for online support, the display device 308 displays a user-side standby screen 600 as shown in Figure 4. This user-side standby screen 600 appropriately includes a call button (button image) 602, which is an operator for calling a staff member of the home center, and an appropriate string of characters (message) 604 that instructs the user to operate (press) the call button 602 if staff support is required.

[0057] The configuration including the design of the user-side standby screen 600 shown in Figure 4 is also just one example and is not limited to the configuration shown in Figure 4. This also applies to other screens displayed on the display device 308 of the user terminal 30, including the user-side response screen 800 which will be described later.

[0058] In other words, a user who wishes to receive online assistance will operate the call button 602 on the user-side waiting screen 600 when receiving such online assistance. When this call operation is performed, a call notification is sent from the user terminal 30 that performed the call operation to the management server 40. At the same time, the terminal ID of the user terminal 30 that performed the call operation is also sent from the user terminal 30 to the management server 40. Then, on the user terminal 30 that performed the call operation, a call screen (not shown) is displayed on the display device 308 instead of the user-side waiting screen 600. This call screen contains appropriate information, including an appropriate string of characters indicating that staff is being called. Hereafter, the user terminal 30 that performed the call operation may be referred to as the "assistance terminal 30a". However, the code "30a" representing the assistance terminal 30a is not specifically shown.

[0059] When the management server 40 receives a call notification from the call-handling terminal 30a, it sends a call report to each operator terminal 20, 20, ... indicating that it has received the call notification. At the same time, the management server 40 also sends the terminal ID of the call-handling terminal 30a to each operator terminal 20, 20, ...

[0060] Each operator terminal 20 receives a call report and the terminal ID of the response terminal 30a from the management server 40, thereby recognizing that a call operation was performed on the response terminal 30a, that is, that a call operation was performed in the room where the response terminal 30a is installed. Then, in the case of an operator terminal 20 in standby mode, the operator-side standby screen 500 displayed on its display device 208 will be in the form shown in Figure 7.

[0061] In the operator-side standby screen 500 shown in Figure 7, a suitable call mark 506 is placed on the room display area 502 corresponding to the responding terminal 30a among the room display areas 502, 502, ... that is, on the room display area 502 corresponding to the room in which the call operation was performed. In addition, a suitable decoration is applied to the periphery of the room display area 502 to indicate that the call operation has been performed, for example, a red border is added.

[0062] Figure 7 shows an example of the operator's standby screen 500 when a call operation is performed in "Room 5" (Sales Floor 5), which has room number "5". In Figure 7, a thick solid border is added to the periphery of the room display area 502 corresponding to "Room 5", indicating that a red border has been added to the periphery of the room display area 502. In Figure 7, a call mark 506 is placed near the upper left corner of the room display area 502, but the position where the call mark 506 is placed is not limited to this.

[0063] Upon seeing the operator-side standby screen 500 as shown in Figure 7, the operator of the operator terminal 20, which is in a standby state, can intuitively recognize that a call operation has been made in "Room 5," that is, in the room corresponding to the room display area 502 which has a call mark 506 and a red border around its edge. The operator of the operator terminal 20 can then select the room where the call operation was made as the online counterparty by operating the room display area 502 corresponding to the room where the call operation was made. In other words, the room display area 502 corresponding to the room where the call operation was made functions as an operator that can accept an operation to select the room to be the online counterparty, a counterparty selection operation. When any of the room display areas 502 corresponding to the room where the call operation was made is operated, the operator-side response screen 700 as shown in Figure 8 is displayed on the display device 208 instead of the operator-side standby screen 500. Figure 8 shows an example of the operator-side response screen 700 when the room display area 502 corresponding to "Room 5" in the operator-side standby screen 500 shown in Figure 7 is operated.

[0064] As shown in Figure 8, the operator-side response screen 700 includes an enlarged room display area 702, which is a relatively large rectangular area. This enlarged room display area 702 is positioned, for example, towards the upper right of the operator-side response screen 700. The enlarged room display area 702 displays the user image of the room selected as the online counterparty, in this case "Room 5". The user image of "Room 5" includes the image 704 of the user present in "Room 5". An evaluation indicator 706 is also placed at an appropriate position in the enlarged room display area 702, for example, near its lower right corner. This evaluation indicator 706 is overlaid on the user image displayed in the enlarged room display area 702. This evaluation indicator 706 will be explained in detail later. Furthermore, a string 708 representing the location of the room selected as the online counterparty is placed near the enlarged room display area 702, for example, at a position slightly to the left of the bottom of the enlarged room display area 702.

[0065] In addition, at an appropriate location on the operator-side response screen 700, for example to the left of the enlarged room display area 702, P (a natural number less than N), in this case four, relatively small rectangular areas called reduced room display areas 710, 710, ... are arranged in a vertical line. These four reduced room display areas 710, 710, ... correspond to four rooms other than the room selected as the counterparty for online response. Each reduced room display area 710, 710, ... displays an image of the user of the corresponding room. At the same time, a string 712 representing the location of the room corresponding to each reduced room display area 710 is displayed near the reduced room display area 710, for example, slightly to the left and below the reduced room display area 710.

[0066] The order in which each of the reduced room display areas 710, 710, ... is arranged follows, for example, according to the room number of each room corresponding to each reduced room display area 710, 710, ..., specifically in order from top to bottom as the room number increases. Furthermore, if the number of rooms other than the room selected as the online counterparty is greater than 4, that is, if the value obtained by subtracting 1 from M (M-1) is greater than the value of P, although not shown in the diagram, the area of ​​the operator-side response screen 700 where each of the reduced room display areas 710, 710, ... is located can be scrolled vertically. This allows the reduced room display areas 710, 710, ... corresponding to each room other than the room selected as the online counterparty, in other words, the user images of each room other than the room selected as the online counterparty, to be displayed on the operator-side response screen 700.

[0067] In addition, three controls 714, 716, and 718 are arranged horizontally at an appropriate position on the operator-side response screen 700, for example, slightly to the right of the lower part of the enlarged room display area 702. Of these three controls 714, 716, and 718, the leftmost control 714 is a [Back] button that instructs the system to end the display of the operator-side response screen 700 and to display the operator-side standby screen 500 on the display device 208 again, in other words, to return to that state. The central control 716 is a [Start Response] button that instructs the system to start the online response with the room selected as the other party to the online response. Furthermore, the rightmost control 718 is a [End Response] button that instructs the system to end the online response.

[0068] In other words, when the [Back] button 714 is pressed, the operator-side standby screen 500 is displayed on the display device 208 again, replacing the operator-side response screen 700, thus returning to that state. Then, when the [Start Response] button 716 is pressed, the system becomes ready to conduct an online response with the room selected as the other party for the online response. And when the [End Response] button 718 is pressed while an online response is in progress, the online response ends.

[0069] Even on the operator terminal 20, which is displaying the operator-side response screen 700, user images are acquired from each room in a time-division multiplexing manner, or more precisely, from each user terminal 30, 30, ... installed in each of those rooms in a time-division multiplexing manner. Then, the user image acquired from the room selected as the online counterpart (response terminal 30a) among the rooms (each user terminal 30, 30, ...) is displayed in the enlarged room display area 702. At the same time, the user images acquired from each room corresponding to each reduced room display area 710, 710, ... are displayed in each of those reduced room display areas 710, 710, ...

[0070] In particular, when the [Start Response] button 716 on the operator-side response screen 700 is pressed, the operator terminal 20 that pressed the [Start Response] button 716 becomes, so to speak, a response-responsible terminal responsible for online responses. Then, a response start notification is sent from the operator terminal 20 that has become a response-responsible terminal to the management server 40. At the same time, the terminal ID of the operator terminal 20 that has become a response-responsible terminal is sent from the operator terminal 20 that has become a response-responsible terminal to the management server 40. From this point onward, the operator terminal 20 that has become a response-responsible terminal will be described with the designation "20a". However, the designation "20a" will not be specifically illustrated.

[0071] When the management server 40 receives a response start notification from the responding terminal 20a, it sends a response start command to the responding user terminal 30a, which is the counterparty device of the responding terminal 20a. At the same time, the management server 40 also sends the terminal ID of the responding terminal 20a to the responding user terminal 30a.

[0072] As a result, two-way communication is initiated between the responding terminal 20a and the receiving terminal 30a, which is the counterpart device of the responding terminal 20a. For example, one-to-one two-way communication using the P2P (Peer to Peer) method is initiated. In order to realize this two-way communication using the P2P method, appropriate communication programs are incorporated into the operator-side responding software 216a of each operator terminal 20, 20, ... including the responding terminal 20a. In addition, similar communication programs are incorporated into the user-side responding software 316a of each user terminal 30, 30, ... including the receiving terminal 30a. Furthermore, in order to realize two-way communication using the P2P method, appropriate elements such as a signaling server, STUN server, and TURN server (not shown) are provided, but since these are publicly known, detailed explanations of each are omitted.

[0073] In this P2P-based two-way communication, the user's image and voice from the user terminal 30a are transmitted from the user terminal 30a to the operator terminal 20a. On the other hand, the avatar data generated by the operator terminal 20a and the operator's voice are transmitted from the operator terminal 20a to the user terminal 30a. The avatar data is the data that forms the basis of the avatar 800a, which will be described later, acting as a doppelganger (proxy) of the operator of the operator terminal 20a, or the operator. This avatar data is generated based on the operator image of the operator terminal 20a, and movements corresponding to the operator image, that is, movements corresponding to the operator, are given to the avatar 800a. For this reason, when generating the avatar data, a known motion detection process (including processing by machine learning such as deep learning) is applied to the operator image, and the results of this motion detection process are reflected in the avatar data.

[0074] Now, focusing on the response terminal 20a, the operator-side response screen 700 continues to be displayed on the display device 208. The user image of the response user terminal 30a is then displayed in the enlarged room display area 702 of the operator-side response screen 700. In addition, although not visible in the figures including Figure 8, the speaker 212 of the response terminal 20a outputs audio corresponding to the user voice of the response user terminal 30a, that is, the voice of the user of the response user terminal 30a, in other words, the voice of the response user.

[0075] As mentioned above, one-to-one bidirectional communication is conducted between the responding terminal 20a and the responding user terminal 30a using the P2P method. However, between the responding terminal 20a and each user terminal 30, 30, ... other than the responding user terminal 30a, each user image is continuously transmitted to the responding terminal 20a using time-division multiplexing. Then, the user images of each user terminal 30, 30, ... corresponding to each reduced room display area 710, 710, ... from the user images received from each user terminal 30, 30, ... other than the responding user terminal 30a are displayed in the respective reduced room display areas 710, 710, ...

[0076] On the other hand, focusing on the call handling terminal 30a, instead of the aforementioned calling screen, the user-side call handling screen 800, as shown in Figure 9, is displayed on the display device 308. This user-side call handling screen 800 includes an avatar 800a. As mentioned above, the avatar 800a is a representation of the call handling staff based on avatar data and moves in accordance with the call handling staff. In addition, although not visible in the figures including Figure 9, the speaker 312 of the call handling terminal 30a outputs audio corresponding to the operator's voice on the call handling terminal 20a, that is, the voice of the call handling staff. Note that instead of the avatar 800a, an image of the operator, which is a photograph of the call handling staff, may be displayed.

[0077] Therefore, the operator will conduct an online conversation with the user via the operator-side interaction screen 700. At that time, the operator can visually recognize the user from the user image, including the user's image 704, displayed in the enlarged room display area 702. In addition, the operator can recognize the user's voice from the sound output by the speaker 212. Furthermore, in addition to the user image, the enlarged room display area 702 of the operator-side interaction screen 700 may also display a slightly smaller avatar, similar to that on the user-side interaction screen 800, that is, an avatar representing the operator themselves, for example, by overlaying it on a part of the user image.

[0078] The user then receives online support from the support staff via the user-side support screen 800. At this time, the user can recognize the movements of the support staff from the movements of the avatar 800a. In addition, the user can recognize the voice of the support staff from the sound output from the speaker 312. As mentioned above, the sound output from the speaker 312 may be processed by a voice changer. The voice changer is installed, for example, on the user support terminal 30a (user terminal 30), but it may also be installed on the support staff terminal 20a (operator terminal 20).

[0079] Incidentally, the online response system 10 according to this first embodiment is equipped with a real-time evaluation function. According to this real-time evaluation function, when a response staff member is providing online support to a user, the user provides an evaluation (of satisfaction) of the response staff member, and the evaluation result is presented to the response staff member in real time. Moreover, according to this real-time evaluation function, the user's evaluation of the response staff member is estimated based on the user's image and voice during the online response. In other words, the user's evaluation of the response staff member is not done by methods that the user consciously performs, such as questionnaires or written comments, but is done without the user's awareness, so to speak, internally (secretly). This ensures a highly reliable evaluation. Furthermore, since the evaluation result is presented to the response staff member in real time, the response staff member can experience an objective evaluation of themselves. From these points, various effects can be expected, such as improvement in the response staff member's response skills.

[0080] To explain this real-time evaluation function in more detail, as mentioned above, this real-time evaluation function estimates the evaluation of the user of the interaction with the service provider based on the user's image and voice while the interaction is taking place online. This estimation is performed by the user's terminal 30a.

[0081] For example, if an online interaction is taking place and the user's evaluation of the service provider is to be estimated based on the user's image, the evaluation will be estimated based on the user's facial expression recognized from the user's image, as well as the user's actions, which are separate from the facial expression.

[0082] In particular, when estimating a user's evaluation of the service provider based on their facial expression, the user's emotions are detected (inferred) from their facial expression. Here, the user's emotions are classified into five categories, such as joy, fear, sadness, disgust, and anger. If the detected emotion is joy, for example, it is estimated that the user has a high evaluation of the service provider, which is a positive evaluation. Conversely, if the detected emotion is anything other than joy, that is, fear, sadness, disgust, or anger, it is estimated that the user has a low evaluation of the service provider, which is a negative evaluation.

[0083] Furthermore, the detection of the emotions of the user being interacted with, based on their facial expressions, is achieved using publicly known technologies. While a detailed explanation is omitted here, the publicly known technologies mentioned here include those disclosed in publications such as: "Hiroshi Kobayashi, Fumio Hara: Recognition of basic human facial expressions using neural networks, Transactions of the Society of Instrument and Control Engineers, Vol. 29, No. 1, 112 / 118 (1993)", "Yosuke Kotani, Tsuneyasu Honma, Masao Sakai, Kenichi Abe: Facial expression recognition using neural networks, Bulletin of the Faculty of Medicine and Health Sciences, Tohoku University 13(1):23~32, 2004", and "Daiki Nishime, Satoshi Endo, Yoshiaki Toma, Koji Yamada, Yuhei Akamine: Acquisition of facial expressions and analysis of facial features using convolutional neural networks, Transactions of the Japanese Society for Artificial Intelligence, Vol. 32, No. 5, FZ (2017)". These can be used as appropriate. In addition, for example, the technology disclosed in Japanese Patent Publication No. 2020-163660 can also be used. In any of the technologies used, it is preferable to use a neutral state, where the user's facial expression is not particularly expressive, as the baseline, and to detect the user's emotions based on changes in their facial expression from this baseline state. Then, based on the detected emotions of the user, their evaluation of the service provider is estimated.

[0084] Furthermore, when estimating a user's evaluation of the service provider based on the user's actions (gestures) separate from their facial expressions, known techniques are used to detect the user's actions, for example, by applying known motion detection processing (including machine learning processing such as deep learning) to the user's image. Here, the user's actions include movements of the user's neck, the direction of their face, bowing, waving, and other actions that express the user's emotions.

[0085] For example, regarding the head movements of the person being answered, if the head movement is vertical, that is, if the person nods, and especially if the number of times the person nods exceeds a predetermined threshold number of nods, it is presumed that the person being answered has a high evaluation of the person answering the call, i.e., a positive evaluation. On the other hand, if the head movement is horizontal, that is, if the person shaking their head from side to side, and especially if the number of times the person shaking their head from side to side exceeds a predetermined threshold number of side-to-side head shakes, it is presumed that the person being answered has a low evaluation of the person answering the call, i.e., a negative evaluation.

[0086] Furthermore, regarding the orientation of the user's face, if the face is facing camera 314, and especially if this state continues for a predetermined attention threshold time, it is presumed that the user has a high evaluation of the person handling the interaction, i.e., a positive evaluation. On the other hand, if the user's face is turned away from camera 314, and especially if this state continues for a predetermined non-attention threshold time (second threshold time), it is presumed that the user has a low evaluation of the person handling the interaction, i.e., a negative evaluation.

[0087] Furthermore, regarding bowing, if the user bows, especially if the number of bows exceeds a predetermined threshold, it is presumed that the user has a high evaluation of the service provider, i.e., a positive evaluation. If the user does not bow, no special evaluation is given.

[0088] Furthermore, regarding waving motions, for example, if the waving motion involves waving the hand horizontally with the palm facing the camera 314, and especially if the number of such wavings exceeds a predetermined threshold number of waving motions, a positive evaluation will be given. On the other hand, if the waving motion involves waving the hand horizontally without facing the camera 314, or if it is a motion like shaking something off, and especially if the number of such wavings exceeds a predetermined threshold number of shaking motions, a negative evaluation will be given.

[0089] In addition, when an online interaction is taking place, if the user's evaluation of the service provider can be estimated based on the user's voice, the evaluation will be estimated based on the tone of the user's voice (pitch, intonation, volume, speaking speed, tempo, etc.), the words included in the user's voice, and the sounds resulting from the user's actions that can be recognized from the user's voice.

[0090] For example, when estimating a user's evaluation of a service provider based on the tone of their voice, the user's emotions are detected from the tone of their voice. In this detection of emotions based on the tone of the user's voice, the emotions are classified into five categories: joy, fear, sadness, disgust, and anger. If the detected emotion is joy, for example, it is estimated that the user has a high evaluation of the service provider, i.e., a positive evaluation. Conversely, if the detected emotion is anything other than joy, i.e., fear, sadness, disgust, or anger, it is estimated that the user has a low evaluation of the service provider, i.e., a negative evaluation.

[0091] Furthermore, the detection of emotions based on the tone of the user's voice is achieved using publicly known technologies. While a detailed explanation is omitted here, the publicly known technologies referred to here include those disclosed in "Daiki Mori: Understanding Emotions and Attitudes from Voice, IEICE Journal Vol. 101, No. 9, 2018" and Japanese Patent Publication No. 2021-12285, and these can be used as appropriate. In any case, it is preferable that the tone of the user's voice at the beginning of the online interaction is used as a reference, and the emotions of the responding user are detected based on the change in the tone of the user's voice from this reference state. Then, based on the detected emotions of the responding user, the user's evaluation of the responding staff member is estimated.

[0092] Furthermore, when estimating a user's evaluation of a service provider based on words contained in the user's voice, the user's emotions are detected using publicly known technology, such as a large-scale language model, and the evaluation is then estimated. For example, if a user's voice contains predetermined words expressing gratitude to the service provider, such as "thank you" or "that was helpful," and especially if the number of times such words appear exceeds a predetermined number of occurrences (the first occurrence count), the user's evaluation of the service provider is estimated to be high, i.e., a positive evaluation. On the other hand, if a user's voice contains predetermined words that imply reprimanding the service provider, such as "I don't understand what you're saying" or "explain it properly," and especially if the number of times such words appear exceeds a predetermined number of occurrences (the second occurrence count), the user's evaluation of the service provider is estimated to be low, i.e., a negative evaluation.

[0093] Furthermore, if an evaluation of the customer service representative can be estimated based on the sounds of the customer service representative's actions recognized from the customer's voice, an evaluation is given according to the type of sound. For example, if the customer service representative's sound is clapping, a positive evaluation is given. On the other hand, if the customer service representative's sound is an abnormal sound such as tapping on the customer service terminal 30a, a negative evaluation is given. The detection of these sounds is achieved using known technology. Examples of such technologies include those disclosed in "Kazuhiro Ouchi, Kazuhiko Kawahara, Yu Kamamoto, Takashi Sato, Akira Omoto, and Takehiro Moriya: Proposal of a clapping sound detection method for improving clapping detection performance in APRICOT systems, IEICE Technical Report, IEICE Technical Report Vol. 119, No. 334, 2019", Japanese Patent Publication No. 2012-058340, Japanese Patent Publication No. 2013-229657, and Japanese Patent Publication No. 2014-055783.

[0094] Furthermore, an evaluation of the customer service representative by the customer service representative may be estimated based on the combination of the customer service representative's image and voice. For example, if the customer service representative's voice contains the word "I see," and the evaluation based on the customer service representative's image is positive, a positive evaluation will be derived from this combination. On the other hand, even if the customer service representative's voice contains the word "I see," if the evaluation based on the customer service representative's image is negative, no evaluation will be derived from this combination. Also, for example, if the customer service representative's voice is not included in the voice recording (there is no utterance by the customer service representative), and the evaluation based on the customer service representative's image is positive, no evaluation will be derived from this combination. Conversely, if the customer service representative's voice is not included in the voice recording, and the evaluation based on the customer service representative's image is negative, a negative evaluation will be derived from this combination. This is just one example, but an evaluation of the customer service representative by the customer service representative may be estimated based on the combination of the customer service representative's image and voice, in an appropriate manner following this example.

[0095] In this way, the user-operated terminal 30a estimates the user's evaluation of the service provider. The user-operated service provider software 316a incorporates an estimation program to realize this estimation. An evaluation estimation result notification, which represents the estimation result from the user-operated terminal 30a—that is, a notification indicating, for example, whether a positive or negative evaluation was given—is sent from the user-operated terminal 30a to the management server 40. At the same time, the terminal ID of the user-operated terminal 30a is also sent from the user-operated terminal 30a to the management server 40.

[0096] When the management server 40 receives an evaluation estimation result notification from the user terminal 30a, it derives a user evaluation value representing the user's evaluation of the service provider based on the evaluation estimation result notification. Specifically, for example, if the evaluation estimation result notification represents a positive evaluation, a user evaluation value of "+1" is derived, and if the evaluation estimation result notification represents a negative evaluation, a user evaluation value of "-1" is derived. The management server 40 then sends a user evaluation notification containing the derived user evaluation value to the service provider terminal 20a. The service provider terminal 20a to which this user evaluation notification is sent, that is, the service provider terminal 20a which is the counterparty device of the user terminal 30a, is identified based on the terminal ID of the user terminal 30a.

[0097] When the customer service terminal 20a receives a user evaluation notification from the management server 40, it updates the display of the evaluation indicator 706 based on the user evaluation value included in the user evaluation notification.

[0098] Referring again to Figure 8, the evaluation indicator 706 includes five roughly rectangular indicators 706a, 706a, ... which resemble lamps. These indicators 706a, 706a, ... are arranged, for example, in a single vertical column. Each indicator 706a, 706a, ... is assigned a unique number from "1" to "5" in order from the bottom to the top. This number represents the level of evaluation given by the customer to the service provider, with a higher number indicating a higher evaluation.

[0099] When the operator-side response screen 700, which includes such an evaluation indicator 706, is first displayed, that is, when online support is initiated, only the bottom two of the indicator elements 706a, 706a, ... of the evaluation indicator 706 are lit, while the top three are off. In other words, when online support is initiated, the user's evaluation of the support staff is at level "2". This level "2" represents "average". Incidentally, a level "1" represents "below average". A level "3" represents "good", a level "4" represents "very good", and a level "5" represents "excellent".

[0100] For example, if the user evaluation value included in the user evaluation notification received from the management server 40 is "+1", the evaluation level of the responding user towards the responding staff is increased by "+1", meaning it goes up by one level. On the other hand, if the user evaluation value is "-1", the evaluation level of the responding user towards the responding staff is decreased by "-1", meaning it goes down by one level. Based on the result, the display of the evaluation indicator 706 is updated, that is, the display (on / off) of each indicator 706a, 706a, ... is updated. The maximum evaluation level is "5", and the evaluation level will not go above this maximum. The minimum evaluation level is "1", and the evaluation level will not go below this minimum.

[0101] Furthermore, the evaluation indicator 706 includes a string 706b that represents the evaluation level in text. This string 706b is placed above each indicator 706a, 706a, ... For example, if the evaluation level is "1", the string 706b will be "Poor", which means "below average" in English. If the evaluation level is "2", the string 706b will be "Average", which means "average" in English. Furthermore, if the evaluation level is "3", the string 706b will be "Good!", which means "good" in English. If the evaluation level is "4", the string 706b will be "Very Good!", which means "very average" in English. If the evaluation level is "5", the string 706b will be "Excellent!", which means "great" in English. As can be inferred from this, if the evaluation level is "3" or higher, an exclamation mark "!" is added to the end of the string 706b. Figure 8 shows an example of the operator-side response screen 700 when the evaluation level is "4".

[0102] In response to the display of such evaluation indicators 706, that is, from the display patterns of each indicator 706a, 706a, ..., the customer service representative can perceive an objective evaluation of themselves by the customer service representative. Moreover, the customer service representative's evaluation of the representative is estimated based on the customer's image and voice during the online interaction. In other words, the customer service representative's evaluation of the representative is not conducted through methods that the customer service representative consciously engages in, such as surveys or comments, but rather unconsciously, so to speak, internally. This ensures a highly reliable evaluation. As a result, various effects can be expected, such as an improvement in the customer service representative's ability to handle customer interactions.

[0103] Then, while an online interaction is taking place, the [End Interaction] button 718 on the operator-side interaction screen 700 is pressed, and the online interaction ends. At that time, an interaction evaluation screen 900, as shown in Figure 10, is displayed on the display device 208. The interaction evaluation screen 900 is, for example, a modal dialog and is displayed in the center of the operator-side interaction screen 700 so as to cover a part of it. Alternatively, the interaction evaluation screen 900 may be displayed so as to cover the entire operator-side interaction screen 700, that is, it may be displayed in place of the operator-side interaction screen 700.

[0104] As shown in Figure 10, the response evaluation screen 900 includes a string 902 representing the user's overall evaluation of the response staff member. This overall evaluation follows, for example, the final display mode of the evaluation indicator 706, that is, the display mode of the evaluation indicator 706 immediately before the [End Response] button 718 on the operator-side response screen 700 is pressed. Near the string 902, for example below the string 902, is a string 904 representing a supplementary message (comment) corresponding to the content of the overall evaluation expressed by the string 902. Furthermore, an [OK] button 906 is placed at an appropriate position on the response evaluation screen 900, for example below the string 904. This [OK] button 906 is an operator used to indicate that the response staff member has recognized the content expressed by strings 902 and 904, in other words, to instruct the response staff member to end the display of the response evaluation screen 900.

[0105] In other words, upon displaying the response evaluation screen 900, the response staff member can intuitively recognize the overall evaluation of their response by the user. The response staff member can then close the response evaluation screen 900 by pressing the [OK] button 906 on the screen.

[0106] Figure 10 shows an example of the customer service evaluation screen 900 when the overall evaluation of the customer service representative by the customer is level "4". In this example, the content of the string 904 is "Keep up the good work!". Incidentally, if the overall evaluation of the customer service representative by the customer is level "1", the content of the string 904 will be, for example, "If you are unsure, consult with your leader!". If the overall evaluation of the customer service representative by the customer is level "2", the content of the string 904 will be, for example, "Let's try to improve a little more!". If the overall evaluation is "3", the content of the string 904 will be, for example, "Keep it up, let's strive for further improvement!". And if the overall evaluation of the customer service representative by the customer is level "5", the content of the string 904 will be, for example, "Excellent, perfect!".

[0107] When the [OK] button 906 on the response evaluation screen 900 is pressed, the display of the response evaluation screen 900 ends. At the same time, the aforementioned operator-side standby screen 500 is displayed again on the display device 208 in place of the operator-side response screen 700. Furthermore, a response completion notification is sent from the response terminal 20a to the management server 40. At the same time, the terminal ID of the response terminal 20a is attached to the management server 40.

[0108] When the management server 40 receives a call completion notification from the call handling terminal 20a, it sends a call completion command to the call user terminal 30a, which is the counterparty device of the call handling terminal 20a. The call user terminal 30a to which this call completion notification is sent, that is, the call user terminal 30a which is the counterparty device of the call handling terminal 20a, is identified based on the terminal ID of the call handling terminal 20a that is received along with the call completion notification.

[0109] As a result, the P2P-based bidirectional communication between the response terminal 20a and the response user terminal 30a ends, meaning the online response is terminated. Then, on the response user terminal 30a, the aforementioned user-side standby screen 600 is displayed on the display device 308 again, replacing the user-side response screen 800.

[0110] Furthermore, in the aforementioned operator-side standby screen 500, room display areas 502 corresponding to rooms other than the room from which the call operation was performed can also be operated, meaning they function as operators. For example, if any room display area 502 corresponding to a room that is in a standby state, where no call operation has been performed and no online response has been made, is operated, the room corresponding to the operated room display area 502 will be selected as the online response partner. Then, the operator-side response screen 700 is displayed on the display device 208 instead of the operator-side standby screen 500.

[0111] Furthermore, on the operator-side standby screen 500, a room display area 502 corresponding to a room that is currently receiving an online call is marked with a call-in mark 510 (see Figure 17), which will be described later. In addition, appropriate decorations are applied to the periphery of the room display area 502, for example, a green border is added. When a room display area 502 corresponding to any of the rooms currently receiving an online call is operated, the centralized monitoring screen 1200, which will be described later, is displayed on the display device 208 instead of the operator-side standby screen 500. These configurations will be described in detail later as a different embodiment (second embodiment) from this first embodiment.

[0112] Furthermore, to provide additional information about the operator-side response screen 700, when the operator-side response screen 700 is first displayed, that is, before online response is initiated, the [End Response] button 718 is in a state where it cannot be operated, for example, it is grayed out. On the other hand, when online response is in progress, the [Back] button 714 and the [Start Response] button 716 are in a state where they cannot be operated, for example, they are grayed out.

[0113] Furthermore, in the operator-side response screen 700 before an online response is made, each of the reduced room display areas 710, 710, ... on the operator-side response screen 700 can also accept operations, that is, it functions as an operator. For example, if any of the reduced room display areas 710 corresponding to a room that is not currently receiving an online response is operated, the room corresponding to the operated reduced room display area 710 is selected as the online response partner, and the partner is changed. Then, the user image of the changed partner is displayed in the enlarged room display area 702, replacing the user image that was previously displayed in the enlarged room display area 702. At the same time, the user images displayed in each of the reduced room display areas 710, 710, ... are changed (organized) as appropriate. In other words, each of the reduced room display areas 710, 710, ... on the operator-side response screen 700 before an online response is made functions as an operator that can accept a partner change operation to change the room that will be the online response partner.

[0114] In addition, although not shown in the diagram, the reduced room display area 710 corresponding to the room on which the call operation was performed is marked with a call mark similar to that on the operator-side standby screen 500 shown in Figure 7. Furthermore, the periphery of the reduced room display area 710 is decorated in the same way as on the operator-side standby screen 500 shown in Figure 7, for example, by a red border. Therefore, if any of the reduced room display areas 710 corresponding to the room on which the call operation was performed is operated on the operator-side response screen 700 before the online response is performed, the room corresponding to the operated reduced room display area 710 is selected as the online counterparty, meaning the counterparty is changed.

[0115] Furthermore, although not shown in Figures 8 and 10, the reduced room display area 710 of the operator-side response screen 700 that corresponds to a room currently receiving an online response is marked with a response mark 720 (see Figures 19 and 20), which will be described later, indicating that the room is currently receiving an online response. In addition, the periphery of the reduced room display area 710 is appropriately decorated, for example, by adding a green border. When the reduced room display area 710 corresponding to any room currently receiving an online response is operated on the operator-side response screen 700 before an online response is made, the centralized monitoring screen 1200, which will be described later, is displayed on the display device 208 instead of the operator-side response screen 700. These configurations will also be described in detail later as a different embodiment (second embodiment) from this first embodiment.

[0116] In the online response system 10 according to this first embodiment, the management server 40 has an operator terminal management table 1000 as shown in Figure 11 in order to realize online responses. This operator terminal management table 1000 is a table for managing each operator terminal 20, 20, ... and is included in the aforementioned response management database 410b (see Figure 5).

[0117] The operator terminal management table 1000 stores information representing each operator terminal 20, such as its IP address, terminal ID, operator ID (which is the identification information of the operator using the operator terminal 20), and the status of the operator terminal 20, using the management number in the operator terminal management table 1000 as the key. In particular, there are three states for the status of the operator terminal 20: "standby," "responding," and "stopped." "Standby" indicates that the operator terminal 20 is in the standby state described above. "Responding" indicates that the operator terminal 20 is in an online response state, that is, that the operator terminal 20 is operating as a response-responding terminal 20a. Furthermore, when the operator terminal 20 is in the "responding" state, the terminal ID of the response-using terminal 30a as the other party's device is stored along with the information representing the status of the operator terminal 20. Furthermore, "Stopped" indicates that the operator terminal 20 is in a state where it cannot respond online, for example, by being powered off or by the aforementioned operator-side response software 216a not being started.

[0118] In addition, the management server 40 has a user terminal management table 1100 as shown in Figure 12. This user terminal management table 1100 is a table for managing each user terminal 30, 30, ... and this user terminal management table 1100 is also included in the response management database 410b.

[0119] The user terminal management table 1100 stores information representing each user terminal 30, such as its IP address, terminal ID, location, and status, using the management number in the user terminal management table 1100 as the key. In particular, there are four statuses for the user terminal 30: "Standby," "Calling," "Responding," and "Stopped." "Standby" indicates that the user terminal 30 is in the standby state described above. "Calling" indicates that the user terminal 30 has performed the call operation described above (more precisely, it has sent a call notification to the management server 40 and is waiting to receive a response start command from the management server 40). "Responding" indicates that the user terminal 30 is receiving an online response, that is, the user terminal 30 is operating as a response user terminal 30a. Furthermore, when the user terminal 30 is in the "handling" state, the terminal ID of the responding terminal 20a, which is the other party's device, is stored along with the information indicating the status of the user terminal 30. "Stopped" indicates that the user terminal 30 is unable to perform online responses, for example, if the user terminal 30 is powered off or if the aforementioned user-side response software 316a is not running. In addition, the room number of the installation location of the user terminal 30 is stored along with the installation location (text information).

[0120] Furthermore, in order to enable online responses, the management server 40, or more precisely, the CPU 402a of the management server 40, executes response control tasks according to the response control program included in the response control software 410a. Simultaneously, each operator terminal 20, or more precisely, the CPU 202a of the operator terminal 20, executes operator-side response tasks according to the operator-side response program included in the operator-side response software 216a. And each user terminal 30, or more precisely, the CPU 302a of the user terminal 30, executes user-side response tasks according to the user-side response program included in the user-side response software 316a. This enables online responses. Note that the response control tasks, operator-side response tasks, and user-side response tasks are not directly related to the essence of the present invention, so detailed explanations of each are omitted.

[0121] In addition, to realize the real-time evaluation function, each user terminal 30, or more precisely, the CPU 302a of the response user terminal 30a, executes a user evaluation task according to the user evaluation program included in the user-side response software 316a. The flow of this user evaluation task is shown in Figure 13. The user evaluation task is executed in response to the start of an online response. When an online response is started, the user-side response screen 800 (see Figure 9) is displayed on the display device 308 of the response user terminal 30a, replacing the user-side standby screen 600 (see Figure 4) that was previously displayed on the display device 308.

[0122] In addition, the CPU 302a of the user-facing terminal 30a executes an evaluation event detection task according to the evaluation event detection program included in the user-side user-facing software 316a. This evaluation event detection task is for detecting events that constitute elements of the user's evaluation of the service provider, in other words, evaluation events, based on the user's user image and user voice. These evaluation events include the user's facial expression (and the emotions based on it), the user's neck movements, and the orientation of the user's face, all obtained from the user's user image. The tone of the user's voice (and the emotions based on it) and the words contained in the user's voice are also included in the evaluation events. The procedure for detecting these evaluation events is as described above. Therefore, a detailed explanation of the evaluation event detection task will be omitted.

[0123] As shown in Figure 13, according to the user evaluation task, the CPU 302a of the user terminal 30a first determines in step S1 whether any evaluation event has been detected by the evaluation event detection task. If any evaluation event is detected (S1: YES), the CPU 302a proceeds to step S3. On the other hand, if no evaluation event is detected (S1: NO), the CPU 302a proceeds to step S7, which will be described later.

[0124] In step S3, the CPU 302a estimates the evaluation given by the user to the service provider based on the evaluation event detected in step S1, that is, whether a positive or negative evaluation was given. Then, the CPU 302a proceeds to step S5.

[0125] In step S5, CPU 302a sends an evaluation estimation result notification, which represents the estimation result from step S3, to the management server 40. At the same time, CPU 302a sends its own terminal ID (the user terminal 30a) to the management server 40. Then, CPU 302a proceeds to step S7.

[0126] In step S7, the CPU 302a determines whether the online interaction has been completed. If the online interaction has not been completed (S7: NO), the CPU 302a returns to step S1. On the other hand, if the online interaction has been completed (S7: YES), the CPU 302a terminates the user evaluation task. Once the online interaction is completed, the user-side waiting screen 600 is displayed on the display device 308 instead of the user-side interaction screen 800.

[0127] Furthermore, in order to realize the real-time evaluation function, the management server 40, or more precisely, the CPU 402a of the management server 40, executes a user evaluation control task according to the user evaluation control program included in the response control software 410a. The flow of this user evaluation control task is shown in Figure 14. The user evaluation control task is executed in response to the start of an online response. The user evaluation control task is also executed in correspondence for each pair of the response terminal 20a that performs the online response and the response user terminal 30a that receives the online response. Furthermore, when an online response is started, the status of the response terminal 20a in the operator terminal management table 1000 (see Figure 11) transitions from "standby" to "responding," and at the same time, the status of the response user terminal 30a in the user terminal management table 1100 (see Figure 12) transitions from "calling" to "responding" (after transitioning from "standby" to "calling" in response to receiving a call notification).

[0128] As shown in Figure 14, according to the user evaluation control task, the CPU 402a of the management server 40 first determines in step S101 whether it has received an evaluation estimation result notification from the user terminal 30a. If an evaluation estimation result notification has been received (S101: YES), the CPU 402a proceeds to step S103. On the other hand, if an evaluation estimation result notification has not been received (S101: NO), the CPU 402a proceeds to step S107, which will be described later. When an evaluation estimation result notification is received from the user terminal 30a, the terminal ID of the user terminal 30a is also received.

[0129] In step S103, CPU 402a derives a user evaluation value representing the user's evaluation of the service provider based on the evaluation estimation result notification received in step S101. For example, if the evaluation estimation result notification represents a positive evaluation, CPU 402a derives a user evaluation value of "+1". On the other hand, if the evaluation estimation result notification represents a negative evaluation, CPU 402a derives a user evaluation value of "-1". Then, CPU 402a proceeds to step S105.

[0130] In step S105, the CPU 402a sends a user evaluation notification, including the user evaluation value derived in step S105, to the customer service terminal 20a. The customer service terminal 20a to which this user evaluation notification is sent, that is, the customer service terminal 20a which is the counterparty device of the customer service terminal 30a, is identified based on the terminal ID of the customer service terminal 30a, which was received in step S101 along with the evaluation estimation result notification. Then, the CPU 402a proceeds to step S107.

[0131] In step S107, CPU 402a determines whether the online interaction has been completed. If the online interaction has not been completed (S107: NO), CPU 402a returns to step S101. On the other hand, if the online interaction has been completed (S107: YES), CPU 402a terminates the user evaluation control task. When the online interaction is completed, the status of the interaction terminal 20a in the operator terminal management table 1000 changes from "in action" to "standing," and the status of the interaction user terminal 30a in the user terminal management table 1100 also changes from "in action" to "standing."

[0132] Furthermore, in order to realize the real-time evaluation function, each operator terminal 20, or more precisely, the CPU 202a of the response terminal 20a, executes a user evaluation presentation task according to the user evaluation presentation program included in the operator-side response software 216a. The flow of this user evaluation presentation task is shown in Figure 15. Note that the user evaluation presentation task is executed in response to the start of an online response.

[0133] According to this user evaluation presentation task, the CPU 202a of the customer service terminal 20a first determines in step S201 whether it has received a user evaluation notification from the management server 40. If it has received a user evaluation notification (S201: YES), the CPU 202a proceeds to step S203. On the other hand, if it has not received a user evaluation notification (S201: NO), the CPU 202a proceeds to step S205, which will be described later.

[0134] In step S203, CPU 202a updates the display of the evaluation indicator 706 on the operator-side response screen 700 (see Figure 8) based on the user evaluation value included in the user evaluation notification received in step S201. For example, if the user evaluation value is "+1", CPU 202a increases the evaluation level represented by the evaluation indicator 706 by "+1", that is, updates the display of the evaluation indicator 706 accordingly. On the other hand, if the user evaluation value is "-1", CPU 202a decreases the evaluation level represented by the evaluation indicator 706 by "-1", that is, updates the display of the evaluation indicator 706 accordingly. Initially, when the user evaluation presentation task is executed, the evaluation level represented by the evaluation indicator 706 is "2", meaning the evaluation indicator 706 is in that state. After executing step S203, CPU 202a proceeds to step S205.

[0135] In step S205, CPU202a determines whether the online interaction has been completed. If the online interaction has not been completed (S205: NO), CPU202a returns to step S201. On the other hand, if the online interaction has been completed (S207: YES), CPU202a proceeds to step S207.

[0136] In step S207, the CPU 202a displays the response evaluation screen 900 (see Figure 10) on the display device 208, specifically displaying the response evaluation screen 900 so as to cover a portion of the operator-side response screen 700. Then, the CPU 202a proceeds to step S209.

[0137] In step S209, CPU 202a waits for the [OK] button 906 on the response evaluation screen 900 to be pressed (S209: NO). When the [OK] button 906 on the response evaluation screen 900 is pressed (S209: YES), CPU 202a proceeds to step S211.

[0138] In step S211, the CPU 202a terminates the display of the response evaluation screen 900. With this, the CPU 202a terminates the user evaluation presentation task. Subsequently, the operator-side standby screen 500 (see Figures 6 and 7) is displayed on the display device 208 again, replacing the operator-side response screen 700.

[0139] As described above, according to the online response system 10 of this first embodiment, particularly the real-time evaluation function, when a response staff member is providing online support to a user, the user provides an evaluation of the response staff member, and the results of this evaluation are presented to the response staff member in real time by the evaluation indicator 706. Moreover, according to this real-time evaluation function, the user's evaluation of the response staff member is estimated based on the user's image and voice during the online response. In other words, the user's evaluation of the response staff member is not done by methods that the user consciously performs, such as questionnaires or comments, but rather is done unconsciously, so to speak, internally. This ensures a highly reliable evaluation. Furthermore, since the evaluation results are presented to the response staff member in real time by the evaluation indicator 706, the response staff member can experience an objective evaluation of themselves, or in other words, can get an overview of their own response abilities. From these points, various effects can be expected, such as improvement in the response abilities of the response staff member.

[0140] As mentioned above, the configuration, including the design of each screen displayed on the display device 208 of the operator terminal 20, is not limited to that described in this first embodiment. In particular, the operator-side response screen 700, and especially the evaluation indicator 706, may be configured as shown in Figure 16, for example. The evaluation indicator 706 shown in Figure 16 includes a rounded rectangular indicator element 706c that resembles a lamp. This indicator element 706c is accompanied by an appropriate string of characters 706d representing the level of evaluation given by the user to the response staff member, as well as an appropriate mark 706e corresponding to the level of evaluation. With an evaluation indicator 706 configured in this way, the response staff member can intuitively recognize the user's evaluation of them.

[0141] Incidentally, Figure 16 shows an example of an operator-side response screen 700 including an evaluation indicator 706 when the evaluation level is "4". Therefore, when the evaluation level is other than "4", the appearance of the evaluation indicator 706, and more specifically the appearance of the string 706d and mark 706e, will be different from those shown in Figure 16. In addition, the indicator 706c may be given a color or pattern corresponding to the evaluation level, and its size may be changed (for example, so that it becomes larger as the evaluation level increases).

[0142] In this first embodiment, the evaluation level was set to five stages from "1" to "5," but other numbers of stages may be used. Furthermore, the evaluation procedure is not limited to the procedure described in this first embodiment. The evaluation level may also be presented in an auditory manner, such as audio, rather than visually using the evaluation indicator 706. Additionally, negative evaluations may not be presented to the customer service representative, and only positive evaluations may be presented to them.

[0143] Furthermore, the system may be configured to record (save) the evaluation results for each customer service representative as appropriate. These recorded evaluation results can be useful, for example, for evaluating the performance of each operator or for improving the quality of service.

[0144] Furthermore, in this first embodiment, the evaluation of the responding user towards the responding staff was estimated based on the user image and user voice of the responding user during the online interaction, that is, based on two types of data: user image and user voice. However, it is not limited to this. For example, the evaluation of the responding user towards the responding staff may be estimated based on either of the two types of data: the user image and user voice of the responding user during the online interaction.

[0145] In this first embodiment, when an online interaction is taking place, if the user's evaluation of the service provider is to be estimated based on the user's image, the evaluation is estimated based on the user's facial expression recognized from the user's image, and the user's actions, separate from the facial expression. In particular, the user's actions include, but are not limited to, the user's neck movements, face orientation, bowing, and waving. For example, the user's evaluation of the service provider may be estimated based on any (or part) of these elements. Furthermore, the user image is a moving image, but may also be a still image, or a combination of moving and still images.

[0146] In addition, in this first embodiment, when an online interaction is conducted, the user's evaluation of the service provider is estimated based on the user's voice, the words contained in the user's voice, and the sounds of the user's actions, such as noises and applause, contained in the user's voice. However, this is not limited to this. For example, the user's evaluation of the service provider may be estimated based on any (or a part) of these elements.

[0147] Furthermore, in this first embodiment, the evaluation of the responding user by the responding user was estimated by the responding user terminal 30a, but this is not limited to this. For example, the evaluation of the responding user by the responding user by the responding user may be estimated by the management server 40, or the evaluation may be estimated by the responding terminal 20a.

[0148] Furthermore, in this first embodiment, bidirectional communication is performed between the response terminal 20a and the response user terminal 30a using a P2P method, but this is not limited to this. In other words, bidirectional communication may be performed between the response terminal 20a and the response user terminal 30a using a method other than P2P.

[0149] In addition, the management server 40 does not necessarily have to be provided. In this case, for example, each operator terminal 20, 20, ... may be configured to handle the same processing as the management server 40, or one of the operator terminals 20, 20, ... (a specific operator terminal 20) may be configured to handle the same processing as the management server 40.

[0150] In this first embodiment, the camera 314 of the customer service terminal 30a is an example of a camera according to the present invention, and the microphone 210 of the customer service terminal 30a is an example of a microphone according to the present invention. The CPU 302a of the customer service terminal 30a, more specifically the CPU 302a that executes steps S1 and S3 of the evaluation event detection task and the user evaluation task (see Figure 13) described above, is an example of an estimation means according to the present invention. Furthermore, the evaluation indicator 706 of the operator-side customer service screen 700 is an example of the first evaluation information according to the present invention. The CPU 202a of the customer service terminal 20a, more specifically the CPU 202a that executes step S203 of the user evaluation presentation task (see Figure 15), cooperates with the display device 208 of the customer service terminal 20a to constitute the first presentation means according to the present invention. The display device 208 of the customer service terminal 20a is also an example of a display means according to the present invention.

[0151] [Second Example] Next, a second embodiment of the present invention will be described with reference to Figures 17 to 24.

[0152] This second embodiment is based on the online response system 10 according to the first embodiment, and is further equipped with a centralized monitoring function. With this centralized monitoring function, when an online response is being conducted with one operator acting as the response person and another user acting as the response user, any operator in standby mode can centrally monitor (view) the status of the online response. Furthermore, the centralized monitoring function includes an operator-to-operator evaluation function as a sub-function. With this operator-to-operator evaluation function, the operator performing the centralized monitoring, so to speak, the centralized monitor, can evaluate the response person while performing the centralized monitoring, and the evaluation results are presented to the response person in real time. In other words, the response person is evaluated by an operator other than the response person, and the evaluation results are presented to the response person in real time. Various effects can be expected from such a centralized monitoring function, including the operator-to-operator evaluation function, such as improving the response person's response ability.

[0153] Specifically, in the operator terminal 20 which is in standby mode, the operator standby screen 500 is displayed on its display device 208 as described above. However, when any room is receiving an online response, as shown in Figure 17, an appropriate "receiving online response" mark 510 is added to the room display area 502 corresponding to the room receiving the online response. In addition, an appropriate decoration is added to the periphery of the room display area 502 to indicate that an online response is taking place, for example, a green border is added.

[0154] Figure 17 shows an example of the operator-side standby screen 500 when "Room 1" (sales area 1) is receiving online support, and "Room 5" (sales area 5) is also receiving online support. In Figure 17, a thick dashed border is added to the periphery of the room display area 502 corresponding to "Room 1," and a thick dashed border is also added to the periphery of the room display area 502 corresponding to "Room 5," indicating that a green border has been added to the periphery of each room display area 502. Furthermore, in Figure 17, a support mark 510 is placed near the upper left corner of the room display area 502, but the position where the support mark 510 is placed is not limited to this.

[0155] Upon displaying the operator-side standby screen 500 as shown in Figure 17, the operator (operator terminal 20) in standby mode can intuitively recognize that "Room 1" and "Room 5," that is, the rooms corresponding to the room display area 502 which has a call-in mark 510 and a green border around its edge, are currently receiving online calls. The operator in standby mode can then select the status of the online calls being received by a room as the target of centralized monitoring by operating one of the room display areas 502 corresponding to the room receiving an online call. In other words, the room display area 502 corresponding to the room receiving an online call functions as an operator that can accept a centralized monitoring target selection operation to select the target of centralized monitoring. When one of the room display areas 502 corresponding to the room receiving an online call is operated, that is, when a centralized monitoring target selection operation is accepted, the centralized monitoring screen 1200 shown in Figure 18 is displayed on the display device 208 instead of the operator-side standby screen 500. Figure 18 shows an example of the centralized monitoring screen 1200 when the room display area 502 corresponding to "Room 5" of the operator-side standby screen 500 shown in Figure 17 is operated.

[0156] As shown in Figure 18, the centralized monitoring screen 1200 includes an enlarged room display area 1202 similar to that in the operator-side response screen 700 (see Figure 8) described above. This enlarged room display area 1202 displays user images of the room that is currently receiving online support, in this case "Room 5". The user images for "Room 5" include images 1204 of the users present in "Room 5". In addition, a suitable string 1206 indicating that centralized monitoring is currently being performed is placed at an appropriate position in the enlarged room display area 1202, for example, near its upper right corner. A suitable mark may be placed in place of, or in addition to, this string 1206. Furthermore, a string 1208 indicating the location of the room receiving online support is placed near the enlarged room display area 1202, for example, slightly to the left of the bottom of the enlarged room display area 1202. In addition, a string 1210 representing the online support staff member is placed near string 1208, for example, below string 1208. In Figure 18, string 1210 indicates that the support staff member is "A".

[0157] In addition, to the left of the enlarged room display area 1202, four (P) reduced room display areas 1212, 1212, ... are arranged in a vertical line, similar to those on the operator-side response screen 700. Each of these reduced room display areas 1212, 1212, ... displays an image of the user of the corresponding room. Furthermore, a string 1214 representing the location of the room corresponding to each reduced room display area 1212 is displayed near each reduced room display area 1212, for example, slightly to the left and below the reduced room display area 1212. Then, the reduced room display area 1212 corresponding to a room that is currently receiving online support is marked with a support mark 1216, similar to that on the operator-side response screen 700 shown in Figure 17, for the reduced room display area 1212 corresponding to "Room 1". In addition, the periphery of the reduced room display area 1212 corresponding to "Room 1" is modified appropriately to indicate that it is receiving an online response, for example, by adding a green border.

[0158] In Figure 18, a thick dashed border is added to the periphery of the reduced room display area 1212 corresponding to "Room 1," indicating that a green border has been added to the periphery of the reduced room display area 1212. In Figure 18, a "handling" mark 1216 is placed near the upper left corner of the reduced room display area 1212, but the position where the "handling" mark 1216 is placed is not limited to this. Furthermore, although not shown in the illustration, the reduced room display area 1212 corresponding to the room on which the call operation was performed is marked with a call mark similar to that on the operator-side standby screen 500 shown in Figure 7. In addition, the periphery of the reduced room display area 1212 with the call mark is decorated appropriately, similar to that on the operator-side standby screen 500 shown in Figure 7, for example, by adding a red border.

[0159] Furthermore, if the number of rooms other than those receiving online support is greater than 4, that is, if the value obtained by subtracting 1 from M (M-1) is greater than the value of P, then, although not shown in the diagram, the area of ​​the centralized monitoring screen 1200 where each reduced room display area 1212, 1212, ... is located can be scrolled vertically. This allows the reduced room display areas 1212, 1212, ... corresponding to each room other than those receiving online support, in other words, the user images of each room other than those receiving online support, to be displayed on the centralized monitoring screen 1200.

[0160] In addition, an evaluation button 1218 is placed in an appropriate position on the centralized monitoring screen 1200, for example, below the enlarged room display area 1202. This evaluation button 1218 includes three controls 1218a, 1218b, and 1218c. These three controls 1218a, 1218b, and 1218c are arranged in a vertical column. The top control 1218a of these three controls 1218a, 1218b, and 1218c is the [Nice!] button, which instructs the user to give the highest possible rating to the online responder, for example, a rating of "Nice!". The middle control 1218b is the [Keep it up!] button, which instructs the user to give a relatively high rating to the responder, for example, a rating of "Keep it up!". The control element 1218c at the very bottom is the [Calm Down!] button, which instructs the user to convey a message of encouragement to the customer service representative, such as "Calm down!", indicating that the user cannot give the representative a high rating.

[0161] Furthermore, a [End Monitoring] button 1220 is placed in an appropriate location on the centralized monitoring screen 1200, for example, near the lower right corner of the centralized monitoring screen 1200, that is, to the right of the evaluation button 1218, to indicate that centralized monitoring should be terminated.

[0162] Even when the operator terminal 20 is displaying the centralized monitoring screen 1200, user images are acquired from each room in a time-division manner, similar to when the operator terminal 20 is displaying the operator-side standby screen 500. More precisely, user images are acquired in a time-division manner from each user terminal 30, 30, ... installed in each of those rooms. The user images acquired from the room receiving online support (support terminal 30a) among each room (each user terminal 30, 30, ...) are displayed in the enlarged room display area 1202. In addition, user images acquired from each room corresponding to each reduced room display area 1212, 1212, ... are displayed in each reduced room display area 1212, 1212, ... From this point forward, the operator terminal 20 displaying the centralized monitoring screen 1200 may be referred to as the "centralized monitoring terminal 20b". However, the symbol "20b" representing the centralized monitoring terminal 20b will not be specifically illustrated.

[0163] Furthermore, the centralized monitoring terminal 20b acquires the operator's voice from the handling terminal 20a related to the handling of the call that is the subject of centralized monitoring, and also acquires the user's voice from the handling terminal 30a, which is the counterpart device of the handling terminal 20a. Although not apparent from the figures, including Figure 18, the speaker 212 of the centralized monitoring terminal 20b outputs voice corresponding to the operator's voice acquired from the handling terminal 20a, that is, the voice of the handling staff, as well as voice corresponding to the user's voice acquired from the handling terminal 30a, that is, the voice of the user being handled.

[0164] Therefore, the central monitor, who is the operator of the central monitoring terminal 20b, can centrally monitor the status of the online interaction between the responder and the user by receiving the central monitoring screen 1200 displayed on the display device 208 of the central monitoring terminal 20b, and the voices of the responder and the user being interacted with from the speaker 212 of the central monitoring terminal 20b. The central monitor can then evaluate the responder by operating the evaluation button 1218 on the central monitoring screen 1200.

[0165] Referring again to Figure 17, when a centralized monitoring target selection operation is performed on the operator-side standby screen 500 shown in Figure 17, that is, when one of the room display areas 502 corresponding to a room that is receiving an online response is operated, a centralized monitoring start notification is sent from the operator terminal 20 that received the centralized monitoring target selection operation to the management server 40. At the same time, the terminal ID of the operator terminal 20 that received the centralized monitoring target selection operation and the room number of the room related to the response that is the subject of centralized monitoring (i.e., the room display area 502 that was operated) are sent from the operator terminal 20 to the management server 40.

[0166] Then, in the operator terminal 20 that has received the centralized monitoring target selection operation, the centralized monitoring screen 1200 is displayed on the display device 208 instead of the operator-side standby screen 500 as described above. As a result, the operator terminal 20 that has received the centralized monitoring target selection operation becomes the centralized monitoring terminal 20b.

[0167] In response, when the management server 40 receives a centralized monitoring start notification, it sets the status of the operator terminal 20 that sent the centralized monitoring start notification in the aforementioned operator terminal management table 1000 (see Figure 11) to "Centralized Monitoring in Progress". At this time, the management server 40 associates the terminal ID of the handling terminal 20a related to the response subject to centralized monitoring with the information representing the status "Centralized Monitoring in Progress". As can be seen from this, in this second embodiment, in addition to the three states "Standby", "Handling", and "Stopped", the operator terminal management table 1000 also includes the state "Centralized Monitoring in Progress". Furthermore, the management server 40 sends a centralized monitoring start report notification to the handling terminal 20a related to the response subject to centralized monitoring.

[0168] The operator terminal 20 that sends the centralized monitoring commencement notification is identified from the terminal ID of the operator terminal 20 received along with the centralized monitoring commencement notification. The handling terminal 20a related to the response subject to centralized monitoring is identified by referencing the aforementioned user terminal management table 1100 (see Figure 12) based on the room number of the room related to the response, which is received along with the centralized monitoring commencement notification. Hereafter, the handling terminal 20a related to the response subject to centralized monitoring may be referred to as the "monitored terminal 20c". However, the code "20c" representing the monitored terminal 20c will not be specifically illustrated.

[0169] Furthermore, for example, when the evaluation button 1218 on the central monitoring screen 1200 is operated on the central monitoring terminal 20b, that is, when any of the three operators 1218a, 1218b, and 1218c is operated, an evaluation content notification indicating the evaluation content corresponding to that operation is sent from the central monitoring terminal 20b to the management server 40. At the same time, the terminal ID of the central monitoring terminal 20b is also sent from the central monitoring terminal 20b to the management server 40.

[0170] When the management server 40 receives an evaluation notification from the central monitoring terminal 20b, it sends the evaluation notification as an operator-to-operator evaluation notification to the monitored terminal 20c. The monitored terminal 20c to which this operator-to-operator evaluation notification is sent is identified by referencing the operator terminal management table 1000 based on the terminal ID of the central monitoring terminal 20b, which is received from the central monitoring terminal 20b along with the evaluation notification. In addition, the management server 40 sends the terminal ID of the central monitoring terminal 20b to the monitored terminal 20c.

[0171] When the monitored terminal 20c receives an operator-to-operator evaluation notification and the terminal ID of the central monitoring terminal 20b from the management server 40, it updates the display of the operator-to-operator evaluation indicator 722, as described below, based on the operator-to-operator evaluation notification and the terminal ID of the central monitoring terminal 20b.

[0172] In other words, even in the monitored terminal 20c, the operator-side response screen 700 is displayed on its display device 208. Specifically, the operator-side response screen 700 includes an operator-to-operator evaluation indicator 722, as shown in Figure 19, at an appropriate position in the enlarged room display area 702. This operator-to-operator evaluation indicator 722 is overlaid on the user image displayed in the enlarged room display area 702 and is positioned, for example, near the lower left corner of the enlarged room display area 702.

[0173] More specifically, the operator-to-operator evaluation indicator 722 includes a plurality of circular indicators 722a, 722a, ... These indicators 722a, 722a, ... correspond to the operators of each operator terminal 20, 20, ... other than the monitored terminal 20c, and are arranged, for example, side by side. Then, as described above, when the central monitor evaluates the responding personnel, an indicator 722b appears corresponding to the evaluation content, that is, corresponding to the operator-to-operator evaluation notification received from the management server 40. This indicator 722b is, for example, in the shape of a speech bubble, and is accompanied by a string 722c and a mark 722d corresponding to the operator-to-operator evaluation notification.

[0174] Figure 19 shows an example of the operator-side response screen 700, including the operator-to-operator evaluation indicator 722, when the central monitor is operator "C" and the [Nice!] button 1218a on the central monitoring screen 1200 is operated by operator "C". Furthermore, for each indicator 722a, 722a, ... corresponding to the central monitor, the indicator 722a corresponding to the central monitor who most recently performed an evaluation of the responding staff, or in other words, an operator-to-operator evaluation, is appropriately modified, for example, by adding a blue border around the periphery of that indicator 722a. In Figure 19, the blue border around the periphery of the indicator 722a corresponding to the central monitor "C" who most recently performed an operator-to-operator evaluation is shown by adding a thick solid border around the periphery of that indicator 722a. Furthermore, the speech bubble-shaped indicator 722b appears to be ejected from the indicator 722a corresponding to the central monitoring operator who most recently performed the operator-to-operator evaluation, that is, from the indicator 722a labeled with the letter "C" in Figure 19.

[0175] When this inter-operator evaluation indicator 722 is displayed, the customer service representative can experience an objective evaluation of themselves by other operators. This is expected to have various effects, such as improving the customer service representative's ability to handle customer interactions.

[0176] Furthermore, with the centralized monitoring function including the operator-to-operator evaluation function in this second embodiment, multiple operators can act as centralized monitors and arbitrarily (randomly) evaluate one customer service representative. An example of the operator-side response screen 700 in this case is shown in Figure 20. Figure 20 shows an example of the operator-side response screen 700 including the operator-to-operator evaluation indicator 722 when, after the state shown in Figure 19, that is, after operator "C" has performed an operator-to-operator evaluation as a centralized monitor, operators other than the customer service representatives "A," "B," "C," "D," and "E" have performed an operator-to-operator evaluation as centralized monitors, and more specifically when the [Keep it up!] button 1218b on the centralized monitoring screen 1200 is operated.

[0177] In other words, when a new inter-operator evaluation is performed, the previously displayed bubble-shaped indicator 722b moves upward (shifts), and then a new bubble-shaped indicator 722b corresponding to the content of the newly performed inter-operator evaluation appears in the position where the previously displayed indicator 722b was located. Note that the indicator 722a (marked with the symbol "...") at the right end of the inter-operator evaluation indicator 722 shown in Figure 20 corresponds to operators other than the responding personnel "A," "B," "C," "D," and "E." Although not shown in the illustration, when the indicator 722a at the right end of the inter-operator evaluation indicator 722 is operated, appropriate information, such as an indicator, representing the operator corresponding to the new bubble-shaped indicator 722b, that is, the central monitoring person who performed the new inter-operator evaluation, appears. From this point onward, each time a new inter-operator evaluation is performed, the appearance of the inter-operator evaluation indicator 722 changes in the same manner, that is, the display of the inter-operator evaluation indicator 722 is updated.

[0178] Incidentally, in the initial operator-side response screen 700 when online support is initiated, only the circular indicators 722a, 722a, ... of the operator-to-operator evaluation indicator 722 are displayed. Each time an operator-to-operator evaluation is performed, the display of the operator-to-operator evaluation indicator 722 is updated in the same manner as described above.

[0179] Referring again to Figure 18, in the centralized monitoring screen 1200 shown in Figure 18, a "receiving a call" mark 1216 is placed on the reduced room display area 1212 corresponding to a room that is currently receiving an online call—in this case, the reduced room display area 1212 corresponding to "Room 1"—and a green border is placed around the periphery of the reduced room display area 1212 corresponding to "Room 1". Each reduced room display area 1212, 1212, ... including the reduced room display area 1212 corresponding to "Room 1" functions as an operator that can accept operations. For example, when the reduced room display area 1212 corresponding to "Room 1," that is, the reduced room display area 1212 corresponding to a room that is currently receiving an online call, is operated, the status of the online call received by the room corresponding to the operated reduced room display area 1212 is selected as the target of centralized monitoring, that is, the target of centralized monitoring is changed. Then, the user image that was previously displayed in the enlarged room display area 702 is replaced with the user image of the room corresponding to the operated reduced room display area 1212, which is then displayed in the enlarged room display area 702. At the same time, the user images displayed in each reduced room display area 1212, 1212, ... are changed (organized) as appropriate. In other words, the reduced room display area 1212 corresponding to a room that is currently receiving online support functions as an operator capable of receiving a centralized monitoring target change operation to change the target of centralized monitoring.

[0180] Furthermore, when a reduced room display area 1212 corresponding to a room other than the one currently receiving an online response is operated, the room corresponding to the operated reduced room display area 1212 is selected as the online counterparty. In this case, the operator-side response screen 700 is displayed on the display device 208 instead of the centralized monitoring screen 1200. In particular, the user image of the room selected as the online counterparty is displayed in the enlarged room display area 702 of the operator-side response screen 700. In other words, the reduced room display area 1212 corresponding to a room other than the one currently receiving an online response functions as an operator capable of receiving a counterparty selection operation to select a room to be the online counterparty.

[0181] Furthermore, referring again to Figure 19, in the operator-side response screen 700 shown in Figure 19, a response mark 720 indicating that the room is receiving an online response is added to the reduced room display area 710 corresponding to the room receiving an online response—in this case, the reduced room display area 710 corresponding to "Room 1." In addition, a green border is added to the periphery of the reduced room display area 710 corresponding to "Room 1," that is, the periphery of the reduced room display area 710 corresponding to the room receiving an online response. The same applies to the operator-side response screen 700 shown in Figure 20.

[0182] Figures 19 and 20 show examples of the operator-side response screen 700 when online responses are being made. However, if the reduced room display area 710 corresponding to any room that is currently receiving online responses is operated on the operator-side response screen 700 before online responses are made, that is, before the [Start Response] button 716 is operated, the centralized monitoring screen 1200 will be displayed on the display device 208 instead of the operator-side response screen 700. In other words, on the operator-side response screen 700 before online responses are made, the reduced room display area 710 corresponding to a room that is currently receiving online responses functions as an operator that can accept a centralized monitoring target selection operation for selecting a target for centralized monitoring.

[0183] Furthermore, referring again to Figure 18, when the [End Monitoring] button 1220 is operated on the centralized monitoring screen 1200 shown in Figure 18, the operator-side standby screen 500 is displayed on the display device 208 instead of the centralized monitoring screen 1200. Then, a centralized monitoring completion notification is sent from the centralized monitoring terminal 20b to the management server 40. At the same time, the terminal ID of the centralized monitoring terminal 20b, which is the source of the centralized monitoring completion notification, is also sent from the centralized monitoring terminal 20b to the management server 40. With this, the centralized monitoring rig by the centralized monitoring terminal 20b is terminated. The centralized monitoring terminal 20b then becomes the operator terminal 20 in standby mode.

[0184] The management server 40 recognizes that centralized monitoring by the centralized monitoring terminal 20b has ended when it receives a notification from the centralized monitoring terminal 20b that the centralized monitoring has ended. The management server 40 then updates the operator terminal management table 1000 to reflect the status of the operator terminal 20 (centralized monitoring terminal 20b), which was the source of the centralized monitoring start notification, as "standing".

[0185] Furthermore, when centralized monitoring is being performed by the centralized monitoring terminal 20b, if the online interaction being monitored is terminated, that is, if the [End Interaction] button 718 is operated on the monitored terminal 20c, the centralized monitoring is also terminated, or rather, forcibly terminated. In this case, the centralized monitoring terminal 20b displays an interaction termination screen (not shown) on the display device 208 for a certain period of time (for example, a few seconds to tens of seconds) instead of the centralized monitoring screen 1200. After that, the operator-side standby screen 500 is displayed on the display device 208 again. The fact that the online interaction being monitored has been terminated is notified to the centralized monitoring terminal 20b from the management server 40. The interaction termination screen may also be, for example, a modeless dialog. In this case, the dialog as the response termination screen is displayed so as to cover a part of the centralized monitoring screen 1200.

[0186] To realize the centralized monitoring function, including the operator-to-operator evaluation function, in this second embodiment, the response control software 410a of the management server 40 includes a centralized monitoring control program. In addition, the operator-side response software 216a of each operator terminal 20 also includes a centralized monitoring program.

[0187] In particular, in order to realize the operator-to-operator evaluation function, each operator terminal 20, or more precisely, the CPU 202a of the operator terminal 20 acting as a centralized monitoring terminal 20b, executes an operator-to-operator evaluation task according to the operator-to-operator evaluation program included in the centralized monitoring program. The flow of this operator-to-operator evaluation task is shown in Figures 21 and 22. The operator-to-operator evaluation task is executed in the operator terminal 20 acting as the centralized monitoring terminal 20b in response to the acceptance of the aforementioned centralized monitoring target selection operation, that is, in response to the operation of a room display area 502 corresponding to one of the room display areas 502, 502, ... of the operator-side standby screen 500 that is currently receiving online responses.

[0188] According to this operator-to-operator evaluation task, the CPU 202a of operator terminal 20, which acts as a centralized monitoring terminal 20b, first sends a centralized monitoring start notification to the management server 40 in step S301. At the same time, the CPU 202a sends its own terminal ID (centralized monitoring terminal 20b) and the room number of the room related to the response that is the subject of centralized monitoring to the management server 40. In response, the management server 40 updates the operator terminal management table 1000 in the manner described above. After executing step S301, the CPU 202a proceeds to step S303.

[0189] In step S303, the CPU 202a replaces the operator-side standby screen 500 that had been displayed on the display device 208 with the centralized monitoring screen 1200 on the display device 208. Then, the CPU 202a proceeds to step S305.

[0190] In step S305, it is determined whether the online interaction that is the subject of centralized monitoring has been completed. If the online interaction has not been completed (S305: NO), CPU 202a proceeds to step S311, which will be described later. On the other hand, if the online interaction has been completed (S305: YES), CPU 202a proceeds to step S307.

[0191] In step S307, the CPU 202a displays the aforementioned (not shown) call completion screen on the display device 208 for a certain period of time, instead of the centralized monitoring screen 1200. Then, the CPU 202a proceeds to step S309.

[0192] In step S309, the CPU 202a displays the operator-side standby screen 500 on the display device 208. With this, the CPU 202a terminates the operator-to-operator evaluation task.

[0193] In contrast, if the process proceeds from step S305 to step S311, CPU 202a determines whether any operation has been accepted in step S311. If no operation has been accepted (S311: NO), CPU 202a returns to step S305. On the other hand, if any operation has been accepted (S311: YES), CPU 202a proceeds to step S313.

[0194] In step S311, the CPU 202a determines whether the operation received in step S311 is an operation to the [End Monitoring] button 1220 on the centralized monitoring screen 1200, in other words, whether it is a centralized monitoring termination operation. If the operation received in step S311 is a centralized monitoring termination operation (S313: YES), the CPU 202a proceeds to step S315. On the other hand, if the operation received in step S311 is not a centralized monitoring termination operation (S313: NO), the CPU 202a proceeds to step S317, which will be described later.

[0195] In step S315, CPU 202a sends a centralized monitoring completion notification to the management server 40. At the same time, CPU 202a sends its own terminal ID (centralized monitoring terminal 20b) to the management server 40. In response, the management server 40 updates the operator terminal management table 1000 in the manner described above. After executing step S315, CPU 202a proceeds to step S309.

[0196] In contrast, if the process proceeds from step S313 to step S317 as described above, the CPU 202a determines in step S317 whether the operation received in step S311 is an operation on the evaluation button 1218 of the centralized monitoring screen 1200, that is, whether it is an operation on any of the operators 1218a, 1218b, and 1218c. If the operation received in step S311 is an operation on the evaluation button 1218 (S317: YES), the CPU 202a proceeds to step S319. On the other hand, if the operation received in step S311 is not an operation on the evaluation button 1218 (S317: NO), the CPU 202a proceeds to step S321, which will be described later.

[0197] In step S319, CPU 202a sends an evaluation content notification to the management server 40 corresponding to the operation of the evaluation button 1218 received in step S311. At the same time, CPU 202a sends its own terminal ID (centralized monitoring terminal 20b) to the management server 40. Then, CPU 202a returns the process to step S305.

[0198] In response to this, when the CPU 202a proceeds from step S317 to step S321, in step S321 it determines whether the operation received in step S311 is an operation on one of the reduced room display areas 1212 of the centralized monitoring screen 1200, in other words, whether it is a different room selection operation. If the operation received in step S311 is a different room selection operation, that is, if one of the reduced room display areas 1212 is operated (S321: YES), the CPU 202a proceeds to step S323. On the other hand, if the operation received in step S311 is not a different room selection operation (S321: NO), the CPU 202a proceeds to step S331, which will be described later.

[0199] In step S323, the CPU 202a determines whether the alternative room selection operation received in step S311 is an operation on the reduced room display area 1212 corresponding to the room currently being answered. If the alternative room selection operation received in step S311 is an operation on the reduced room display area 1212 corresponding to the room currently being answered, that is, if the reduced room display area 1212 corresponding to the room currently being answered is operated (S323: YES), the CPU 202a proceeds to step S325. On the other hand, if the alternative room selection operation received in step S311 is not an operation on the reduced room display area 1212 corresponding to the room currently being answered, that is, if the reduced room display area 1212 corresponding to any room other than the room currently being answered is operated (S323: NO), the CPU 202a proceeds to step S327, which will be described later.

[0200] In step S325, CPU 202a sends a centralized monitoring completion notification to the management server 40. At the same time, CPU 202a sends its own terminal ID (centralized monitoring terminal 20b) to the management server 40. In response, the management server 40 updates the operator terminal management table 1000 in the manner described above. As a result, centralized monitoring is temporarily terminated. Then, CPU 202a returns to step S301 to change the target of centralized monitoring.

[0201] In response, if CPU 202a proceeds from step S323 to step S327, in step S327, it sends a centralized monitoring completion notification to the management server 40. At the same time, CPU 202a sends its own terminal ID (centralized monitoring terminal 20b) to the management server 40. In response, the management server 40 updates the operator terminal management table 1000 in the manner described above. This terminates the centralized monitoring. Then, CPU 202a proceeds to step S329.

[0202] In step S329, the CPU 202a replaces the central monitoring screen 1200 that had been displayed on the display device 208 with the operator-side response screen 700 on the same display device 208. With this, the CPU 202a completes the operator-to-operator evaluation task.

[0203] Furthermore, if the CPU 202a proceeds from step S321 to step S331, it executes processing in step S331 in accordance with the operation received in step S311. The operation received in step S311 includes processing to scroll the area of ​​the centralized monitoring screen 1200 where each reduced room display area 1212, 1212, ... is located in the vertical direction. Then, the CPU 202a returns processing to step S305.

[0204] Furthermore, in order to implement the operator-to-operator evaluation function, the management server 40, or more precisely, the management server 40's CPU 402a, executes operator-to-operator evaluation control tasks according to an operator-to-operator evaluation control program that includes a centralized monitoring control program. The flow of these operator-to-operator evaluation control tasks is shown in Figure 23. The operator-to-operator evaluation control task is executed in response to the receipt of a centralized monitoring start notification from the operator terminal 20, which acts as the centralized monitoring terminal 20b. The operator-to-operator evaluation control task is also executed in correspondence for each pair of operator terminal 20 acting as the centralized monitoring terminal 20b and operator terminal 20 acting as the monitored terminal 20c (responding terminal 20a).

[0205] According to this operator-to-operator evaluation control task, the CPU 402a of the management server 40 first determines in step S401 whether it has received an evaluation content notification from the central monitoring terminal 20b. If an evaluation content notification has been received (S401: YES), the CPU 402a proceeds to step S403. On the other hand, if an evaluation content notification has not been received (S401: NO), the CPU 402a proceeds to step S405, which will be described later. When an evaluation content notification is received from the central monitoring terminal 20b, the terminal ID of the central monitoring terminal 20b is also received.

[0206] In step S403, CPU 402a sends the evaluation content notification received in step S401 as an operator-to-operator evaluation notification to the monitored terminal 20c. The monitored terminal 20c to which this operator-to-operator evaluation notification is sent is identified by referring to the operator terminal management table 1000 based on the terminal ID of the central monitoring terminal 20b, which is received from the central monitoring terminal 20b along with the evaluation content notification. At the same time, CPU 402a sends the terminal ID of the central monitoring terminal 20b to the monitored terminal 20c. Then, CPU 402a proceeds to step S405.

[0207] In step S405, CPU 402a determines whether the online interaction has been completed. If the online interaction has been completed (S405: YES), CPU 402a terminates the inter-operator evaluation control task. Subsequently, the operator terminal management table 1000 is updated in the manner described above. On the other hand, if the online interaction has not been completed (S405: NO), CPU 402a proceeds to step S407.

[0208] In step S407, the CPU 402a determines whether it has received a notification from the central monitoring terminal 20b that central monitoring has ended, that is, whether central monitoring by the central monitoring terminal 20b has ended. If central monitoring has ended (S407:YES), the CPU 402a terminates the inter-operator evaluation control task. On the other hand, if central monitoring has not ended (S407:NO), the process returns to step S401.

[0209] Furthermore, in order to implement the operator-to-operator evaluation function, each operator terminal 20, or more precisely, the CPU 202a of the monitored terminal 20c, executes an operator-to-operator evaluation presentation task according to the operator-to-operator evaluation presentation program included in the centralized monitoring program. The flow of this operator-to-operator evaluation presentation task is shown in Figure 24. Note that the operator-to-operator evaluation presentation task is executed in response to receiving a centralized monitoring start report notification from the management server 40.

[0210] According to this inter-operator evaluation presentation task, the CPU 202a of the monitored terminal 20c first determines in step S501 whether it has received an inter-operator evaluation notification from the management server 40. If it has received an inter-operator evaluation notification (S501: YES), the CPU 202a proceeds to step S503. On the other hand, if it has not received an inter-operator evaluation notification (S501: NO), the CPU 202a proceeds to step S505, which will be described later. When an inter-operator evaluation notification is received from the management server 40, the terminal ID of the centralized monitoring terminal 20b is also received.

[0211] In step S503, CPU 202a updates the display in response to the inter-operator evaluation notification received in step S501. Then, CPU 202a proceeds to step S505.

[0212] In step S505, the CPU 202a determines whether the online interaction has ended, that is, whether the [End Interaction] button 718 on the operator-side interaction screen 700 has been pressed. If the online interaction has ended (S505: YES), the CPU 202a terminates the operator-to-operator evaluation presentation task. Subsequently, the operator-side waiting screen 500 is displayed on the display device 208 in place of the operator-side interaction screen 700. On the other hand, if the online interaction has not ended (S505: NO), the CPU 202a returns the process to step S501.

[0213] Thus, according to the online response system 10 of this second embodiment, particularly the centralized monitoring function including the operator-to-operator evaluation function, when an online response is being conducted with one operator acting as the responder and another user acting as the user, any operator in standby mode can centrally monitor the status of the online response. Furthermore, the centralized monitor performing the centralized monitoring can evaluate the responder while conducting the centralized monitoring, and the evaluation results are presented to the responder in real time via the operator-to-operator evaluation indicator 722. Therefore, the responder can experience objective evaluations of themselves by other operators. This is expected to have various effects, such as improving the responder's response skills.

[0214] It should be noted that the configuration, including the design of the centralized monitoring screen 1200 shown in Figure 18, is merely an example, and the configuration of the centralized monitoring screen 1200 is not limited to this. For example, although three evaluation buttons 1218a, 1218b, and 1218c are provided as evaluation buttons 1218, meaning there are three levels of inter-operator evaluation, other numbers of levels may also be used. In addition, in the enlarged room display area 1202 of the centralized monitoring screen 1200, in addition to the user image, an avatar similar to that in the user-side response screen 800 (see Figure 9), that is, an avatar representing the person handling the response, may be displayed slightly smaller, for example, by overlaying it on a part of the user image.

[0215] Furthermore, the configuration including the design of the operator evaluation indicator 722 in the operator-side response screen 700 shown in Figures 19 and 20 is merely an example, and the configuration of the operator evaluation indicator 722 is not limited to this. In addition, the operator-side response screen 700 shown in Figures 19 and 20 shows an example in which the operator evaluation indicator 722 is placed on the operator-side response screen 700 with the configuration shown in Figure 8, but the operator evaluation indicator 722 may also be placed on the operator-side response screen 700 with the configuration shown in Figure 16. Moreover, the operator evaluation indicator 722, in particular the indicator 722b, may be configured to appear only when a certain number of positive evaluations or more are obtained. In addition, the results of the operator evaluation may be presented not in a visual manner by the operator evaluation indicator 722, but in an auditory manner, such as audio.

[0216] Furthermore, in this second embodiment, the management server 40 does not necessarily have to be provided. In this case, for example, each operator terminal 20, 20, ... may be configured to handle the same processing as the management server 40, or one of the operator terminals 20, 20, ... (a specific operator terminal 20) may be configured to handle the same processing as the management server 40.

[0217] In this second embodiment, the central monitor is an example of a third user according to the present invention. The operator-to-operator evaluation notification, more specifically the content of the evaluation of the responding person by the central monitor included in the operator-to-operator evaluation notification, is an example of second evaluation information according to the present invention. Furthermore, the CPU 202a of the monitored terminal 20c, more specifically the CPU 202a that executes step S501 of the operator-to-operator evaluation presentation task (see Figure 24), cooperates with the communication unit 218 of the monitored terminal 20c to constitute a reception means according to the present invention. Also, the CPU 202a of the monitored terminal 20c, more specifically the CPU 202a that executes step S503 of the operator-to-operator evaluation presentation task, cooperates with the display device 208 of the monitored terminal 20c to constitute a second presentation means according to the present invention.

[0218] [Other application examples] The above embodiments are specific examples of the present invention and do not limit the technical scope of the present invention. The present invention can also be applied to aspects other than those described in these embodiments.

[0219] For example, in the extreme case, only one user terminal 30 may be provided. That is, multiple operator terminals 20, 20, ... may be configured to correspond to one user terminal 30.

[0220] Furthermore, while the online customer service system 10 for a home improvement store was used as an example in each embodiment, the present invention can be applied to other uses as well.

[0221] Furthermore, the present invention is not limited to the form of an information processing device, but can also be provided in the form of an information processing method, and in the form of an information processing program.

[0222] Furthermore, the present invention can also be provided in the form of a computer-readable, non-transient recording medium on which an information processing program is recorded. In this case, the terminal assignment control program recorded on the recording medium is read by a computer and executed by the computer, thereby achieving the same effects as in the present invention. The recording medium referred to here includes semiconductor media and disk media. In addition, embedded (internal) media that are incorporated into devices such as ROMs and hard disk drives, rather than portable media, can also be applied as recording media in this invention. [Explanation of symbols]

[0223] 10… Online response system 20 ... Operator terminal 30 ... User terminal 40 ... Management Server 202 ... Control Unit 202a ... CPU 202b... Main memory section 206 … Input device 208 … Display device 212 ... Speaker 216 … Auxiliary storage unit 216a ... Operator-side response software 302 ... Control Unit 302a ... CPU 302b... Main memory section 310... Mike 314… Camera 316 … Auxiliary storage unit 316a ... User-side response software 402 ... Control Unit 402a ... CPU 402b … Main memory section 410 … Auxiliary storage unit 410a ... Customer service control software 410b ... Customer Service Management Database 700 ... Operator-side response screen 702 … Enlarged Room Display Area 704 ... User's image 706… Evaluation indicator 1200 … Centralized monitoring screen 1202 … Enlarged Room Display Area 1204 ... User's image 1218 … Rating button

Claims

1. An information processing device used by a first user for the first user to communicate with a second user online, An estimation means for estimating the second user's evaluation of the first user based on at least one of the following: an image of the second user captured by the camera of the space where the second user is present, including an image of the second user captured by the camera while the aforementioned conversation is taking place; and sound detected by the microphone of the space where the second user is present, including the voice of the second user detected by the microphone; An information processing device comprising a first presentation means for presenting to the first user, while the conversation is taking place, first evaluation information representing the evaluation of the first user by the second user estimated by the estimation means.

2. The estimation means is, When estimating the evaluation of the first user by the second user based on the captured image, the evaluation of the first user by the second user is estimated based on at least one of the facial expression of the second user recognized from the captured image and the actions of the second user recognized from the captured image. The information processing device according to claim 1, wherein, when estimating the evaluation of the first user by the second user based on the detected sound, the evaluation of the first user by the second user is estimated based on at least one of the tone of the second user's voice recognized from the detected sound, the words contained in the second user's voice recognized from the detected sound, and the sound of action caused by the second user's actions recognized from the detected sound.

3. The system further includes a display means for displaying the captured image, The information processing device according to claim 1, wherein the first presentation means presents the first evaluation information by displaying the first evaluation information on the display means in conjunction with the display of the captured image.

4. A receiving means for receiving input of second evaluation information representing an evaluation of the first user by a third user who is listening to the aforementioned conversation, and The information processing apparatus according to claim 1, further comprising a second presentation means for presenting the second evaluation information received by the reception means to a first user.

5. An information processing method for a first user to have an online conversation with a second user, An estimation step of estimating the second user's evaluation of the first user based on at least one of the following: an image of the space where the second user is present, including an image of the second user captured by the camera while the aforementioned conversation is taking place; and the sound detected by the microphone in the space where the second user is present, including the voice of the second user detected by the microphone; An information processing method comprising a first presentation step of presenting to the first user, while the conversation is taking place, first evaluation information representing the evaluation of the first user by the second user estimated by the estimation step.

6. An information processing program for a first user to have an online conversation with a second user, An estimation procedure for estimating the second user's evaluation of the first user based on at least one of the following: an image of the second user captured by the camera of the space where the second user is present, including an image of the second user captured by the camera while the aforementioned conversation is taking place; and the sound detected by the microphone of the space where the second user is present, including the voice of the second user detected by the microphone; An information processing program that causes a computer to execute a first presentation procedure, which presents to the first user, while the conversation is taking place, first evaluation information representing the evaluation of the first user by the second user estimated by the estimation procedure.

Images